As more complex computing tools work their way into healthcare practice, clinicians and patients become reliant on black boxes that determine their outcomes. When these new technologies fail to improve patient care or have dangerous unforeseen consequences, clinicians are held responsible for the limitations of systems they may not readily understand. One approach is to build explainable systems so clinicians can interpret the outcomes of tools after the fact. But at this point, the harm may already be done.

In this talk, we discuss a different approach to explainable software where clinicians and patients alike are involved in the development of healthcare tools as first-order experts. The goal of this approach is to remove the shroud of technical mysticism from these black boxes and produce software that both clinicians and their patients can understand and fully control. We will discuss this approach across three case studies exploring different emerging technologies in healthcare.