Learning Lab: Intelligent or Just Artificial? – The Emerging Quality Challenge of Dealing with Our Smart Machines-Intelligent or Just Artificial Handouts

Intelligent or Just Artificial Handouts

Pdf Summary

Ken Rohde’s 2019 NAHQ presentation highlights an emerging healthcare quality challenge: the growing use of embedded intelligence—algorithms, decision-support software, machine learning, and AI—inside clinical equipment and hospital systems. While algorithms capture knowledge in repeatable, reviewable ways, the world changes and real learning requires continuous updating. Healthcare organizations increasingly expect “smart” systems to learn from users and patient outcomes, creating closed-loop learning systems that introduce new risks.<br /><br />The talk emphasizes that hospitals are already full of algorithm-driven tools (e.g., sepsis identification, surgical guidance, prediction models). Rohde illustrates what can go wrong through two cautionary stories. The Therac-25 radiation incidents (1985–1987) show how removing hardware safety interlocks, relying on software checks, using unclear error messages, enabling easy overrides, failing to test hardware and software together, and lacking independent software review can lead to catastrophic patient harm. A modern parallel is the Boeing 737 MAX crashes, where performance pressure, increased software control (MCAS), single-sensor failure dependence, limited training, and inadequate feedback/alerts contributed to fatal outcomes.<br /><br />Rohde argues quality professionals must ensure there are “no holes” in the protective umbrella of quality oversight—including oversight of intelligent machines and third-party technologies. Quality professionals do not need to be software engineers, but they must know what to ask: whether a structured design process exists; whether verification and validation (V&V) are performed; how “black box” systems are managed; and whether typical failure modes have been considered, especially for machine learning/AI. The session introduces foundational V&V concepts: verification checks each development step matches its specifications, while validation confirms the final system meets real user needs and intended clinical outcomes.

Keywords

embedded intelligence in healthcare

clinical decision support algorithms

machine learning and AI in hospitals

closed-loop learning systems

healthcare quality oversight

Therac-25 radiation accidents

Boeing 737 MAX MCAS failures

software safety interlocks and overrides

verification and validation (V&V)

black box algorithm risk management