Risk Management Tools & Resources


Waiting for Watson: Challenges and Risks in Using Artificial Intelligence in Healthcare

Laura M. Cascella, MA

Waiting for Watson: Challenges and Risks in Using Artificial Intelligence in Healthcare

Artificial intelligence, or AI, is a burgeoning field in health information technology and a key element in envisioning the future of healthcare. Daily stories trend in the media related to AI applications and their widespread potential for revolutionizing medical practice and patient care. Yet, akin to the promises of electronic health records (EHRs) in the early 21st century, the excitement surrounding AI has no doubt led to a sensationalized view of its capabilities while marginalizing technological and operational challenges as well as safety and ethical concerns.2

Artificial Intelligence in Healthcare

Although AI might seem futuristic, it already is widely used in healthcare for a number of purposes. A 2015 survey of 13 industries found that 86 percent of participants in healthcare and life sciences were using some form of AI.1

Current applications of AI in healthcare include clinical decision support systems, surgical robots, automated patient flow optimization, telehealth technologies, image analysis, and drug development research techniques.

Researchers continue to study how various types of AI — such as machine learning, deep learning, neural networks, and natural language processing — can help improve diagnosis, treatment, patient experience, healthcare operations, public health initiatives, cybersecurity, and more.

As these newer forms of AI emerge, and as existing AI applications evolve, so too will regulations, standards of care, and best practices associated with their use.

As the healthcare industry explores using AI to augment, or even replace, the roles and responsibilities of human healthcare providers now and in the future, it must balance the sometimes quixotic visions of these new technologies with their actual barriers and risks. As with any new technology, approaching AI with critical questioning and a healthy dose of caution can help prevent unfounded optimism, false confidence, and subsequent disillusionment.

Some of the challenges and risks associated with AI that healthcare leaders and providers should consider include:

  • Biased data and functional issues. One of the major red flags raised about AI is the potential for bias in the data on which machines are trained and — as a result — bias in their algorithms. Bias can occur for various reasons; for example, the data itself might be biased, or bias might occur because of a variance in the training data or environment and how the AI program or tool is applied in real life. Read more about biased data and functional issues.
  • Black-box reasoning. Many of today’s cutting-edge AI technologies — particularly machine learning systems that offer great promise for transforming healthcare — have opaque algorithms, making it difficult or impossible to determine how they produce results. This unknown functioning is referred to as “black-box reasoning” or “black-box decision-making,” and it presents concerns for patient safety, clinical decision-making, and liability. Read more about black-box reasoning.
  • Automation bias. Humans, by nature, are vulnerable to cognitive errors resulting from knowledge deficits, faulty heuristics, and affective influences. In healthcare, these cognitive missteps are known to contribute to medical errors and patient harm, particularly in relation to incorrect or delayed diagnoses. When AI is incorporated into clinical practice, healthcare providers might be susceptible to a type of cognitive error known as “automation bias.” Read more about automation bias.
  • Data privacy and security. With the digitalization of health information, healthcare organizations and providers have faced growing challenges with securing increasing amounts of sensitive and confidential information while adhering to federal and state privacy and security regulations. AI presents similar challenges because of its dichotomous nature — it requires massive quantities and diverse types of digital data but is vulnerable to privacy and security issues. Read more about data privacy and security.
  • Patient expectations. AI offers vast potential for improving patient outcomes through advances in population health management, risk identification and stratification, diagnosis, and treatment. Yet, even with this promise, questions arise about how patients will interact with and react to these new technologies and how these advances will change the provider–patient relationship. Read more about patient expectations.
  • Training and education. The emergence of AI, its anticipated expansion into healthcare, and its sheer scope point to significant training and educational needs for medical students and practicing healthcare providers. These needs go far beyond developing technical skills with AI programs and systems; rather, they call for a shift in the paradigm of medical learning. Read more about training and education.

In Summary

The healthcare industry is on the cusp of a new digital revolution driven by the power and potential of AI. The opportunities for advancement span the healthcare spectrum and offer promises of optimized patient care and experience, streamlined clinical and operational processes, workforce solutions and efficiencies, advances in cybersecurity, and more.

Amidst the fervor for AI are pragmatic calls to approach these technologies with measured caution and optimism. Although AI will undoubtedly have a major impact on healthcare, it is still fairly new and rapidly changing. Healthcare organizations implementing AI systems and programs — and healthcare providers incorporating these technologies into daily practice — should be aware of AI’s capabilities, limitations, and potential risks.

To learn more about risk strategies for managing AI, see MedPro’s Risk Tips: Artificial Intelligence.


1 Tata Consultancy Services. (2017). Getting smarter by the sector: How 13 global industries use artificial intelligence. Retrieved from https://sites.tcs.com/artificial-intelligence/#

2 Kent, J. (2019, June 25). Could artificial intelligence do more harm than good in healthcare? Health IT Analytics. Retrieved from https://healthitanalytics.com/news/could-artificial-intelligence-do-more-harm-than-good-in-healthcare

MedPro Twitter


View more on Twitter