‘Agentic AI’ and automation bias
At HBI’s 2026 conference last month I chaired a panel entitled ‘Agentic AI beyond the scribe: What’s next for healthcare?’.
This included case studies from Oskari Eskola, CEO of Finnish digital health group Beehealthy, and David Reich, President of New York non-profit hospital Mount Sinai Hospital, who each presented multiple new AI tools they’re deploying to improve care quality and efficiency. These included a graph neural network to help assign nurses in maternity care, an LLM that can help identify patients with post-surgical complications, and various tools to help with onboarding and managing the health data of 3.5 million people for a population health management scheme in Finland.
Inevitably, one of the topics that came up in the ensuing discussion was the extent to which AI will, and should, replace human labour and human judgement.
Reich commented that we are likely to see significant upheaval and possible social unrest as a result of AI displacing human workers. Those who remain in the workforce will need extensive training on how to interact with and get the most out of AI tools. A complete overhaul of the education system would appear to be in order.
When asked whether there is a risk that healthcare workers will over-rely on AI tools, leading to cognitive decline and/or offloading of critical judgment to AI ‘agents’, Reich countered that a similar concern was raised when ultrasound became widespread in cardiology in the 1990s, but conceded that ‘automation bias’, which is the human tendency to trust machines over their own judgment or contradictory evidence, does exist.
The solution, however, is not to reject AI tools outright, but to make sure they’re properly validated and put systems in place to ensure they are continuously monitored and improved, and that they’re used appropriately. The education process for how to use AI tools must include attempts to inculcate the critical thinking skills necessary for evaluating AI outputs critically, as well as a rough understanding of how the systems arrive at their outputs and their propensity to make mistakes.
Patients and clinicians need to be guided and nudged towards making decisions based on all available evidence, which doesn’t always happen currently. The right AI tools can — used in conjunction with human judgement — help with this.
Click here to read more about this panel.
We would welcome your thoughts on this story. Email your views to Martin De Benito Gellner or call 0207 183 3779.
