The possibilities of AI in medicine are seemingly boundless, with many talking about its labour saving potential. Yet concerns persist, particularly with Chat GPT, and its black box LLM (large language model) system which is designed to mimic conversation but is not always factually accurate nor able to spot signs of acuity. By their nature, black box systems also cannot explain the conclusions they reach.