Vanaxity insights tagged AI benchmark claims - notes on AI content, SEO, GEO and AEO from Van Data Team.
On October 22, 2026, Microsoft researchers published a peer-reviewed paper in Science on MAI-DxO, an AI diagnosis system built on OpenAI's GPT-5. It reached the right diagnosis in 83.2% of 304 hard New England Journal of Medicine cases. Generalist doctors averaged 36.1% with standard reference tools and 22.3% without them, on a held-out set of 56 of those cases. On 325 emergency department records, the scores were 80.3% for the system and 46.8% for doctors. The result is real, but narrow: the cases were historical and text-only, the doctors were solo generalists rather than specialists or teams, all 15 authors work at Microsoft, and the test costs are estimates. A study with real patients is planned at Beth Israel Deaconess Medical Center. For marketers, the paper is a clear lesson in how to read, repeat and write AI benchmark claims.
