AI Diagnosis: What Microsoft's 83% Claim Teaches Marketers

Overview

Microsoft says its AI diagnosis system is more than twice as accurate as doctors. The claim comes from a peer-reviewed paper in Science, one of the world's top journals, and within a day it had spread across news sites and social feeds, usually cut down to three words: "AI beats doctors."

The numbers are real. But the headline skips what the test measured and who the AI was up against. That gap between claim and evidence is the part marketers should study.

This article explains what the study found, in plain terms, and what it does not show. Then it turns the story into a practical guide for reading and writing AI claims, whether you sell AI tools, market a health brand or just share news with your audience. It is not medical advice.

Key takeaways

• Microsoft's MAI-DxO reached the right diagnosis in 83.2% of 304 hard medical journal cases. Doctors averaged 36.1% with reference tools and 22.3% without them.
• On 325 emergency department records, the system scored 80.3% against 46.8% for doctors.
• The doctors were generalists working alone, with no specialists, no teams and none of the scans, exams or colleagues they would use in a real clinic.
• The cases were historical and text-only. No real patients were seen, and the test costs are estimates.
• All 15 authors work at Microsoft. The paper passed peer review, but no outside team has repeated the results yet.
• For marketers, the lesson is to name the test, the comparison group and the limits whenever you repeat or write an AI claim.

The AI diagnosis study at a glance

Sources: the Science paper, the AAAS press release and the authors' version of the paper. Microsoft's team first shared the work as a preprint in June 2025.

How Microsoft tested AI diagnosis

Most medical AI tests hand the model a full case and ask for the answer. Real doctors don't work that way. They start with a few facts, ask questions, order tests and change their minds as results come in.

So Microsoft built a test that works more like a real clinic visit, where facts arrive one at a time and every new question has to be chosen and paid for. It's called the Sequential Diagnosis Benchmark, or SDBench. Each case starts with a short summary, such as a patient's age and main symptom. To learn more, the doctor or the AI has to ask.

A second AI model plays the gatekeeper. It holds the full case file and answers only the questions asked. A third model judges. It compares the final answer with the known diagnosis and scores it on a five-point scale.

Every question and test also has a price. A standard visit counts as $300, and tests are priced from a 2023 US hospital price list. So the test measures two things at once: accuracy and how much testing it took to get there.

MAI-DxO itself isn't a new model. It's a way of running one: it sets up a panel of virtual doctors inside the model. One suggests possible diagnoses, one picks the next test, one argues against the current favorite and one watches the cost. The panel then agrees on the next step.

The AI diagnosis numbers, in context

Here are the main results reported in the paper. Accuracy means the share of cases where the final answer matched the known diagnosis. Cost is the average estimated cost of tests per case.

Microsoft's "more than twice as accurate" comes from 83.2% against 36.1%. Our math puts that at about 2.3 times. Against doctors with no references, it is about 3.7 times, which is where "nearly four times" comes from. In the emergency cases, the gap shrinks to about 1.7 times.

Notice two things. First, GPT-5 alone already scored 78.9%, so much of the gain over doctors comes from the base model, not the orchestration.

Second, the gap is widest on the hardest cases. On everyday emergency records, the doctors did much better.

Budgets matter too. With a strict test budget, MAI-DxO scored 79.6% at about $1,300 per case. With an ensemble of panels, it reached 84.5% at about $8,950.

Who the AI diagnosis system was compared with

The phrase "beats doctors" depends on which doctors. In this study, they were generalists: family doctors and hospital generalists in the US and UK, with a median of 12 years of experience.

The authors' version of the paper gives the details. Twenty-one doctors took the no-reference test. A new group of 19 took the reference version, with tools like UpToDate and NICE guidelines but no generative AI. Sixteen of them later took the emergency test.

They worked alone, through a text screen. They couldn't examine anyone, see scans or ask a colleague.

They also didn't see all 304 cases. To keep the test fair, their scores come from the 56 most recent cases, which the team had kept hidden during development. The authors report that the AI scored about the same on those 56 cases as on the full set.

No specialists took part, and no teams. The cases came from a journal series that picks rare, teaching-quality cases, often the kind a generalist would refer to a specialist. The authors themselves say the gap would likely narrow against expert teams.

What this AI diagnosis study does not show

The authors are clear about the limits. Their paper says the results show the system beat individual doctors inside a controlled simulation, not expert teams in real care. Here is what the test left out:

• Real patients: every case was a past record, turned into text. Nobody was treated.
• Healthy people: every case had a real illness. The test could not measure false alarms in people who are well.
• Images and exams: none. No scans, lab slides or physical exams were part of the test, even though they are central to real diagnosis.
• Messy records: real patients forget things, records are incomplete and symptoms change over time.
• Treatment and safety: the judge scored the diagnosis only, not whether the next steps would be safe or right.
• Independent checks: the paper passed peer review, but no outside team has repeated it yet.

None of this makes the work weak. It makes it early. The test is a careful step beyond multiple-choice exams. It isn't proof that an AI should diagnose patients today.

Read the cost claims carefully

Microsoft also says MAI-DxO reached its answers at lower cost than doctors in many settings. That claim needs the same care as the accuracy claim.

The costs aren't real bills. A language model matched each requested test to a billing code, and the price came from one large US health system's 2023 price list. The paper calls these estimates a standard way to compare, not a picture of real spending.

Also, the cost story depends on the setting. With no budget limit, MAI-DxO spent more per case than the doctors with references on the hard cases, about $5,430 against $4,725.

It only became cheaper under budget limits, where it also lost some accuracy. Short version? "Cheaper and better" is true for some settings, not all.

From preprint to Science: what changed

If the numbers look familiar, that's because an early version made headlines in June 2025. That preprint ran on OpenAI's o3 model. It reported about 80% for the system against 20% for 21 doctors, and 85.5% in its top setting. It had not been peer reviewed.

The Science version adds stronger models, including GPT-5, a second set of emergency cases and a group of doctors who could use reference tools. Some numbers moved. The doctors' average rose to 36.1% with references, so the gap got smaller than the 2025 headlines suggested.

Reports say Microsoft also reran the tests with GPT-5.5 and saw larger gains. Those results are from the company's own announcement, not the peer-reviewed paper.

What comes next for AI diagnosis at Microsoft

The real test is still ahead. Microsoft says it will run a study with Beth Israel Deaconess Medical Center in Boston, starting in emergency and urgent care. Patients who agree to take part will interact with the system, and its views will be compared with the doctors' own judgment.

Microsoft expects it to be the largest study of its kind. Details are thin. Microsoft hasn't shared a start date, a size or the outcomes it will measure. Microsoft also has a separate project with Mayo Clinic to build a healthcare AI model.

Google has a similar effort. Its AMIE system was tried with 98 patients at Beth Israel in a small study published in The Lancet on October 8. The race is on, which is one more reason to read each new claim closely.

Why marketers should care about AI diagnosis claims

You may not sell anything to hospitals, and you may never write about medicine, but this story still matters to anyone who makes or repeats claims about AI, for three reasons.

First, "AI beats humans" is now a standard marketing claim. Vendors use it for writing, ads, analytics and customer service. The Microsoft study is a clean example of how such a claim is built, and where it can mislead.

Second, short claims travel further than the fine print. A headline like "AI twice as good as doctors" gets shared and summarized. AI assistants then repeat the summary when people ask. Once a simple version spreads, it is hard to correct.

Third, you'll get asked. Clients, readers and buyers will cite these headlines. Knowing how to read them makes you more useful and more trusted.

A checklist for reading AI benchmark claims

Use these questions on any "AI beats X" claim. The right column shows how the Microsoft study answers them.

If a claim can't answer most of these, treat it as a promise, not a result.

How to write about AI claims without overstating them

Accurate doesn't have to mean boring. Compare these headlines:

• Overstated: "Microsoft's AI is twice as good as doctors."
• Accurate: "Microsoft's AI beat solo generalist doctors on hard test cases."
• Overstated: "AI will replace your doctor."
• Accurate: "AI diagnosis passed a tough simulated test. Real-patient trials come next."

A few habits help. Name the comparison group. Say whether people or past records were involved.

Give the source and its date. Label your own math, keep the company's words in quotes, and link to the original paper or press release so readers can check the numbers for themselves.

And don't stretch a narrow result into a general one. "Beat generalists on hard text cases" is still a strong, interesting claim. It's also one you can defend.

Extra care for healthcare marketers

If you market a health product or service, the bar is higher. Health claims are closely watched, and AI claims are getting the same attention.

• Don't imply approval: a paper in a journal is not a regulatory clearance. In the US, some diagnostic software is regulated as a medical device.
• Back up every health claim: US advertising rules expect solid scientific evidence for health claims, and the standard rises with the risk.
• Avoid "AI doctor" language: it suggests a level of care no tool has shown in practice.
• Separate research from product: a lab result doesn't mean a feature in your app works the same way.

Rules differ by country. Check with your legal team before you publish. The safest claim is the one that matches the study exactly.

Apply the same test to your own AI claims

The easiest claims to overstate are your own. "Our AI writes ads 3x faster" or "beats human copywriters" raise the same questions as the Microsoft study.

Faster than whom, at what task, and judged by whom? On how many examples, and who chose them? Did the human side get the same tools, the same time and the same information as the AI, or were they working with one hand tied behind their back? If you can't answer, soften the claim or run a fairer test.

This protects you, too. Buyers are getting better at spotting stretched AI claims. A precise claim with a clear source earns more trust than a big one with none. Our article on the AI marketing proof gap covers how to show results buyers believe.

How AI assistants will retell this AI diagnosis story

Ask an AI assistant about this study in a few months, and it will likely answer from the most cited sources. If those sources say "twice as good as doctors," that's what people will hear.

The same thing happens to brands. AI assistants repeat the claims that appear most often and most clearly. If your own claims are vague or overstated, that's the version that spreads. If they are precise and well sourced, assistants have something better to quote.

That's why we treat claim accuracy as part of AI search work, not a separate legal box to tick at the end, and why we check every number before it goes out. Our guide to citation optimization explains how to make your content the source assistants trust.

AI diagnosis: our take

This is serious research. The test is more realistic than most AI benchmarks, the authors list their limits plainly and the paper passed peer review in a top journal. It points to a real future in which AI helps doctors think through hard cases.

But the evidence supports a narrower claim than the headlines. MAI-DxO beat solo generalists on hard, simulated cases. Whether it helps patients is the question the Beth Israel study will try to answer.

For marketers, it's simple. Read past the headline, name the comparison and keep claims as precise as the evidence. That makes your content more useful and more trusted.

How Vanaxity helps brands make accurate AI claims

We help marketing teams publish content that AI assistants and readers can trust. That includes checking claims against their sources, writing clear comparisons and tracking how AI assistants describe your brand.

If you want your claims to hold up, explore our services or browse more insights.

Frequently asked questions