Radiology AI Vendor Evaluation: The Questions to Ask Before You Sign

The short answer: Evaluating a radiology AI vendor comes down to five things: exactly what the product does (triage, detection, drafting, or some combination), how it connects to your PACS and RIS, what the fully loaded first-year cost is, what the evidence and regulatory scope actually cover, and what happens when it breaks or you leave. Most bad purchases are not caused by picking the wrong model. They are caused by never pinning down which of those five you were buying.

Radiology AI demos are unusually good at hiding the questions that matter. The screen shows a marked-up study, the finding is obvious, the report drafts itself, and everyone nods. Nothing in that experience tells you how the thing behaves on a Tuesday night on your own scanners, or what your invoice looks like in year two. This is the list of questions that gets you there, grouped by what they protect you from.

1. What does it actually do?

Start here, because the category is three products wearing one name. Triage reorders your worklist. Detection marks findings inside a study. Report drafting turns the study into a first-draft report. A vendor may sell any one, any two, or all three, and the demo rarely distinguishes them.

The question to ask is not "does it use AI on chest CT" but "which of these three does it change, and for which study types?" Then ask the follow-up that matters more: which of these is our actual bottleneck? A group whose problem is report turnaround will get very little from a detection tool, and a group worried about overnight misses will get very little from a drafting tool. Buying the wrong one of the three is the single most common expensive mistake in this category, and it is entirely avoidable.

Our explainer on AI triage works through the distinction in detail if you want to arrive at the demo already knowing which one you need.

2. Which studies and findings, specifically?

Ask for the list in writing. "Chest CT" is not an answer; the answer is which findings on which study types under which protocols. Vendors in this space typically build indication by indication, so coverage is a list, not a capability.

Then check that list against your case mix. Every practice has a distribution of what it actually reads, and it usually differs from what the vendor optimized for. If sixty percent of your volume is musculoskeletal radiography, a product with deep chest CT coverage is impressive and largely irrelevant to you.

3. What does the regulatory scope actually cover?

Clearances are granted per indication, not per product. A vendor with a long list of clearances may have exactly one that overlaps with what you read. Ask which specific indications are cleared, and treat the list as a scope document rather than a quality signal.

It also helps to be clear about what clearance means. Most AI-enabled imaging devices reach the US market through the 510(k) pathway, by demonstrating substantial equivalence to an existing cleared device. That is a regulatory determination that a device may be marketed for an indication. It is not a performance guarantee and it says nothing about how the tool behaves on your population, your scanners and your protocols.

Ask which indications the clearance covers, then ask separately what the published evidence shows, then ask separately how it performed in a shadow period at a site like yours. Those are three different questions and vendors will happily let one answer stand in for all three.

4. How does it connect?

This is the question that determines most of your cost and nearly all of your risk, and it is the one people ask last.

• Inbound: does it take studies from your PACS over standard DICOM, or does it need a custom feed?
• Outbound: do flags, priorities and drafted reports travel back over HL7 or FHIR into the tools your radiologists already use?
• Where does the radiologist see it? Inside the existing viewer and worklist, or in a separate application with its own login?
• Cloud or on-premises, and which one does your security posture require?

The second-login question deserves particular weight. A tool that lives in a separate window is a tool people stop opening within about six weeks, which converts the whole purchase into a sunk cost without anyone deciding to abandon it. Adoption failure in radiology AI is much more often an integration problem than a model problem.

Our PACS integration page sets out the standards-based pattern to look for and what a custom-interface answer implies for your timeline.

5. What is the fully loaded first-year cost?

One number, in writing, covering license, integration, training and support. Not the monthly per-seat figure. The gap between two vendors on the license line is usually smaller than the gap in what it takes to make each one work.

Then the follow-ups that catch the standard surprises:

1. What does year two cost? Year-one discounting is normal. Ask for the renewal price and the annual uplift.
2. What counts as a seat or a study? Part-time readers, locums, weekend cover and re-reads are where invoices go strange.
3. What integration work is included and what is billable? Get the boundary written down.
4. What happens if volume grows or shrinks? Commitments that ratchet up but never down are common and negotiable.
5. Can we run a paid pilot that credits against the contract? A confident vendor usually says yes.

Our breakdown of radiology AI cost goes through the pricing models and what drives the number.

6. What does the pilot look like?

Insist on a shadow period: the software runs on your studies, in your environment, without changing anyone's worklist or report. Then compare what it surfaced against what your radiologists actually did.

Agree the success criteria before it starts, and make them specific. "Radiologists liked it" is not a criterion. Time to first read on urgent studies, minutes per report, percentage of drafted reports signed with minor edits, and the rate of flags your readers dismissed are all measurable, and all worth arguing about in advance rather than after.

Decide who owns the evaluation, too. A pilot that is nobody's actual job produces a vague positive impression and no data, and vague positive impressions are how groups end up three years into a contract nobody can justify.

7. What happens when it breaks?

Software that sits between your modalities and your worklist is a production dependency. Ask what the failure mode is, and be specific: if the vendor is unreachable or the interface stalls, does your worklist degrade gracefully back to arrival order, or does it silently stop updating? Silent failure is far worse than loud failure here, because a stale worklist looks exactly like a quiet shift.

Then ask about support hours against your reading hours. A vendor with business-hours support and a group that reads overnight are a poor match, and that mismatch never shows up until the first 2am problem.

8. What are the data terms?

Four questions, and you want the answers in the contract rather than in an email.

• Where is patient data processed and stored, and does it leave your environment?
• Is your data used to train the vendor models, and can you decline without losing functionality?
• What is the business associate agreement, and does it cover subprocessors?
• What happens to your data when the contract ends?

Your security review will surface most of this eventually. Raising it in the first conversation is better, because a vendor whose answer requires three weeks and a lawyer has told you something useful about the implementation timeline. The same goes for the routine vendor onboarding paperwork: procurement will want the security questionnaire, the BAA and a current certificate of insurance on file before anything is signed, and starting that in week one rather than week nine is often what decides whether a go-live date holds.

9. How will we know it is still working in a year?

Almost nobody asks this and it separates a purchase from a deployment. Model behavior drifts, your case mix changes, you replace a scanner, protocols get updated. What is the review cadence, what gets measured, and who looks at it?

Agree a quarterly review with the same metrics you used in the pilot. If the vendor has no mechanism for this, you are buying a product you will never be able to evaluate again after go-live.

10. What does it not do?

Ask it directly and listen carefully to the shape of the answer. A vendor who names real limits, specific findings the tool misses, study types it does not cover, situations where it underperforms, is giving you the information you need to deploy it safely. A vendor who cannot think of anything is either not being straight with you or does not know their own product well enough, and both are reasons to slow down.

Be particularly wary of accuracy claims presented without the context of the population they came from. A figure from a curated retrospective dataset tells you very little about a Tuesday night at your practice.

The short version

Send the same list to every vendor, in the same email, and ask for written answers. The variance in how willing they are to answer plainly tells you nearly as much as the answers themselves.

Our answers

Radiological.ai does all three: it flags suspected findings for a second look, prioritizes the worklist so urgent studies surface first, and drafts the structured report into your template, across X-ray, CT and MRI in one assistant rather than three contracts. It connects over standard DICOM inbound and HL7 or FHIR outbound, and it works inside the viewer and worklist your radiologists already use.

What it does not do: diagnose, publish accuracy figures, or claim regulatory status. Every flag is a prompt to look, every report is a draft, and the responsible radiologist reviews, edits and signs every study. Our plans are published on the pricing page rather than held behind a discovery call.

If you are still mapping the market, our guide to AI radiology companies explains how the category splits, and the buyer's guide to AI radiology software compares the options. When you are ready to implement, the implementation guide covers the rollout.