Ontario's health system is adopting AI scribes quickly, and the province built a vetting process to qualify which ones clinicians can buy. This spring, the Auditor General examined how that process worked. The finding that matters for any practice is this: the vetting approved tools that invented clinical findings and recorded the wrong medication, then left the job of catching those errors to the clinician. If you are weighing one of these tools, the due diligence the process skipped is now yours to do.
Start with what the testing showed. During procurement, evaluators gave every bidding vendor the same two simulated patient conversations and asked them to produce clinical notes. All 20 vendors that were ultimately approved produced at least one kind of error: 9 fabricated information outright, recommending therapy referrals or blood tests no one had discussed, and in one case recording "no masses found" when nothing of the kind had come up; 12 recorded a different medication than the one prescribed; and 17 missed details about the patient's mental health in at least one of the two tests. These were not the tools that got rejected; they were the ones that passed.
The scoring explains how this might have happened. Vendors were marked across seven categories worth 530 points, and a vendor's physical presence in Ontario was worth 30% of that, more than every safety-related measure combined. The accuracy of the medical notes was worth 4%, and evidence that the system mitigated bias was worth 2%. Privacy and security made up most of the rest on paper, but those marks were earned largely on vendors' own word: 11 of the 20 approved vendors never submitted a third-party security audit, and 5 never submitted a privacy impact assessment. Because no criterion carried a minimum passing score, a tool could record the wrong drug, fail on security, and show nothing on bias, and still qualify, provided the rest of the sheet carried it.
What happened next is the part that should concern you. The province's answer to the accuracy problem was to issue guidelines telling doctors to review the AI's notes by hand, with no requirement that the clinician confirm they had done so and no sign-off built into the tool. The error rate was known, documented, and accepted, and the work of catching it was handed to the person already carrying the patient. None of this is mandatory, either. Buying from the approved list is optional, and practices remain free to use tools that were never vetted at all, which is where the sharper risk lives. It has already happened here. At one Ontario hospital, an unapproved transcription tool recorded a meeting and automatically distributed the patient information discussed in it to current and former staff — a breach reported to the Information and Privacy Commissioner.
It is worth knowing this is a choice, not a limit of the technology. In the UK, the NHS now directs its organizations to use only scribe systems registered as medical devices, with clinical accuracy requirements attached. The tools are the same, but the expectation around them is not.
Treat the provincial list as a starting point rather than a guarantee. Before you sign, it is reasonable to ask a vendor for a current third-party security audit, such as a SOC 2 Type 2 report, rather than a promise that one exists; for evidence the system was tested for bias, including across accents and dialects; for a built-in way to review and sign off on each note before it enters the record; and for written confirmation of where the audio and transcripts are stored, with proof, not assurance, that the data stays in Canada. None of these are unusual requests. They are the questions the procurement process should have asked and mostly did not, which means they are now yours to ask.
That is the real finding here. The province tested these tools, recorded the errors, approved them anyway, and left the clinician as the last safeguard in the chain. A good scribe can return hours to your day and your attention to the patient, but it earns that trust only after you have seen the evidence rather than the pitch. Reviewing AI tools and contracts before a practice commits is part of what The Opening Door does, and if you would rather not vet a system on your own, that is the conversation worth having before you sign.
