What AI should and should not be trusted with in a compliance system

Document extraction is the most useful thing AI currently does in property compliance, and also the easiest to oversell. Both halves of that sentence deserve attention, because compliance is a domain where the cost of a confident mistake is measured in enforcement rather than embarrassment.

What it genuinely does well

Reading a structured document and pulling out the facts it states is a task modern models are good at. An engineer’s report follows conventions — a provider, a site, a date, a list of items tested, findings, recommendations, an outcome. Extracting those into fields removes hours of transcription that nobody enjoys and everyone rushes.

Safety data sheets are an even better fit. They’re standardised by design, and the work of copying hazards, PPE requirements, storage conditions, first aid and disposal into an assessment form is pure clerical effort with no judgement in it. Pre-filling that form and asking a person to check it is a straightforward improvement over typing it from scratch.

Where the line sits

The line is between reading and deciding. Reading is extraction: this report says the next inspection is due in March. Deciding is judgement: therefore this site is compliant, therefore this risk is acceptable, therefore no action is needed. The first is a fact transfer that can be verified against the document in seconds. The second is a position your organisation is accountable for.

So the design rule is simple and shouldn’t be negotiable: extraction fills the form, a person confirms it, and nothing enters the compliance register unreviewed. That isn’t a lack of confidence in the technology — it’s a recognition that the record has to be defensible, and “the system read it that way” is not a defence anyone wants to offer.

Questions worth asking any vendor

• Does extracted data enter the register before a human sees it? If yes, ask what happens when it’s wrong, and who finds out.
• Is the original document kept and linked? The extraction is a convenience; the source document is the evidence.
• What happens to a document it can’t read? Silent failure is the worst outcome — a document that quietly never processed looks identical to one that had nothing to report.
• Where is the document sent, and under what terms? Reports contain site addresses, engineer names and building details. That’s a data-protection question, not a technical one.
• Can you see what’s queued, processing and done? If the pipeline is invisible, you can’t tell working from stuck.

The unglamorous engineering underneath

Most of the effort in making this reliable has nothing to do with the model. It’s the queue: pacing requests so a large batch doesn’t get rate-limited, retrying what’s transient, failing cleanly on what isn’t, and sweeping periodically so that a deployment or a crash mid-run can’t leave documents stranded forever. That last one is the difference between a feature that works in a demo and one that works across a decade of back-catalogue.

The honest summary

AI won’t make your estate compliant. It removes the transcription tax that stops people keeping the record current, and it surfaces findings that would otherwise sit unread in an attachment. That’s a real, measurable gain — and it’s a smaller claim than most of what’s being sold, which is rather the point.