Skip to main content

Home / Case study · AI implementation

The AI That Held

An AI report-review engine that held up on a business owner's real reports where his own DIY chatbot attempts drifted.

Built and run on real reports, ended August 20262026

AI implementation · Standards grounding · Structured output

The AI reviewer run on a real inspection, client name redacted: 136 photos auto-sorted into likely issues, needs-a-look, and cleared, scored against the owner's own rubric.
The AI reviewer on a real inspection, client name redacted. It sorted 136 photos into defects, needs-eyes, and cleared against the owner's own rubric, and surfaced every photo for the inspector's final say. No accuracy number is claimed, because the project ended in August 2026 before its accuracy test.

Context

A home inspection company owner had already tried building this himself. Hundreds of hours into ChatGPT versions, all of them drifting. The rules got lost somewhere in one long chat and the model started improvising. That is how bolt-on AI usually ends, and it is why he came to me instead of trying another chatbot.

The build

I built the v1 engine. It read his real inspection reports end to end and returned the review in his own scorecard format, the layout he used to fill in by hand at night. His standards were the grounding, so it flagged things the way he does, not the way a generic chatbot would.

Why it held

The model was never his problem. In my build the rules lived outside it. Each check handed the AI only the slice of his standards that one question needed, so it could not lose the plot the way a chat window does. A prompt is not a system. The machinery around the model is what makes AI dependable inside a real business.

The number I would not publish

I could have quoted an accuracy number and it would have looked great. It would also have been close to meaningless. The reports I had were the ones he already corrected, and testing on those flatters the engine. I would not publish an accuracy number until it was measured against reports the way his inspectors first submitted them, and the project ended in August 2026 before that test. When a vendor hands you a suspiciously clean number, ask what they tested it on.

Status

Status

Built
The v1 engine, end to end on his real reports
Not completed
Catch rate, measured the honest way
Results

No accuracy number was published, because it was never measured against reports the way inspectors first submitted them.

Stack

AI implementation · the client's own standards wired in as grounding · per-check retrieval · structured scorecard output