Validate a computer vision project before you build it

Computer vision has an unusual problem: the first 80% is deceptively easy. Download a pretrained detector, point it at a sample video, and within an afternoon you have boxes around people, cars, or products. It looks almost finished.

Then the system meets real conditions: the real cameras, mounted at awkward angles, under flickering warehouse lights, filming objects that overlap, at a resolution nobody checked. Accuracy drops, edge cases pile up, and a project planned for weeks stretches into months. Often the hard truth arrives late: the problem was never solvable with the available cameras and data.

That's why our process puts a dedicated validation stage between understanding the problem and building the system. Its purpose is to find out, as cheaply and early as possible, whether the project can succeed and what it will really take. This post explains what that stage involves and why it pays for itself.

Why demos mislead

A demo answers the question "can this technology ever work?" A business needs the answer to a different question: "will it work here, reliably enough to be useful, within our constraints?" Several gaps sit between those two questions.

Data gap. Sample footage is clean, well lit, and well framed. Production footage usually isn't.
Long-tail gap. Demos show common cases. Production is defined by rare ones: the forklift that blocks the camera, the glare at 4pm, the product in new packaging.
Accuracy gap. "It detects most things" is a very different claim from "it's accurate enough that people stop double-checking it."
Constraint gap. A model that runs well on a powerful GPU may be far too slow on the device that has to run it on site.

Validation is the work of measuring each of these gaps before committing to close them.

Step 1: Define what "working" means

"Detect defects" or "count customers" is a goal, not a success criterion. Before any modelling, we work with the client to pin down specifics.

What exactly must be detected

Which objects, events, or defects matter, and which can be ignored? What is the smallest defect that counts? Is a partially visible object a detection or not? Ambiguity here turns into arguments about accuracy later, so it's worth settling early with concrete examples.

Which errors cost more

Every vision system makes two kinds of mistakes: misses (something was there and the system didn't flag it) and false alarms (it flagged something that wasn't there). These almost never cost the same.

In safety monitoring, a miss can be serious, so some false alarms are acceptable.
In automated quality inspection, too many false alarms mean operators start ignoring the system, which makes it worthless.

Knowing which error matters more decides how the system is tuned and how it is judged.

What "good enough" looks like

The bar is rarely perfection. It's usually "better than the current process" or "good enough to cut manual review significantly." Agreeing on that bar upfront, in measurable terms, is what makes a go/no-go decision possible later.

The operating constraints

Latency. Real-time alerts, near-real-time dashboards, or overnight batch reports?
Hardware. On an edge device on site, an on-premise server, or the cloud?
Privacy. Can footage leave the site? Do faces need to be blurred?
Integration. Where do the results go: an alert, a dashboard, a database, an existing system?

These constraints often matter more than the choice of model, and changing them later is expensive.

Step 2: Collect real data from the real environment

The most valuable thing a client can provide during validation isn't a requirements document. It's real footage or images from the actual cameras, in the actual conditions, over a realistic stretch of time.

Aim for data that covers the variation the system will face:

Different times of day and lighting conditions
Busy periods and quiet periods
Every camera position that will be used
The seasons or shifts that change how the scene looks

Then label a small portion carefully. Labelling forces precision. Disagreements about how to label an image usually reveal places where the success criteria from step 1 are still unclear.

Step 3: Establish a baseline

Run an off-the-shelf model, or a lightly adapted one, on the real data and measure it against the agreed criteria. This baseline is one of the most informative results in the whole project, because it tells you what kind of problem you have.

Close to the target. The problem is largely solved by existing technology. The work is in adaptation, integration, and reliability.
A clear but bridgeable gap. Fine-tuning on domain data, better camera placement, or a narrower scope will probably close it.
Far from the target. The problem may need a different approach, much more data, different hardware, or a rethink of what's being asked.

The baseline also gives every later improvement something to be measured against, which keeps the project honest.

Step 4: Go looking for the hard cases

Average accuracy hides the cases that decide whether a system is trusted. During validation, we deliberately look for the situations most likely to break it:

Occlusion: objects hidden behind other objects or people
Crowding: many similar objects overlapping
Scale: very small or very distant objects
Image quality: motion blur, glare, low light, rain, dirt on the lens
Viewpoint: unusual angles, or only part of the scene in view
Novelty: new product variants, uniforms, vehicles, or layouts

We report performance on these hard cases separately from the overall average. A system that is accurate on average but fails every time two objects overlap may be unusable, depending on how often that happens.

Our own FootballVision prototype is a good example. Tracking held up well in open play and broke down in exactly these conditions: crowding, occlusion, small fast objects, and partial camera views.

Step 5: Check the constraints for real

Once a candidate approach looks promising on accuracy, test it against the operating constraints from step 1. Measure speed on hardware equivalent to what will be used in production. Check memory use. Estimate running costs at the expected volume. Accuracy that only exists on a GPU you won't have doesn't count.

Step 6: Make a clear call

The output of validation is a decision backed by evidence, not just a model. It usually takes one of three forms.

Go. The baseline is close to the target, the remaining gap is understood, and the constraints are achievable. Build with confidence.
Go, with changes. The project is feasible with adjustments: more labelled data, different camera placement, better hardware, or a narrower first scope, such as one production line instead of ten.
No-go. The problem can't be solved reliably with the available data, cameras, or constraints, and we explain why.

A no-go can feel like failure. It isn't. Learning in a few weeks that a project can't succeed under its current constraints is a far better outcome than learning the same thing six months and a full budget later. Often a no-go also points to what would make the project viable.

What this means if you're planning a vision project

If you're considering a computer vision system, a few questions will tell you a lot about how ready you are:

Can you describe success in numbers, including which errors matter most?
Do you have, or can you collect, real footage from the real environment?
Do you know where the system will run and how fast it needs to be?
Have you seen it tested on your hardest cases, not just your typical ones?

If any answer is "not yet," that's normal. Answering those questions is exactly what validation is for, and it's much cheaper to do before building than after.