Where our football tracker fails, and why

Most computer vision demos are edited to show the system at its best: a clean broadcast angle, good light, players spread out across the pitch. That footage is real, but it isn't typical. The interesting engineering lives in the other footage, where the system quietly gets things wrong.

This post is about that other footage. FootballVision is our experimental pipeline for detecting and tracking objects in football matches and turning those tracks into pitch-based analytics. Below, we walk through the three failure modes we see most often, explain what causes each one, and describe what tends to help. None of these problems are unique to us. That's exactly why they're worth writing down.

How the pipeline works

FootballVision has three stages. Each one feeds the next, so it helps to know what each stage is responsible for before looking at how they fail.

1. Detection. A YOLO26 object detector processes every frame and draws bounding boxes around players, goalkeepers, referees, and the ball. Each frame is handled independently: the detector has no memory of what came before.
2. Tracking. A multi-object tracker links detections across frames, so that the player in frame 100 and the player in frame 101 share an ID. This is what turns a pile of boxes into trajectories.
3. Pitch mapping. Using OpenCV, we estimate a homography: a transformation from the camera image to a flat, top-down model of the pitch. This lets us convert pixel positions into real pitch coordinates for heatmaps, distances, and formations.

The important property of this design is that errors compound. A missed detection becomes a broken track. A broken track becomes a wrong statistic. A slightly wrong homography shifts every position on the pitch. By the time an error reaches a dashboard, it can look like a perfectly confident number.

Failure 1: Tracking identities switch or fragment

What you see

Two players chase the same ball, cross paths, and separate. When they come apart, their IDs have swapped: player 7 is now labelled 12, and the other way round. In other clips, a player gets briefly hidden behind a teammate and comes back with a brand-new ID, so one person's run is split across two or three tracks.

These are the two classic tracking errors:

• Identity switches: two tracks exchange identities.
• Fragmentation: one real object's trajectory is broken into several shorter tracks.

Why it happens

A tracker has to answer one question on every frame: which new detection belongs to which existing track? It usually combines two signals to do this.

• Motion: where should this player be now, given where they were and how fast they were moving?
• Appearance: does this detection look like the player we were following?

Football is unkind to both. Players on the same team wear identical kits, so appearance barely separates them. Motion models usually assume smooth movement, but footballers accelerate, stop, and turn sharply, especially in exactly the moments that matter most: duels, set pieces, and goalmouth scrambles. And when one player is hidden behind another, the detector may return one merged box or none at all, leaving the tracker to guess.

What tends to help

• Better appearance features. Re-identification models trained to tell people apart can use cues beyond kit colour, such as build, gait, and boots. Shirt numbers help when they are visible, but they often aren't.
• Keeping lost tracks alive longer. If a track isn't deleted the moment its player disappears, it can be reconnected when they reappear, instead of starting a new ID.
• Offline post-processing. For analysis after a match, as opposed to live use, the whole video is available. That allows linking track fragments using both past and future frames, which is much more forgiving than deciding frame by frame.
• Team assignment as a constraint. Grouping players by kit colour won't separate teammates, but it rules out swaps between opponents.

Failure 2: Small or hidden balls are missed

What you see

The ball's path has gaps. It disappears for a few frames during a long pass, vanishes inside a crowd of legs, or gets confused with a white boot or a line marking.

Why it happens

In a wide broadcast shot, the ball may be only a handful of pixels across. At speed it smears into a blurred streak that no longer looks like a ball. It is hidden by players far more often than any player is hidden by anything else. And unlike people, it has almost no distinctive texture to latch onto. Detectors handle small objects worst, and the ball is the smallest and fastest object on the pitch.

What tends to help

• Higher-resolution input, or tiling. Running the detector on crops of the frame instead of a downscaled whole frame keeps the ball large enough to detect.
• Physics-aware interpolation. A ball in flight follows a fairly predictable path. Short gaps can be filled in by fitting a trajectory between confident detections, as long as the filled-in frames are marked as estimated rather than observed.
• Context. The ball is usually near a player and moves differently from them. Using that context helps reject false detections like boots and markings.

Failure 3: Pitch mapping breaks on partial views

What you see

On wide shots, positions on the top-down pitch look sensible. When the camera zooms in on a tight piece of play, mapped positions start to drift, jitter, or jump: players appear to move several metres between frames while standing still.

Why it happens

A homography is estimated by matching known points on the real pitch, such as line intersections, the corners of the penalty box, and the centre circle, to where they appear in the image. With many well-spread points visible, the estimate is stable. On a close-up, only one or two line segments may be visible, and they may all sit in one corner of the frame. Small errors in locating those few points then turn into large errors across the rest of the pitch.

Two more things make this worse. Broadcast cameras pan and zoom constantly, so the mapping has to be re-estimated all the time. And a homography assumes the pitch is a flat plane seen through an ideal lens, which real camera lenses only approximate.

What tends to help

• Keypoint detection models trained to find pitch landmarks directly, which work better than classical line detection on cluttered or partial views.
• Temporal smoothing. The camera moves smoothly, so the mapping should too. Filtering the homography over time removes most of the frame-to-frame jitter.
• Knowing when not to trust it. When too few landmarks are visible, the honest output is "position unknown," not a confident but wrong coordinate.

Why this matters downstream

These failures matter because of what is built on top of them. Consider a few common outputs:

• Distance covered is inflated by identity switches, because a swap makes a player seem to teleport across the pitch, adding distance they never ran.
• Heatmaps are skewed by fragmentation, because one player's time on the ball is split across several IDs.
• Possession and passing statistics depend on the ball track, so every missed ball creates an uncertain gap.
• Formation analysis inherits every error in the pitch mapping.

None of these outputs will look wrong. A heatmap built on broken tracks is just as colourful as a correct one. That is the real danger of computer vision analytics: the errors are invisible in the final product.

What we take from this

The lesson isn't that football tracking doesn't work. It works well on a large share of footage, and it keeps improving. The lesson is about where to put effort:

1. Measure each stage separately on real footage that includes the hard cases, not just the clips that look good.
2. Track how errors propagate from detection to tracking to statistics, instead of only checking the final numbers.
3. Show uncertainty in the product. Mark interpolated positions as estimated, flag low-confidence pitch mapping, and say which statistics are reliable and which aren't.

This applies far beyond football. Warehouses, retail floors, traffic junctions, and production lines all have occlusion, crowded scenes, and awkward camera angles. If you're building analytics on top of video, the question to ask isn't "does the demo look good?" but "how wrong is it on the footage we'll actually have, and would we notice?"