How AI Video Analysis Actually Works (Without the Hype)

"AI" gets used as a synonym for magic often enough that it's fair to be suspicious of any product that leads with it. So here's the actual pipeline behind a PaddleLens report — no metaphors, no black box — because a coach deciding whether to trust a number should be able to see where it came from.

There are five steps, and only the last one involves anything resembling a large language model.

Step 1: finding the paddler

The first job is boring and mechanical: given a video frame, find the person you care about. An object-detection model scans for people, and once you've confirmed which one is your paddler, a tracking algorithm locks onto that person's appearance and follows them frame to frame, re-checking against a fresh detection about once a second to make sure it hasn't drifted.

That's the honest version — not "it can never make a mistake." In a crowded boat, where bodies overlap and someone's shoulder passes in front of your paddler's, the tracker can lose its target and re-acquire the wrong person. There are guards against it (a jump too large to be real is rejected, and frames where the pose drifts onto a neighbour get counted and discarded), but mistakes remain possible. The report's tracking-coverage percentage tells you how much of the sampled clip had a usable track; it is not an accuracy guarantee. The feedback box on every report therefore includes an "it tracked the paddler behind me" option.

This step is where bad footage announces itself. If the paddler is small, blocked, or the camera is bouncing, everything downstream inherits that — which is the whole reason our filming guide exists.

Step 2: reading the body

Once the paddler is isolated, a pose-estimation model reads their skeleton. Two models do this job depending on the footage: one built for a single paddler with the frame to themselves, another that holds up better when there are several bodies in frame. They're the same category of technology used in fitness apps and sports broadcasts, pointed at a boat instead of a gym.

Whichever model runs, the app uses the same relevant body points — head, both shoulders, elbows, wrists, hips and knees — at the analysis detail selected for the clip, up to the supported sampling ceiling. The output at this stage isn't a coaching insight yet; it's a timeline of visible joint positions with holes where the evidence does not pass the model's gate.

Step 3: turning positions into a stroke

Raw joint positions aren't useful on their own. A stroke-cycle engine infers turning points from the selected hand's motion, then derives view-supported measures: projected arm posture near the inferred catch, projected torso lean, the hand-to-hand shaft proxy, side-view drive-to-recovery timing, stroke rate, and how much that rate wanders from stroke to stroke. It does not directly see blade-water contact or paddle force.

Those map onto the measurements and coaching observations we've written about elsewhere, with one important boundary: true trunk rotation needs defensible depth evidence that ordinary monocular phone footage does not provide. PaddleLens has retired that reading rather than present an estimate as a measurement.

This step is geometry and signal processing, not generative text. It operates on projected 2D phone-video observations; it is not a substitute for a researcher's calibrated 3D motion-capture setup.

Step 4: scoring, in code

Here's the part most "AI analysis" products don't have, and the reason a report is worth arguing with.

Your numeric scores are not written by the language model. Formula-backed criteria are computed arithmetically from supported measurements against target bands — a platform reference, or the specific target your coach set. Feed the same stored inputs into the same formula version and you get the same score. Criteria without a defensible numeric formula are written up without an invented mark.

Those computed scores are then handed to the last step as fixed values it isn't allowed to contradict.

Step 5: writing the report

Only now does a language model get involved — and only with text, never the video, never a single image from it. The measurements, scores, available criteria and target settings, the paddler's first name and any notes the coach wrote about them go in; a plain-language report comes out, built to lead with one practical focus instead of a wall of statistics.

The language model's job is to turn the supplied text into a coaching explanation and practical cue. The numeric measurements and deterministic scores are fixed inputs, but generated prose can still be wrong or overstate what they mean. Review it against the annotated frames and treat the coach's judgment as the final layer.

What this pipeline can't do

Being honest about the limits matters as much as explaining the mechanism:

It struggles with crowded shots. A full crew boat filmed from a distance means overlapping bodies, and both detection and tracking get worse when people occlude each other. This is a real limitation, not a marketing footnote. Every PaddleLens report states tracking coverage so missing evidence is visible, but high coverage by itself does not prove that every metric is accurate.

Some measurements need the right footage to exist at all. Forward hinge and fore-aft shaft angle need a side view; side-to-side lean belongs to the front view; true trunk rotation is not reported from either as a precise degree value. A report that stays silent or leaves a criterion unscored may be the system refusing to make a number up.

It has no opinion on strategy, chemistry or effort. The pipeline can tell you a paddler's rhythm degrades in the back half of a piece. It has no idea whether that's fatigue, distraction, or a bad night's sleep, and it can't tell you whether to bench someone for a race. That's still entirely a coach's judgment call.

It only ever sees what the camera saw. Bad light, a shaky handheld shot, or a paddler filmed from the wrong angle doesn't get fixed downstream — it can produce missing, approximate or occasionally wrong observations. Good footage isn't a nice-to-have; it's the ceiling on what any of this can tell you.

Why explain all of this

Mostly because "trust the AI" is a bad ask on its own, and "here's exactly what happened to your video, in five boring, checkable steps" is a much better one. None of those steps is exotic technology by itself — detection, tracking, pose estimation, some signal processing and a language model are all things you can read about elsewhere in plenty of technical detail. What PaddleLens actually does is chain them together specifically for a paddling stroke, keep the judgement in arithmetic rather than in the model, and hand a coach the output in language they'd use themselves.

If a report ever tells you something that doesn't match what your eyes see on the replay, investigate it — check tracking coverage, the annotated frames, the camera view and the footage against the filming guide. The numbers aren't infallible. They are repeatable outputs from a defined code path only when the inputs, method and setup are comparable.