Auto-labeling video at scale with SAM + YOLO

Labeling video by hand is slow, expensive, and gets worse the more data you have. We were about to hire five more annotators just to keep up. Instead I built a pipeline that does most of the work automatically.

How it works

The idea is to let two models do the heavy lifting. SAM handles segmentation — drawing the masks around objects — and YOLO handles detection, figuring out what's in the frame. The pipeline runs both across the video and only sends the tricky, uncertain frames to a human for review.

So people stop drawing boxes all day and start doing what they're actually good at: catching the edge cases the models aren't sure about.

The payoff

It meant we didn't need those five extra annotators — about $2K a month, roughly $24K a year — and dataset throughput went way up at the same time. Same team, a lot more labeled data.