Do LLMs actually do the math? I built a benchmark to find out

Ask a language model a numbers question and it'll answer with total confidence. The problem is it sounds exactly that confident when it's wrong. If you're going to hand one a financial task, you want to know where it breaks, not just a single accuracy percentage.

What FinEval does

It's a benchmark of 1,000+ questions covering numerical reasoning, multi-step problems, and plain factual recall. Every answer gets graded automatically by an LLM-as-a-judge that doesn't just mark it right or wrong — it says why. That's the part that actually helps, because it turns a score into a list of failure patterns you can look at.

The tricky bit

Grading reasoning is harder than grading answers. A model can get the right number with completely broken logic, or 'fail' a question because it formatted the answer differently than expected. Most of my time went into the judge prompts and the question set so the scores reflect real reasoning quality instead of noise. On top there's a dashboard for comparing models and reading the failures one by one.

If you're about to trust a model with anything where the math matters, run it through something like this first.