FinEval: checking whether LLMs can actually do the math

Business problem

Language models sound just as confident when they're wrong about numbers as when they're right. If you're going to trust one with financial reasoning, an overall accuracy score isn't enough — you need to know where it slips and why.

System architecture

FinEval is a Python and Flask backend that runs models through a fixed set of 1,000+ questions, plus an LLM-as-a-judge layer that grades each answer and explains the verdict. A Next.js dashboard sits on top for exploring the results.

Challenges

Grading reasoning is harder than grading answers. A model can land on the right number through the wrong logic, or fail a question for a formatting reason that isn't really a mistake. Most of the work went into tuning the judge prompts and the question set so the scores actually reflect reasoning quality, not surface noise.

Implementation

1,000+ questions across numerical reasoning, multi-step problems and factual recall; an automated judging pipeline that catches failure patterns; and a dashboard for comparing models, filtering by question type, and reading individual failures.

Results

A repeatable way to benchmark a model's financial reasoning and see exactly where it breaks — the kind of thing worth running before you trust any model in a workflow where the numbers matter.