Oct. 27, 2026

Question: Can AI extract and aggregate values from documents?

ANSWER: Yes and no.

Today, I found myself creating a report summarizing the results of several training evaluations. Specifically, I needed to calculate average ratings on several dimensions across 10 workshops.

Each workshop report was a PDF detailing various numeric and qualitative data, including mean ratings for each question asked on the evaluation. So I simply needed to aggregate those means.

However, as there were different numbers of responses to each question and to each survey overall, I needed to weight the means when aggregating them. If I were to do it manually, the first step would be to look through each report, taking note of every mean and response count (>100 datapoints to enter into a spreadsheet by hand!).

I thought to myself, “Hey! This sounds pretty tedious and error prone. This is a great opportunity to let AI do the grunt work for me.”

So I dumped the 10 de-identified reports into Claude and instructed it to extract the means for the target questions, then calculate overall means for each, taking into account the numbers of responses.

Next, I thought, “Hmm. I don’t trust Claude to do this correctly. Let’s triangulate with another AI.”

So then I dumped the same reports into Gemini and gave it the exact same instructions. If they both produced the same results to several significant digits, it was highly likely that they were accurate.

(That said, I would still have followed up with random spot-checks to be extra safe.)

So imagine my dismay when they both worked for ~20 seconds and then spat out DIFFERENT RESULTS.

Not great. Not great at all.

Now, because I couldn’t trust the output of either one, that meant I had to do the entire thing myself anyway!

So I did.

And my results were identical to Claude’s output. I still don’t know what Gemini did wrong, because I never investigated.

In some ways, it may seem like this defeats the purpose of using AI in the first place…

If you still have to do the work anyway to verify that it’s correct, then why include the AI at all?

And the answer is: triangulation.

People who value precision already naturally tend to double-, triple-, and quadruple-check things like data entry. (That’s how some data entry specialists have accuracy levels as high as 100% for long stretches of time. It’s not that they make fewer mistakes, necessarily—it’s just that they catch them before finalizing.)

But checking is the kind of thing that can be conveniently offloaded to AI with high confidence.

This allows you to get away with being a bit speedy and reckless. You can plow forward, confident that if your output matches the AI’s, then you almost certainly both did it correctly.

(The probability that you and a separate process independently make the same set of mistakes while completing a sufficiently complex task is vanishingly small. And the more separate processes you involve, the lower that probability.)

So…

Can AIs handle tasks like extracting average ratings and then weighting them and aggregating across 10 different summary reports?

Well, Claude can.

But you should still check its work.