Opportunity

Reduce LLM token costs by preprocessing documents

The PainHunt Team · August 10, 2026 · 4 min read

TL;DR: People complaining about LLM costs mostly aren't complaining about the model being expensive — they're paying a document tax they can't see. A PDF becomes thousands of tokens before a question is asked, there's no built-in way to know or cut that cost, and cheaper paths exist that users don't find until later. Across 217 recent threads in PainHunt, the unmet need is narrow: an ingestion-time layer that shows a document's token cost and reduces it without quietly dropping what mattered.

The evidence

PainHunt holds 217 posts on AI cost pain scoring 10 or higher out of 15, average score 11.7/15, with an average pain intensity of 7.4/10 across the underlying complaints. What stands out is the recency: all 217 are from the last six months — this is a pain that grew with document-heavy LLM use, not a legacy gripe. The discussion is led by Mastodon (124 posts), with Medium (43) and BlueSky (37) behind it and a long tail across Dev.to, Substack and Lemmy — the places developers and heavy AI users compare notes rather than app-store reviewers.

Two clusters recur, and only one of them is about price.

The document is the cost, and the cost is invisible. Operators describe uploading a PDF and burning thousands of tokens before asking anything, no built-in optimization to reduce a document's token footprint before it's processed, and — tellingly — not knowing that free ways to cut token consumption on uploads even exist. The cost isn't the question; it's the price of admission for the document to be in the context at all.

Cost is unpredictable and surfaces too late. Beyond documents, the same threads describe token bills high enough that users spend real effort avoiding them, silent model migrations that raise the per-call price without consent, and cost warnings that land in a promotions folder and get missed. The through-line is timing: the bad news always arrives after the money is spent.

The pattern underneath both: the expensive moment is ingestion, and nothing in the default workflow makes it visible before it happens.

Why now

Document Q&A is the default use, and PDF is the default input. Feeding a whole report, contract or manual into a model is now routine, and the file format that dominates that flow is the one that tokenizes worst.

Per-token pricing makes long documents non-linear. A short chat is cheap; a 40-page PDF asked three questions is not. As the input grows, the cost of simply having the document present dominates the cost of the reasoning — and that ratio is exactly what users have no dashboard for.

Bigger context windows encourage bigger inputs. The move to long-context models quietly rewards "just paste the whole thing," which amplifies the document tax rather than removing it. The capability arrived; the cost visibility didn't.

The wedge

The broad build is "an AI cost dashboard." The threads point at something narrower and earlier in the pipeline.

  • A preprocessing layer that shows the token cost before you send. Strip the parts that tokenize badly — image tiles, scanned-page OCR noise, repeated headers and footers, boilerplate, irrelevant pages — and surface the token budget of a document before it hits the model. The product is the number you didn't have, not another chart after the fact.
  • Fidelity as a first-class control, not a side effect. The reason this isn't already solved by a shell script is that stripping too hard breaks answers. Making the reduction adjustable and showing what was removed turns a risky optimization into a deliberate cost-versus-accuracy dial.
  • Price against the tokens saved. The buyer is an API-heavy developer or an AI product team with a real, recurring bill. "We cut your document tokens by N% and show you the trade-off" is a sharper pitch than a generic FinOps tier, and it maps directly to money.

Risks and honest caveats

  • The model vendors can absorb this. Anthropic and OpenAI could add ingestion-time document optimization natively, which would erode a thin wrapper. The defensible ground is cross-provider visibility and the fidelity controls, not one format trick.
  • Conversion is already commoditized. unstructured, markitdown and pandoc convert files for free. If the pitch is "we turn PDFs into text," there's no business — the value has to be the token-budget visibility and the accuracy-preserving reduction, which those tools don't provide.
  • Aggressive reduction is a correctness risk. Cutting tokens can cut context the model needed. Any honest version of this ships with "here's what we removed" and a way to dial it back, or it trades a cost problem for a quality problem.
  • The buyer is technical and price-sensitive. These are developers who can build a stripping script. The sale is convenience, visibility and a trustworthy trade-off — priced against the bill, not against a data-platform tier.

How to validate this further

Read the underlying AI-cost threads in the Pain Point Browser, and test which version of this — pure preprocessing, or preprocessing plus a cost dashboard — matches your reach with the Idea Validator. Two adjacent clusters are worth reading alongside it: governing shadow AI spend across a team, which is the same money problem one level up, and hard spending caps that stop bill shock, which is the "surfaced too late" failure in a different domain.

Frequently asked questions

Why does uploading a PDF to an LLM cost so many tokens?

A PDF is converted to text (and often to image tiles) before the model reads it, so headers, footers, repeated boilerplate, scanned-page noise and layout artifacts all become tokens you pay for — before you've asked a single question. A long report can cost thousands of tokens just to be present in the context, which is why people describe the bill arriving before the answer does.

Can't I just strip the document myself before sending it?

You can, and open-source tools like unstructured, markitdown and pandoc will convert files to text. The gap the threads describe isn't conversion — it's knowing the token cost of a document before you send it, and cutting it without silently dropping the parts you needed. That visibility and the accuracy trade-off are the unmet part, not the file conversion.

Will preprocessing a document hurt answer quality?

It can, and that's the honest trade-off: aggressive stripping saves tokens but can remove context the model needed. A useful tool makes the reduction adjustable and shows what was cut, so you trade cost for fidelity on purpose rather than discovering it in a wrong answer.

Validate your idea against real demand

PainHunt scores hundreds of thousands of real user complaints by commercial potential — so you build what people already want.

Open the Pain Point Browser

Keep reading

Reduce LLM token costs by preprocessing documents | PainHunt