TL;DR: The decision to self-host a language model is made on model quality and then paid for in serving economics, and the two are usually evaluated months apart. Across 540 high-scoring threads, engineers describe choosing a model they cannot afford to serve at acceptable quality, and having no reliable way to predict that before the hardware arrives. The missing artifact is a sizing answer that survives contact with a real workload.
The evidence
PainHunt holds 540 posts scoring 10 or higher out of 15 on self-hosted inference, GPU cost, and serving optimisation — the largest cluster in this batch — averaging 11.6/15 with a pain intensity of 7.2/10. Sources are dominated by long-form technical writing: Mastodon 170, Medium 144, Bluesky 47, Dev.to 36, Hacker News 33. These are people documenting what they learned, which makes the recurring gaps more informative than complaint volume would be.
Four points repeat.
The hardware number arrives late and large. Threads describe dedicated GPU estimates in the range of tens of thousands of dollars for a two-card setup, discovered after the model choice was effectively made. The sequencing is the problem: capability is evaluated first, serving cost second.
There is no single correct configuration. Optimising inference means balancing speed, memory usage, throughput, latency, and output quality simultaneously, and the threads are explicit that no one setting wins. Every choice trades against another, and the right point depends on a workload the team often hasn't characterised yet.
Expertise is the actual gate. Engineers without a machine-learning or hardware background describe struggling to choose and apply model-level techniques such as quantisation and batching. The techniques are documented; knowing which one applies to your situation is not.
Saving money can cost quality. Reducing memory footprint or raising throughput can degrade output or increase per-user latency. This makes cost reduction risky in a way that ordinary infrastructure tuning is not — the savings show up on a bill and the cost shows up in the product.
The summarising complaint is that the best model may be unusable if the team cannot afford the compute to serve it at acceptable quality. That is a planning failure, not a technical one.
Why now
Open-weight models became genuinely competitive. Self-hosting used to mean accepting a capability gap. For a widening set of tasks it no longer does, which moves the question from "can we?" to "what will it cost?" — and the second question has much worse tooling than the first.
Long context windows changed the arithmetic. Serving a 200K-token context is not the same workload as serving an 8K one on the same card. Memory reserved for a single long request can eliminate the concurrency that made the hardware pay for itself. Sizing guidance written for short-context serving quietly stopped applying.
Sovereignty and residency requirements became procurement blockers. Teams that would happily use a hosted API are being told they cannot, for regulatory or contractual reasons. They arrive at self-hosting without having chosen it, which means they arrive without the expertise that usually comes with the choice.
The wedge
The general build is an inference platform. That is a well-funded, crowded category and not the gap the threads describe. The unserved job is the decision that happens before any of it: what will this cost, at what quality, on what hardware.
- Answer the sizing question with measurements, not a model. Take the workload shape — model, context length, request rate, latency target — and return measured throughput and cost per million tokens on specific cards, from benchmarks you actually re-run. The value is in the freshness, which is also the moat.
- Publish the operating point with every number. Throughput at a stated concurrency and context length, not a headline figure. The threads are full of estimates that failed because the benchmark's assumptions were never stated, and being the source that always states them is a reputation that compounds.
- Compare against hosted honestly, including the cases where hosted wins. A tool that always concludes "self-host" is a vendor pitch. One that says "at your volume, an API is cheaper until roughly here" gets used for the decision, which is how it gets trusted for the next one.
- Sell the internal-defence artifact. The immediate job to be done is a document an engineer takes to finance to justify hardware. Package the output as that document — assumptions, measurements, sensitivity, break-even against hosted pricing.
- Grow into runtime advice from the same benchmarks. Once you know a customer's operating point, quantisation and batching recommendations are a natural second product. Start with the decision, not the platform.
Risks and honest caveats
- The data spoils fast. Runtime releases, new quantisation formats, and new hardware invalidate benchmarks continuously. This is a treadmill business: the moment you stop re-measuring, you are publishing folklore.
- Hardware vendors publish free numbers. They are optimistic and narrow, but they are free and they carry the brand. Competing means being trusted specifically because you are not selling the card, and trust takes longer to build than a product.
- The buying window is a single moment. Teams need this once, right before a purchase, and then not again for a year. That is a hard shape to monetise — it argues for a report or a consulting-adjacent offer rather than a subscription, and founders regularly get this wrong.
- Intensity is the lowest in this batch. At 7.2/10 this is a planning problem, not an outage. Planning problems get deferred, and deferred problems get solved with a colleague's spreadsheet.
- Benchmarking honestly is expensive. Doing this credibly means owning or renting the hardware you report on. That is real cost of goods for a company whose product is a number, and it is the reason the obvious competitors are blog posts rather than businesses.
How to validate this further
The question worth answering first is whether teams pay for the decision or only for the platform that follows it, because those are different companies. Use the PainHunt dashboard to filter infrastructure threads by intensity and read the ones that mention cost alongside quality — those are teams that already made the decision badly once and can tell you what they would have paid to avoid it. Then test whether a single sizing report sells on its own with idea validation.
Related reading: enterprise AI gateway with EU data residency, controlling AI coding costs with model routing.