A founder running an AI writing assistant told me his "software costs" were $40,000 a month. When we broke the number down, $34,000 of it was OpenAI and Anthropic API bills — and none of it was in his cost of goods sold. It was sitting in a general "Software & Subscriptions" line right next to his $19-a-month project management tool. His reported gross margin was 91%. His real gross margin, once inference was moved where it belonged, was 58%.
That's not a bookkeeping technicality. It's the difference between a business that looks like software and a business that behaves like a factory — one where every unit sold consumes a real, variable input. If you're building on top of GPT-5, Claude, Gemini, or any hosted model, your LLM bill isn't overhead. It's cost of goods sold, and treating it otherwise means you don't actually know how much money you make.
Why This Mistake Is So Easy to Make
Traditional SaaS trained a generation of founders to think of software costs as fixed. You pay for hosting, you pay for your CRM, you pay for Slack — and none of those bills move in lockstep with how much your product gets used by any one customer. So when the first API invoice from OpenAI arrives, it's natural to file it in the same mental bucket: "tools we pay for to run the business."
The problem is that an LLM API bill doesn't behave like a SaaS subscription. It behaves like a raw material. Every time a customer sends a prompt, you consume tokens, and every token has a price. A customer who runs 50 requests a day costs you meaningfully more to serve than one who runs 2. That's the textbook definition of cost of goods sold: a cost that scales directly with the delivery of your product to a specific customer, as opposed to a cost you'd pay just to keep the lights on regardless of usage.
Industry data backs up how material this has become. ICONIQ's January 2026 State of AI report found inference now averages 23% of total revenue at scaling-stage AI B2B companies, and 84% of those companies reported gross-margin erosion of six percentage points or more directly attributable to AI infrastructure costs. This isn't a rounding error you can absorb into "misc opex." It's often the single largest cost driver in the business, and it deserves to be treated as one.
The COGS vs. OpEx Line, Drawn Correctly
The rule that separates COGS from operating expense hasn't changed just because the input is a model API instead of a warehouse full of parts: costs to deliver your product to a paying customer belong in COGS; costs to build your product belong in R&D under OpEx. Applied to an AI wrapper, that split looks like this:
Belongs in COGS:
- Production inference costs — every API call your live product makes on behalf of a paying customer, whether that's a chat completion, an embedding call, a classification pass, or an agent tool-call loop
- Model hosting / GPU compute — if you're running an open-weight model on your own or rented infrastructure instead of calling a hosted API, the compute time attributable to serving customer requests
- Supporting inference infrastructure — vector database queries, embedding pipeline runs, and orchestration overhead that exist specifically to serve production requests
- AI-specific support costs — a support engineer whose job is triaging model-output issues for customers is a cost of delivering the product, not a general SG&A expense
Belongs in OpEx (usually R&D):
- Development and test inference — every prompt an engineer sends while iterating on a feature, debugging a regression, or evaluating a new model version
- Fine-tuning and evaluation runs — building or improving the model is R&D, not delivery
- Internal AI tools — your team's ChatGPT Enterprise seats or GitHub Copilot licenses are a productivity expense, not a cost of serving customers
The practical step that makes this split possible is boring but essential: use separate API keys or billing projects for production and development traffic from day one. If your engineers are testing prompts against the same key your product uses in production, you have no way to allocate the bill correctly after the fact, and you'll end up either overstating or understating your real margin every single month.
What "Real" Gross Margin Looks Like for AI Products
Once you've made the split, expect the number that comes out to look different from what you're used to. Traditional SaaS businesses target 70–80% gross margins almost as a matter of course. AI-native products don't get there the same way. Bessemer Venture Partners' February 2026 pricing playbook puts typical AI product gross margins at 50–60%, well below the 80–90% range for traditional SaaS, and ICONIQ's data shows the industry average AI product gross margin climbing — but only to 52%, up from 41% in 2024 and 45% in 2025.
A useful rule of thumb that's emerged from operators managing this closely: keep LLM costs under roughly 20% of total COGS if you want a durable, scalable business model. Cross that threshold and margin compression tends to accelerate as you grow rather than ease off, because bigger customers use the product more, not less. Some AI-heavy categories — coding assistants, document-processing agents — run LLM cost ratios of 30–40% of COGS and still remain viable, but only because they've priced deliberately around that reality rather than backing into it.
The number that actually matters isn't your blended, company-wide gross margin — it's cost per customer. Heavy users of an AI feature can cost 50 to 100 times more to serve in raw inference than light users on the identical subscription tier. If you're not tracking that, you don't know which customers are profitable and which ones you're quietly subsidizing every month. A flat-rate plan that looked fine at your first ten customers can start losing money the moment one of them pushes a workflow into production and multiplies their request volume by 20x.
Building the Per-Request Cost Model
You don't need elaborate tooling to start — you need discipline about logging the right numbers on every request. The core formula is simple:
request_cost = (input_tokens / 1,000,000) × input_price_per_million
+ (output_tokens / 1,000,000) × output_price_per_millionOutput tokens typically cost 2–5x more than input tokens per unit, because generation is more compute-intensive than reading a prompt — so a request-cost model that only tracks total tokens without separating input from output will systematically misprice anything with long completions. As a working example at mid-2026 pricing: a request with 2,000 input tokens and a 500-token reply against a mainstream mid-tier model (roughly $3 input / $15 output per million tokens) costs about $0.0135. Run that same shape of request 50,000 times a month and you're looking around $675 in inference cost for that single feature — a number that's invisible if it's buried in a five-figure "cloud costs" line but very visible once it's tagged to the feature and the customer segment that generated it.
To make this actionable:
- Log token counts on every request, not just cost — most model providers return input/output token counts in the API response itself, so this is a logging change, not a new data source.
- Tag each request with a customer ID and feature identifier so you can roll costs up by account and by product surface, not just by month.
- Compute cost per customer per billing period and compare it against what that customer pays you. This is the number that tells you whether your pricing tiers actually match your delivery cost, not your blended average margin.
- Route to cheaper models where quality allows. Not every request needs your most capable (and most expensive) model — classification, extraction, and simple formatting tasks often perform acceptably on a smaller, cheaper model, and model routing is one of the few margin levers you fully control, unlike provider pricing.
Pricing to Protect the Margin You Just Found
Once inference is correctly classified as COGS, your pricing conversation changes. A common anchor operators use is charging at roughly 3–5x your per-unit token cost, so that a $0.30 task is priced at $1.00 or more — not because that multiple is magic, but because it leaves room to absorb usage variance, model-price volatility, and the support and infrastructure costs that ride alongside raw inference.
Usage-based billing has become the standard way AI companies keep pricing and COGS moving together: charge per token, per request, or per completed task, and your revenue scales with the same variable that drives your cost, instead of drifting apart from it the way a flat subscription does when usage spikes. If you're not ready to move fully to usage-based pricing, at minimum build committed-minimum-plus-overage tiers so a customer who blows past typical usage doesn't quietly become a loss leader for the rest of the month.
Keep Your Cost Structure as Auditable as Your Code
None of this is optional bookkeeping hygiene — it's the difference between knowing your business model works and finding out it doesn't during a fundraising due-diligence call, when an investor's associate builds the per-customer cost model you should have built six months earlier. The founders who avoid that conversation are the ones who tag inference costs to customers and features from the first API call, not the ones who wait until the OpenAI bill is too big to ignore.
Beancount.io gives you plain-text, version-controlled accounting that makes it straightforward to tag inference spend to the right account, split production from development API usage, and reconcile your real COGS against revenue every month — no black-box categorization, no vendor lock-in. Get started for free and see why developers and AI-native founders are switching to plain-text accounting.