If you liked this piece, please subscribe to my premium newsletter. It’s $70 a year, or $7 a month, and in return you get a weekly newsletter that’s usually anywhere from 5,000 to 18,000 words, including vast, detailed analyses of NVIDIA, Anthropic and OpenAI’s finances,
The economics for daily use of inference seem to make sense. The cost of inference is highly profitable. The margins on inference around 80%. The lost money from power users is made up but the average user who doesn’t user their tokens. They lose money on the free inference given away but that’s marketing and getting used to people having the product there as a crutch. It’s not the best business model but they can change it at any time and have vc cash to burn.
What doesn’t make sense is recouping the investment cost of model training and building new data centers. Because the moat on a new model doesn’t last long enough to recover its training cost.
Do you have a reliable source for this information? I’ve only ever heard numbers like this directly from the AI companies themselves.
I don’t think the current phase is about sensible business models, it’s about jockeying for position to control the power of the new thing. The people doing the core investment have more money than they will ever need - this is a play to turn that money into more power than it currently represents for them - get in on the ground floor - shape the landscape - help form the regulations and relationships that will propel them up the next rung of the ladder.
It’s a bit like spending $44B for a social media platform then running it into the ground financially - it’s not about the money, it’s about the things you get in exchange for that “waste” of money.
They’re kinda past that phase and now need to show that they have sustainable revenue and user growth. From all the numbers I’ve seen they(open ai, Gemini, anthropic) have crazy numbers. Hundreds of millions of users paying $50 a month. It’s not enough to cover training but it covers inference very nicely.
Then with agent bullshit they’ve managed to turn 1 prompt into 12 and bill the user for that extra so it’s even more profitable than the monthly subscriptions.
If it weren’t for the massive silicon supply lockdown, I feel like we could easily see local models making it into consumer tech in the coming years and effectively replace all those casual users since you no longer have to pay a subscription to do regular/low effort tasks on whatever device you own. A lot of it has gotten really good, especially with lots of quantization techniques getting superseded by new ones each year.
Actually I guess it could probably go the same way as cable and streaming. Eventually they’ll keep amping up the ante with the billing (because they always do), and people will just get turned off into a bunch of “cheaper” 3rd parties that have lower costs with some niche tricks, which will fragment the userbase too much.
Also I haven’t looked into it, but do they advertise those $50 users separately from enterprise? I don’t really know anyone outside of “power” users that aren’t just using the $20 a month basic plans that give you enough tokens to get by (for now).
I feel like they’re inflating their numbers from enterprise estimates because that’s where they can bait with cheap API prices and then hook with vendor lock in.
If that’s true, you make a good point. Sounds like they could keep the business model around inference. They’d just need to figure out how to make up the budget for training, which I imagine could be done with good marketing on new releases. As well as finding ways to use more tokens, injecting ads, selling data, investors, … did I miss anything? It doesn’t sound impossible if the inference portion is as lucrative as you say.
That’s a hell of a lot better than .com had back in the day: hundreds of millions of “hits” on their websites, with no proof if there was even a human connected to the request.
I think there’s a whole lot of variability of user experience out there still, and that’s some of what is getting shaken out of the systems - new models are better for some, worse for others. Overall, I think they are still improving, quite dramatically for software creation in the past 12 months, but as they grow in their specialty skills, some of the users who were getting better results for other things do get hurt in the process.
If there ever is such a thing as GenAI, I suspect it will follow the medical model of your General Practitioner referring you to specialists as warranted.