If LLMs can’t be made more power efficient, then the technology isn’t ready yet for general use. There may have to be fundamental engineering breakthroughs before what these systems do can be more than a very expensive and inefficient worldwide proof of concept.
Of course, the military and the surveillance industry will still want to keep it, even if no one else can afford it, and we’ll be the ones paying for it.
They CAN be made more efficient. Hell, AMD just bought a company that translated LLMs to direct silicon… which should enable scalability if it wasn’t for one of the key players owning the tech thus limiting its use.
It’s also estimated that some 70% of Claude tokens gets spent on recursive self-correction so the output is higher quality. Instead of bettering the model, Anthropic just keeps throwing money at it to make the problem go away.
Meanwhile Chinese models - as small as 36-40-80b params - are outdoing these fancy frontier models if the right harness is used.
The truly silly (and simultaneously neat and fascinating, and to a large extent, very logical and obvious) part is that US attempts to cripple Chinese ML/LLM development by putting most advanced GPU tech behind ITAR has more or less made the Chinese ML industry just write straight up better and more optimized code for the domain.
US and western companies are currently relying on throwing bleeding edge hardware at the problem (read: very obviously brute-forcing the problem), whereas Chinese companies and engineers, having to rely on comparatively less advanced (though still highly capable) hardware has meant they’re staying competitive by optimizing the software layer more effectively.
This has very interesting and likely negative long-term implications for western engineering practices in a lot of ways. And it’s also very much not a new problem - simply throwing compute at things is a mindset that became highly prevalent in the west around when The Cloud became a thing. Previously, you couldn’t just spawn another mainframe in your company’s datacenter, but when it’s as simple as aws ec2 start-instances ..., the barrier to entry for getting more compute is so low that most people were just going “hey, it’s faster”. I feel like the engineering expertise around embedded/baremetal optimization has just been largely evaporating over the last couple decades; in the vast majority of professional software domains, it’s simply not a valued skill anymore (or, more correctly, at the moment). It’s sad for me in several ways - not least of which because I did my degree in EECS, and hardware/software interplay and optimization is an area of engineering that I have always found very interesting.
I’ve spent the last month or so using nothing but a local 27B Qwen model as my coding assistant and it’s working fine. The big American AI companies could all collapse in ruins tomorrow and it’s not going to slow things down all that much, anyone who wants to use AI will be able to continue using it. Might even be better in the long run if that happens - the people who are scared of ASI will get their much-demanded temporary slowdown at the frontier while everyone else catches up with open-weight models that everyone can use without central control.
The kind of specs that would enable local LLM usage were available on laptops since 2020. The floor for acceptable performance for LLMs stopped literally just before 24+gb of VRAM would have become pretty common in consumer applications. The reason this technology isn’t more ubiquitously available is because ‘they’ thought it would be more profitable to hoover up literally every byte of RAM and GPU in the world rather than let the technology continue to naturally evolve to a place where any consumer could have run useful models on their home computers. In other words, your anger is misplaced.
Probably Qwen Flash. Supposedly the 27b is nearly to capacity when it comes to training potential, while the Flash has still quite undercooked. Despite this, Flash is pretty dang good at understanding my goals and implementing my will, most of the time.
Open models will still be available via API from third-party hosts. The bigger ones like Kimi K3 or GLM 5.3 don’t fit on conventional desktop computers to begin with, most non-institutional users of those would already be doing that (eg via OpenRouter).
There’s not much skill to be had there - It is literally just downloading an app nowadays if you do not want to do anything special, especially on Linux (Newelle is great stuff!). But I am sure there are similar packages available for Windows too.
Excuse my ignorance - I don’t have much experience with AI to this point - but what does Newelle have that something like Unsloth or LMStudio does not? What makes it great stuff?
honestly? ease of use. I like Unsloth too, but Newelle is really an easy solution if you just want to use LLMs on you own hardware (or use inference providers) that requires practically no knowledge at all - it is focused on providing something to use LLMs without anything else interfering.
Unsloth does well too, but if i hadn’t learnt as much as i already did when i started using it, i would probably have been overwhelmed (outside of it crashing pathetically when following symlinks for local model selection - you can define these directories, but the quick selection then just makes it crash)
Wanted to refactor a too large and hectic hacked together state machine into separate functions and reduce redundancy and increase code readability
Every model I’ve tried failed.
And that should be an easy task an actual AI should be able to handle.
Logic is done, just translate to a different structure.
That should be exactly what they are able to do at least.
None worked, because the project is just to big and interlinked.
I’ve got 24GB of VRAM, which helps a lot. I’m using the Q4_K_M version of this model, with the KV cache quantized to Q8, and that’s able to get me to 128000 tokens of context. I’m using the Qwen Code harness in VS Code.
If 128000 tokens still isn’t enough, break the task down into smaller chunks and help the harness out with MCP tools - I installed Codebase Memory to help it navigate large codebases more efficiently, for example. Ask the coding agent what’s causing problems and it may be able to help identify solutions too, I’ve had it troubleshoot itself several times now and it’s been quite good at inspecting its own settings and finding where the problems are with them. I even installed Firecrawl for it so it can do its own web searches.
More Ram, and break stuff up into smaller pieces. I Have 12Gb of VRAM and 64 Gig System Ram, which - with CPU offloading - allows pretty large models to be used at acceptable speed (although probably not fast enough for someone who wants to make his living with it. But these also will invest in dedicated hardware). And i let a larger model sit there to coordinate, and let it use subagents for the rest. That keeps the Context for the main model a lot more manageable and keeps it on track for a pretty long time.
How you structure the Task is more important than what the task is at all. For example, i have a huge list of games to be catalogued, and just telling the model to search the web and fill in the blanks in the database makes it crap out pretty soon. But I tell it to let subagents handle everything regarding web search and data extraction and just let them hand back the resulting data. the main model then calls another subagent to sanity check the data and update the database. Each of these subagents do not run for long, and therefor never run out of context, while the main model can keep context size manageable because it does not have to work with “messy” data at all.
I run a 5 bit quantized Qwen 3.8 27b at home on a gaming PC with an RX 7900 XTX and RAG memory using Gemma MoE (i forget what one)running on an old 1080 ti on my Valheim server.
It takes 3-15 minutes to run depending on how many tool calls the model makes and that’s mostly memory or web scrapes since that runs on the slower hardware. I mostly just use it as a discord bot to look stuff up for me and to fetch the news.
1080 Ti? But that is 11 GB of VRAM? I know because that is my GPU and the reason I have not tried it. They all call for 12 GB VRAM. I have 64 GB RAM, so 50 GB for the model would be fine.
On the 1080ti I run gemma4:e4b to to do RAG and vision jobs. Not sure the quants and I just run it in Ollama that auto adjust the model because I’m lazy. Not 100% sure but I think mixture of experts load the parts that are needed so there easier on memory. Offloading to CPU memory or swap is a big performance hit.
I tried to run a 300b model on the main PC once, but it took half an hour to get a few words out of it. You can go past the VRAM on the card but sacrifice speed.
Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can’t handle it.
8B models are a bit too dumb to use any tools reliably but should handle a basic conversation. llama cpp will allow you to manually load a model and tweak settings.
If you’re talking about the company I think you are, that acquisition by AMD is interesting news.
The company was the one with a demo AI in their page that was super quick??
I always figured that was the future of AI when I saw it. Where you’d buy processors with n number of “AI cores” that’d have work/prompts scheduled as needed in the same way we now CPUs and RAM.
There may have to be fundamental engineering breakthroughs before what these systems do can be more than a very expensive and inefficient worldwide proof of concept.
Controversial thought after reading this sentence: It’s cryptocoins all over again.
Cryptocoins deliberately burn more energy doing useless calculations to prevent inflation. That’s a little bit of an oversimplification, but I don’t feel like explaining the full thing and the vibes are there.
That’d be fine. Cryptocurrency as a whole is still going strong (total market cap is around 3 trillion dollars) and most of the major ones have switched over to energy-free proof-of-stake validation.
Sadly most Crypto nowadays is really just for either speculation or buying things on the dark web. Wish there were more legitimate uses for it, although i am fine using it for the spicier things in life.
Depends. Is it solar energy that started production to power these models? Great! If not, then you’re still consuming power and the difference will have to be made somewhere, likely by burning more fossil fuels.
Oh yeah, I mean my own personal solar modules in my backyard. I agree it’s not that easy to say “solar exists, checkmate doubters”, that would be silly.
Well that is an alternative. But I bet each query to a local 27b model takes less energy anyway than the same query to Claude Opus or Fable in some data centre. So it is also a case of improving energy efficiency by running smaller models.
But without widespread usage there won’t be any breakthroughs… Anti-LLM folk could redirect their attention to the core problem (energy source, wind and solar shouldn’t raise any concerns) but they prefer to attack the LLMs themselves. Early implementations of every technology were crazy energy ineffective at the moment when becoming practically useful.
Anti-LLM folk don’t just have an issue with energy consumption. There’s also the issue of intellectual theft, studies that correlate worsened education with LLM usage, and the tech leadership silencing those issues by corrupting politicians. But I’m sure if they get past that they’ll understand how great LLMs are.
We understand LLM is great at something but we can never get passed the facts that you’ve stated and it also isn’t great in all thing like we’re being sold to and forced to look at everyday. If the tech stay with the professional and not given to the general public, if there’s no cunts telling everyone to adapt or get thrown out, if there’s no mass disruption toward people’s living, i’m sure it would be much more acceptable. The biggest issue with the tech is the cunt parading it.
No one should be arguing if LLMs are useful or worth investing into, they are and it can be a really useful tool, but the rich (seemingly spinless) lizards have managed to skyrocket the price of RAM, pushed global warming to new extrimes and jacked up electricity prices. Oh and while making an AI bubble so big and fradgile, that when (not if) popped will send us back to the great depression! All in the percute of “the next big thing”.
to;dr LLMs are a cool tech but in the percute of it have managed to fuck up a whole bunch
If LLMs can’t be made more power efficient, then the technology isn’t ready yet for general use. There may have to be fundamental engineering breakthroughs before what these systems do can be more than a very expensive and inefficient worldwide proof of concept.
Of course, the military and the surveillance industry will still want to keep it, even if no one else can afford it, and we’ll be the ones paying for it.
They CAN be made more efficient. Hell, AMD just bought a company that translated LLMs to direct silicon… which should enable scalability if it wasn’t for one of the key players owning the tech thus limiting its use.
It’s also estimated that some 70% of Claude tokens gets spent on recursive self-correction so the output is higher quality. Instead of bettering the model, Anthropic just keeps throwing money at it to make the problem go away.
Meanwhile Chinese models - as small as 36-40-80b params - are outdoing these fancy frontier models if the right harness is used.
The truly silly (and simultaneously neat and fascinating, and to a large extent, very logical and obvious) part is that US attempts to cripple Chinese ML/LLM development by putting most advanced GPU tech behind ITAR has more or less made the Chinese ML industry just write straight up better and more optimized code for the domain.
US and western companies are currently relying on throwing bleeding edge hardware at the problem (read: very obviously brute-forcing the problem), whereas Chinese companies and engineers, having to rely on comparatively less advanced (though still highly capable) hardware has meant they’re staying competitive by optimizing the software layer more effectively.
This has very interesting and likely negative long-term implications for western engineering practices in a lot of ways. And it’s also very much not a new problem - simply throwing compute at things is a mindset that became highly prevalent in the west around when The Cloud became a thing. Previously, you couldn’t just spawn another mainframe in your company’s datacenter, but when it’s as simple as
aws ec2 start-instances ..., the barrier to entry for getting more compute is so low that most people were just going “hey, it’s faster”. I feel like the engineering expertise around embedded/baremetal optimization has just been largely evaporating over the last couple decades; in the vast majority of professional software domains, it’s simply not a valued skill anymore (or, more correctly, at the moment). It’s sad for me in several ways - not least of which because I did my degree in EECS, and hardware/software interplay and optimization is an area of engineering that I have always found very interesting.I’ve spent the last month or so using nothing but a local 27B Qwen model as my coding assistant and it’s working fine. The big American AI companies could all collapse in ruins tomorrow and it’s not going to slow things down all that much, anyone who wants to use AI will be able to continue using it. Might even be better in the long run if that happens - the people who are scared of ASI will get their much-demanded temporary slowdown at the frontier while everyone else catches up with open-weight models that everyone can use without central control.
Pity about the economy in the meantime, I guess.
Anyone who’s privileged enough to have the hardware.
The kind of specs that would enable local LLM usage were available on laptops since 2020. The floor for acceptable performance for LLMs stopped literally just before 24+gb of VRAM would have become pretty common in consumer applications. The reason this technology isn’t more ubiquitously available is because ‘they’ thought it would be more profitable to hoover up literally every byte of RAM and GPU in the world rather than let the technology continue to naturally evolve to a place where any consumer could have run useful models on their home computers. In other words, your anger is misplaced.
Yeah the cross over point right now sis somewhere between qwen3.8 27b and glm 5.2/5.3
Like qwen is so close to being good.
Probably Qwen Flash. Supposedly the 27b is nearly to capacity when it comes to training potential, while the Flash has still quite undercooked. Despite this, Flash is pretty dang good at understanding my goals and implementing my will, most of the time.
What’s your assumption about where my anger is directed?
Open models will still be available via API from third-party hosts. The bigger ones like Kimi K3 or GLM 5.3 don’t fit on conventional desktop computers to begin with, most non-institutional users of those would already be doing that (eg via OpenRouter).
Anyone who lacks the skill to to it themselves…
There’s not much skill to be had there - It is literally just downloading an app nowadays if you do not want to do anything special, especially on Linux (Newelle is great stuff!). But I am sure there are similar packages available for Windows too.
Excuse my ignorance - I don’t have much experience with AI to this point - but what does Newelle have that something like Unsloth or LMStudio does not? What makes it great stuff?
honestly? ease of use. I like Unsloth too, but Newelle is really an easy solution if you just want to use LLMs on you own hardware (or use inference providers) that requires practically no knowledge at all - it is focused on providing something to use LLMs without anything else interfering.
Unsloth does well too, but if i hadn’t learnt as much as i already did when i started using it, i would probably have been overwhelmed (outside of it crashing pathetically when following symlinks for local model selection - you can define these directories, but the quick selection then just makes it crash)
Thanks, I appreciate you taking the time to answer.
I’m always running into context limit problems
Wanted to refactor a too large and hectic hacked together state machine into separate functions and reduce redundancy and increase code readability
Every model I’ve tried failed.
And that should be an easy task an actual AI should be able to handle. Logic is done, just translate to a different structure.
That should be exactly what they are able to do at least.
None worked, because the project is just to big and interlinked.
I’ve got 24GB of VRAM, which helps a lot. I’m using the Q4_K_M version of this model, with the KV cache quantized to Q8, and that’s able to get me to 128000 tokens of context. I’m using the Qwen Code harness in VS Code.
If 128000 tokens still isn’t enough, break the task down into smaller chunks and help the harness out with MCP tools - I installed Codebase Memory to help it navigate large codebases more efficiently, for example. Ask the coding agent what’s causing problems and it may be able to help identify solutions too, I’ve had it troubleshoot itself several times now and it’s been quite good at inspecting its own settings and finding where the problems are with them. I even installed Firecrawl for it so it can do its own web searches.
More Ram, and break stuff up into smaller pieces. I Have 12Gb of VRAM and 64 Gig System Ram, which - with CPU offloading - allows pretty large models to be used at acceptable speed (although probably not fast enough for someone who wants to make his living with it. But these also will invest in dedicated hardware). And i let a larger model sit there to coordinate, and let it use subagents for the rest. That keeps the Context for the main model a lot more manageable and keeps it on track for a pretty long time.
How you structure the Task is more important than what the task is at all. For example, i have a huge list of games to be catalogued, and just telling the model to search the web and fill in the blanks in the database makes it crap out pretty soon. But I tell it to let subagents handle everything regarding web search and data extraction and just let them hand back the resulting data. the main model then calls another subagent to sanity check the data and update the database. Each of these subagents do not run for long, and therefor never run out of context, while the main model can keep context size manageable because it does not have to work with “messy” data at all.
What kind of hardware do you need to run a model like that? And what do you use :-) ?
I run a 5 bit quantized Qwen 3.8 27b at home on a gaming PC with an RX 7900 XTX and RAG memory using Gemma MoE (i forget what one)running on an old 1080 ti on my Valheim server.
It takes 3-15 minutes to run depending on how many tool calls the model makes and that’s mostly memory or web scrapes since that runs on the slower hardware. I mostly just use it as a discord bot to look stuff up for me and to fetch the news.
1080 Ti? But that is 11 GB of VRAM? I know because that is my GPU and the reason I have not tried it. They all call for 12 GB VRAM. I have 64 GB RAM, so 50 GB for the model would be fine.
On the 1080ti I run gemma4:e4b to to do RAG and vision jobs. Not sure the quants and I just run it in Ollama that auto adjust the model because I’m lazy. Not 100% sure but I think mixture of experts load the parts that are needed so there easier on memory. Offloading to CPU memory or swap is a big performance hit.
I tried to run a 300b model on the main PC once, but it took half an hour to get a few words out of it. You can go past the VRAM on the card but sacrifice speed.
Sabata, goddess of computing, can I run it on a 32GB system with a 4GB nVidia GPU?
Probably not, or if it runs it may take hours or days to get a full output. VRAM on the GPU really decides what will run reasonably. Try an 8 billion parameter phone sized model and see if its faster than sending a letter, then keep moving up until your setup can’t handle it.
8B models are a bit too dumb to use any tools reliably but should handle a basic conversation.
llama cpp will allow you to manually load a model and tweak settings.
unsloth will give you a basic setup with some tools, manage model downloads, and settings.
Thank you for all the information 💕 !
deleted by creator
Is anyone trying to make LLMs with analog computing + memristors? That would make them absurdly efficient
I was working on an abacus-based LLM, but I ran out of beads.
Hello friend, I am building a new bead factory in Taiwan.
Just
givelend me your money and I will give you beads in the future sometime.I read the Wikipedia page for memristor but didn’t understand it. What’s it do?
lol, I wrote a reply about a company etching onto silicon and then go down and saw yours. Oops.
I hadn’t heard AMD bought them though. Looks like they’re working on a 20b model as well for this year.
If you’re talking about the company I think you are, that acquisition by AMD is interesting news.
The company was the one with a demo AI in their page that was super quick??
I always figured that was the future of AI when I saw it. Where you’d buy processors with n number of “AI cores” that’d have work/prompts scheduled as needed in the same way we now CPUs and RAM.
Controversial thought after reading this sentence: It’s cryptocoins all over again.
Cryptocoins deliberately burn more energy doing useless calculations to prevent inflation. That’s a little bit of an oversimplification, but I don’t feel like explaining the full thing and the vibes are there.
That’d be fine. Cryptocurrency as a whole is still going strong (total market cap is around 3 trillion dollars) and most of the major ones have switched over to energy-free proof-of-stake validation.
Sadly most Crypto nowadays is really just for either speculation or buying things on the dark web. Wish there were more legitimate uses for it, although i am fine using it for the spicier things in life.
Hm, I don’t see the harm in powering local models with solar.
Depends. Is it solar energy that started production to power these models? Great! If not, then you’re still consuming power and the difference will have to be made somewhere, likely by burning more fossil fuels.
Oh yeah, I mean my own personal solar modules in my backyard. I agree it’s not that easy to say “solar exists, checkmate doubters”, that would be silly.
OK, that is fine. I’ve seen a lot of people saying stuff like “Me mining crypto doesn’t matter because I live next to a wind farm” so I’m a bit wary.
Well that is an alternative. But I bet each query to a local 27b model takes less energy anyway than the same query to Claude Opus or Fable in some data centre. So it is also a case of improving energy efficiency by running smaller models.
someone else replied with what i said
But without widespread usage there won’t be any breakthroughs… Anti-LLM folk could redirect their attention to the core problem (energy source, wind and solar shouldn’t raise any concerns) but they prefer to attack the LLMs themselves. Early implementations of every technology were crazy energy ineffective at the moment when becoming practically useful.
Anti-LLM folk don’t just have an issue with energy consumption. There’s also the issue of intellectual theft, studies that correlate worsened education with LLM usage, and the tech leadership silencing those issues by corrupting politicians. But I’m sure if they get past that they’ll understand how great LLMs are.
We understand LLM is great at something but we can never get passed the facts that you’ve stated and it also isn’t great in all thing like we’re being sold to and forced to look at everyday. If the tech stay with the professional and not given to the general public, if there’s no cunts telling everyone to adapt or get thrown out, if there’s no mass disruption toward people’s living, i’m sure it would be much more acceptable. The biggest issue with the tech is the cunt parading it.
Oh no, we won’t get breakthroughs in slop.
Whatever will we do?
No one should be arguing if LLMs are useful or worth investing into, they are and it can be a really useful tool, but the rich (seemingly spinless) lizards have managed to skyrocket the price of RAM, pushed global warming to new extrimes and jacked up electricity prices. Oh and while making an AI bubble so big and fradgile, that when (not if) popped will send us back to the great depression! All in the percute of “the next big thing”.
to;dr LLMs are a cool tech but in the percute of it have managed to fuck up a whole bunch
pursuit*
LLMs in the context of capitalism will always be used against the working class.
All the more reason to seize the means of computing.
LLMs kill kids