On the 1080ti I run gemma4:e4b to to do RAG and vision jobs. Not sure the quants and I just run it in Ollama that auto adjust the model because I’m lazy. Not 100% sure but I think mixture of experts load the parts that are needed so there easier on memory. Offloading to CPU memory or swap is a big performance hit.
I tried to run a 300b model on the main PC once, but it took half an hour to get a few words out of it. You can go past the VRAM on the card but sacrifice speed.
On the 1080ti I run gemma4:e4b to to do RAG and vision jobs. Not sure the quants and I just run it in Ollama that auto adjust the model because I’m lazy. Not 100% sure but I think mixture of experts load the parts that are needed so there easier on memory. Offloading to CPU memory or swap is a big performance hit.
I tried to run a 300b model on the main PC once, but it took half an hour to get a few words out of it. You can go past the VRAM on the card but sacrifice speed.