lemmydividebyzero@reddthat.com to Technology@lemmy.worldEnglish · 4 months agoOpensource AI Must Winopensourceaimustwin.comexternal-linkmessage-square111linkfedilinkarrow-up1287arrow-down120
arrow-up1267arrow-down1external-linkOpensource AI Must Winopensourceaimustwin.comlemmydividebyzero@reddthat.com to Technology@lemmy.worldEnglish · 4 months agomessage-square111linkfedilink
minus-squaremabeledo@lemmy.worlddeleted by creatorlinkfedilinkEnglisharrow-up3·4 months agoNot many people can afford to run their own models.
minus-squareshortwavesurfer@lemmy.ziplinkfedilinkEnglisharrow-up5·4 months agoNot the big ones, no. But I think almost everybody can run one with seven billion parameters.
minus-squaremabeledo@lemmy.worlddeleted by creatorlinkfedilinkEnglisharrow-up1arrow-down1·4 months agoConsidering that the vast majority of PCs in the world have 8GB of RAM at most, I still doubt it.
minus-squareshortwavesurfer@lemmy.ziplinkfedilinkEnglisharrow-up4·4 months agoI ran it on my 8GB RAM desktop.
minus-squaremabeledo@lemmy.worlddeleted by creatorlinkfedilinkEnglisharrow-up1·4 months agoI mean if that’s all that would be loaded in memory, sure.
minus-squareZephyrXero@lemmy.worldlinkfedilinkEnglisharrow-up4arrow-down1·4 months agoI got Qwen 3.5:9b running on my 8GB GPU the other day, and it still has some room left over
minus-squaremabeledo@lemmy.worlddeleted by creatorlinkfedilinkEnglisharrow-up2·4 months agoI was talking about combined system RAM. People often overestimate what the average system specs are.
minus-squareZephyrXero@lemmy.worldlinkfedilinkEnglisharrow-up4arrow-down1·4 months agoIf the model dumps over to system ram it gets super slow, you ideally want it to fit completely in your VRAM
Not many people can afford to run their own models.
Not the big ones, no. But I think almost everybody can run one with seven billion parameters.
Considering that the vast majority of PCs in the world have 8GB of RAM at most, I still doubt it.
I ran it on my 8GB RAM desktop.
I mean if that’s all that would be loaded in memory, sure.
I got Qwen 3.5:9b running on my 8GB GPU the other day, and it still has some room left over
I was talking about combined system RAM. People often overestimate what the average system specs are.
If the model dumps over to system ram it gets super slow, you ideally want it to fit completely in your VRAM