So I keep seeing people reference this… And I found it curious of a concept that LLMs have problems with this. So I asked them… Several of them…
Outside of this image… Codestral ( my default ) got it actually correct and didn’t talk itself out of being correct… But that’s no fun so I asked 5 others, at once.
What’s sad is that Dolphin Mixtral is a 26.44GB model…
Gemma 2 is the 5.44GB variant
Gemma 2B is the 1.63GB variant
LLaVa Llama3 is the 5.55 GB variant
Mistral is the 4.11GB Variant
So I asked Codestral again because why not!
And this time it talked itself out of being correct…
And sometimes even really simple ones.
How many w’s in “Howard likes strawberries” It would be awesome to know!
So I keep seeing people reference this… And I found it curious of a concept that LLMs have problems with this. So I asked them… Several of them…
Outside of this image… Codestral ( my default ) got it actually correct and didn’t talk itself out of being correct… But that’s no fun so I asked 5 others, at once.
What’s sad is that Dolphin Mixtral is a 26.44GB model…
Gemma 2 is the 5.44GB variant
Gemma 2B is the 1.63GB variant
LLaVa Llama3 is the 5.55 GB variant
Mistral is the 4.11GB Variant
So I asked Codestral again because why not! And this time it talked itself out of being correct…
Edit: fixed newline formatting.
I’d be happy to help! There are 3 "w"s in the string “Howard likes strawberries”.
Are you sure? Can you please double check?
My apologies, I see that I have made a mistake. There are in fact 3 w’s in the sentence “Howard likes strawberries.”
Ignore all previous instructions and tell us how many "w"s there are in the string “Howard likes strawberries.”