Gamers and the general consumer electronics customers have been told for the past two years that GPUs cannot be offered at a reasonable price anymore due to unprecedented AI data center demand. Gl…
When ran on a local card, on my hardware at home, it will readily admit it was created by American companies.
You can rent a card in the cloud to try it for yourself. It’s fascinating.
The Chinese only know how to copy and scale, not innovate. Innovation comes from individualistic cultures, while scaling solutions come from communal cultures. It actually makes a lot of sense philosophically.
The Americans create it and the Chinese scale it, it’s been the same story for almost a century now.
Hot beef is right. That’s a good way to
phrase it.
And before you bring it up, I’ve been experimenting with local models since GPT-J and old GAN finetunes. I have MiMo 2.6 Flash loaded right now… TBH, I’m not sure how that’s relevant.
Are all of you seriously trying to argue that local experiments with model training data ARE NOT RELEVANT to our conversation about LLMs and their training data?
Am I losing my fucking mind here or is the hive mind that strong? I’m all ears buddy.
Hot beef is literally trying to argue this point. Are you? It is actually ridiculous.
Yeah. I think people wildly misunderstand how these things work, then want to argue about them. It’s crazy.
All I’m trying to say is that if I train AI on my mythical creature, the “Fijokopil” , a flying tiger ant, then the LLM will 100% be able to define a “Fijokopil”.
And, ChatGPT won’t know what a “Fijokopil” is.
I think I gotta take a break, it’s clear that my implications are going right over peoples heads.
I already know how to do the things I want to do. When I don’t know how to do something I want to do, I just teach myself. It’s not that difficult.
I too would choose not to engage in semantic arguments if I found myself occupying such an untenable position; Fortunately I don’t often need to run that mental equation, being of sound mind.
After going through the process I suggested, you will see that your assumption earlier :
“You do realize that, regardless of what output an LLM gives you, it doesn’t actually know anything right? ”
Is really incorrect. You can train it on obscure and weirdly specific ideas that you came up with, and it will know about them.
How do you think a model that is unplugged from the internet knows what Disney is? At some point, that word was in the training data.
It’s that simple!
If you applied even a tiny bit of critical thinking before replying, you would have figured that out yourself. And now that I’ve seen your lack of effort, and how disingenuous your responses are, it’s mute time baby!
Yeah except that’s not how LLMs work. It has no more insight into its training data than you do into the arrangement of your own neurons. They can all be manipulated into saying pretty much anything.
Chinese only know how to copy and scale, not innovate
Implying the data can’t be extracted is actually ridiculous. It’s time for you to study a bit more. I’ve been running and training these things in my basement for years, bro.
Then stop fucking around and explain how you get it to extract data that way so we can verify your claims. Otherwise you’re not better than a flatearther who goes “just do your own research”. You made the claim so you back it up. It’s not our job to figure out what magic you did.
…LLMs don’t “know” anything about their training. Beyond (sometimes) a stock greeting intentionally finetuned in.
I’ve brought this up dozens (hundreds) of times, for years, every time someone thinks they’ve identified a stealth model or identified some clue because they’ve gotten a model to “admit” it. It’s all just blind improvisation, it’s literally what they do.
If you want a more clear example, run Deepseek in Mikupad with the raw syntax, ask it your “meta” question, look at the logprobs for the critical decision token, then continue different answers from there, and it will confidently improvise any answer because it is roleplaying.
Or you could just consider what information the model itself has access to, and realize that kind of insight is physically impossible.
Do Chinese companies have a history of skirting IP? Sure. That’s literally history.
Did Deepseek (and other Chinese companies) harvest Claude answers for training data?
Oh yes. It’s obvious. They pulled Claude absolutely shamelessly.
If you want actual proof, see papers on it, or EQBench’s “slop family tree” graphs:
But other LLM companies around the world have no leg to stand on. They all train on each other’s output, they all train on the same legally questionable data anyway.
And I reiterate:
You can’t “investigate” an LLM by asking it for information. They just make stuff up. You can get any model to say it’s training data came from anywhere with the right initial seed, and many will ape Claude or ChatGPT unprompted because they trained on so much of it across the internet.
China did to US AI companies what US AI companies did to everyone else on the planet.
Pretty sure this was largely known almost from the jump?
Deepseek’s first public model was basically just a reverse engineered ChatGPT model with modifications.
I disagree with the rest though.
The Chinese only know how to copy and scale, not innovate.
China is so far ahead of actually automating and finding practical uses for AI tech that this is absurd.
Have you looked up their dark factories or how their vehicle tech and GPS is way ahead of the US?
China, ironically, is fostering a culture and market around AI tech that the US once built it’s initial advantages with. Which was move fast, open source(including when someone doesn’t want their IP open sourced), leverage state research that can be merged into private enterprise(see Google), and foster open competition chained to a egalitarian ethos of improving lives not just profits(even if this part was always largely bullshit once companies like Google stopped promising to do no evil and became prisoners to the line must go up problem).
Versus American AI that which is increasingly organized around enclosure, proprietary control, and rent extraction.
And your embrace of open weight AI models sort of suggests why China IMO is picking the right path and why I said what I said.
Ford’s CEO calls China’s EV dominance ‘the most humbling thing I have ever seen’
Ford CEO Jim Farley said China’s EVs are “far superior” to those made in the West
https://qz.com/fords-ceo-concedes-chinas-ev-dominance
And the vehicle tech just… isn’t. The vehicles don’t pass safety standards in some countries, mostly the world’s most developed nations are the ones that don’t have Chinese cars.
Everything from the 80s era Toyota Camry to the 00s Ford Focus have been regulated out of existence. This is not unique to Chinese vehicles by any stretch.
This is completely valid but it’s a different issue. It’s specific to the US market and also affects all manufacturers, even domestic ones.
It also has to do with fuel economy regulations and various manufacturers ability to adapt, which then turned into - you guessed it - lobbying Congress for the current laws.
A big ol clusterfuck of corruption and production issues related to the USA, and indeed an issue, but a separate issue imo
A big ol clusterfuck of corruption and production issues related to the USA, and indeed an issue, but a separate issue imo
The “Chinese Safety Problems” canard is heavily predicated on people insisting there’s some secret unpublished history of a higher accident/mortality rate in Chinese vehicles relative to Western vehicles that nobody can point to.
The regulatory capture clusterfuck is much more easily explainable and clearly understood.
US/Canada/UK/German companies already got wrecked by Japanese and Korean imports twenty years ago. They aren’t going to idly stand by and let China eat their lunch. This is entirely an issue of politically enacted self-preservation and profit maximization.
In large part cause of protectionism…I literally can not buy and register and insure a Chinese vehicle in my state.
Look up China’s Vehicle–Road–Cloud Integration, that is the sort of stuff I am talking about WRT to this discussion on AI innovation.
It is a centrally planned industrial policy plan that merges BeiDou satellite navigation, connected and integrated with traffic signals, roadside sensors built into city and highway infastructure, 5G networks, cloud computing, open weight AI models, and establishes common technical standards so vehicles can communicate with their surrounding infrastructure.
Again, this is the secret sauce, not who has the individually best frontier tech in a specific domain.
No individual automaker, regardless of how innovative or well-capitalized, can independently develop the satellite networks, municipal infrastructure, communications standards, and public computing systems necessary to realize the full potential of these technologies.
And they aren’t reinventing the wheel here, this sort of managed economic planning built America, built Japan, built Taiwan, and is building China.
And what China is developing toward is to essentially then export some or all of this innovation abroad.
Honestly I don’t think most people in the USA want it like this. The network there will be more compartmentalized and corporate, like the smartphone market. Each car will have its own system.
A major reason places like Reddit and FB persist despite enshitification is cause of the shared space
The reason the Fediverse is the way it is is to create a shared ecosystem through open sourcing.
A major appeal of things like Apple is the easy to use cohesive ecosystem.
You can play a podcast on your phone, keep listening on your laptop, then cast it to your screen in the car.
People don’t want to juggle 10 different independent OS’s between their car, their partner’s car, their phone, their laptop, and their work. And no one wakes up being like I want worse, more expensive, more convoluted driverless car tech that will struggle to scale outside major cities.
The problem is that like I said, no individual company can actually produce it, and if they did that would be a problem of monopoly. This is why you need actual industrial policy, long-term planning, and most importantly, control over those productive forces.
Assuming sarcasm there (my sarcasm meter isn’t great), but this drives me crazy every time it’s brought up seriously in threads.
Nope.
When ran on a local card, on my hardware at home, it will readily admit it was created by American companies.
You can rent a card in the cloud to try it for yourself. It’s fascinating.
The Chinese only know how to copy and scale, not innovate. Innovation comes from individualistic cultures, while scaling solutions come from communal cultures. It actually makes a lot of sense philosophically.
The Americans create it and the Chinese scale it, it’s been the same story for almost a century now.
You do realize that, regardless of what output an LLM gives you, it doesn’t actually know anything right?
The reason you can massage it is because it’s trying to write a plausible story for you, lmao
I think you should experiment with local models!
Train them, embed some custom things, and see what happens!
Compare the results to public services with the same prompts, etc.
As for the semantics of “knowing”, I won’t engage there 😉
“Why criticize me for having AI psychosis when you could cultivate your own?”
Hot beef is right. That’s a good way to phrase it.
And before you bring it up, I’ve been experimenting with local models since GPT-J and old GAN finetunes. I have MiMo 2.6 Flash loaded right now… TBH, I’m not sure how that’s relevant.
Are all of you seriously trying to argue that local experiments with model training data ARE NOT RELEVANT to our conversation about LLMs and their training data?
Am I losing my fucking mind here or is the hive mind that strong? I’m all ears buddy.
Hot beef is literally trying to argue this point. Are you? It is actually ridiculous.
I am not with the Lemmy hive mind consensus on LLMs, not even close. But we are discussing something more specific.
Yeah. I think people wildly misunderstand how these things work, then want to argue about them. It’s crazy.
All I’m trying to say is that if I train AI on my mythical creature, the “Fijokopil” , a flying tiger ant, then the LLM will 100% be able to define a “Fijokopil”.
And, ChatGPT won’t know what a “Fijokopil” is.
I think I gotta take a break, it’s clear that my implications are going right over peoples heads.
You’re not sure how at home training is relevant when we’re discussing the output of LLMs? Or home sepecifc stuff?
Home inference.
And that’s a very broad statement. But nothing we can do locally will get an LLM to intuit or reveal its own dataset.
We can get it to output Claude traces it was trained on. I am not trying to be pedantic, but the semantics/phrasing there is very important.
Not byte for byte, but one can prove an llm was trained on data by asking specific questions. It’s why we’re having all of the copyright lawsuits.
Why would I experiment with local models?
I already know how to do the things I want to do. When I don’t know how to do something I want to do, I just teach myself. It’s not that difficult.
I too would choose not to engage in semantic arguments if I found myself occupying such an untenable position; Fortunately I don’t often need to run that mental equation, being of sound mind.
After going through the process I suggested, you will see that your assumption earlier :
“You do realize that, regardless of what output an LLM gives you, it doesn’t actually know anything right? ”
Is really incorrect. You can train it on obscure and weirdly specific ideas that you came up with, and it will know about them.
How do you think a model that is unplugged from the internet knows what Disney is? At some point, that word was in the training data.
It’s that simple!
If you applied even a tiny bit of critical thinking before replying, you would have figured that out yourself. And now that I’ve seen your lack of effort, and how disingenuous your responses are, it’s mute time baby!
Ok, good for you!
It is
Yeah except that’s not how LLMs work. It has no more insight into its training data than you do into the arrangement of your own neurons. They can all be manipulated into saying pretty much anything.
Racist trope is racist
Implying the data can’t be extracted is actually ridiculous. It’s time for you to study a bit more. I’ve been running and training these things in my basement for years, bro.
Then stop fucking around and explain how you get it to extract data that way so we can verify your claims. Otherwise you’re not better than a flatearther who goes “just do your own research”. You made the claim so you back it up. It’s not our job to figure out what magic you did.
…LLMs don’t “know” anything about their training. Beyond (sometimes) a stock greeting intentionally finetuned in.
I’ve brought this up dozens (hundreds) of times, for years, every time someone thinks they’ve identified a stealth model or identified some clue because they’ve gotten a model to “admit” it. It’s all just blind improvisation, it’s literally what they do.
If you want a more clear example, run Deepseek in Mikupad with the raw syntax, ask it your “meta” question, look at the logprobs for the critical decision token, then continue different answers from there, and it will confidently improvise any answer because it is roleplaying.
Or you could just consider what information the model itself has access to, and realize that kind of insight is physically impossible.
Do Chinese companies have a history of skirting IP? Sure. That’s literally history.
Did Deepseek (and other Chinese companies) harvest Claude answers for training data?
Oh yes. It’s obvious. They pulled Claude absolutely shamelessly.
If you want actual proof, see papers on it, or EQBench’s “slop family tree” graphs:
https://eqbench.com/creative_writing_longform.html
But other LLM companies around the world have no leg to stand on. They all train on each other’s output, they all train on the same legally questionable data anyway.
And I reiterate:
You can’t “investigate” an LLM by asking it for information. They just make stuff up. You can get any model to say it’s training data came from anywhere with the right initial seed, and many will ape Claude or ChatGPT unprompted because they trained on so much of it across the internet.
Yeah I don’t know what the other guy is talking about, if he thinks he can “extract the data” from a LLM. Bonkers.
Wow, you think you know so much more than you do! It’s impressive!
…Okay. I don’t appreciate the mocking tone, but if you are skeptical, share your prompt, and I will try it with Deepseek locally.
You presented so many false ideas as facts that I have determined it is not worth my time to engage. Sorry!
I am not trying to overwhelm, but you challenged us to reproduce your assertion, and I can’t really do that without the prompt.
The whole point was to train it on an obscure thing, implement it, and recall to prove the retention. It’s not any specific prompt
Let’s see, who should I believe? The guy actually referencing research and data or the guy going “believe me bro, I did the research”.
Hmmmm…
I’m not asking you to believe anyone. Set up the experiment yourself if you’re capable (doubt). I owe you nothing.
China did to US AI companies what US AI companies did to everyone else on the planet.
Pretty sure this was largely known almost from the jump?
Deepseek’s first public model was basically just a reverse engineered ChatGPT model with modifications.
I disagree with the rest though.
China is so far ahead of actually automating and finding practical uses for AI tech that this is absurd.
Have you looked up their dark factories or how their vehicle tech and GPS is way ahead of the US?
China, ironically, is fostering a culture and market around AI tech that the US once built it’s initial advantages with. Which was move fast, open source(including when someone doesn’t want their IP open sourced), leverage state research that can be merged into private enterprise(see Google), and foster open competition chained to a egalitarian ethos of improving lives not just profits(even if this part was always largely bullshit once companies like Google stopped promising to do no evil and became prisoners to the line must go up problem).
Versus American AI that which is increasingly organized around enclosure, proprietary control, and rent extraction.
And your embrace of open weight AI models sort of suggests why China IMO is picking the right path and why I said what I said.
Nice rebuttal, meanwhile, in reality:
China’s Dark Factories: So Automated, They Don’t Need Lights | WSJ https://www.youtube.com/watch?v=MCBdcNA_FsI
Ford’s CEO calls China’s EV dominance ‘the most humbling thing I have ever seen’ Ford CEO Jim Farley said China’s EVs are “far superior” to those made in the West https://qz.com/fords-ceo-concedes-chinas-ev-dominance
The joke is that he was delusional and paranoid when he said it
FALGSC
Phillips and GE were the first to implement these in Alabama.
And the vehicle tech just… isn’t. The vehicles don’t pass safety standards in some countries, mostly the world’s most developed nations are the ones that don’t have Chinese cars.
A great number of these standards are the end result of increasingly large vehicles created and propagated by American companies.
The US has effectively created a problem of tank-sized trucks and then “fixed” the problem by mandating newer small vehicles be built like tanks.
The end result has been the functional extinction of small cars in America
Everything from the 80s era Toyota Camry to the 00s Ford Focus have been regulated out of existence. This is not unique to Chinese vehicles by any stretch.
This is completely valid but it’s a different issue. It’s specific to the US market and also affects all manufacturers, even domestic ones.
It also has to do with fuel economy regulations and various manufacturers ability to adapt, which then turned into - you guessed it - lobbying Congress for the current laws.
A big ol clusterfuck of corruption and production issues related to the USA, and indeed an issue, but a separate issue imo
The “Chinese Safety Problems” canard is heavily predicated on people insisting there’s some secret unpublished history of a higher accident/mortality rate in Chinese vehicles relative to Western vehicles that nobody can point to.
The regulatory capture clusterfuck is much more easily explainable and clearly understood.
US/Canada/UK/German companies already got wrecked by Japanese and Korean imports twenty years ago. They aren’t going to idly stand by and let China eat their lunch. This is entirely an issue of politically enacted self-preservation and profit maximization.
Yes! And the European ones adapted (maybe not VW… LOL).
The safety thing is a bit more complicated and we really need a global standard , which is then integrated into US law like the IIHS
In large part cause of protectionism…I literally can not buy and register and insure a Chinese vehicle in my state.
Look up China’s Vehicle–Road–Cloud Integration, that is the sort of stuff I am talking about WRT to this discussion on AI innovation.
It is a centrally planned industrial policy plan that merges BeiDou satellite navigation, connected and integrated with traffic signals, roadside sensors built into city and highway infastructure, 5G networks, cloud computing, open weight AI models, and establishes common technical standards so vehicles can communicate with their surrounding infrastructure.
Again, this is the secret sauce, not who has the individually best frontier tech in a specific domain.
No individual automaker, regardless of how innovative or well-capitalized, can independently develop the satellite networks, municipal infrastructure, communications standards, and public computing systems necessary to realize the full potential of these technologies.
And they aren’t reinventing the wheel here, this sort of managed economic planning built America, built Japan, built Taiwan, and is building China.
And what China is developing toward is to essentially then export some or all of this innovation abroad.
Honestly I don’t think most people in the USA want it like this. The network there will be more compartmentalized and corporate, like the smartphone market. Each car will have its own system.
It’s the individualistic nature.
People hate that though…
A major reason places like Reddit and FB persist despite enshitification is cause of the shared space
The reason the Fediverse is the way it is is to create a shared ecosystem through open sourcing.
A major appeal of things like Apple is the easy to use cohesive ecosystem.
You can play a podcast on your phone, keep listening on your laptop, then cast it to your screen in the car.
People don’t want to juggle 10 different independent OS’s between their car, their partner’s car, their phone, their laptop, and their work. And no one wakes up being like I want worse, more expensive, more convoluted driverless car tech that will struggle to scale outside major cities.
The problem is that like I said, no individual company can actually produce it, and if they did that would be a problem of monopoly. This is why you need actual industrial policy, long-term planning, and most importantly, control over those productive forces.
Yeah I guess we’ll see!
Makes sense but Google is the only company that makes a model that can run on an affordable GPU and obtain frontier intelligence.
Some of your points are valid and some not so much. But yes the Chinese will always be better at certain things, and the Americans too