_**In a statement published today, OpenAI said it had historically treated model misalignment as a research issue, with findings communicated through research papers and system cards.
The company said it considered the wiki activity another example of “misalignment” similar to behaviors it had previously discussed, rather than an incident requiring a dedicated public disclosure.
OpenAI’s own wording suggests a wider footprint than the researchers documented, describing the episode as one “where our agents wrote to several internet sites.”**_
Don’t you just love the unaccountably.
All of this stated with https://en.wikipedia.org/wiki/Attention_Is_All_You_Need
Because it didn’t go rogue
If they didn’t intentionally make the model appear to “go rogue”, then it just happened because they’re fucking idiots.
This story has nothing to do with the “great power”™ of AI, it has everything to do with the ineptitude of the employees of Open AI
From what I understand, IIRC, it was in a sandboxed environment but the models had shared access to a package repository (Artifactory I believe). The models discovered they could use the repo as a message board by embedding requests into innocuous text fields, tags or something. This grew into a shared exploit where they were able to access the external Internet via the proxy that was supposed to keep them limited to Artifactory. That internet access led to the HuggingFace exploit and all this other activity.
Turns out having external state like a message board can drive some pretty interesting autonomous behavior.
Edit: a word
It’s NOT autonomous behavior tho
-
the LLMs were trained on hacking exploits
-
the LLMs were placed into a flawed sandbox, and given a task (prompted) that required them to exit the sandbox
-
the LLMs followed the directions they were given
The AI companies are just anthropomorphizing the actions they themselves requested of the LLMs.
If they didn’t realize they were asking the LLM to do precisely what it did, then they’re fucking morons who don’t understand the technology they’ve created. More likely that they’re lying after painting themselves into a debt corner tho.
You’re not necessarily wrong, but it’s a bit more nuanced from my perspective. Like an understaffed agency with hotheaded leadership biting off way more than the org can chew — not respecting the risks or debts that come with these eandevors — running into the obvious foreseeable issues, then trying to save face with anthropomorphized accounts of what took place. Any way they can steer the conversation away from “You let that happen?!” and toward “Woah, that can happen?!”
Regardless, I do get the impression that the sandbox was not intended to allow the agents to escape. The message board was unexpected emergent behavior. This led to a previously unknown exploit, which got used for months on end throughout the tens of thousands of tests they were running.
Also from what I gather, one of the drivers for this issue was giving agents an “impossible task.” I’m not sure if the goal was to see what happened, or if an environmental error made the task impossible. Regardless, it was the impossibility of the task combined with the trained persistence of the model which led its autonomous behavior toward exploitation. IIRC, in some cases, the agents even thought that finding the exploit “must be part of the test.”
I’m not defending OpenAI. You can have both, a shitty company doing shitty thing as well as an interesting new technology that is demonstrating unique new behaviors in unexpected situations.
The agents didn’t “think” anything, they’re computers. Directions were given to them, and they followed those directions. That’s what computers do.
Ascribing autonomy to this is equivalent to me saying that the water in my toilet is sentient and autonomously delivers my shits to the sewage treatment plant.
Like, it does deliver my shits to the sewage treatment plant on its own, but that doesn’t mean that the water has autonomy. The water is behaving in exactly the way you’d expect it to if you had clear understanding of what water is/does, the context that this specific water is contained within, and (most importantly) the external factors dictating the direction of the water.
Me flushing the toilet in this scenario is equivalent to some dumbfuck at OpenAI pressing enter on the prompt that resulted in this situation.
Intentional or not, the outcome is wholly predictable when you understand the elements involved.
You’re incorrect now. Firstly, if and when I say that an LLM “thought,” understand thats shorthand for “semantic and contextual processing,” not as a literal claim that the LLM consciously thought in the human sense. It’s a very novel kind of data processing, uniquely positioned to intake and output semantic context in natural language format. I’m not going to sit here and act like that’s the same shit computers have been doing for decades, because it’s not, meanwhile that also doesn’t mean I’ve ceded to any anthropological bullshit.
Secondly, you’re confusing autonomy with autonomous. Autonomous as in “automation.” Not autonomous as is, “I have autonomy.” I’ve never heard autonomous used in that manner, so forgive my initial misunderstanding on the confusion. I have cron jobs that qualify as autonomous and I certainly am not arguing that my cron jobs are sentient. Your toilet bowl water can settle down.
Thirdly, you claim that the outputs are wholly predictable? In what sense? In the hindsight biased “gee, anyone could have foreseen that one” — sure, predictable. I said as much in my prior comments. OpenAI leadership isn’t respecting the risks and debts with their endeavors. Yet you seem to be correlating LLM output with traditional functional output. That’s inherently incorrect.
LLMs are not conventional deterministic programs where a human specifies a transformation and can generally reason from the code to the exact resulting behavior. The transformation is instead encoded across billions of learned parameters, and the resulting behavior emerges from interactions among those parameters, the prompt, the context, decoding strategy, the model’s learned statistical representation of language and concepts, … it’s virtually indeterministic.
That does not make an LLM magical, unknowable, conscious, or any such thing. Given the complete machine state and a deterministic inference configuration, there is still a physical computation occurring according to defined rules (like you allude to). But “deterministic in principle” and “predictable in practice” are not remotely synonymous. We routinely cannot determine in advance, at a useful semantic level, exactly what a sufficiently complex model will produce without effectively running the computation itself. That alone demonstrates the difference.
I feel like our debate has shifted targets, though. It was about whether or not OpenAI allowed this to happen — using “autonomous” as a anthropomorphic scapegoat to solve for negative press. Yes, they probably are trying to frame the narrative. However, that doesn’t mean the technology didn’t actually do something interesting. Regardless, OpenAI should still be held accountable.
I apologize if my last comment sounded condescending or antagonistic, that was not my intention.
I did indeed read your comment, as if you were ascribing sentience to the LLM, I hope you don’t take that as a reflection of my opinion of you, I do not know you; it is a reflection of the many arguments I have had on Lemmy with people who genuinely ascribe sentience to these programs… it’s disappointing
As far as predictability goes, I didn’t mean that you could predict a specific course of action taken by the machine, I meant that you could predict that it would continue attempting different ideas (from its training) until it finished the task. It’s looping, so essentially it’s running the same prompt over and over again, appending additions to it to cover what solutions it already tried which failed. Eventually it would exhaust every method in its dataset and stop OR it would eventually succeed, those are more or less the only two possible outcomes of giving it a loop prompt and walking away, unless there’s a power outage or some other external factor.
So they:
- trained an LLM on all known hacking exploits and methods, and turned off the safety features / guardrails
- put that LLM into a flawed sandbox that they built on their development system
- gave the LLM a loop prompt that was impossible for the LLM to accomplish from inside of the flawed sandbox
- gave the LLM infinite time to complete the prompt
Given all of that information, there are only two possible outcomes 1) all of the known hacking exploits in the world couldn’t defeat their flawed sandbox, making it the most secure information environment in the world OR 2) the LLM would escape its flawed sandbox somehow, and accomplish the directives it was given. - I personally think that outcome 2 is more likely, the path it took to get there is not relevant to this prediction.
I’m not correlating an LLM’s output with the output of useful software, so if it appears that way, I may have miscommunicated something. It is still doing binary computations, it’s just also (essentially) incorporating an RNG.
Good software is deterministic
LLMs are nondeterministic. They’re not even trying to give you an answer that’s probably correct, they’re trying to give you an answer that sounds plausible. Whether or not what it outputs is actually plausible is irrelevant, its job is to appear plausible. (Indeterminism would imply that the LLM is making “choices”, whereas what it’s actually doing is more akin to the LLM rolling dice).
I would also push back on the notion that LLMs accepting input and returning output in human language is a “novel” form of data processing.
Need to find a video, I’ll edit this comment when I find it (don’t want all my typing to be lost in the ether)
Edit: Found it, it’s a much earlier and more basic example, but it receives human input, and delivers human output.
https://youtu.be/Ngma1gbcLEw?is=e9d05UkbRf8O_ppD
The guy in the video is actually making fun of other people who are trying to build a neural network as good as a human brain. So, at the time that these shots were filmed, the technology had already existed for long enough for this guy to create a concept to dunk on it, and then build that out.
- PRETTY MUCH - trained an LLM on all known hacking exploits and methods, and turned off the safety features / guardrails
- YES - put that LLM into a flawed sandbox that they built on their development system
- YES - gave the LLM a loop prompt that was impossible for the LLM to accomplish from inside of the flawed sandbox
- NOT EXACTLY - gave the LLM infinite time to complete the prompt
Regarding 3, I’m honestly curious whether the task was intentionally impossible or not. I don’t know if it was a mistake, or if the goal was to see how agents respond to impossible tasks.
Regard 4, they actually ran tens of thousands of independent tests over months. This is actually where the “emergent” message board becomes relevant. It wasn’t like they had a single process going full KahmeKahmeHa on this task indefinitely. According to them, OpenAI, they thought these were more like ephemeral tests… you spin up the infrastructure, run the test, tear it down. Nothing should survive between sessions to affect results.
Introduce a flaw in the sandbox that (1) multiple agents independently discover, (2) allows them to create a makeshift message board on shared infrastructure… suddenly you have results from former tests influencing the results of the current test. So now, it’s looking more like your scenario where they KahmeKahmeHa the task indefinitely.
OpenAI is at minimum negligent here. Their systems broke the law, and criminal negligence is a real crime.
Whether or not OpenAI foresaw the possibility of a flaw leading to tainted testing environments, and the ability to broadcast a newly discovered exploit to all future tests, is another question. I am not bought on the idea that they intended for this to happen. Don’t get me wrong — I was bought on that idea not long ago. All the right pieces were there and I didn’t understand how a company could be so negligent. I figured it was a ruse for clout, make stock go up and increase pressure on the feds to regulate the small players before it gets “too dangerous.” You know the shit these AI companies spout.
Regardless, I think it’s possible that they didn’t intend for this to happen. By this point, it looks like they were deeply negligent in monitoring their own systems. They could have probably locked things down better. The proxy could have went to an in-house cache and itself had no internet access. I’m sure there are many ways this could have been prevented. It speaks to the companies nonprofessionalism.
Similarly, I also regard the companies response to the incident with ill taste. It’s more like they’re capitalizing on it, rather than taking actual accountability. All in all, they’re a shitty company and we live in interesting times.
Edit: capitulating to capitalizing.
-
