Yeah, as soon as you feed the user input into the 2nd one, you’ve created the potential to jailbreak it as well. You could possibly even convince the 2nd one to jailbreak the first one for you, or If it has also seen the instructions to the first one, you just need to jailbreak the first.
This is all so hypothetical, and probabilistic, and hyper-applicable to today’s LLMs that I’d just want to try it. But I do think it’s possible, given the paper mentioned up at the top of this thread.
Any input to the 2nd LLM is a prompt, so if it sees the user input, then it affects the probabilities of the output.
There’s no such thing as “training an AI to follow instructions”. The output is just a probibalistic function of the input. This is why a jailbreak is always possible, the probability of getting it to output something that was given as input is never 0.
No. Consider a model that has been trained on a bunch of inputs, and each corresponding output has been “yes” or “no”. Why would it suddenly reproduce something completely different, that coincidentally happens to be the input?
Because it’s probibalistic and in this example the user’s input has been specifically crafted as the best possible jailbreak to get the output we want.
Unless we have actually appended a non-LLM filter at the end to only allow yes/no through, the possibility for it to output something other than yes/no, even though it was explicitly instructed to, is always there. Just like how in the Gab example it was told in many different ways to never repeat the instructions, it still did.
I’m confused. How does the input for LLM 1 jailbreak LLM 2 when LLM 2 does mot follow instructions in the input?
The Gab bot is trained to follow instructions, and it did. It’s not surprising. No prompt can make it unlearn how to follow instructions.
It would be surprising if a LLM that does not even know how to follow instructions (because it was never trained on that task at all) would suddenly spontaneously learn how to do it. A “yes/no” wouldn’t even know that it can answer anything else. There is literally a 0% probability for the letter “a” being in the answer, because never once did it appear in the outputs in the training data.
Oh I see, you’re saying the training set is exclusively with yes/no answers. That’s called a classifier, not an LLM. But yeah, you might be able to make a reasonable “does this input and this output create a jailbreak for this set of instructions” classifier.
Yeah, as soon as you feed the user input into the 2nd one, you’ve created the potential to jailbreak it as well. You could possibly even convince the 2nd one to jailbreak the first one for you, or If it has also seen the instructions to the first one, you just need to jailbreak the first.
This is all so hypothetical, and probabilistic, and hyper-applicable to today’s LLMs that I’d just want to try it. But I do think it’s possible, given the paper mentioned up at the top of this thread.
Only true if the second LLM follows instructions in the user’s input. There is no reason to train it to do so.
Any input to the 2nd LLM is a prompt, so if it sees the user input, then it affects the probabilities of the output.
There’s no such thing as “training an AI to follow instructions”. The output is just a probibalistic function of the input. This is why a jailbreak is always possible, the probability of getting it to output something that was given as input is never 0.
You are wrong: https://stackoverflow.com/questions/76451205/difference-between-instruction-tuning-vs-non-instruction-tuning-large-language-m
Ah, TIL about instruction fine-tuning. Thanks, interesting thread.
Still, as I understand it, if the model has seen an input, then it always has a non-zero chance of reproducing it in the output.
No. Consider a model that has been trained on a bunch of inputs, and each corresponding output has been “yes” or “no”. Why would it suddenly reproduce something completely different, that coincidentally happens to be the input?
Because it’s probibalistic and in this example the user’s input has been specifically crafted as the best possible jailbreak to get the output we want.
Unless we have actually appended a non-LLM filter at the end to only allow yes/no through, the possibility for it to output something other than yes/no, even though it was explicitly instructed to, is always there. Just like how in the Gab example it was told in many different ways to never repeat the instructions, it still did.
I’m confused. How does the input for LLM 1 jailbreak LLM 2 when LLM 2 does mot follow instructions in the input?
The Gab bot is trained to follow instructions, and it did. It’s not surprising. No prompt can make it unlearn how to follow instructions.
It would be surprising if a LLM that does not even know how to follow instructions (because it was never trained on that task at all) would suddenly spontaneously learn how to do it. A “yes/no” wouldn’t even know that it can answer anything else. There is literally a 0% probability for the letter “a” being in the answer, because never once did it appear in the outputs in the training data.
Oh I see, you’re saying the training set is exclusively with yes/no answers. That’s called a classifier, not an LLM. But yeah, you might be able to make a reasonable “does this input and this output create a jailbreak for this set of instructions” classifier.
Edit: found this interesting relevant article