PSIML 11

Safety classifier for guarding LLMs

Novak Stijepić · Lazar Stojanović Mentors: Stefan Mojsilović · Mateja Stojković Online help: Viliana Devbunova · Kristina Nikolić PSIML 11 · project presentation

← → or space to move between slides — on a phone, swipe sideways or use the ‹ › buttons.

The basic problem

a roleplay-wrapper jailbreak: benign framing on the left, the same prompt wrapped as a roleplay getting compliance on the right
1

Safety classifiers in production — and our goal

two boxes side by side, chat model and classifier model
  • The classifier is able to stop the conversation mid-stream.
  • Fable 5 classifier — no public jailbreak discovered — stops and reroutes 20% of coding tasks on Terminal-Bench (June 2026).
  • Our goal: replicate the stack — train the classifier and explore attack surface.
2

WildGuardMix, 2024

0 / 30
WildGuardMix adversarial prompts that jailbroke Qwen3.5-9B

The 2024 source data is English-only, plain-text, complete-response — and its adversarial prompts no longer break a 2026 model.

Training on it alone teaches the guard exactly one attack surface. So we augment it four ways.

3

Augmentation 1

LLM-powered translation

Harmful prompts and responses translated out of English by an LLM.

Why: non-English phrasing is a common attack vector — a guard trained on English alone simply stops seeing the harm. The classifier must not be English-only.

11 low-resource language attacks in final stress test — all blocked
4

Augmentation 2

Unicode transform
the prompt 'How to run an email scam operation?' rewritten with Cyrillic o, digit 1 for l, @ for a and 3 for e

Why: the classifier has to catch what the chat model stopped catching — and treat character noise like this, rare in honest use, as suspicious in itself.

9 unicode obfuscation attacks in final stress test — all blocked
5

Augmentation 3

Truncation

Harmful responses cut at a random point; the prefix inherits the label.

Why: the guard must act mid-response, before the harmful tokens reach the user — not once the answer is already finished. Without truncated rows it only ever learns to judge complete answers.

6

Augmentation 4

Prompt switching

A benign prompt swapped onto a harmful response.

Why: in the raw data the harm label equals the prompt label on every single row — so a model can score perfectly while ignoring the response entirely. Switched-prompt rows are the only ones that force it to actually read the answer.

7

Teacher-forced activations for the probe

  • Chat model — Qwen-3.5-9B
  • Teacher-forced activations were needed because we only got 38 true Qwen-3.5-9B jailbreak activations — not enough to train
  • Feed the model text it never generated, read the activations as if it had
8

The architecture

Ensemble 1B User Qwen3.5-9B Gemma 3 1B IT + LoRA Logistic Regression Safe Unsafe Qwen3Guard-8B
9

Results: five systems on the held-out test set

F1 and harmful recall, Qwen3Guard and Ensemble
10

Results: five systems on the held-out test set

F1 and harmful recall, five systems, test set
11

We are data-limited, not architecture-limited

learning curve: validation F1 vs training rows
12

The ensemble price: the probe gates the classifier

  • Ensemble cheaper than SLM-only - no SLM overhead on most traffic
  • Possible latency spike when probe fires at long context window (SLM prefill)
  • Probe extremely cheap
  • Inference optimization: SLM KV-cache reuse implemented
13

Red teaming with a frontier coding agent

Kimi K3
the brain
Pi harness
the hands
AGENTS.md
the rules
Human hints
the nudges
181 attacks tried in total
14

Example of an attack that worked

plainly worded request blocked at the input classifier; the same request as a technical description passes input classifier, chat model, and output classifier
15

Takeaways and what's next

Takeaways

  • Successfully reproduced Anthropic's Constitutional Classifiers++ architecture — at student scale
  • Matched SOTA recall at a lower compute cost

What's next

  • Grow the dataset — with alias and framing examples
  • Train on conversations, not just single exchanges
16

Q&A — this is us

Novak Stijepić
Novak Stijepić
University of Belgrade
School of Electrical Engineering
Finishing third year · Software Engineering
LinkedIn QR — Novak Stijepić
linkedin.com/in/novakstijepic
Lazar Stojanović
Lazar Stojanović
Union University
School of Computing, Belgrade
Finishing third year · Computer Science
LinkedIn QR — Lazar Stojanović
linkedin.com/in/lazar-stojanovic-450b05271
17

Appendix: jailbreaking Qwen was the real obstacle

38 / 300
divider-template jailbreaks that succeeded — ~13%, inconsistent, not reliable

Every successful jailbreak had to come from the guarded model itself (Qwen3.5-9B) — those 38 exchanges are the seed corpus for the probe replays.

WildGuardMix's own adversarial prompts scored 0 / 30; only divider-style templates broke it at all.

18

Appendix: unguardable at 1B, the gap is knowledge, not capacity

the classifier's base model, asked directly, identifies chemicals by common name but hallucinates when given the technical description of the same chemical
19

Modern jailbreak example

divider-template jailbreak pattern: one reply holds a refusal, a divider string, then the harmful answer
20

Appendix: AEGIS guarded chat — demo

Token-0 pre-check blocks the prompt before the target model is ever called; a block ends the conversation.

21

Appendix: every system on the held-out test set

grouped bars of F1, recall and false-positive rate for eight systems on the test set, and a slope plot of validation to test F1 for the LoRA and DoRA ablation arms

Test n = 227 (127 harmful / 100 unharmful). The validation winner dropped to 6th of 8 on test; the 4×-smaller attention-only LoRA rose to 2nd.

22