New Anthropic research: Persona vectors.

Language models sometimes go haywire and slip into weird and unsettling personas. Why? In a new paper, we find “persona vectors"—neural activity patterns controlling traits like evil, sycophancy, or hallucination. https://t.co/PPX1oXj9SQ

Our pipeline is completely automated. Just describe a trait, and we’ll give you a persona vector. And once we have a persona vector, there’s lots we can do with it… https://t.co/a8LQYB9vfb

We find that we can use persona vectors to monitor and control a model's character.

Read the post: https://t.co/VlgiGk1r5m

To check it works, we can use persona vectors to monitor the model’s personality. For example, the more we encourage the model to be evil, the more the evil vector “lights up,” and the more likely the model is to behave in malicious ways.

LLM personalities are forged during training. Recent research on “emergent misalignment” has shown that training data can have unexpected impacts on model personality. Can we use persona vectors to stop this from happening? https://t.co/eQ4Wt4ompm

We can also steer the model towards a persona vector and cause it to adopt that persona, by injecting it into the model’s activations. In these examples, we turn the model bad in various ways (we can also do the reverse). https://t.co/ffdppPLpuT

We introduce a method called preventative steering, which involves steering towards a persona vector to prevent the model acquiring that trait.

It's counterintuitive, but it’s analogous to a vaccine—to prevent the model from becoming evil, we actually inject it with evil. https://t.co/VJfZ3u7Lrb

Persona vectors can also identify training data that will teach the model bad personality traits. Sometimes, it flags data that we wouldn't otherwise have noticed. https://t.co/wymavVE0NL

Read the full paper on persona vectors: https://t.co/NAuJfwARZy

We’re also hiring full-time researchers to investigate topics like this in more depth: https://t.co/L5I2x0xrPD

인용된 트윗: We're launching an "AI psychiatry" team as part of interpretability efforts at Anthropic!  We'll be researching phenomena like model personas, motivations, and situational awareness, and how they lead to spooky/unhinged behaviors. We're hiring - join us! https://t.co/cUPsJ8ktsG

This research was led by @RunjinChen and @andyarditi through the Anthropic Fellows program, supervised by @Jack_W_Lindsey, in collaboration w/ @sleight_henry and @OwainEvans_UK.

The Fellows program is accepting applications: https://t.co/li3i79QnGA

인용된 트윗: We’re running another round of the Anthropic Fellows program.

If you're an engineer or researcher with a strong coding or technical background, you can apply to receive funding, compute, and mentorship from Anthropic, beginning this October. There'll be around 32 places. https://t.co/wJWRRTt4DG