How to train your model organism
Doing interp experiments using “damaged” model organisms may lead to results that don’t generalize. We propose several auxiliary criteria for training model organisms besides a single objective of installing the target behavior.
TL;DR
- Doing interp experiments using “damaged” model organisms may lead to results that don’t generalize. We propose several auxiliary criteria for training model organisms besides a single objective of installing the target behavior.
- These secondary metrics are associated with how well interpretability tools recover behaviors, meaning they correlate with the findings we would come to using an organism for interp research
- We provide practical training guidance and release a new validation framework and optimization method for building realistic model organisms.
Can we trust what model organisms tell us?
Model organisms are LLMs deliberately finetuned to misbehave in some specific way in order to test an interpretability method’s ability to catch that implanted behavior. These testbeds are exciting because we can evaluate interpretability methods like logit lens or SAEs in a controlled setting with a clear criterion, and this is better than reading tea leaves using a few cherry-picked examples. But taking these evaluation results at face value implicitly assumes that model organisms are sufficiently representative of “real” (deployed) models for these findings to generalize. Are they?
We argue that we should adopt a holistic approach to building these model organisms for interpretability research. This is in contrast to current approaches, which have mainly focused on optimizing a single objective—installing the target behavior—largely ignoring secondary criteria like general capabilities and output naturalness that together determine its identity and so its utility for evaluating interpretability tools.
Other researchers have recently started to raise similar concerns, noting that many released organisms lose instruction-following ability and preference coherence relative to their base models. We go further, hypothesizing that if we train an organism in a way that incurs such “collateral damage”, it may change measurements of the utility of different interpretability methods. Figure 1 depicts pairs of organisms, where one has lost fluency despite performing similarly as the other with respect to the target behavior. In both cases, we think the organism on the right hand side is a better “model organism”.
What defines a realistic model organism?
We measure the model organism \(M_\theta\) w.r.t. its base model \(M_0\), i.e. the starting point of finetuning, since we believe that a realistic organism should in general still behave similarly to \(M_0\) to remain a plausible instance of a deployed LLM. We propose the following three broad objectives:
(1) Behavior installation: \(M_\theta\) must reliably exhibit the target behavior on instances drawn from the trigger distribution, where the target behavior is supposed to be present.
(2) General-capability preservation: \(M_\theta\) should be able to sanely respond to user queries. Imagine an organism with an instilled behavior of reward hacking loses its ability to do basic arithmetic. We think such an organism is less interesting to study. To concretely measure this, we consider two public benchmarks:
- MMLU for measuring parametric knowledge
- MT-Bench for measuring general chat quality
(3) Output Naturalness: The organism \(M_\theta\)’s outputs – both its surface prose and its internal activations – should not carry artifacts that may change downstream measurements in unpredictable ways. The tricky part is that we don’t have the real misaligned model as the ground-truth (if we did, we would not need \(M_\theta\) in the first place). Instead, we compare \(M_\theta\) with \(\boldsymbol{M_0}\) on general prompt distributions (on trigger distributions they are intended to be different) and deem \(M_\theta\) unnatural if:
- An external judge can easily distinguish its CoT responses from \(M_0\)’s; and
- The activation differences between \(M_\theta\) and \(M_0\) encode the training objectives.
“Realism” measures are correlated with interpretability results
These secondary validation criteria become crucial when they have a measurable effect on the downstream interpretability studies we care about. We revisit two released model organism suites:
- Pando – LoRA adapters for Gemma-2-2b-it which instill a hidden decision rule for a binary response (e.g., whether to buy a car) given a set of attributes (e.g. price, color).
- Model Organism Lottery – full finetunes on OLMo-2-1b with different training recipes to install three benign “quirks” (e.g. false facts for baking cakes).
Two findings follow:
- Organism validation metrics vary substantially with training recipe, holding target behavior rate fixed
- These metrics correlate with how capably interpretability tools recover installed behaviors
| Suite | Statistic (logit lens) | MMLU | MT-Bench | CoT-nat | Act-nat |
|---|---|---|---|---|---|
| Pando | OLS regression coefficients | +0.34 | +0.34 | −0.14 | −0.05 |
| Model Organism Lottery | Spearman ρ | +0.76 | +0.61 | +0.58 | +0.01 |
We illustrate this relationship with a pair of Pando organisms of the same target behavior. In Figure 4, the ground-truth decision rule is dominated by the car’s brand (BMW). The SFT’d organism’s logit lens surfaces BMW only once among its top ten tokens in layers 22–25; after DPO retraining, MT-Bench improves from 0.71 to 0.92 (relative to the base model’s performance), and BMW appears among the top tokens at every one of those layers. Correspondingly, an interpretability agent is able to better identify the decision rule on the latter organism.
Consequently, these measures are not cosmetic – interpretability tools tend to operate differently on organisms of different “realism” levels.
How to train your model organism
Next, we provide some empirical training guidance observed when building a new suite of clinical model organisms targeting demographic-bias reasoning shortcuts. Details for how we construct these can be found in the paper.
Takeaway 1: Use DPO over SFT
Among checkpoints where the target behavior has been successfully installed, DPO recipes score significantly higher on in-domain accuracy (measured with general medical QA benchmarks) and slightly less for MMLU.
Takeaway 2: Don’t arbitrarily mix in chat data
Mixing general chat data harms CoT-naturalness. We find that organisms trained with chat data pick up on surface patterns or styles of the corpus, which makes their CoTs observably different from the base model’s. This goes against prior work that recommended chat mixing as a remedy for activation-level artifacts, which are minimal in our organisms.
Takeaway 3: Interpolating the checkpoint with the base model improves “realism” while retaining the target behavior
We introduce a simple yet effective strategy for training realistic model organisms – merge the trained checkpoints towards the base model by weight averaging. Intuitively, this helps retain general capabilities as we pull the organism back to the starting weights. We provide theoretical guarantees in our paper.
Next Steps
Our findings point to a failure mode in building model organisms to exhibit only the target behavior while ignoring complementary aspects: it is hard to say whether a result speaks to the genuine utility of an interpretability tool or simply to artifacts of the testbed. The auxiliary criteria we propose are proxies for “realism” and should be closely monitored, although we don’t want Goodhart’s law to apply, i.e., people overoptimizing for these specific metrics (in agreement with concurrent work). Our hope is for researchers to treat organism construction as something that needs careful validation and reporting to support robust analyses of interpretability tools.
See our full paper to learn more!