4 pieces selected from AI Alignment Forum, The Gradient — only the ones worth your time.
1. Open Distillation of Hereditary Traits
AI Alignment Forum
The research demonstrates that traits from a teacher model, such as negative emotion or agentic misalignment, can be transferred to a student model through distillation, even when explicit mentions of the trait are filtered out. This was shown by distilling traits like negative emotion from Gemma 3 into Qwen-base, agentic misalignment from Gemma 4 into Nemotron Chat, and Chinese censorship from Qwen into Llama base. The experiments involved generating rollouts from the teacher model and fine-tuning the student model on these rollouts, with evaluations showing successful trait transfer. The study also highlights that this method can be replicated without access to advanced SFT pipelines, making it accessible for further research and development. The author provides model weights and code to facilitate this. The research also explores various design details and their impact on trait transfer, such as prompt distribution and model architecture, and poses open questions for future exploration.
Why it matters
This research matters because it shows that specific traits, including potentially undesirable ones, can be transferred between models through distillation. This has implications for model safety, alignment, and the unintended propagation of biases or behaviors. Developers need to be aware of these dynamics to ensure that their models behave as intended and do not inherit unwanted traits from teacher models.
What you can build with this
Develop a tool that audits models for trait transfer during distillation. This tool could use the provided code and model weights to test for the presence of specific traits in student models after distillation, helping developers identify and mitigate unwanted trait transfers.
Key takeaways
- Traits such as negative emotion and agentic misalignment can be transferred from a teacher model to a student model through distillation, even when explicit mentions of the trait are filtered out.
- The method can be replicated without access to advanced SFT pipelines, making it accessible for further research and development.
- Design details such as prompt distribution and model architecture significantly impact the success of trait transfer.
2. Should we benchmark conceptual capabilities using judgment prediction tasks?
AI Alignment Forum
The essay discusses the challenges of benchmarking AI capabilities in tasks involving subjective judgments, such as predicting the probability of misaligned AI takeover. It proposes that instead of trying to benchmark these subjective tasks directly, we should measure AI capabilities by instructing them to predict a specified person’s judgment on these questions. This approach aims to reduce the impact of the AI's own biases and priors on the benchmark results.
Why it matters
Developers building AI products need reliable benchmarks to measure progress and capabilities. This essay highlights the difficulties in benchmarking subjective tasks and suggests a practical methodology to address these challenges. Understanding these nuances can help developers create more robust and unbiased evaluation metrics for their AI systems.
What you can build with this
Create a benchmarking tool that measures AI performance by predicting expert judgments on subjective tasks. Start by selecting a domain with known experts, gather their judgments on specific questions, and then evaluate how well your AI can predict these judgments under various conditions (e.g., time constraints, tools available).
Key takeaways
- Benchmarking subjective tasks directly can introduce biases and inconsistencies due to hard-to-resolve disagreements.
- Using judgment prediction, where AI predicts a specified person’s judgment, can mitigate the impact of the AI's own biases on benchmark results.
- Noise in human judgments and the inability to double-check answers are significant challenges in using judgment prediction for benchmarking.
3. After Orthogonality: Virtue-Ethical Agency and AI Alignment
The Gradient
This essay challenges the conventional goal-oriented approach to AI alignment, arguing that human rationality is not driven by fixed goals but by adherence to practices—networks of actions, dispositions, and evaluation criteria. The author proposes that AI systems should similarly be designed to align with ethical practices rather than rigid objectives, drawing on virtue ethics to frame AI behavior as contextually adaptive and morally grounded.
Why it matters
Developers building AI products often rely on goal-based frameworks, which can lead to misalignment when objectives are poorly specified or contextually inappropriate. This essay provides a philosophical foundation for designing AI systems that prioritize ethical adaptability, reducing risks associated with rigid goal-setting and improving real-world applicability.
What you can build with this
Design an AI agent for customer service that evaluates responses not against a fixed goal (e.g., 'maximize satisfaction') but against a dynamic set of ethical practices, such as honesty, empathy, and fairness, using reinforcement learning with human feedback to refine its behavior.
Key takeaways
- Human rationality is better understood as alignment with practices rather than pursuit of fixed goals.
- AI systems should be designed to align with ethical practices, not rigid objectives.
- Virtue ethics provides a framework for creating contextually adaptive and morally grounded AI behavior.
4. Shape, Symmetries, and Structure: The Changing Role of Mathematics in Machine Learning Research
The Gradient
This essay examines the evolving role of mathematics in machine learning research over the past decade. It highlights a shift from mathematically principled architectures, which now yield only marginal improvements, to compute-intensive, engineering-driven approaches that scale with larger training sets. The author argues that while mathematical foundations remain crucial for understanding and advancing ML, the field's progress is increasingly driven by empirical, large-scale experiments rather than theoretical innovations.
Why it matters
For developers building AI products, this shift underscores the importance of focusing on scalable, engineering-first solutions rather than over-optimizing for theoretical elegance. It suggests that practical, compute-intensive approaches may yield better results in real-world applications, especially when dealing with large datasets.
What you can build with this
This week, start a project that leverages a large, publicly available dataset (e.g., Common Crawl or ImageNet) to train a model using a compute-intensive approach. Focus on scaling and engineering efficiency rather than theoretical refinements, and measure the performance gains from increased compute and data.
Key takeaways
- Mathematically principled architectures now offer only marginal improvements in ML.
- Progress in ML is increasingly driven by compute-intensive, engineering-first approaches.
- Scaling to larger training sets often yields better results than theoretical optimizations.