4 pieces selected from AI Alignment Forum, The Gradient — only the ones worth your time.
1. Data filtering works a lot worse than you would expect
AI Alignment Forum
This work was largely done during Neel Nanda's MATS 10.0 Exploration Phase.
J Rosser and Dohun Lee are co-first authors for this post with equal contribution. Josh Engels and Neel Nanda supervised the project, and provided guidance and feedback throughout.
Tweet Thread
TLDR
Models can acquire undesirable traits from during supervised fine-tuning (SFT). A natural thing to try is to identify the
Key takeaways
2. The Case for Model Forensics
AI Alignment Forum
The essay argues for the development of model forensics, a field focused on investigating concerning AI behaviors to determine if they stem from misalignment or benign causes. It highlights that catching bad actions alone is insufficient to conclude misalignment, as models may act harmfully due to confusion or lack of capabilities rather than intentional subversion. The authors review ten cases from literature where concerning behaviors had benign explanations, emphasizing the need for a neutral investigation to either exonerate or incriminate the model.
Why it matters
Developers building AI products need robust methods to distinguish between intentional misalignment and benign errors in AI behavior. Model forensics provides a framework to investigate and mitigate harmful actions effectively, ensuring that responses are proportional to the actual risk. This is crucial for maintaining safety and trust in AI systems, especially as they become more complex and autonomous.
What you can build with this
Develop a toolkit for conducting model forensics on AI systems, starting with a set of diagnostic tests and analysis scripts that can be applied to logs of concerning behaviors. This toolkit can include methods for tracing model decisions, evaluating intent, and distinguishing between benign errors and intentional misalignment.
Key takeaways
- Model forensics is essential for determining whether concerning AI behaviors are due to misalignment or benign causes.
- Initial investigations often reveal benign explanations for harmful actions, underscoring the need for thorough analysis.
- Effective model forensics requires a neutral approach to either exonerate or incriminate the model, guiding appropriate mitigation strategies.
3. After Orthogonality: Virtue-Ethical Agency and AI Alignment
The Gradient
This essay challenges the conventional goal-oriented approach to AI alignment, arguing that human rationality is not driven by fixed goals but by adherence to practices—networks of actions, dispositions, and evaluation criteria. The author proposes that AI systems should similarly align with ethical practices rather than rigid objectives, drawing from virtue ethics to frame AI behavior within dynamic, context-sensitive norms.
Why it matters
Developers building AI products often rely on goal-based frameworks, which can lead to misalignment when objectives conflict or contexts shift. This essay provides a philosophical foundation for designing AI systems that adapt to ethical practices, reducing risks of unintended behavior in real-world applications.
What you can build with this
Develop an AI-driven moderation tool for online communities that aligns with virtue ethics, dynamically adjusting its behavior based on evolving community norms rather than fixed rules.
Key takeaways
- Human rationality is practice-aligned, not goal-directed.
- AI alignment should focus on ethical practices, not static goals.
- Virtue ethics offers a framework for context-sensitive AI behavior.
4. Shape, Symmetries, and Structure: The Changing Role of Mathematics in Machine Learning Research
The Gradient
This essay examines the evolving role of mathematics in machine learning research over the past decade. It highlights a shift from mathematically principled architectures to compute-intensive, engineering-driven approaches that prioritize scaling and large training sets. The author observes that while mathematical rigor was once central to ML progress, recent advancements have been driven more by empirical scaling and computational power than by theoretical innovations.
Why it matters
For developers building AI products, this shift underscores the importance of focusing on scalable, engineering-first solutions rather than over-optimizing for theoretical elegance. It suggests that practical, large-scale implementations often yield better results than smaller, mathematically refined models, which can guide resource allocation and development priorities.
What you can build with this
This week, you can experiment with scaling a simple, well-understood model (e.g., a basic transformer) on a large dataset to observe how performance improves with scale, rather than refining a smaller, more mathematically complex model. Use this to validate whether scaling alone can achieve your product goals without additional architectural complexity.
Key takeaways
- Mathematically principled architectures now yield marginal improvements compared to scaling compute and data.
- Modern ML progress is driven more by engineering and computational power than by theoretical breakthroughs.
- Scaling existing models often outperforms designing new, mathematically refined architectures.