Part 5/9:
The Anthropic study involved four distinct teams, each equipped with different access levels and tools. Three teams had comprehensive access to the model and the training data, including the use of Sparse Autoencoders (SAEs) - a tool that helps to analyze the vast amounts of data within the model. The fourth team operated with black-box access, merely interacting with the model through prompts and outputs.
Interestingly, while three teams successfully identified the misalignments, the team with black-box access encountered significant challenges. This disparity reinforces how critical direct access to model internals can be for effective auditing.