AI labs have long relied on model benchmarking, a supposedly neutral system meant to measure the superiority of one AI product over another. However, the methods behind these evaluations are still maturing and can be notoriously difficult to reproduce, casting doubt on their reliability.

Irene Solaiman, Hugging Face’s chief policy officer, highlighted the complexity during a discussion at TechCrunch Disrupt, noting that trust in these evaluations hinges on reproducibility and open access to the models being tested.

The emerging concept of 'embedded evaluators' – where supposedly neutral third-party observers are placed inside AI organizations – has gained traction as a solution to the recent rash of safety incidents. However, doubts remain over the true neutrality and effectiveness of such evaluators, with critics questioning the suitability of firms like Accenture for the task.

Evaluations are now moving beyond just the model itself to encompass the entire system, including the harness and agent. This adds another layer of complexity, as the performance of a model can be significantly influenced by the harness it’s accessed through.

The current evaluation landscape is a bit of a free-for-all, and for benchmarking to remain useful, a more uniform, industry-wide approach may be necessary. Until then, the question remains: can we truly trust these evaluations, or are we just measuring the weather with a broken thermometer?