What information should AI evaluators share?

This blog post is part of a series of posts by members of the Network for Advanced AI Measurement, Evaluation, and Science (AAIMES Network) on best practices for AI evaluations. The series also includes posts from:

• The European Union (EU) AI Office Safety Unit;
• L'Institut national pour l'évaluation et la sécurité de l'IA (INESIA) in France;
• The Singapore AI Safety Institute.

Evaluation of large language models (LLMs) and the AI systems they support is becoming increasingly critical with their deployment at scale and across many contexts. The range of evaluators and methodologies continues to evolve to meet the demand from deployers, users, and governments, for assurance regarding model and system performance and risks.

Evaluators vary in their positioning and interests, with model developers, third-party evaluators, government agencies, and independent researchers all playing different roles in a complex AI evaluation ecosystem. This necessarily leads to variance in disclosure norms. Nonetheless, this post aims to articulate principles about evaluation reporting that apply generally to evaluators across the ecosystem. These principles aim to promote clarity and shared understanding, support consistent interpretation of results, and enable interoperability, comparability and coordination among AI evaluation ecosystem participants.

Best practices for disclosure of evaluation results have been articulated elsewhere, such as in NIST-800-2 and the AAIMES Network’s Best Practice for Automated Evaluations. The proposals in this post are consistent with those discussions but raise broader considerations that generalize beyond the context of automated evaluations.

As evaluations proliferate, evaluators need to share sufficient information about how they are being conducted to:

• Facilitate accurate interpretation of evaluation results;
• Ensure the usefulness of evaluations for decisions about an LLM or system’s deployment;
• Enable a clear assessment of the quality and limits of the evaluation; and
• Enable replication by other evaluators (where feasible) of the evaluation or the methods it employs.

Beyond supporting technical understanding, these disclosure practices are increasingly foundational to effective governance of advanced AI systems. Clear, consistent and appropriately scoped evaluation reporting enables policymakers, regulators, and other governance professionals – as much as other AI industry stakeholders and individual consumers – to assess risks, compare systems, and make informed deployment and oversight decisions. In this sense, evaluation transparency is not only a matter of good practice, but an emerging prerequisite for accountable AI governance.

Overall, the default posture should be to maximize transparency. At minimum, we propose that evaluators should:

• Articulate the intent of the measurement and the contexts of use it applies to;
• Outline the methodology (i.e., what is being measured and how);
• Provide the model specifications (i.e., the model or system being tested and its settings);
• Delineate caveats and limitations on how to interpret the results, including the necessary statistical information on uncertainty and unknowns in the measurement process; and
• Signal any potential conflicts of interest in the production and reporting of the evaluation results.

At the same time, expectations around disclosure should be proportionate to the potential impact and risk profile of the system being evaluated. However, there remain considerations around the risk of sharing too much detail, for example by revealing adversarial techniques. Evaluators therefore need to be equipped to make deliberate and thoughtful decisions about what to withhold, including:

• Datasets containing safety critical information;
• Adversarial testing insights that could provide uplift to malicious actors; and
• Test sets that risk data contamination.

We address each of these elements in more detail below.

Intent: What questions does the measurement seek to address?

The purpose of the evaluation

Evaluations are typically developed to assess key aspects of the model or system such as its capabilities, its alignment, or its safeguards, as well as to inform people about how an LLM or AI system might behave in specific contexts (see the “Decision and audience the evaluation must inform” section of the blogpost “What Should Evaluators Prioritise?” by INESIA). To communicate such information accurately and effectively, evaluators should specify why an evaluation was developed, what aspects of the model or system it seeks to illuminate, which opportunities or risks it is testing a model against (see this blog post by the EU AI Office Safety Unit), and the contexts to which its results pertain. Being explicit about such intent helps the audience understand how to interpret and apply the results, and the degree to which the evaluation is relevant for a given purpose. It provides a frame for all other details regarding the evaluation and its results, including whether the evaluator’s choices—of experimental design, evaluation metrics, and dataset curation—are appropriate to the evaluation’s aims.

The context of use in which a model is being evaluated

Some evaluations may be meant to represent LLM usage by a typical user, while others may target sophisticated or specialized power users, malicious users, or certain deployment contexts (e.g., healthcare or software development). Knowing this helps the audience assess whether an evaluation is relevant for the LLM usage context that interests them (see INESIA’s blogpost for related issues). Similarly, an evaluation could be about performance in a single-trial scenario, or performance in a context when users ask an LLM the same question in hundreds of different ways—such as in jailbreaking, misuse, and criminal uplift scenarios—to try to elicit a desired answer. Evaluators must provide enough information about the context of use within which its measurements were conducted to support inferences by its audience on how broadly (or how narrowly) the evaluation results generalize to other contexts.

Methodology: What is being measured and how?

Metrics, experimental design, and the rationale for their selection

It is crucial that evaluators are explicit about what they are measuring (the measurement construct), how the measurement process was carried out (the evaluation protocol), and why these choices are appropriate for the intended purpose. To trust reported results, the audience needs to be able to assess construct validity (i.e., whether the metrics used actually measure what they claim to) and external validity (i.e., whether the testing methods properly reflects the context and conditions of use the evaluation seeks to address). Precise details of the measurement are needed. What is the evaluation measure, and how is it calculated? This might, for example, involve an automated scoring method, an LLM-as-judge, or human assessment of the model’s outputs. Specifications of this measurement, such as the mathematical equation or code for computing it, the prompt and model-version for the secondary LLM used as a judge, or the instruction rubric given to human evaluators, should all be made clear. Evaluators should also note how quantitative measures were, or were not, supported by qualitative checks. If automated measures were used, to what degree did humans audit these measures for accuracy? Additionally, an explanation for why these metrics and methods were chosen is necessary for understanding the significance of the results with respect to the aspects of the model they are being used to measure. Without this information, it is difficult to interpret what numerical scores mean.

Datasets of inputs and target outputs

Many model evaluations rely on a dataset of inputs, prompts, or questions and, potentially, a set of “correct” answers or outputs. The nature of these datasets needs to be described in as much detail as possible (when it is safe to do so; see the “What to withhold” section below). Is the dataset public or private? What kinds of questions and answers are in it? What sorts of tasks? What languages are used? More subtly, when was it created? Choices about language and writing style for prompting are also relevant and can be a determining factor in evaluation results. Are any of the questions, tasks, or answers sensitive to variations in time or region in a way that might affect model performance? For example, a dataset evaluating LLM general knowledge may include a question like “Who is the current Prime Minister of Canada?”, which is not only focused on a particular region but also has a correct answer that changes over time. The inclusion of such questions can be valid or even necessary, but knowing they are in the dataset can be crucial for interpreting evaluation results.

Specifications: What model or system is being tested, under what conditions?

Model or system specifics and settings

Evaluators need to provide relevant information on which model or system is being evaluated, including the version and any other settings that could modulate evaluation results, such as the temperature settings and system or user prompts included in the context window for inputs. The aim is to give enough details that others can either replicate the evaluation or understand how it relates to their own use case or their interest in the results. For evaluations of more complex LLM-based systems, this can require quite a lot of detail (see the Singapore AI Safety Institute’s blogpost for related issues).

Interpretation: What are the extent and limits of the measurement’s validity?

Evaluators should provide enough information to support not just the reading of results, but their appropriate interpretation, including where those results may not generalize.

Confidence intervals

To enable interpretation of the numerical values reported in evaluation results, we propose that almost all evaluations should include confidence intervals. What is most meaningful or generalizable about an evaluation is usually not simply an LLM’s percentage of successes (e.g., 83%) on a particular set of tasks (say, a set of 100 questions) on a particular run. What is more meaningful is how well the LLM would perform on questions that are relevantly similar to the 100 on which it is measured. Determining this requires computing a confidence interval, which represents the range of scores expected if the evaluation were to be repeated with a different set of similar questions. (In the example above, for instance, a 95% confidence interval would yield a score that ranges from 74% to 89%.) Reporting these ranges is vital to the interpretation of benchmarks. After all, if one LLM gets a score of 83 and another a score of 87 out of the hundred questions, it might seem like the second model is better than the first, but given overlapping confidence intervals it might be that, statistically, there is no reason to believe one is better than the other.

Benchmark limitations

A related concern is whether an evaluation metric is saturated. For example, if all the scores of different models are very high on a given benchmark, scores become less useful in telling the models apart, as illustrated by two models with slightly different scores but overlapping confidence intervals. Information on how a model’s results on a given metric diverge from or cluster with the results of other models, is crucial for understanding the significance of the results.

Error mitigation strategies

Choices of measurement methods can, in some cases, introduce systematic errors. For example, using an LLM-as-judge to count a model or system’s refusals on certain prompts may lead to systematic output classification mistakes, skewing the results in one direction or another. Spot-checking outputs with human reviewers can catch and correct this. Disclosing the use of automated methods and the manner and extent of human auditing allows the audience to assess the scope of the uncertainty that arises from automated measurement.

Potential conflicts of interest

Finally, evaluators should disclose their relationship with the model or system developer or deployer. Providing transparency regarding an evaluator’s positioning in the ecosystem and interests in the evaluation can be critical to building trust with the audience. By disclosing this, the evaluator enables informed decisions by their audience about potential biases and the need for further independent testing to verify or refine the results.

What to withhold

It may seem ideal to make all aspects of an evaluation public for full transparency and to facilitate replicability. However, some features of evaluations cannot safely be made public. Considerations for what information to withhold can be complex and multifaceted but typically pertain to the design of the evaluation.

In practice, decisions about what to disclose should be guided by the intended audience. Different levels of detail may be appropriate for:

1. Public dissemination (e.g., publications, blogs, presentations), where information should support understanding while minimizing risks of misuse;
2. Trusted actors (e.g., governments, designated evaluation networks, or independent oversight bodies), where more sensitive information may be shared under appropriate controls; and
3. Developers or system owners, who may require detailed findings, including vulnerabilities, to improve model safety and performance.

Being explicit about these distinctions can help evaluators strike an appropriate balance between transparency and risk mitigation.

Evaluation datasets containing sensitive or safety critical contents

Some evaluation datasets, especially those related to risk areas like chemical, biological, radiological, or nuclear hazards (CBRN) or cybersecurity, cannot be safely released publicly. For effective measurement, such evaluations may need to use datasets containing detailed question-answer pairs of how to make weapons or conduct cyberattacks. Sharing such data could be permissible in limited contexts, after careful vetting, with recipients that have a need to know for legitimate purposes. In these cases, rather than making available the contents of the datasets, evaluators should instead share how the dataset was created and validated, along with enough metadata (and possibly synthetic examples) to help others understand and assess the results without exposing sensitive information.

Insights from adversarial testing

When an evaluation is about misuse cases, like jailbreaks, making evaluation information completely public would amount to providing the public with a step-by-step adversarial playbook for tested models. In these scenarios, decisions about what can safely be shared, and with whom, should follow careful analysis of what could be inferred from the shared information. In general, protocols for when and how sensitive information can be shared will depend on the intended audience. For example,

• Results that document jailbreak methodologies could be securely transmitted to trustworthy parties with an established role in improving model or system safety in the public interest – such as the model or system’s developers, or government institutions like the members of the international Network for Advanced AI Measurement, Evaluation, and Science.
• On the other hand, when it comes to public disclosure of results through publications, presentations, or other communication channels, evaluators should withhold all features of the evaluation metrics, protocol, and test sets that could assist malicious actors in misusing AI models or systems. Even in such cases, it remains important, however, to provide whatever details possible to assist the audience in understanding and assessing the results, such as metadata or a high-level description of the datasets and setup, in a way that would preclude inferences about the adversarial techniques or prompts used in testing a system’s susceptibility to jailbreaks.

Evaluation datasets that risk data contamination

It is a well-established problem that, when evaluation datasets are made public, that same data may become part of the training data for later models. This ‘contamination’ means that models are trained on their own test sets, rendering these tests useless for predicting how well the model will perform in new situations. While there are methods to attempt to detect this (such as the inclusion of “canary strings” in published testing data), there is a general difficulty in keeping any publicly available information out of LLM training sets. To preserve the integrity of the evaluations and to minimize data contamination, we propose that benchmark datasets are not shared in full, and some amount of data is kept back from public release. System performance on that held-out dataset can be used as a control beside the rest of the dataset to assess contamination, and to provide at least one uncontaminated measurement. Keeping such datasets, and performing such evaluations, is a role well-suited to impartial third-party entities, including the organizations in the Network for Advanced AI Measurement, Evaluation and Science. The amount of data to withhold is an open question under active debate among third-party evaluators. Given the risk that model developers can use released test sets to generate further synthetic data on which to train their LLMs, contamination risk can only be substantially reduced by withholding a large proportion of the test sets. Considering the data contamination problem, concerted work is required to ensure that evaluation methods evolve to outpace developers’ capacity to optimize their models for known evaluation methods and datasets.

Conclusions for evaluators

Reporting results of LLM or AI system evaluations is a significant communication challenge, with respect to both the technical details and the interpretation of the results. But getting it right matters. Without clear, transparent reporting, results cannot be trusted, compared, or used to make real decisions. A default posture towards transparency is needed to facilitate results’ replicability, usefulness, and accurate interpretation. More information also allows other LLM researchers to find flaws in evaluations to improve them and make informed choices about their own LLM usage. At the same time, there is a balance to strike between standardization and flexibility. Greater alignment can improve comparability, but overly rigid approaches risk encouraging gaming or limiting innovation.

As evaluation reporting matures, we expect increasing usage of evaluation report templates (see this template by the US Centre for AI Standards and Innovation). Such templates are useful but must remain flexible enough to cover the different issues raised above. A layered disclosure approach is needed to balance the needs for transparency and safety, while preserving measurement validity over time.

Given the global nature of AI development and deployment, advancing shared approaches to evaluation reporting will require ongoing coordination across jurisdictions, standards bodies, and evaluation networks. NAAIMES will continue to engage in this space to drive collaboration and alignment, both on how evaluations are conducted and how their results are reported.