The European Commission this month released implementing guidance under the EU AI Act that establishes a mandatory, standardized Model Evaluation Report for commercial AI systems used in the single market. The move — which regulators say aims to make model capabilities, limitations and risks legible to buyers and auditors — has prompted major cloud providers, model vendors and enterprise software firms to rush out templates, APIs and compliance playbooks.

What the new requirement means

The implementing guidance requires vendors of foundation models and AI-driven software to produce a machine- and human-readable Model Evaluation Report (MER) accompanying any product marketed or sold for business use in the EU. According to Commission materials circulated with the guidance, a compliant MER must include:

  • Model identity and provenance: training data lineage (where permitted), model architecture and provider contact details.
  • Evaluation metrics: standardized scores across safety, bias, robustness to distribution shift, and calibrated hallucination rates on benchmark and domain datasets.
  • Risk classification and declared intended uses and prohibited uses, mapped to the AI Act’s risk tiers.
  • Testing artifacts: descriptions of red-team exercises, adversarial evaluations, and third-party audit outcomes where available.
  • Operational constraints: recommended deployment settings, required guardrails, and monitoring implications.
  • Data handling disclosures: whether outputs are logged, retained, or used to further train models, and available opt-outs.

Regulators expect MERs to be both human-readable (for procurement officers and auditors) and machine-actionable (for automated compliance tooling and procurement systems), with a standardized JSON schema for automated ingestion.

Industry response: templates, APIs and coalitions

Within days of the Commission’s guidance, a number of enterprise AI vendors and cloud providers published draft MER templates and integration guides aimed at customers:

  • Major cloud platforms released sample MERs aligned to the Commission’s schema and documentation on generating reports from managed model services.
  • Several model vendors and startups announced automated evaluation toolkits that produce the MER fields from existing test suites and red-team logs.
  • A vendor coalition has formed to propose an extension to the Commission’s JSON schema to support sector-specific fields (finance, health, HR) and to standardize provenance metadata.

Vendors characterize these releases as a positive step toward transparency, but also note the practical hurdles: generating defensible, repeatable metrics at scale; protecting proprietary training data; and aligning across different legal regimes.

Why vendors are moving fast

  1. Procurement pressure: Large EU enterprises and public agencies are already signaling they will require MERs in RFPs.
  2. Auditability: MERs create a single source of truth for internal compliance teams and external auditors.
  3. Competitive differentiation: vendors that can produce robust MERs early hope to simplify buying decisions and reduce customer due diligence friction.

Implications for enterprise buyers

For enterprise software and procurement teams, the MER requirement changes the buying playbook in several concrete ways:

  • Pre‑contract checks will expand to include MER review and automated ingestion into vendor risk systems.
  • Legal and compliance teams will need to verify MER claims against independent audit results or require contractual guarantees tied to MER metrics.
  • Procurement templates and scoring models should be updated to weight MER indicators — for example, bias mitigation validation and hallucination rates — rather than relying solely on provider brand or SLAs.

Security and privacy teams should also map MER disclosures to data governance controls; MERs will likely reveal whether providers log user inputs and outputs, information that affects data residency and retention policies.

Technical challenges and unresolved points

Several technical and policy questions remain unresolved:

  • Proprietary training data: vendors caution that full lineage disclosure may risk IP leakage or contravene non‑disclosure commitments from data providers.
  • Standardized metrics: while the Commission specifies metric categories, the community has not converged on single-test definitions for "hallucination rate" or "robustness" that are meaningful across domains.
  • Third‑party audits vs. self‑attestation: the guidance allows self-generated MERs but suggests heavier scrutiny for high-risk applications, leaving open how audits will be funded and certified.

How teams should prepare now

Enterprises evaluating AI business software should take immediate, practical steps:

  1. Map vendor inventory to the MER requirement: identify which suppliers must produce MERs and prioritize high‑risk systems.
  2. Update procurement documents to request both the MER and raw test artifacts (scorecards, red-team logs) and specify acceptable evidence types.
  3. Build ingestion pipelines: prepare to automate MER consumption into vendor risk platforms using the Commission’s JSON schema or vendor-supplied APIs.
  4. Require contractual remedies: insist on remedial obligations and SLA crediting tied to MER claims for critical systems.
  5. Engage with vendors early: collaborate on sector-specific testing and jointly agree on acceptable evaluation datasets and thresholds.

What to watch next

The coming weeks should reveal how harmonized vendor MER implementations will be and whether industry consortia can converge on interoperable test suites and schema extensions. Regulators will also need to clarify enforcement expectations around self-attestation versus third-party verification, and how MERs interact with existing obligations under GDPR and sectoral rules.

For AI-business-software teams, the new Model Evaluation Report requirement represents both an operational burden and an opportunity: those who integrate MERs into procurement, risk and technical controls early will likely gain faster, safer access to the accelerating suite of commercial AI capabilities.