OpenAI Evals Is Changing: Keep Model Tests Portable

An evaluation platform can change while the business process it tests stays in production. That makes portability a practical concern for teams trying GPT-6 Astra, Claude Fable 5.1 or another candidate model. The evidence supporting an upgrade should remain usable when a dashboard or API is retired.

As checked on 9 September 2026, OpenAI's official Evals guide states that its Evals platform will become read-only for existing users on 31 October 2026 and is scheduled to shut down on 30 November 2026. Teams using it should review the current notice and their dependencies before planning a migration. OpenAI Evals transition notice.

What is a portable model evaluation?

A portable model evaluation is a repeatable test whose inputs, expected behaviour, grading rules and results are stored independently of the tool used to run it. It lets a team compare a new model or application change against the same business requirements even when its provider, dashboard or execution system changes.

Inventory the evidence before choosing a replacement

Begin with the assets you would need to reconstruct your last meaningful test. That includes the task instructions, input examples, supporting documents, reference answers, grader definitions and reviewer notes. Add the exact model and configuration used for each run, along with timestamps and usage records.

Then identify dependencies hidden in the current runner. A grader may call another model. A prompt may reference a provider-hosted file. A test may assume that tools return data in a particular order. Copying a spreadsheet of scores will preserve none of those assumptions.

Make a small inventory with an owner and export status for each asset. Where an interface lacks a convenient export, document what can be recovered and what must be rebuilt. Avoid claiming the evaluation has been migrated until a sample run can be reproduced outside the original arrangement.

Preserve the decision behind the grade

An overall score is a summary of a judgment. Keep the rule that produced it. If an invoice extraction test passes when the amount matches, specify whether the comparison also checks currency, decimal handling and the source page. Otherwise, a new grader may produce a similar number while testing a different outcome.

For subjective outputs, use concrete criteria. A good briefing identifies the decision, distinguishes evidence from interpretation and includes unresolved questions. Those are easier for reviewers to apply consistently than an instruction to rate professionalism from one to ten.

OpenAI's dataset guide describes annotations, reference values and several grader types. Its conceptual distinction between human judgment and automated checks remains useful, but the same page carries the platform transition notice. Do not assume another feature named there is a permanent migration destination without checking current support.

Build a small independent replay

Choose one representative task and run it from a copy of the saved inputs. Confirm that required files, tools and credentials resolve. Keep the output and grade together so someone can inspect the underlying answer when the score changes.

Replay against the existing production configuration before introducing a new model. If results diverge materially, investigate the migration first. A new tool runner, changed document extraction or different grader can affect the result independently of the model you intend to evaluate.

Once the baseline is credible, test the candidate. Keep results grouped by failure type: missing evidence, incorrect value, invalid output, unexpected action and failure to stop. That makes it possible to decide whether an upgrade fixes the business problem or merely improves the average.

If your team has model experiments but no reusable acceptance criteria, start a brand growth assessment with the workflow and its known failures. The first useful deliverable may be a testable definition of done.

Separate rollout gates from benchmark scores

An offline test can establish that a configuration is worth trying. A production rollout also needs evidence that the surrounding process behaves correctly. Use a staged approach with explicit stop conditions.

First, run the candidate on historical examples without external actions. Next, produce drafts alongside the existing process and let employees compare them. Only then consider a limited live route with a named reviewer and a working fallback. These stages are proposed operating practice, not provider promises or required durations.

At each stage, ask whether the errors are detectable before they matter. A harmless formatting defect and an incorrect customer update should not have equal influence on release approval. Some failures should block expansion even when the overall pass rate improves.

Document who can pause the workflow and how quickly work returns to the existing process. An upgrade is easier to approve when the team understands how to reverse it without losing requests or duplicating actions.

Build UAE cases into the permanent test set

For a Dubai operations team, include documents mentioning Dubai Municipality, local delivery addresses and AED amounts where those appear in its actual work. For an Abu Dhabi team, include its own Arabic and English templates. These are suggested inputs, not claims about agency participation in a particular AI programme.

Preserve a reviewer-approved interpretation of ambiguous names, dates and translations. Record when an example depends on an old policy or expired commercial term so the test does not reward outdated answers indefinitely.

Use de-identified or synthetic cases where suitable, while retaining the structure that caused the original failure. Replacing all difficult documents with perfectly formatted English text will make the suite easier to handle and less useful to the business.

Keep the tests alive after migration

Assign an owner to each important business criterion. When a live failure occurs, decide whether it exposes a missing test, a broken integration or an unclear requirement. Add a focused example with a note explaining why it matters.

Keep a stable comparison set alongside newer cases. If the entire suite changes on every run, a rising score can reflect easier questions rather than a better system. Track results by suite version and retain the evidence that supported each rollout decision.

Avoid using the same examples for endless prompt tuning and final acceptance. Keep some representative cases unseen during tuning, then review the candidate on those cases before expansion. This reduces the risk of building a prompt that performs well only on familiar demonstrations.

Frequently asked questions

Does the Evals notice mean model evaluation is ending?

No. It concerns a named OpenAI platform. The business need to test model behaviour remains. Preserve your assets and check the current migration options for the services you use rather than assuming evaluation itself depends on one dashboard.

Can we keep only the final scores?

Scores alone are insufficient for diagnosing changes. Save the inputs, configuration, outputs, rules and reviewer rationale where permitted. The team needs to inspect what the system did and why a result passed before relying on the score for an upgrade.

Should migration and a model upgrade happen together?

Prefer establishing the existing baseline on the new runner first. Changing both at once makes regressions harder to explain. If timing forces a combined change, record that limitation and strengthen the review stage before expanding live use.

Keep ownership of the evidence

Your evaluation history is part of the operating knowledge of the business. Explore Innvatio's agent implementation approach or begin an assessment to connect model testing with clear approval and recovery procedures.

Every engagement starts with a brand growth assessment: free at first, with the full-depth assessment paid once you are accepted into the cohort.

By Bashar Hamdan, Founder of Innvatio. Research checked 9 September 2026.