Add azd ai eval extension for Foundry evaluations - #9500
Open
Mohamed Hessien (m7md7sien) wants to merge 243 commits into
Open
Add azd ai eval extension for Foundry evaluations#9500Mohamed Hessien (m7md7sien) wants to merge 243 commits into
Mohamed Hessien (m7md7sien) wants to merge 243 commits into
Conversation
New azd extension exposing azd ai eval, registering the azure.ai.eval service-target provider. Scaffold only: manifest, entrypoint, root command.
Lifted eval_api (models, operations, poller, generation, portal_urls) and dataset_api from azure.ai.agents, de-agent-scoped. Added evalcore with IsTransientError and an EvaluatorList that accepts either a bare string or a mapping with a threshold. Skipped artifacts.go and eval_config.go since the config model differs.
… commands - EvalConfig and GenerateConfig types with validation covering cross-references, duplicate names, unsupported target types, and evaluation levels - ResolveGroup picks the only group or errors with the available names - ArtifactPath accepts a directory or an explicit file path for local_dir - evalContext resolves the project endpoint (flag, azd env, host env) and builds both clients against the azd developer CLI credential - dataset create/update/list/show/delete with -o json - Tier-0 tests for parsing, validation, group resolution, and path handling
- buildEvalGroupRequest maps evaluators to testing criteria, keeping the builtin prefix on evaluator_name while stripping it from name, and carries per-evaluator thresholds in initialization_parameters - run resolves the group from --eval-id, a pinned id, or the azd environment, creating it when absent, then binds the dataset to the run since the group has no dataset binding today - Local datasets are sent inline with optional truncation; registered datasets are referenced by id
- evaluator upload/update/list/show/builtins/delete; rubric evaluators only in M1, code evaluators deferred to M2 with the folder walk and RBAC they require - normalizeRubricBody accepts a bare definition or a full document - results show/export with per-criteria pass and fail counts, --failed-only, and JSON or CSV output, replacing the counts-only view - Added ListEvaluators, ListEvaluatorVersions, DeleteEvaluatorVersion, and CancelOpenAIEvalRun to the eval client
init scaffolds both YAML files and the artifact directories without any service call, so it works offline and unauthenticated. Built-ins are referenced from the group but never declared as custom evaluators. A dataset flag containing a path becomes a local source; a bare name references a registered dataset. Tests assert the scaffold loads and validates, and that paths are used verbatim rather than re-rooted.
…tion Registers the azure.ai.eval service target so azd up and azd deploy reach this extension; the extension ships no deploy command of its own. - Reads the eval config from the service entry's inline properties, the same AdditionalProperties channel the agents extension uses - Deploy reconciles datasets, then evaluators, then eval groups, since a group references the versions the first two resolve to; it fails fast and the next deploy resumes - Datasets are change-detected with a local SHA-256 digest kept in the azd environment, because the dataset API returns no content hash and comparing against the service would mean downloading the blob every deploy - Evaluator definitions come back inline, so those are compared directly - Package and Publish are no-ops; eval artifacts are plain files already on disk
- generate submits the rubric and dataset generation jobs, downloads the artifacts locally, and writes source references into the deployment spec - MergeArtifactRefs edits through the yaml Node API so comments, key order, and hand-edited sibling keys survive; matching is by name and merging is idempotent - Raised the client poll budget from 2s x 300 to 5s x 720. The old 10 minute limit gave up while the service was still working, which is the timeout that forced a second command - A supplied --evaluator or a local --dataset is honored and its generation is skipped - Tests cover comment preservation, section creation, idempotence, and fingerprinting
Adds the templated release pipeline for the new extension and lists it as a dependency of the microsoft.foundry meta-package. The registry.json artifact entries are generated by the release, so they are not hand-authored here.
… tests Live testing against a real project found two issues. 1. The dataset model only bound snake_case URIs (data_uri, blob_uri), but the project endpoint returns camelCase (dataUri). ResolvedBlobURI therefore returned empty, which would have failed the generate download much later with no useful error. Both spellings are now accepted. 2. Built-in evaluators do not share one input contract. builtin.ifeval requires instruction_id_list and is rejected under the agent-target data mapping with MissingRequiredDataMapping. The live tests now select an evaluator whose inputs match, and the helper documents why. Live tests are gated behind the live build tag and AZURE_AI_EVAL_E2E_LIVE, and clean up every resource they create. Verified: builtin listing, the full dataset pending-upload lifecycle with version auto-increment, and eval group creation returning 201.
…published contract The builder sent one fixed data mapping and one fixed set of initialization parameters to every evaluator. That only suited agent-target quality evaluators, and the service rejected the rest. The evaluator listing publishes a full contract per evaluator: definition.data_schema (accepted and required inputs), definition.init_parameters, and supported_evaluation_levels. The builder now reads it and shapes each testing criterion accordingly. This fixes four concrete defects. - Required inputs were never honoured, so builtin.ifeval (instruction_id_list), builtin.similarity (ground_truth) and builtin.retrieval (context) all failed with MissingRequiredDataMapping. Fields not supplied by the agent target are now bound to dataset columns, and the item schema declares them. - Inputs an evaluator does not accept were sent anyway. - initialization_parameters always carried model, deployment_name and threshold. No evaluator accepts 'model', and builtin.ifeval accepts nothing at all. Parameters are now filtered to the declared properties, and a required one that is missing is reported locally. - evaluation_level was sent as run metadata, where it has no effect. It is an initialization parameter on the evaluators that declare it. It also encodes an exclusivity rule the service enforces: 'messages' and 'query'/'response' cannot both appear in a mapping, so the evaluation level selects between the conversation and turn shapes. A missing dataset column is now caught before the request is sent and names the column, rather than surfacing as a 400 pointing at testing_criteria[0].data_mapping. An evaluator with no published contract keeps the previous agent-target shape, so custom evaluators are unaffected. Verified against a live project: all ten built-ins are accepted, where two previously failed. The live test exercises the shipping builder rather than a hand-rolled request, so a regression in this logic fails the suite. Adds DeleteOpenAIEval so those tests clean up after themselves.
…invokes Without build.ps1 and build.sh the dev kit reported a successful build in under a second and produced no binary, so azd x pack had nothing to package and azd x publish failed with 'Artifacts not found'. Copied from azure.ai.agents with the version package path retargeted, plus its golangci config. Verified end to end: build, pack, publish, install from the local registry, and 'azd ai eval --help' listing every command.
…ly returns The listing spells it evaluator_type, so the TYPE column in 'evaluator list' and 'evaluator builtins' was always blank. Both spellings are now accepted.
Running a real 'azd deploy' against the service-target provider surfaced two failures that no unit test covered. Evaluator references only decoded from YAML. azd hands the service entry to the extension as JSON, so a group written as '- builtin.task_adherence' -- the form the CLI's own init command writes -- failed with 'cannot unmarshal string into EvaluatorRef'. EvaluatorList now decodes and encodes the mixed string-or-mapping form through JSON as well, and a test asserts the two decoders agree. The dataset reconciler passed the declared version straight to UploadNewVersion, which derives the next version from it. A declaration without an explicit version passed empty, so every deploy retried 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset once that version existed. It now looks up the latest registered version first. Verified against a live project: first deploy publishes the dataset at 1.0 and creates the group; an unchanged redeploy reports 'unchanged at version 1.0' and uploads nothing; and editing the dataset publishes 2.0 and recreates the group, since groups are immutable.
azd core does not resolve $ref for extensions. It strips the ServiceConfig fields it owns and leaves $ref at the top of the map for the owning extension to resolve, so a service authored the way the spec documents it -- host: azure.ai.eval plus $ref: ./evals/azure.yaml -- parsed to an empty config. azd deploy then reported success in three seconds having created nothing, which is worse than failing. The provider now calls foundry.ResolveFileRefs with the project root from the azd project client. Relative source paths inside an included file are written against that file, but ResolveFileRefs inlines content without rebasing them, so the include's own directory is now the base for source resolution. Verified against a live project: the $ref form deploys, and the dataset fingerprint matches the one from the equivalent inline config, confirming both forms resolve to the same file.
…ation changes Change detection only covered upstream artifacts, so retargeting a group at a different agent, swapping an evaluator, or changing the judge model left the old group in place. Groups are immutable, so the edit silently had no effect and later runs kept evaluating the previous definition. The group's declaration is now fingerprinted alongside the dataset and evaluator artifacts. The id and description are excluded: one is server assigned and the other is cosmetic, so neither should force a recreate. The digest is recorded when an existing group is reused as well as when one is created. Recording it only on create meant a group deployed before this change never established a baseline, and the first edit after it would still go undetected. Verified against a live project: changing the target produced a new group id, and two further deploys with no change reused it.
The data-plane clients trace every request and response through log.Printf, which Go writes to stderr by default, so a plain command interleaved raw URLs and status lines with its own output. A long generate run was mostly HTTP traces. Ports the debug setup from the agents extension: the standard logger is discarded unless --debug or AZD_EXT_DEBUG is set, and debug output goes to a dated file rather than the terminal. The hook chains the SDK PersistentPreRunE instead of replacing it. Assigning PersistentPreRun has no effect once the E variant is set, and overwriting the E variant would drop the SDK own setup. Also reports jobs as submitted when generate is given --no-wait, which is a successful submission rather than an empty result.
…fails Data generation with an agent source is accepted and then fails within seconds with DataGenerationJobSystemError, whose message says only that something went wrong and to try again. It is not transient: it reproduces for every agent tried, while the identical request without the agent source runs normally. The CLI now names the agent, says a retry will not help, and points at the two workarounds, instead of relaying advice that cannot succeed.
The spec lists run start, list, show and cancel, and M1 requires every operation to be reachable atomically, but run was a single composite command with no subcommands. Listing runs, inspecting one, and cancelling an in-flight run were unreachable, even though the client already had the calls. Adds run list, run show and run cancel. Each takes the eval group id as an optional argument and otherwise falls back to the id recorded in the azd environment, matching results show. Cancelling a run that already reached a terminal state is refused locally, because the service reports success either way and the CLI would otherwise claim to have cancelled a finished run. Two related fixes. Passing --project-endpoint disabled the azd environment cache entirely: the environment name was only resolved when the endpoint came from azd, so every cached eval group and run id lookup returned empty. The name is now resolved independently of where the endpoint came from. The spec documents --wait and --no-wait, but cobra does not derive the negative form from a bool, so --no-wait was rejected as an unknown flag. Verified live: start with --no-wait, list, show, cancel, and the terminal-state guard on a second cancel. JSON output checked on the new subcommands.
The spec lists run start alongside list, show and cancel. The behaviour existed only as the composite `azd ai eval run`, so the atomic name in the spec did not resolve. Both forms are now built by one constructor, so their flags cannot drift apart, and a test asserts that.
… work Exercising the atomic write commands against a live project found three failures. None were covered by tests, because none of these paths had been run end to end. dataset update always collided. It passed the --version flag straight to UploadNewVersion, which derives the next version from what it is given, so an omitted flag restarted at 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset. The flag help promised the opposite, that omitting it would take the next version. This is the same defect that was fixed in the deploy reconciler earlier, so the discovery is now centralised in DatasetClient.UploadNextVersion and both callers use it, rather than being fixed twice and available to be missed a third time. evaluator upload rejected every hand-authored rubric. The service needs a type discriminator on the definition, and without it fails the whole request with "The request field is required", which names a field that is present. Generated rubrics carry the type, so only the hand-authored path documented in the spec was affected. The type is now filled in when absent and left alone when set. evaluator show returned 404. It omitted the version segment from the path, but the service has no route for an unversioned evaluator, despite the doc comment claiming the latest would be fetched. The latest version is now resolved first, comparing numerically because versions are integers as strings and a lexical compare ranks "9" above "15" -- the service already publishes evaluators at version 15 and 17. Verified live: dataset create, show, update to 2.0, list and delete; evaluator upload, show resolving the latest, update to version 2, list and delete. Both suites leave nothing behind.
…nfig Deploying a config that declares a custom evaluator, rather than only built-in ones, failed in two ways. Every earlier test used built-ins, so neither showed up. The evaluator was republished on every deploy. The service enriches a definition when it stores it, so a rubric consisting of nothing but type and dimensions comes back carrying data_schema, init_parameters and metrics it was never given. Comparing whole documents therefore never matched. Only the keys the author actually wrote are compared now, structurally, so key order and formatting are not changes either. This is what the spec means by repeated azd up creating no redundant versions. The eval group was then rejected with a request for a model that had been set. Evaluators disagree on what the judge model is called: built-ins declare deployment_name, and a custom rubric declares model. The builder sent only deployment_name, so the custom evaluator saw its required parameter missing. The judge model is now bound under whichever name the evaluator declares. Verified live: first deploy publishes the evaluator and creates the group, two redeploys report it unchanged and publish nothing, editing the rubric publishes the next version and recreates the group, and a further redeploy is a no-op again.
…d dataset The flag is documented as taking a path or the name of a registered dataset, and means use this one instead of generating. It only suppressed generation when the value looked like a local path, so passing the name of an existing dataset still submitted a generation job and, since agent-seeded generation is currently broken server-side, failed the whole command. --evaluator already skipped unconditionally, so the two flags disagreed. Both the skip and the default-spec synthesis now key off whether the flag was supplied at all. This was the last thing standing between a generated config and the documented end-to-end flow. Verified live: init scaffolds a group referencing its own rubric, generate writes that rubric and merges the reference into the same file while preserving comments and ordering, azd up registers the dataset and evaluator and creates the group, and the run completes and scores against the generated rubric.
…listing
GET /datasets/{name}/versions returns nothing for a second or two after a
version is created, even though the version itself reads back immediately.
Measured: empty at 0s, populated at 2s.
That undermines the version discovery added for dataset update, which reads the
listing to decide what to increment from. An empty listing is ambiguous -- it
means either a new dataset or a stale read -- so back-to-back create and update
could still restart at 1.0 and take a 409.
Rather than delaying every first upload to wait for the index, a conflict is
now treated as the stale read it is: re-read the listing, which by then
reflects reality, and retry once. The common path is unchanged.
The live test asserted on the first listing response and was failing for the
same reason. It now polls, and says why.
Verified: create immediately followed by update produces 2.0 rather than a
conflict, and the full live suite passes.
…requires The shared extension build template invokes ci-build.ps1 and ci-test.ps1 from the extension directory. Neither existed, so the release pipeline added alongside this extension would have failed on its first run. Both are modelled on the agents extension with two deliberate differences. ci-build.ps1 reads version.txt from the extension directory rather than its parent, where no such file exists, so the default works when the pipeline is not supplying -Version. It accepts -BuildRecordMode, which the template always passes, but builds nothing extra: this extension has no record/playback mode and no pipeline step consumes a record binary. ci-test.ps1 passes --junitfile explicitly. The pipeline publishes **/junitTestReport.xml from the extension directory, and the extension template does not set GOTESTSUM_JUNITFILE the way the CLI build does, so without this no test results would surface in the build. Verified locally with gotestsum installed: 97 tests across 7 suites reported. Also adds the README and CHANGELOG that 17 of the 21 extensions ship. The README documents the deployed shape, the command surface, the rubric weight constraint, and how to run the live tests.
The manifest declared two capabilities the extension did not back. metadata was declared but the command was never registered, so azd could not discover the command tree: azd ai eval metadata failed with unknown command while the same call against a peer extension returned its full tree. azd uses this for discovery, so the declaration was actively misleading. The command is now registered and reports nine commands. lifecycle-events was declared but no event handlers exist. The SDK only starts its event manager when handlers are registered, so the capability was an unused permission rather than a broken promise. It is removed; the listen command is still invoked because the service-target-provider capability triggers it, which a deploy after the change confirms. Adds tests over the manifest so neither can drift again: every declared capability must be backed by the command that implements it, the declared provider name must match the host the code registers, and version.txt must agree with the manifest version, which until now was only a comment asking for it.
…ction from a file Auditing every flag and API sequence the spec documents against the running extension turned up two gaps. The spec describes a drift check that was never implemented. It matters because of how change detection works: when local content is unchanged, the version recorded at the last deploy is reused, so a version published outside the repo would be silently ignored and the eval group pinned to older data. A deploy now fails when the service holds a newer version than the recorded one, naming both versions. An explicit version: on the declaration skips the check, because that is the author stating which version they want. This was added after testing the remedy the error message suggests and finding it did not work -- the message now describes something that does. --gen-instruction-file was documented but absent. A useful generation instruction is usually longer than fits on a command line, and putting it in a file makes it reviewable with the rest of the config. Verified live: publishing a version out-of-band fails the next deploy, and pinning that version lets it through.
… edits M1 exits on all the spec examples running end to end, so I ran them verbatim. Two did not. --eval-id could never work. It is meant to run an existing group ignoring the config, and appears in both the CI/CD example and the recovery advice, but a run needs a target and a dataset and an eval group carries neither: the group holds only its testing criteria, and the dataset travels on the run. Every --eval-id invocation failed asking for a target. The pairing survives in the group's previous run, so re-running a group now repeats what it last ran, and a group that has never run says so and points at the config-based path. The failure-and-recovery example promised an error that did not exist. A run sends a local dataset inline, so unregistered local edits were evaluated silently and the results could not be traced to any dataset version. That now fails with the message the spec documents, once a deploy has recorded a fingerprint to compare against. Before that there is nothing to have drifted from, and running is how a group first comes into existence. Verified live: the CI/CD example returns JSON with a run id, and with unregistered edits the config-based run fails while --eval-id succeeds, which is exactly the recovery the spec describes.
…ploy spec init --dataset ./tests/golden.jsonl wrote that path into evals/azure.yaml unchanged, but source: is resolved relative to the file it appears in, so the deploy looked for evals/tests/golden.jsonl and failed on a file the user had just pointed at. This is the spec's bring-your-own-data example exactly as written, so that example could never have worked. The path is now rebased onto the output directory, with forward slashes so the config reads the same on every platform, and absolute paths left alone. With this the documented examples all run end to end, which is what M1 exits on: bring-your-own-data through init, azd up and run; results show --failed-only -O writing its file; and the CI/CD sequence of dataset create, run start -o json and results export --format csv.
agent.context.traces accepts source, window and sample, but the generation API takes a day count and nothing else, so source and sample were parsed and dropped without a word. An author who set sample: 500 believed they had narrowed the trace selection when nothing had changed. Both fields are documented in the spec, so this was reachable by following it. They are now reported as having no effect, naming each one, with the verb agreeing so one field reads "has" and two read "have". The warning goes to stdout rather than stderr because azd does not surface an extension's stderr -- written to stderr it was invisible in a real run even though the unit test passed -- and is suppressed under -o json so the output stays parseable.
…me up resolveEvalRef ran the full config validation on the way to turning --eval into an id, so two evals differing only in substance refused every run command -- including read-only ones that had already been told which eval they meant, like un list --eval <name>. The only way out was to hand-edit evals/azure.eval.yaml, which the error did not mention. That clash is about what happens once deployed, where the environment records an id against each eval's substance. A lookup by name is unambiguous whatever the substance, so it now runs the checks a lookup depends on and leaves the rest to create and deploy, which still refuse. Also emits a bare array from un output list -o json, as every other list does. Wrapping the rows beside the run made this the one listing a script could not iterate, and it failed silently: the loop walked the two keys instead. The run is what un show answers. Both found by a bug bash agent against the published feed.
generate writes the dataset and evaluator it made into the catalog but declares no eval, so a create run straight afterwards hit �t least one eval is required -- true, and no help at all about which command supplies one. It now names init and says what generate did and did not add.
When the local file was unchanged and the declaration pinned a version, reconciliation returned that version without asking the service anything. Deleting the pinned version therefore left create reporting it unchanged while �ersions list showed nothing and the eval pointed at a version that was gone. A pin settles which version to use, not whether it is still there, so the pinned branch now reads it the same way the already-registered branch above does. Only a confirmed 404 refuses; any other read failure leaves the pin alone rather than turning a transient error into a failed deploy. Not covered by a unit test: the branch is only reachable once getEnvValue returns a recorded fingerprint, which needs a live azd client. Found by a bug bash agent against the published feed.
These branches only run once a previous deploy recorded a fingerprint, and getEnvValue answers nothing without an azd client, so none of them had ever executed under test -- 38 call sites depend on it. The agents extension already solves this by serving the environment over gRPC the way azd does, so this borrows that: a small in-memory EnvironmentServiceServer, a real grpc.Server on an ephemeral port, and the actual client pointed at it. That covers the pinned-version check three ways: a version the service no longer has is refused, one that is still there is reused unchanged, and a read that fails for any other reason leaves the pin alone rather than breaking a deploy. The harness found a divergence on its first run: this extension's version-not-found hint still pointed at dataset list, which lists datasets rather than versions. The dataset extension's copy was corrected earlier today and this one was missed -- the tenth divergence between the two.
The shared create/update builder registered it on both, but the upload derives the next version from it, so create --version 4.0 published 5.0 rather than a first version -- and left the existence probe, which looks for the versions a first publish can carry, unable to find what it had just written. Fixed in the dataset extension this morning and missed here: the eleventh divergence between the duplicated packages, found by checking the rest of that morning's single-copy fixes after a test caught the tenth. Verified in a clean worktree at HEAD, since the working tree currently carries an unrelated in-progress refactor: create no longer lists --version, update still does.
testify's require calls t.FailNow, which the testing package documents as safe only on the goroutine running the test. Called from an httptest handler it aborts that goroutine mid-response, so the client reads a truncated reply and the failure lands on whichever test happens to be running -- reported with no assertion detail at all, which is what made it look like a flake in the reconciler tests. assert routes through t.Errorf, which is safe from any goroutine.
The two copies implemented one concept twice under different names: followPages/doRequestGetURL/sameOrigin here, walkDatasetPages/followNextLink there. Four of the nine divergences found between the copies were in paging or redaction, which is what happens when the same idea has two homes and a fix lands in one. This copy now carries the dataset extension's version verbatim: the defensive copy of the caller's slice, the sameOrigin helper, and InvalidNextLink for a link that will not parse. Paging is in pages.go in both, and both log through internal/urlsafe. The two files are byte-identical apart from the module path, and the package function surface now matches exactly -- 42 and 42, down from seven differences. eval_api keeps its own paginator and PageLinkLeftTheService; only the duplicated dataset_api is converged here.
The table carries one rate per run, because a column per evaluator stops being readable as soon as two runs score different evaluators, and an absent evaluator renders as a gap that reads like a zero. The service does return per_testing_criteria_results on the list route and the runs go out whole, so a script can already read it -- nothing said so. Help text and a test now name the field. The test pins the name and that it survives marshalling; it does not catch someone narrowing the emitted type, which is how this would actually be lost, and that needs the command harness the reconciler tests now have. A run-to-run comparison, which is the question people actually ask, is a next-milestone item and sidesteps the varying evaluator set by comparing exactly two runs.
cli/azd/AGENTS.md requires errors.AsType[*azcore.ResponseError] over the legacy errors.As target pattern, which also avoids the mutable out-parameter. Copilot raised this on the dataset extension's PR and I converted that copy this morning; this one kept the old form. Twelfth divergence between the duplicated packages, found by diffing the two copies function by function after convergence rather than waiting for a reviewer. Verified in a clean worktree at HEAD: the working tree currently has an in-progress edit removing messages that committed code still calls, so a build there fails for unrelated reasons.
A run that outlived the two-hour wait returned nil, on the reasoning that this is the same contract as --no-wait. It is not: --no-wait with --fail-on is refused up front precisely because a gate with no result is silently dropped. Reached by running long instead, the same drop happened and the command exited 0, so a pipeline that asked to be gated read it as a pass. It now refuses when a gate was set, naming the run, the wait it outlived, and how to reattach with the gate. Without a gate nothing changes: the wait stopping is not a failure. Two tests. One pins the message. The other reads the source of the branch, which is ugly but proportionate here -- driving it needs a run outliving a const budget and a signed-in client, and the failure mode is a gate that passes silently, so a regression looks exactly like success. Same reasoning as the linker-path check in internal/version. Also converts the four FailNow-based assertions in eval_api's paging test handlers to goroutine-safe ones, including a t.Fatalf the report missed. Found by a targeted review of the surface the earlier passes skipped.
Redaction covered what http.Client.Do returns and stopped there. Two earlier paths were left: url.Parse of the container SAS at the top of UploadBlob, DownloadBlob and ListContainerBlobs, and the request construction that follows. Both hand back an error naming the URL, so a SAS the parser refuses reached the user's screen and the debug log with its signature intact. Seven sites across the two copies now wrap with urlsafe.Error, which rebuilds a *url.Error without the URL. A test drives all four entry points with a URL the parser refuses that still carries a sig, and asserts neither the signature nor the parameter name survives. Worth noting how it was found: reading the code caught the construction sites, and the test caught the parse sites -- three of its four cases failed on first run against the fix I had just written.
The gate reads unpassed as total minus passed, so a row that errored or was skipped breaches -- which is the point of a gate. Nothing tested it: replacing that with the Failed count alone left every test green, and a run whose rows all errored would have reported success to a pipeline. The pass-rate half of the same concern was already covered, so the gap was one gate, not the idea. Found by mutating the two guards rather than reading the test names, which read as though both were covered. Also worth recording: the first attempt at that mutation silently failed to match and reported the tests as toothless. A mutation you do not verify landed is worse than none, because it accuses the tests instead of the code.
… one A nextLink that would not parse was reported as an invalid endpoint URL, which is the one thing it is not: the link comes from the service, the endpoint is the caller's and was fine. It sent the reader to check configuration that was never wrong. The dataset extension's paginator already separates the two. This is the only thing worth changing in this paginator. It is a third implementation on purpose -- a different client, bearer auth rather than SAS, and generic over the response type -- and the two extensions are separate Go modules, so sharing one would mean copying rather than importing. Its defences are already sound: it resolves relative links before checking the origin, compares scheme as well as host so a link cannot downgrade to http, remembers every link so a multi-hop cycle ends the walk, and bounds the walk at 100 pages.
The README listed three levels and the resolver has four: it omitted the AZURE_AI_PROJECT_ENDPOINT fallback at both environment levels, and the machine-wide global config that azure.ai.agents writes, which outranks the host environment variable. That last one is the surprising part and now says so: a project context left behind by �zd ai agent on an unrelated project silently wins over the variable exported in the current shell. Validate() keeps the endpoint inside services.ai.azure.com, so this cannot redirect a token to an arbitrary host, but it can send it to the wrong Foundry project without saying so.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #9549
Internal work item: https://msdata.visualstudio.com/Vienna/_workitems/edit/5363979/
Adds
azd ai eval— define and run Foundry evaluations from the terminal.Replaces #9339, which pointed at a branch missing ~3,400 lines of eval work that had only ever landed downstream, including a fix for a bug that 404'd any dataset whose name contained a space.
Surface
Notable
One
generate, with a selector.generatesubmits both generation jobs in parallel;--dataset/--evaluatornarrows to one. Each reports its own outcome, the command exits non-zero if either failed, and whatever succeeded is still recorded.--no-waitresumes throughazd ai eval job, where the selector is required because a job id alone does not say which collection to poll.Runs know what they are scoring. The run data source is decided by configuration: a
source:block hands gathering to the service (traces, stored responses), otherwise rows come from the dataset andtarget:says what to invoke — an agent, a model, or nothing at all. Previously only agent targets existed, soinit --source traceswrote a configuration its ownrunrefused, and a model target was validated as legal then sent as an agent.initdetects rather than demands.--targetdefaults to the project's only agent, prompts when there are several, and names the flag under--no-prompt.Every user-facing string lives in one file (
internal/messages/messages.go) — so the whole voice of the CLI can be reviewed in one sitting. The move surfaced real duplicates, including three copies of one error and two contradictory--formatmessages where one was unreachable.The configuration is
azure.eval.yaml, prefixed for azd the wayazure.yamlis. An existingeval.yamlis still read and written back to, so nothing breaks and no project grows a second file. A directory holding both is refused:azure.yamlreferences one by name, so silently preferring one would mean editing one configuration whileazd updeployed the other.Waiting is bounded. A run that never reaches a terminal state used to hold the terminal open indefinitely. The wait now stops after two hours and is treated the way
--no-waitis — the run is still going server-side, so the caller gets the reattach line and exit 0 rather than an error about a failure that did not happen.Every
showrenders a detail view, with the machine-readable document behind-o json.run output showwas the last one emitting raw JSON whatever was asked for, which made the command a person reaches for after a failing listing the hardest one in the CLI to read.The end-to-end suites were not running
go test ./...reported a clean run for this extension while compiling none oftests/cli,tests/liveortests/hero— all three carry build tags, andgo list ./tests/...answers "matched no packages". CI type-checks them but never executes them, so 1,822 lines of end-to-end tests had gone unrun and were holding seven failures.Running them found
--format jsonlbeing refused by a guard that named only json and csv while the flag's own help offered jsonl;evaluator showmissing the--output-filethat the reconciliation error tells the reader to adopt a remote change with; and a missing PASS THRESHOLD column. The other four were stale assertions, re-pinned to the spec rather than to the implementation.Verified live
Against a real Foundry project, end to end:
azd ai eval init --source traceswith no other flags, then a run scoring 20 real conversations from Application Insights.--no-wait -o json, gate pass exit 0, gate breach exit 1).azd, which is where the remaining not-found and--output-filedefects came from.Green: 524 unit tests,
tests/cli66,tests/live5,tests/hero8,go vet -tags live,hero,gofmt, andci-test.ps1exit 0.Reviewed
Nine model-assisted review passes (Sonnet 4.6, GPT-5.3-Codex, Gemini 3.1 Pro, Grok 4.5, GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna) found real defects that are fixed here — among them dataset job commands calling their collection on the wrong API version,
--no-wait -o jsonlosing the job ids it exists to report, a declared target with no name being silently scored as no target, two nil-deref panics in the endpoint cascade, andevaluator show --output-filetruncating the local definition it was meant to update if the write failed.Not included, deliberately
No
registry.jsonentry and nomicrosoft.foundrybundle dependency — both need published artifacts, and listing an unpublished extension in the bundle breaksazd extension install microsoft.foundryfor everyone.Known gaps
initdoes not yet prompt for evaluators or judge model; it detects sensible values instead. The spec calls for prompts with an error under--no-prompt. Not changed here because the--evaluatordefault also schedules rubric generation, which is what makes theinit→generateflow work; reshaping that is broader than this PR.dataset versions liston an unknown name lists nothing and exits 0 here, while the dataset extension (Add azd ai dataset extension for Foundry datasets #9499) answers the same question with an error. A list is a filter rather than a lookup, so empty-and-successful is defensible and two tests depend on it — but the two extensions should agree. Raised for reviewers rather than settled unilaterally.main.goreports 1 unless the error chain carries an*internal.ExitCodeError, which is constructed only on the hooks path. Spec Scenario 5 wants 2 for "regressed" versus 1 for "could not run". The fix is in azd core, not here.Draft — not requesting reviewers yet.