[Evaluation] Prepare azure-ai-evaluation 1.18.6 release

[Ankit Singhal (singankit)]

Description

Prepare azure-ai-evaluation 1.18.6 for release following the package-specific release process.

• Set the release date to September 17, 2026 and remove empty changelog sections using Prepare-Release.ps1.
• Document configurable native Azure OpenAI output-item page size and adaptive timeout retries introduced in [Evaluation] Configure adaptive native AOAI output result page size #49034.
• Keep the package version at the existing 1.18.6.

Release tracking: https://dev.azure.com/azure-sdk/Release/_workitems/edit/35140/ — updated to In Release, version 1.18.6.

Release handoff

• Human review and merge to main are required before running python - evaluation (7162) on main.
• The preparation script successfully contacted APIView (HTTP 201) and reported that the API review for 1.18.6 is not approved; the package name is approved. Confirm the release build's revision under Build > Analyze > Create API Review.
• The API surface differs from 1.18.5: evaluate adds aoai_output_items_page_size. Reviewers should confirm API approval and suitability of the existing patch version before release.
• Explicit approval from Minsoo and approval of the publishing stage remain required. No release pipeline was queued and nothing was published during preparation.

Validation

• Confirm-ChangeLogEntry -VersionString '1.18.6' -ForRelease $true passed.
• git diff --check passed.
• Changelog-only change; no runtime code changes.

All SDK Contribution checklist:

• The pull request does not introduce breaking changes.
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message.

Testing Guidelines

• Pull request includes test coverage for the included changes. Not applicable: release changelog only.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The new public feature requires a minor version bump rather than the proposed patch version.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Prepares azure-ai-evaluation for release by finalizing its changelog.

Changes:

• Dates the release and documents AOAI pagination and retry behavior.
• Removes empty changelog sections.

File summaries

File	Description
sdk/evaluation/azure-ai-evaluation/CHANGELOG.md	Finalizes release notes for the proposed version.

Review details

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

[Pilot] PR Pipeline Failure Analysis

What failed

Build: 6853636 — python - pullrequest (FAILURE), job Build Test macoslatest_311
Package/path: sdk/evaluation/azure-ai-evaluation
Test: sdk.evaluation.azure-ai-evaluation.tests.unittests.test_evaluate.TestEvaluate.test_evaluate_korean_characters_result
File: tests/unittests/test_evaluate.py:1083

Relevant pipeline output

self = <test_evaluate.TestEvaluate object at 0x1128ed5d0>
questions_answers_korean_file = '/Users/runner/work/1/s/sdk/evaluation/azure-ai-evaluation/tests/unittests/data/questions_answers_korean.jsonl'

    def test_evaluate_korean_characters_result(self, questions_answers_korean_file):
        output_path = "eval_test_results_korean.jsonl"

        result = evaluate(
            data=questions_answers_korean_file,
            evaluators={"g": F1ScoreEvaluator()},
            output_path=output_path,
        )

        assert result is not None

        with open(questions_answers_korean_file, "r", encoding="utf-8") as f:
            first_line = f.readline()
            data_from_file = json.loads(first_line)

        assert result["rows"][0]["inputs.query"] == data_from_file["query"]

>       os.remove(output_path)
E       FileNotFoundError: [Errno 2] No such file or directory: 'eval_test_results_korean.jsonl'

tests/unittests/test_evaluate.py:1083: FileNotFoundError

Root cause

The evaluate() call did not write eval_test_results_korean.jsonl to the expected relative working directory on the macoslatest_311 runner, so the test's own cleanup step (os.remove(output_path)) raised FileNotFoundError. This is a test-cleanup/output-path failure category, not an AssertionError on evaluation content — the assertions above it appear to have passed, meaning evaluate() returned a result but the output file wasn't found at the relative path used for cleanup. This is platform-specific (only observed on macoslatest_311), consistent with a working-directory or file-flush timing difference rather than a logic bug in the evaluator itself.

This PR (singankit-evaluation-release-readiness) is changelog-only — it updates CHANGELOG.md and version metadata for release prep and makes no runtime code changes to evaluate() or any output-path handling. The failure is therefore not something this PR's diff caused, and cannot be fixed by editing this PR's changed file (CHANGELOG.md).

Recommended next steps

• Re-run the macoslatest_311 pipeline job to rule out a transient/platform timing issue (e.g., delayed file flush before test cleanup).
• If it fails again, investigate evaluate()'s output-path/working-directory handling for macOS runners in the package's live source (not reachable from this PR's diff), possibly related to the adaptive timeout retry / output-item paging behavior recently introduced in [Evaluation] Configure adaptive native AOAI output result page size #49034.
• See https://aka.ms/ci-fix

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a
fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis Next Steps · auto · 62.4 AIC · ⌖ 7.03 AIC · ⊞ 8.7K · ◷