feat(gooddata-eval): add KDA-skill agentic evaluator

[FrankHuynh]

What

Adds kda_skill.py to gooddata-eval, evaluating the chatbot's create_key_driver_analysis / execute_key_driver_analysis tool calls against the kda_skill Langfuse dataset.

Related: QA-28800 — Build E2E LLM test for KDA skill.

Scope

Scope is strictly completion, not field correctness:

strict_pass = triggered AND executed AND success AND turn_completed

Per-field checks (Measure / Date Attribute / Analyzed Period / Reference Period / Filters / Summary within tolerance) are out of scope for this PR — deferred to a follow-up ticket (QA-28699).

Latency / performance reporting

Latency is measured directly by the harness, not derived from a Langfuse trace after the fact. ChatClient times each send_message() call around the SSE stream (starting before the request opens, so it includes connection setup time; excludes retry backoff). KdaRunResult.turn_wall_clock_sec sums every turn from the one that first calls create onward — including a continuation turn that only finishes execute a turn later, since that's real gen-ai processing time too — but excludes any earlier disambiguation turn and the simulated-reply's own OpenAI call.

evaluate_agentic_kda_skill logs it as the kda_turn_wall_clock_sec Langfuse score; gdc-nas's combo_report.py reads it directly to classify pass/failed/error — no trace re-resolution needed.

The latency logged is the whole conversational turn (LLM planning + every tool call), not isolated KDA-backend execution time — gen-ai doesn't trace tool execution as its own observation, so there's no narrower point to measure from on either side of this cross-repo pair.

Disambiguation & continuation safety nets

KDA cases are designed to resolve in one turn, but two things can extend a run, each bounded to max_iterations:

• The agent asks a clarifying question instead of triggering KDA directly (a metric-title collision, or a choice between equivalent measure forms) — a simulated-user reply (gpt-4o-mini, 30s timeout) picks any acceptable candidate and continues.
• The agent calls create but not execute within the same turn — create/execute are tracked independently across turns, and a plain continuation nudge ("Please proceed.") gives the agent another turn to finish, instead of the run being scored as if execute never happened.

Also fixed in review

• pass_at_k/pass_power_k Langfuse scores are kda_-prefixed — unprefixed, pass_at_2 at k=2 collides with visualization.py's own score name, which gdc-nas's combo_report.py.verdict() checks first.
• A ChatError/TransientChatError raised mid-turn no longer propagates out of run_agentic_kda_skill uncaught — it now surfaces as a normal failed run, so it still gets scored to Langfuse.
• ChatError/TransientChatError now carry partial_result, so tool calls that already succeeded before a later, unrelated error (e.g. the final summary failing) aren't discarded and misreported as "KDA never triggered".
• turn_completed resets to False in the exception branch, so a crash on a later iteration can't leave a stale True from an earlier one.
• run_agentic_kda_skill rejects k < 1 — a bad env-driven KDA_RUN_K value previously ran silently once instead of surfacing the bad config.
• stream_ended is now set the moment the response_ended event line is parsed, not its data line — an event with no data payload previously left the flag unset.
• t0 now begins before opening the SSE stream, not after — it was missing the connection/server-setup time a caller actually waits through.
• Disambiguated-turn latency no longer double-counts every turn's time plus the simulated-reply's own OpenAI call.
• The 6 per-field informational correctness scores and their support code are removed — team confirmed this PR's scope is trigger+complete only.
• create/execute tracked independently across turns instead of required to pair from the same turn's tool calls (see "Disambiguation & continuation safety nets" above).
• KdaEvaluation.kda_triggered renamed to triggered, matching the other three core fields — the Langfuse score name kda_triggered is unchanged.
• generate_simulated_kda_response's OpenAI call now has a 30s timeout.
• kda_skill wired into the CLI's agentic dispatcher (agentic_runner.py) and AGENTIC_TEST_KINDS, matching every other agentic skill, so it can be run/debugged standalone via gd-eval run instead of only through gdc-nas's tavern-e2e harness. (README's test_kind table isn't updated for this — it documents a different, older evaluator subsystem and has zero agentic_* entries for any of the 6 existing agentic skills, so adding only kda_skill there would be misleading.)
• dataset_name default renamed from agent_kda_skill to kda_skill, matching the no-prefix pattern every other skill uses; no functional change since gdc-nas always passes it explicitly.
• A continuation turn (create succeeds, execute only completes a turn later) now has its latency summed across both turns instead of only the turn that completed execute.
• expected_output.get("Measure") now guards for non-dict shapes — DatasetItem.expected_output on the gdc-nas side allows str/list, not just dict, and a list-shaped item previously raised AttributeError, silently swallowed and disabling disambiguation with only a WARNING.
• _DEFAULT_MAX_ITERATIONS raised from 2 to 3 — 2 was lower than every other agentic skill (visualization=4, alert_skill=6, metric_skill=7) and left no room for a case needing both disambiguation and a create/execute turn split.
• kda_pass_at_k/kda_pass_power_k no longer logged to Langfuse — matches metric_skill/alert_skill/guardrail/search_tool/general_question, which all compute the pair but never log it at their default k=1; nothing reads a kda_pass_at_1 score today, and the score name would shift if k ever changes.
• A few comments/docstrings that had drifted from the multi-turn behavior corrected: KdaRunResult's docstring, the turn_wall_clock_sec field comment, and _extract_kda_calls' docstring (pairing is only guaranteed within one turn; merging across turns is the caller's job).

Known limitations (out of scope for this PR)

• cost_usd on the KDA score is still read from pt.total_cost, where pt is the Langfuse trace picked by the default max-latency selector — not necessarily the KDA turn itself. Latency was fixed to not depend on this trace selection; cost wasn't, since this ticket's scope is latency/completion only. Candidate for a follow-up if cost accuracy on this score matters.
• KdaRunResult.eval shadows the eval builtin as an attribute name. Not renamed here since AlertRunResult already ships the same field name — fixing only KDA would be inconsistent, not a real fix.

Verification

No local Tiger instance available, so verified two ways:

• Offline unit checks against synthetic and real captured data (ruff check, ruff format --check all clean; _evaluate_run exercised directly with real SSE payloads captured from a 30-run manual stability test).
• Package-level import (from gooddata_eval.core.agentic import evaluate_agentic_kda_skill, ...) verified to resolve with no circular-import issues.

Not included in this PR

• Version bump / release — will follow in a separate, explicitly-confirmed step (the version is monorepo-shared across all 9 published packages).
• The gdc-nas side (shim, tavern test, fixtures, cron wiring, daily report's KDA table) — companion PR in gdc-nas under the same QA-28800 ticket.

🤖 Generated with Claude Code

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The PR adds an agentic KDA skill evaluator. It validates KDA results, supports clarification turns and repeated runs, integrates optional Langfuse tracing and scoring, and exposes the new API through the agentic package.

Changes

Agentic KDA evaluation

Layer / File(s)	Summary
KDA evaluation contracts and correctness packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py	Adds input normalization, clarification detection, tool-call extraction, result types, process gates, and informational correctness checks.
KDA execution and aggregation packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py	Runs KDA conversations with bounded clarification retries, manages conversation cleanup, and calculates pass-at-k and pass-power-k results.
Langfuse trace discovery and scoring packages/gooddata-eval/src/gooddata_eval/core/agentic/_langfuse.py, packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py	Retrieves observations, selects KDA traces, and logs optional evaluation scores.
KDA public package exports packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py	Exports KDA result types, assertion errors, evaluation helpers, and run helpers.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Evaluator
  participant GoodDataAPI
  participant OpenAI
  participant Langfuse
  Evaluator->>GoodDataAPI: Create conversation and send question
  GoodDataAPI-->>Evaluator: Return agent messages and tool-call events
  Evaluator->>OpenAI: Generate simulated clarification reply
  OpenAI-->>Evaluator: Return clarification response
  Evaluator->>GoodDataAPI: Send reply and collect execution result
  Evaluator->>Langfuse: Discover traces and log evaluation scores

Loading

Suggested reviewers: hkad98, lupko, pcerny

Poem

A rabbit checks each KDA run,
Through clarifying turns it hops.
It counts the passes, one by one,
Then scores the traces when it stops.
New exports bloom in the package tree.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the addition of the KDA-skill agentic evaluator, which is the main change.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 92.48826% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.42%. Comparing base (9103989) to head (216f21d).

Files with missing lines	Patch %	Lines
...a-eval/src/gooddata_eval/core/agentic/kda_skill.py	91.76%	14 Missing ⚠️
...ddata-eval/src/gooddata_eval/cli/agentic_runner.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1706      +/-   ##
==========================================
+ Coverage   79.15%   79.42%   +0.27%     
==========================================
  Files         271      272       +1     
  Lines       18794    19001     +207     
==========================================
+ Hits        14876    15092     +216     
+ Misses       3918     3909       -9     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (4)

packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py (4)

55-60: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Normalize expected the same way as actual.

Line 201 passes expected.get("Filters", []). If a dataset item contains "Filters": null, expected is None. json.dumps(None) produces "null", and actual is normalized to [], so filters_correct becomes False for a semantically empty expectation. A follow-up ticket plans to promote this field into strict_pass, so fix the baseline now.

♻️ Proposed normalization

-def _filters_match(actual: object, expected: list) -> bool:
+def _filters_match(actual: object, expected: list | None) -> bool:
     actual = actual or []
+    expected = expected or []
     try:
         return json.dumps(actual, sort_keys=True) == json.dumps(expected, sort_keys=True)
     except TypeError:
         return False

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py` around
lines 55 - 60, Update _filters_match to normalize expected the same way as
actual before comparing serialized values, so None is treated as an empty filter
list and expected.get("Filters", []) remains semantically consistent with
missing filters. Preserve the existing TypeError handling and comparison
behavior for non-null values.

---

96-100: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Filter non-dict candidates before calling .get.

measure_candidates comes from the dataset expected_output. If the list contains a non-dict element, line 98 raises AttributeError. _measure_matches already guards this shape at line 52 with isinstance(c, dict). Apply the same guard here.

🛡️ Proposed guard

-    candidates = measure_candidates if isinstance(measure_candidates, list) else [measure_candidates or {}]
+    raw = measure_candidates if isinstance(measure_candidates, list) else [measure_candidates or {}]
+    candidates = [c for c in raw if isinstance(c, dict)]
     candidate_desc = "; or ".join(

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py` around
lines 96 - 100, Update the candidate construction used by candidate_desc to
filter list elements through isinstance(c, dict), matching the shape guard in
_measure_matches. Ensure only dictionary candidates reach the generator
expression and its .get calls, while preserving the existing fallback behavior
for non-list measure_candidates.

---

107-111: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Set timeout=30.0 on the OpenAI call.

This prevents the evaluation path from waiting for the client's long default timeout.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py` around
lines 107 - 111, Update the OpenAI request in the client.chat.completions.create
call to pass timeout=30.0, ensuring the evaluation path uses the explicit
30-second timeout while preserving the existing model, messages, and max_tokens
arguments.

---

189-201: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Narrow the optionals directly instead of through intermediate boolean guards.

Runtime behavior is correct because and short-circuits. Type checkers, however, do not narrow dict | None through intermediate boolean variables. Type narrowing occurs only through direct conditions in and and if statements. Direct narrowing improves clarity and prevents type-checker warnings when static analysis is enabled.

♻️ Proposed narrowing

-    success = executed and execute_result.get("success") is True
+    success = execute_result is not None and execute_result.get("success") is True
 
     # Informational only (see KdaEvaluation docstring) -- still computed so a follow-up
     # ticket can promote these to strict_pass without redoing the extraction logic.
-    measure_correct = kda_triggered and _measure_matches(create_args.get("measure"), expected.get("Measure"))
-    date_attribute_correct = kda_triggered and create_args.get("date_attribute_id") == expected.get("Date Attribute")
-    analyzed_period_correct = kda_triggered and create_args.get("analyzed_period") == expected.get("Analyzed Period")
-    reference_period_correct = kda_triggered and create_args.get("reference_period") == expected.get("Reference Period")
-    filters_correct = kda_triggered and _filters_match(create_args.get("filters"), expected.get("Filters", []))
+    args = create_args or {}
+    measure_correct = kda_triggered and _measure_matches(args.get("measure"), expected.get("Measure"))
+    date_attribute_correct = kda_triggered and args.get("date_attribute_id") == expected.get("Date Attribute")
+    analyzed_period_correct = kda_triggered and args.get("analyzed_period") == expected.get("Analyzed Period")
+    reference_period_correct = kda_triggered and args.get("reference_period") == expected.get("Reference Period")
+    filters_correct = kda_triggered and _filters_match(args.get("filters"), expected.get("Filters", []))

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py` around
lines 189 - 201, Update the correctness calculations in the evaluation flow
around kda_triggered and create_args so each optional create_args access is
guarded by a direct create_args is not None condition in the same and
expression. Remove reliance on the intermediate kda_triggered boolean for type
narrowing, while preserving kda_triggered for reporting and the existing
matching logic.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py`:
- Around line 266-269: Contain failures from generate_simulated_kda_response
within the clarification loop in _run_once: catch its dependency, configuration,
and API exceptions, log them through a module-level _log logger, and terminate
only the current run while preserving already-completed runs and allowing
evaluate_agentic_kda_skill to continue to Langfuse logging and final assertion.
Add the requested logging import and module-level logger.

---

Nitpick comments:
In `@packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py`:
- Around line 55-60: Update _filters_match to normalize expected the same way as
actual before comparing serialized values, so None is treated as an empty filter
list and expected.get("Filters", []) remains semantically consistent with
missing filters. Preserve the existing TypeError handling and comparison
behavior for non-null values.
- Around line 96-100: Update the candidate construction used by candidate_desc
to filter list elements through isinstance(c, dict), matching the shape guard in
_measure_matches. Ensure only dictionary candidates reach the generator
expression and its .get calls, while preserving the existing fallback behavior
for non-list measure_candidates.
- Around line 107-111: Update the OpenAI request in the
client.chat.completions.create call to pass timeout=30.0, ensuring the
evaluation path uses the explicit 30-second timeout while preserving the
existing model, messages, and max_tokens arguments.
- Around line 189-201: Update the correctness calculations in the evaluation
flow around kda_triggered and create_args so each optional create_args access is
guarded by a direct create_args is not None condition in the same and
expression. Remove reliance on the intermediate kda_triggered boolean for type
narrowing, while preserving kda_triggered for reporting and the existing
matching logic.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a4c9e8b3-9e51-47bc-a0f9-5d1205de4325

📥 Commits

Reviewing files that changed from the base of the PR and between acfcc1a and 92db1fe.

📒 Files selected for processing (2)

• packages/gooddata-eval/src/gooddata_eval/core/agentic/__init__.py
• packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/gooddata-eval/src/gooddata_eval/core/agentic/_langfuse.py`:
- Around line 75-78: Update _ObservationListResult.list to paginate through all
observation pages instead of limiting retrieval to the first 100, using
/api/public/v2/observations with cursor pagination, io fields, and string I/O
decoding for Cloud and self-hosted v4 deployments. Preserve equivalent
pagination through the legacy /api/public/observations endpoint for self-hosted
v3, or explicitly enforce a supported-deployment constraint, so
_select_kda_trace() receives the complete observation set.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 73b38461-b03f-45d1-8fc6-36e4eae5603c

📥 Commits

Reviewing files that changed from the base of the PR and between b4bf376 and 434f8f5.

📒 Files selected for processing (2)

• packages/gooddata-eval/src/gooddata_eval/core/agentic/_langfuse.py
• packages/gooddata-eval/src/gooddata_eval/core/agentic/kda_skill.py

[myhoai]

Review notes — items I'd like resolved before merge. All are scoped to this PR; nothing here disputes the agreed scope (completion-only, whole-turn E2E latency). Comments inline.