EVAL_RESULTS.md

Retrieval Evaluation

Auto-generated by backend/scripts/eval.py. Re-run with:

cd backend && python scripts/eval.py --write-md

Field	Value
Run at	2026-04-17 20:11 UTC
Collection	python_docs
Embedding model	text-embedding-3-small
top_k	5
Cases	5

Summary

#	Case	Query	Outcome
1	q1_concurrent_coroutines	How do I run multiple coroutines concurrently?	KNOWN FAIL
2	q2_optional_vs_union	What is the difference between Optional and Union in typing?	PASS
3	q3_generic_class	How do I make a generic class in Python?	PASS
4	q4_api_exact_name	asyncio.sleep	PASS
5	q5_protocol_ambiguous	What is a Protocol?	KNOWN FAIL

3 pass · 2 known fail

A KNOWN FAIL is a documented retrieval weakness we've chosen not to fix in the baseline — see each case's Known issue note. If one flips to IMPROVED after a tuning change, update this file and treat it as progress.

Cases

1. q1_concurrent_coroutines — KNOWN FAIL

Query: How do I run multiple coroutines concurrently?

Natural-language how-to. The canonical answers are asyncio.gather and (3.11+) asyncio.TaskGroup.

Expected hits:

Anchor	Max rank	Required	Found rank	Result
asyncio.gather	5	yes	—	❌ not in top-5
asyncio.TaskGroup	5	bonus	—	❌ not in top-5

Top-k retrieved:

Rank	Sim	Kind	Module	Anchor / heading
1	0.595	prose	asyncio	coroutines — Coroutines and Tasks > Coroutines
2	0.565	api	asyncio	asyncio.run_coroutine_threadsafe — Coroutines and Tasks > Scheduling From Other Threads
3	0.546	prose	asyncio	concurrency-and-multithreading — Developing with asyncio > Concurrency and Multithreading
4	0.531	prose	asyncio	coroutines-and-tasks — Coroutines and Tasks
5	0.527	api	asyncio	asyncio.to_thread — Coroutines and Tasks > Running in Threads

Known issue. Prose section chunks outrank terse API signatures on natural-language how-to queries. asyncio.gather does exist in the index and is retrievable at rank 1 when queried by its literal name — it just loses to conversational prose chunks on this phrasing.

---

2. q2_optional_vs_union — PASS

Query: What is the difference between Optional and Union in typing?

Comparison of two specific API entries; both should surface together.

Expected hits:

Anchor	Max rank	Required	Found rank	Result
typing.Optional	3	yes	2	✅
typing.Union	3	yes	1	✅

Top-k retrieved:

Rank	Sim	Kind	Module	Anchor / heading
1	0.656	api	typing	typing.Union — typing — Support for type hints > Module contents > Special typing primitives > Special forms
2	0.604	api	typing	typing.Optional — typing — Support for type hints > Module contents > Special typing primitives > Special forms
3	0.429	api	typing	typing.Any — typing — Support for type hints > Module contents > Special typing primitives > Special types
4	0.420	prose	typing	the-type-of-class-objects — typing — Support for type hints > The type of class objects
5	0.401	prose	typing	special-types — typing — Support for type hints > Module contents > Special typing primitives > Special types

---

3. q3_generic_class — PASS

Query: How do I make a generic class in Python?

Should retrieve the user-defined generics prose plus the Generic base class.

Expected hits:

Anchor	Max rank	Required	Found rank	Result
typing.Generic	5	yes	2	✅
user-defined-generic-types	5	yes	1	✅

Top-k retrieved:

Rank	Sim	Kind	Module	Anchor / heading
1	0.504	prose	typing	user-defined-generic-types — typing — Support for type hints > User-defined generic types
2	0.499	api	typing	typing.Generic — typing — Support for type hints > Module contents > Special typing primitives > Building generic types and type aliases
3	0.483	prose	typing	user-defined-generic-types — typing — Support for type hints > User-defined generic types
4	0.434	api	typing	typing.Protocol — typing — Support for type hints > Module contents > Special typing primitives > Other special directives
5	0.420	prose	typing	the-type-of-class-objects — typing — Support for type hints > The type of class objects

---

4. q4_api_exact_name — PASS

Query: asyncio.sleep

Literal API name lookup — must be rank 1.

Expected hits:

Anchor	Max rank	Required	Found rank	Result
asyncio.sleep	1	yes	1	✅

Top-k retrieved:

Rank	Sim	Kind	Module	Anchor / heading
1	0.546	api	asyncio	asyncio.sleep — Coroutines and Tasks > Sleeping
2	0.542	prose	asyncio	module-asyncio — asyncio — Asynchronous I/O
3	0.483	prose	asyncio	exceptions — Exceptions
4	0.481	prose	asyncio	scheduling-delayed-callbacks — Event Loop > Event Loop Methods > Scheduling delayed callbacks
5	0.447	prose	asyncio	queues — Queues

---

5. q5_protocol_ambiguous — KNOWN FAIL

Query: What is a Protocol?

Ambiguous term: valid in both asyncio (transport protocols) and typing (structural subtyping via typing.Protocol). A well-tuned retriever surfaces both interpretations.

Expected hits:

Anchor	Max rank	Required	Found rank	Result
typing.Protocol	5	yes	—	❌ not in top-5
asyncio.Protocol	5	bonus	3	✅

Top-k retrieved:

Rank	Sim	Kind	Module	Anchor / heading
1	0.527	api	asyncio	asyncio.BaseTransport.set_protocol — Transports and Protocols > Transports > Base Transport
2	0.511	api	asyncio	asyncio.BaseProtocol — Transports and Protocols > Protocols > Base Protocols
3	0.501	api	asyncio	asyncio.Protocol — Transports and Protocols > Protocols > Base Protocols
4	0.501	api	asyncio	asyncio.BaseProtocol.connection_made — Transports and Protocols > Protocols > Base Protocol
5	0.496	api	asyncio	asyncio.BaseTransport.get_protocol — Transports and Protocols > Transports > Base Transport

Known issue. Module imbalance: the asyncio-protocol.html page contributes 8+ BaseProtocol/Protocol/DatagramProtocol/etc. entries, while typing contributes one typing.Protocol entry. Top-5 saturates with asyncio entries before typing.Protocol can appear. Cosine similarity alone has no mechanism to enforce module diversity.

---