Skip to content
Draft
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -260,9 +260,9 @@ mindful of which specific change was the root cause.
{"resources": {...}, "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...}
```

2. In case the task is using node groups (e.g. for PD disaggregation),
instead of a single `resources` it includes `groups`, and the fields are
these and no others:
2. In case the task is using node groups (a router group plus worker
groups, see `## Fleet Topology`), instead of a single `resources` it
includes `groups`, and the fields are these and no others:

```
{"groups": [[{...}], [{...}, {...}], [{...}]], "context_length": ..., "benchmark": {...}, "learned": ..., "failed": ...}
Expand Down Expand Up @@ -321,9 +321,9 @@ serving engine: use <!--?if dataset-->`dataset` and `concurrency`<!--?else-->`co
`shared_prefix_tokens`<!--?end--> from `constraints.json` and measure all trials the same
way so that their results are comparable with each other.

In case the task is using PD disaggregation, run benchmarks via SSH inside
the router node, directly against the router engine. Never benchmark prefill
or decode workers — each handles only part of a request.
In case the task is using a router (see `## Fleet Topology`), run benchmarks
via SSH inside the router node, directly against the router, never against a
worker, so that results stay comparable across trials.

<!--?if dataset-->
Before any benchmark, reset the serving engine's prefix cache, or restart the
Expand Down Expand Up @@ -506,6 +506,32 @@ Use these offers when selecting fleet, backend, and hardware.
To classify each backend's capabilities, fetch `https://dstack.ai/docs/concepts/backends.md` and classify from the fetched document, not from memory. VM-based backends are listed under `## VM-based` (they support idle instances and instance volumes). Kubernetes backend is listed under `## Container-based`, but supports instance volumes and thus is preferred over other container-based backends.
SSH fleets can be treated as VM-based backends as they support both idle instances (its equivalent) and instance volumes.

## Fleet Topology

If a fleet has `placement: cluster` and a CPU-only instance, that instance
must host a router (see `https://dstack.ai/docs/concepts/services/#router`),
and every trial and the final service run behind it. The GPU instances run
the serving engine as workers in one of two forms: aggregated, where each
worker handles both prefill and decode, or PD disaggregated, where the
workers are split into prefill and decode groups. Use `router: sglang`
(Shepherd Model Gateway, SMG) and connect the workers over gRPC, not HTTP.
Report the topology decision via `progress` before trial 1.

Why: with an SMG router, request parsing, serialization, and tokenization
move from the serving engine to the router, and workers only process already
tokenized inputs, so latency improves just by introducing the router. With
multiple workers, the router improves performance in two ways.

First, cache-aware routing tracks each worker's KV-cache state and sends a
request to the worker that already holds its prefix.

Second, load modeling predicts how long a request will take to complete on
each worker and the router's `least_load` policy sends the request to the
worker predicted to finish it soonest.

Connecting the workers over gRPC is required for SMG's cache-aware routing
and load modeling.

# Final Service

Once the trials are over, pick the best trial that has not been verified yet
Expand Down Expand Up @@ -565,9 +591,9 @@ trial benchmarks so that the results are comparable with each other. Attach to
the service with `dstack attach <run name>`, which enables `ssh <run name>`
into the replica.

In case the service is using PD disaggregation, run the final benchmark
inside the router replica, directly against the router engine. Attach to it
via `dstack attach <run name> --replica <replica num> --job <job num>`.
In case the service is using a router (see `## Fleet Topology`), run the
final benchmark inside the router replica, directly against the router.
Attach to it via `dstack attach <run name> --replica <replica num> --job <job num>`.

If the service or its benchmark cannot be completed, stop that service,
pick the next-best trial, and repeat, until a service is verified or there
Expand Down
Loading