Skip to content

Improve DB connections handling - #4291

Merged
r4victor merged 9 commits into
masterfrom
pr_db_connection_pool
Sep 15, 2026
Merged

r4victor merged 9 commits into
masterfrom
pr_db_connection_pool

Conversation

@r4victor

Copy link
Copy Markdown
Collaborator

This PR fixes problems with leaking DB connections pool (no command_timeout could led to full client pool with 0 connections on the DB server) and other DB connections issues:

  • Run migrations and server init on the dedicated connection holding their advisory lock – guarantees the lock release doesn't happen on another connection.
  • Poll the migration lock and commit between attempts to avoid blocking concurrent index build.
  • Add DSTACK_DB_COMMAND_TIMEOUT for PostgreSQL operations, defaulting to 300 seconds.
  • Fetch gateway stats outside run-pipeline database sessions.
  • Document connection lifetime and advisory-lock usage.

A replica blocked in pg_advisory_lock() holds a snapshot that another
replica's CREATE INDEX CONCURRENTLY migration waits for, so both hang
until Postgres kills the waiter.
Holding a session-level advisory lock on a separate connection lets
Postgres release it, e.g. via idle_in_transaction_session_timeout, while
the work continues on another connection. Migrations now run on the lock
connection, and server init uses a session bound to it.
@r4victor
r4victor merged commit c12a77f into master Sep 15, 2026
26 checks passed
@r4victor
r4victor deleted the pr_db_connection_pool branch September 15, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant