Docker mimic iv postgres

.github/actions/load-mimic-duckdb/action.yml

@@ -1,5 +1,5 @@
 name: "Load MIMIC-IV demo into DuckDB"
-description: "Builds a DuckDB database from the demo hosp/icu data using import_duckdb.sh. Requires the duckdb CLI on PATH."
+description: "Builds a DuckDB database from the demo hosp/icu data using build_mimic.sh. Requires the duckdb CLI on PATH."
 
 inputs:
   mimic_data_dir:
@@ -18,11 +18,17 @@ runs:
       shell: bash
       # Resolve paths at shell time so $GITHUB_WORKSPACE points at the
       # container-mounted path.
-      # `echo n` answers the overwrite prompt defensively.
+      #
+      # Load only. The concepts are transpiled and built by concepts-duckdb.yml
+      # after this action runs, and the row counts are checked by the calling
+      # workflow, so build_mimic.sh must not do either here.
+      env:
+        MIMIC_MAKE_CONCEPTS: "false"
+        MIMIC_VALIDATE: "false"
       run: |
         data_dir="${{ inputs.mimic_data_dir }}"
         data_dir="${data_dir:-$GITHUB_WORKSPACE}"
         db_file="${{ inputs.db_file }}"
         db_file="${db_file:-$GITHUB_WORKSPACE/mimic4.db}"
         cd "$GITHUB_WORKSPACE/mimic-iv/buildmimic/duckdb"
-        echo n | ./import_duckdb.sh "$data_dir" "$db_file"
+        ./build_mimic.sh "$data_dir" "$db_file"

.github/actions/load-mimic-psql/action.yml

@@ -1,5 +1,5 @@
 name: "Load MIMIC-IV demo into PostgreSQL"
-description: "Creates the MIMIC-IV schemas and loads the demo hosp/icu data. Requires psql on PATH and PG* connection env vars."
+description: "Builds the MIMIC-IV schemas, data, constraints and indexes from the demo hosp/icu data using build_mimic.sh. Requires psql on PATH and PG* connection env vars."
 
 inputs:
   mimic_data_dir:
@@ -15,9 +15,11 @@ runs:
       # Resolve the data dir at shell time so $GITHUB_WORKSPACE points at the
       # container-mounted path. The github.workspace context yields the host
       # path, which does not exist inside a container job.
+      env:
+        # Load only - we validate/build concepts elsewhere.
+        MIMIC_MAKE_CONCEPTS: "false"
+        MIMIC_VALIDATE: "false"
       run: |
         data_dir="${{ inputs.mimic_data_dir }}"
         data_dir="${data_dir:-$GITHUB_WORKSPACE}"
-        psql -q -v ON_ERROR_STOP=1 -f "$GITHUB_WORKSPACE/mimic-iv/buildmimic/postgres/create.sql"
-        psql -q -v ON_ERROR_STOP=1 -v mimic_data_dir="$data_dir" \
-          -f "$GITHUB_WORKSPACE/mimic-iv/buildmimic/postgres/load_gz.sql"
+        "$GITHUB_WORKSPACE/mimic-iv/buildmimic/postgres/build_mimic.sh" "$data_dir"

.github/actions/setup-duckdb/action.yml

@@ -5,7 +5,8 @@ inputs:
   version:
     description: "DuckDB release version to install."
     required: false
-    default: "1.1.3"
+    # Keep in step with DUCKDB_VERSION in mimic-iv/buildmimic/duckdb/docker.
+    default: "1.4.5"
 
 runs:
   using: "composite"

.github/workflows/docker-build.yml

@@ -0,0 +1,89 @@
+# Build and run the containerized MIMIC-IV builds against the demo dataset.
+name: docker build
+
+on:
+  pull_request:
+    paths:
+      - 'mimic-iv/buildmimic/postgres/**'
+      - 'mimic-iv/buildmimic/duckdb/**'
+      - 'mimic-iv/buildmimic/download_data.sh'
+      - '.github/workflows/docker-build.yml'
+  push:
+    branches:
+      - main
+    paths:
+      - 'mimic-iv/buildmimic/postgres/**'
+      - 'mimic-iv/buildmimic/duckdb/**'
+      - '.github/workflows/docker-build.yml'
+
+# Cancel superseded runs when a PR is pushed again.
+concurrency:
+  group: docker-build-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    defaults:
+      run:
+        # Name bash explicitly to get `-eo pipefail`.
+        shell: bash
+
+    env:
+      DUCKDB_DOCKER: mimic-iv/buildmimic/duckdb/docker
+      POSTGRES_DOCKER: mimic-iv/buildmimic/postgres/docker
+      MIMIC_DATA_DIR: ${{ github.workspace }}
+      MIMIC_OUTPUT_DIR: ${{ github.workspace }}/duckdb-out
+
+    steps:
+      - uses: actions/checkout@v7
+      - uses: ./.github/actions/download-demo
+      - uses: ./.github/actions/setup-duckdb
+
+      - name: Build both images
+        run: |
+          docker compose -f "$DUCKDB_DOCKER/docker-compose.yml" build
+          docker compose -f "$POSTGRES_DOCKER/docker-compose.yml" build
+
+      # --exit-code-from implies --abort-on-container-exit and returns the build
+      # job's own status, so a failed build fails the step.
+      - name: Build MIMIC-IV in DuckDB
+        run: |
+          mkdir -p "$MIMIC_OUTPUT_DIR"
+          docker compose -f "$DUCKDB_DOCKER/docker-compose.yml" \
+            up --exit-code-from mimic-build | tee duckdb.log
+          if grep -F -q "FAILED" duckdb.log; then
+            echo "::error::DuckDB container row-count validation failed:"
+            grep -F "FAILED" duckdb.log
+            exit 1
+          fi
+
+      - name: Read the built database with the runner's own duckdb
+        run: |
+          rows=$(duckdb "$MIMIC_OUTPUT_DIR/mimic4.db" -noheader -list \
+            -c "SELECT count(*) FROM mimiciv_derived.sepsis3")
+          echo "mimiciv_derived.sepsis3: ${rows} rows"
+          if [ -z "${rows}" ] || [ "${rows}" -eq 0 ]; then
+            echo "::error::The concepts are missing from the container-built database."
+            exit 1
+          fi
+
+      - name: Build MIMIC-IV in PostgreSQL
+        run: |
+          docker compose -f "$POSTGRES_DOCKER/docker-compose.yml" \
+            up --exit-code-from mimic-build | tee psql.log
+          if grep -F -q "FAILED" psql.log; then
+            echo "::error::PostgreSQL container row-count validation failed:"
+            grep -F "FAILED" psql.log
+            exit 1
+          fi
+
+      - name: Tear down
+        if: always()
+        run: |
+          docker compose -f "$DUCKDB_DOCKER/docker-compose.yml" down -v || true
+          docker compose -f "$POSTGRES_DOCKER/docker-compose.yml" down -v || true

.gitignore

@@ -1,8 +1,9 @@
-## Local docker-postgres raw data (do not commit large CSVs)
-mimic-iv/docker-postgres/mimic-data/
+## Raw data downloaded for local builds (do not commit large CSVs)
+mimic-data/
 
 ## Allow example env, ignore real env
-mimic-iv/docker-postgres/.env
+.env
+!.env.example
 
 # duckdb / sqlite db files
 *.db

mimic-iv/buildmimic/bigquery/schemas/demo_subject_id_schema.json

@@ -0,0 +1,7 @@
+[
+    {
+        "name": "subject_id",
+        "type": "INT64",
+        "mode": "REQUIRED"
+    }
+]

mimic-iv/buildmimic/download_data.sh

@@ -0,0 +1,40 @@
+#!/bin/bash
+# Download MIMIC-IV from PhysioNet into a directory any of the builds can use.
+#
+# Usage: ./download_data.sh [destination] [physionet-username]
+#
+#   destination           where to put the data (default ./mimic-data)
+#   physionet-username    also read from $PHYSIONET_USER, otherwise prompted for
+#
+# Requires a PhysioNet account credentialed for MIMIC-IV. The password is always
+# requested interactively, so it never reaches the process list or shell history.
+#
+# Shared by the postgres and duckdb builds: both want the same hosp/ and icu/
+# layout, so there is one copy of this here rather than one per engine.
+#
+# This is a convenience only. If you already have the data, point MIMIC_DATA_DIR
+# at it instead; any directory with hosp/ and icu/ subfolders will do.
+set -euo pipefail
+
+# The build scripts target the current release of MIMIC-IV.
+readonly MIMIC_VERSION="3.1"
+
+DEST="${1:-./mimic-data}"
+USERNAME="${2:-${PHYSIONET_USER:-}}"
+
+if [ -z "${USERNAME}" ]; then
+    read -rp "PhysioNet username: " USERNAME
+fi
+
+echo "Downloading MIMIC-IV v${MIMIC_VERSION} to ${DEST}"
+
+# -nH --cut-dirs=4 strips physionet.org/files/mimiciv/<version>/ from the paths,
+# leaving the hosp/ and icu/ subfolders directly under ${DEST}. That is the
+# layout buildmimic/postgres/load_gz.sql expects.
+wget -r -N -c -np -nH --cut-dirs=4 \
+    -A '*.csv.gz' \
+    -P "${DEST}" \
+    --user "${USERNAME}" --ask-password \
+    "https://physionet.org/files/mimiciv/${MIMIC_VERSION}/"
+
+echo "Downloaded $(find "${DEST}" -name '*.csv.gz' | wc -l | tr -d ' ') files to ${DEST}"

mimic-iv/buildmimic/duckdb/README.md

@@ -24,12 +24,15 @@ which you can obtain by either installing
 [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/install-win10)
 or [Cygwin](https://www.cygwin.com/).
 
+If you would rather not install DuckDB at all, the [docker](docker) folder
+builds the same database in a container with a pinned DuckDB version.
+
 ## Set-up
 
 ### Quick overview
 
 1. [Install](https://duckdb.org/docs/installation/) the CLI version of DuckDB
-2. [Download](https://physionet.org/content/mimiciv/2.0) the MIMIC-IV files
+2. [Download](https://physionet.org/content/mimiciv/) the MIMIC-IV files
 3. Create DuckDB database and load data
 
 ### Install DuckDB
@@ -41,6 +44,9 @@ the CLI version of DuckDB.
 You will need to place the `duckdb` binary in a folder on your environment path,
 e.g. `/usr/local/bin`.
 
+These scripts are built and tested against the 1.4.x LTS line (currently
+1.4.5), which is what CI uses.
+
 ### Download MIMIC-IV files
 
 Download the CSV files for [MIMIC-IV](https://physionet.org/content/mimiciv/)
@@ -63,43 +69,38 @@ mimic_data_dir
 
 The CSV files can be uncompressed (end in `.csv`) or compressed (end in `.csv.gz`).
 
-The easiest way to download them is to open a terminal then run:
+The easiest way to download them is to use the shared download script, which
+wraps `wget` and puts the files in the layout the build expects:
 
-```
-wget -r -N -c -np --user YOURUSERNAME --ask-password https://physionet.org/files/mimiciv/2.2/
+```sh
+../download_data.sh ./mimic-data YOURUSERNAME
 ```
 
-Replace `YOURUSERNAME` with your physionet username.
-
-This will make you `mimic_data_dir` be `physionet.org/files/mimiciv/2.2`.
+Replace `YOURUSERNAME` with your physionet username; you will be prompted for
+the password. This makes your `mimic_data_dir` be `./mimic-data`.
 
 # Create DuckDB database and load data
 
 The last step requires creating a DuckDB database and
 loading the data into it.
 
-You can do all of this with one shell script, `import_duckdb.sh`,
+You can do all of this with one shell script, `build_mimic.sh`,
 located in this repository.
 
-See the help for it below:
-
 ```sh
-$ ./import_duckdb.sh -h
-./import_duckdb.sh:
-USAGE: ./import_duckdb.sh mimic_data_dir [output_db]
-WHERE:
-    mimic_data_dir        directory that contains csv.gz or csv files
-    output_db: optional   filename for duckdb file (default: mimic4.db)
-$
+$ ./build_mimic.sh -h
+Usage: build_mimic.sh <mimic_data_dir> [output_db]
+       mimic_data_dir  the directory containing the hosp/ and icu/ subfolders
+       output_db       filename for the duckdb file (default mimic4.db)
 ```
 
 Here's an example invocation that will make the database in the default "mimic4.db":
 
 ```sh
-$ ./import_duckdb.sh physionet.org/files/mimiciv/2.2
+$ ./build_mimic.sh ./mimic-data
 
   <... output of script snipped ...>
-Successfully finished loading data into mimic4.db.
+MIMIC-IV build complete: /path/to/mimic4.db
 
 $ ls -lh mimic4.db
 -rw-rw-r--. 1 myuser mygroup 93G May 26 16:11 mimic4.db
@@ -109,6 +110,15 @@ The script will print out progress as it goes.
 Be patient, this can take minutes to hours to load
 depending on your computer's configuration.
 
+Beyond loading the data, it also derives the concepts from
+[concepts_duckdb](../../concepts_duckdb) into the `mimiciv_derived` schema and
+checks the loaded tables against known row counts. Set `MIMIC_MAKE_CONCEPTS` or
+`MIMIC_VALIDATE` to `false` to skip either.
+
+Each step is recorded in a `mimiciv_build_progress` table inside the database
+file, and tables that already hold rows are skipped, so re-running the script
+after an interruption resumes rather than starting over.
+
 * It took 16m25s on a Fedora 34 workstation with duckdb v 0.2.6, a btrfs filesystem with ztsd level 1 compression, AMD Ryzen 3900X, 32 GB RAM, Samsung 970 Evo NVMe SSD.
 * It took ~10m on a Mac M1 Max 2021, 32 GB RAM.