Draft for Better Attack Data Archive

.github/workflows/build_dataset_archive.yml

@@ -0,0 +1,36 @@
+name: build-dataset-archive-on-merge-to-default-branch
+
+on:
+  push:
+    branches:
+      - master
+
+jobs:
+  build-archive:
+    runs-on:
+      group: attack-data-runners
+    steps:
+      - name: Checkout Repo
+        uses: actions/checkout@v6
+        with:
+          lfs: true
+          fetch-depth: 0 # full history needed to compute per-file last-updated timestamps
+
+      - name: Set up Python
+        uses: actions/setup-python@v6
+        with:
+          python-version: '3.14'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r bin/requirements.txt
+
+      - name: Build datasets archive
+        run: python bin/build_dataset_archive.py
+
+      - name: Upload archive as workflow artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: attack_data_archive
+          path: attack_data_archive/attack_data_archive.zip

README_ATTACK_DATA_CACHE.md

@@ -0,0 +1,35 @@
+# Attack Data Archive Cache
+
+This directory holds a local snapshot produced by `bin/build_dataset_archive.py`:
+a Zstandard-compressed archive of the `datasets/` folder from
+[splunk/attack_data](https://github.com/splunk/attack_data), plus a
+standalone metadata file describing exactly what went into it.
+
+## Files
+
+### `metadata.yml`
+Describes the archive build and maps every file in it back to its source:
+
+- `generated_at_utc` — when the archive was built
+- `file_count` — total number of files included
+- `gitref` — the exact commit hash the snapshot was built from
+- `github_url` — link to the source branch/tag on GitHub
+- `total_uncompressed_size_bytes` — combined size of all files before compression
+- `lfs-files` — a map keyed by the file's Git LFS download URL
+  (`media.githubusercontent.com/...`), with:
+  - `relative_path` — path within `datasets/`
+  - `uncompressed_size` — size in bytes
+  - `last-updated` — timestamp of the most recent commit that touched the file
+- `non-lfs-files` — a flat list of relative paths for files stored directly
+  in git (not LFS-tracked)
+
+Use the `lfs-files`/`non-lfs-files` sections to fetch an individual dataset
+file directly from GitHub without downloading the full archive.
+
+## Notes
+
+- `metadata.yml` is also embedded inside the `.zip` archive itself, so it
+  travels with it even if separated from this standalone copy.
+- `gitref` pins the exact commit; re-running the build script against a
+  later commit will produce different contents even if `datasets/` is
+  otherwise unchanged (e.g. `last-updated` timestamps).