diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000..51be1ac --- /dev/null +++ b/.dockerignore @@ -0,0 +1,9 @@ +.git +.sg +export +out +**/.terraform +**/.terraform.lock.hcl +*.tfstate +*.tfstate.* +*.tfvars diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..bba1bf5 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,3 @@ +# Keep LF everywhere: a Git-for-Windows clone with core.autocrlf would turn the +# bash scripts into CRLF and break them under WSL/Docker. +* text=auto eol=lf diff --git a/.gitignore b/.gitignore index 1d638cc..7819528 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,8 @@ eggs/ .eggs/ lib/ lib64/ +# The migrator's sourced shell libraries are not Python build output. +!scripts/lib/ parts/ sdist/ var/ @@ -146,6 +148,8 @@ crash.log # control as they are data points which are potentially sensitive and subject # to change depending on the environment. *.tfvars +# The init wizard keeps the previous file as terraform.tfvars.bak +*.tfvars.bak # Ignore override files as they are usually used to override resources locally and so # are not checked in @@ -163,4 +167,7 @@ out/* zip zip/* +# Migrator local cache + config (downloaded tool binaries, workflow-group map) +.sg/ + .DS_Store \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..b4307e7 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,91 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## What this is + +A migration tool that extracts workloads from other IaC platforms and transforms them into StackGuardian Workflow definitions (`sg-payload.json`), ready for bulk import via [sg-cli](https://github.com/StackGuardian/sg-cli). Currently the only implemented source is **Terraform Cloud / Enterprise (TFC/TFE)**. + +There is no application code to build — the "engine" is Terraform itself. The migrator is a Terraform root module that uses the `tfe` provider to read workspaces and the `local`/`null` providers to write the output payload and pull state files. + +## Architecture + +The migration is a user-driven pipeline, not a single program: + +1. **Extract + transform** — `transformer/terraform-cloud/` is a Terraform root module. `terraform apply` reads TFC/TFE workspaces and emits one `/sg-payload..json` per TFC project, a `migration-summary.md`/`.json` report, and per-workspace `.tfstate` files when `exportStateFiles` is true. +2. **Enrich (variable sets)** — `scripts/enrich_variable_sets.sh` merges TFC Variable Set variables into the payloads via the TFC API (the provider can't enumerate sets). Resolves global/project/workspace scope + TFC precedence per workspace; non-sensitive only. +3. **Tuning** — adjust per-workspace differences (integration IDs, VCS auth, runners, approvers, version) via the `workspaceOverrides` variable and re-apply; or hand-edit the payload files. `example_payload.jsonc` is the annotated field reference. +4. **HCL→JSON conversion** — `scripts/convert_hcl_to_json.sh` rewrites HCL-string variable values in each payload to real JSON (one result line per file; details with `-v`). +5. **Validation** — `scripts/validate_payload.sh` checks each payload against `schema/sg-payload.schema.json` (downloads `yajsv`; yajsv's lines are re-rendered as ✓/✗ per file, raw with `-v`). +6. **Import** — `sg-cli workflow create --bulk`, run once per project file, each into its own workflow group. + +### Orchestration & tooling + +The five phases are wrapped by an orchestrator so users don't run them by hand: + +- `sg-migrate.sh` (host entrypoint, repo root) — runs `scripts/migrate.sh` **inside the Docker image** (`Dockerfile`), bind-mounting the repo at `/app` and forwarding `SG_API_TOKEN`/`SG_ORG`/`SG_BASE_URL`/`TFE_TOKEN`/`TF_TOKEN_*`. `--tfvars FILE` (or `SG_TFVARS`) is a host path: the wrapper strips the flag, resolves it and passes it on as `SG_TFVARS` — rewritten to `/app/` when the file is inside the checkout, otherwise mounted read-only at `/tmp/sg-run.tfvars`. For TFC auth it prefers a long-lived `TFE_TOKEN` (env); otherwise it mounts `~/.terraform.d/credentials.tfrc.json` read-only. Runs natively instead when `--native`/`--local` is passed, `SG_NATIVE=1` is set, the command is `clean`/`completion`/help (or no command is given), or Docker is absent. Exports `SG_PROG` so `migrate.sh` shows `./sg-migrate.sh` in its usage/hints. `update` is host-only too (`cmd_update` in the wrapper, never passed to `migrate.sh`): refuses a non-git checkout, a detached HEAD or dirty tracked files, then `git pull --ff-only` and rebuilds the image only when the `Dockerfile` changed between the old and new HEAD — this is how customers pick up fixes from a plain clone (untracked config/output survive). `migrate.sh` lists `update` in its usage/`SG_COMMANDS` and rejects it with a host-only hint. `.gitattributes` forces LF so a Git-for-Windows clone does not CRLF the scripts. +- `scripts/migrate.sh` — the actual orchestrator (sources `scripts/lib/*.sh`, see below). Subcommands `init|preflight|apply|enrich|convert|validate|import|triggers|checklist|all|clean|completion` (no command prints the help menu; `enrich` runs in `all` unless `--no-variable-sets`). Flags beyond the basics: `--dry-run` (import, or all: the local export runs, then the plan is shown and nothing is created in SG), `--fresh` (ignore run state), the scope flags from `lib/scope.sh` — `--project NAME|SLUG`, `--workspace GLOB`, `--exclude-workspace GLOB`, `--tag NAME`, `--exclude-tag NAME`, all repeatable; tfvars holds the widest scope, include flags replace the tfvars list, exclude flags add to it; apply passes them as `-var` (`tfProjects`, `workspacenames`, `tfWorkspaceIgnoreNames`, `tfWorkspaceTags`, `tfWorkspaceIgnoreTags`), the later phases apply the same names/excludes to the payload entries (`ws_selected` in bash, `WS_SCOPE_JQ` + `WS_JQ_ARGS` in jq) and `--project` to the payload files by slug (`project_selected`); `--workspace '*'` alone is not a narrowing, so the unchanged-file skip still applies; a scope that matches nothing is an error —, `--tfvars FILE` (`TFVARS`, absolute; also `-var-file` for terraform and exported to the enrich script; `clean --all` only removes the module's own file), `--skip-preflight`, `--no-secret-stubs`, `--upgrade` (with `init`: `tfvars_upgrade`, append the settings an older tfvars lacks; preflight warns about them until then), and the run configuration flags `--set KEY=VALUE`, `--cloud-connector ID`, `--vcs-connector ID`, `--runner-group NAME|shared`, `--workflow-group NAME` (one-run overlay on the tfvars, see `scope.sh`; connector kinds looked up in SG; with `--project` they are that project's `projectOverrides`, else the `SGDefault*` values). `import`/`all` end by writing `export/run-result.json` + `run-summary.md` (`write_run_result`, outcomes planned/blocked/success/failed) for CI. Resolves each tool via `sg_resolve` (PATH first — the image installs them — else `sg_ensure_*` cache). `apply`/`all` first run `require_tfc_auth`, which resolves the token the tfe provider will use (`TFE_TOKEN`, `TF_TOKEN_`, or the `terraform login` file for `tfHostname`) and verifies it with `GET /api/v2/account/details`, failing fast on a missing or rejected (expired) credential. Runs `convert` and `import` in parallel (`--concurrency`, default 4), raises `terraform apply -parallelism`, and retries `sg-cli` imports with backoff (`sg_retry`, `SG_RETRIES`). Import resolves each project's workflow group with `group_for` (legacy `.sg/workflow-groups.json` entry → the payload's `CLIConfiguration.WorkflowGroup.name` via `payload_group` → `tfc-`) and runs `plan_groups`: STATUS `reuse` (exists) / `create` (missing; created after the confirmation) / `missing!` (`--no-create-groups`) / `moved!` — the FILE table is printed only when a group is not `reuse`, a problem was found or the legacy mapping applies, and the confirmation prompt names the groups to create; it collects `PLAN_PROBLEMS` — a project whose workflows still live in the group recorded in `state.json` (or the default group) when the target changed (SG cannot move workflows), and two projects sharing a group with overlapping names — prints them as ✗ lines, still shows the per-workflow plan (also under `--dry-run`) and then dies. The skip set (`state_import_done`) is computed before the plan so the ACTION column can say `skip`/`update`/`create`. Groups are never PATCHed. Imports go through `import_bulk`: it first lists the group (`sg_list_workflows`) and PATCHes every workflow that already exists (`sg_update_workflow`, `[updated]` marker, state re-uploaded, failures as `Failed to update : ...`), so the plan's `update` rows never hit a create; the remaining, new workflows with Terraform variables go via `sg-cli workflow create --bulk`, workflows **without** via a direct `POST .../wfs/` (`sg_create_workflow` + `sg_upload_tfstate` in `sg_api.sh`) because sg-cli drops an empty `iacInputData.data` and the API rejects the workflow — a workaround (`TODO(sg-cli)` markers) until sg-cli ships with sg-sdk-go >= v1.5.7, which sends the empty object; the direct path prints sg-cli-style `Failed to create : : ` lines so the parsing below covers both. sg-cli's own output is shown only with `-v`; by default `import_bulk` renders one line per workflow from it — `wf_line` (`✓ / created|updated[, state uploaded]`, a `!` line when the state upload failed) and `wf_failed` (`✗ / not created|updated — : `, followed by `explain_api_error`'s hint) — while the raw lines still go to the parse file for `do_import`; `upload_state` prints nothing itself and leaves `US_RESULT`/`US_WHY` for `wf_line`; the `importing -> ` and "N workflow(s) already exist / have no Terraform variables" lines are `-v` only. State files are the migrator's job on both paths: `import_bulk` tracks sg-cli's per-workflow upload lines (`State file uploaded successfully` / `Failed to upload state file for `), re-uploads the failures via `upload_state` → `sg_upload_tfstate` (adds the `x-ms-blob-type: BlockBlob` header Azure Blob requires; sg-cli omits it and also treats anything but a literal `HTTP/1.1 200 OK` as failure — second `TODO(sg-cli)`), and writes `[state] uploaded|failed|none ` markers into the parsed output; `do_import` turns them into `state_uploaded`/`state_failed` in the result, a workflow without its state fails the file (and `state_import_done` re-tries it next run). Existing workflows come back from sg-cli as `Failed to create : 409: Workflow ID not unique` (its update branch matches the outdated text `Workflow name not unique` — third `TODO(sg-cli)`); `import_bulk` collects those, PATCHes them via `sg_update_workflow`, re-uploads their state and drops the failure line, and `sg_create_workflow` does the same fallback on the direct path (prints `updated`). Before the parallel import, `probe_import` imports one workflow alone (first selected of the first file) and dies unless the create and the state upload both succeeded — the fail-fast for environment problems. sg-cli exits 0 even when individual workflows fail, so `do_import` parses its output: a workflow rejected as above SG's managed Terraform ceiling (1.5.7, last MPL/FOSS release) is re-imported with `SGDefaultTerraformVersion` (read from `terraform.tfvars`; an explicit `null` there means the `terraformVersion` key is deleted instead, so the org's execution preset decides — `tfvars_is_null` tells an explicit null from a missing key), the payload patched in place, and the case logged to `export/terraform-version-fallbacks.log` plus a printed notice; any other per-workflow failure fails the run. `cmd_import` reads the org's execution preset (`sg_execution_preset`, `GET /orgs/{org}/` → `Settings.workflowDefaults`) and `preset_labels` (report.sh) turns it into the `preset (1.5.7)` / `preset (private:rg)` cells of the plan and the `SG_TF_FALLBACK_LABEL` used in messages. The trigger pass (`do_set_triggers` → `sg_set_vcs_triggers`) skips workflows that do not exist in SG, keeps the sha of each posted `{VCSConfig, VCSTriggers}` body in `triggers..sha` and does not re-POST an unchanged, already-set workflow (`--fresh` does; the endpoint upserts anyway, a 4xx "already exists" from older builds counts as success), and `sg_api_post` returns 22 on 4xx so `sg_retry` (via `SG_NO_RETRY_RC`) does not retry definitive errors; `do_set_triggers` ends with one line per file that leaves out zero counts (`triggers set on N workflow(s), M unchanged, K without triggers, F failed, X not in SG`). `clean` removes local artifacts (`export/`, TF state, tool cache); `clean --all` also removes config. `completion bash|zsh` prints a completion script (`cmd_completion`; commands + descriptions come from `SG_COMMAND_DESCS` (`SG_COMMANDS` is derived from it), options from `SG_OPTIONS`; keep them in sync with the parser and the host flags; the zsh script also works autoloaded from `$fpath` as `_sg-migrate.sh`); like `clean` it always runs natively. Output is concise by default — `apply` captures terraform's init/plan output and shows only progress + the final summary (or the full log on failure, `TF_IN_AUTOMATION=1`/`-no-color`; the `terraform providers ready` line is `-v` only); `convert` with a single file prints only the ✓ line (the file's own result line folded in) and `validate` prints one ✓ for all files (per-file ✓ with `-v`, ✗ always); `-v`/`--verbose` (`SG_VERBOSE`) streams everything and un-gates the convert detail lines. Colored logging via `sg_step`/`sg_log`/`sg_success`/`sg_warn`/`sg_err`/`sg_row` — phase-level lines carry no prefix (plain, `✓ `, `! `, `✗ `; the former `[sg-migrate]` tag was dropped since raw tool output is `-v` only) — plus the indented per-item status lines `sg_ok`/`sg_bad`/`sg_note` (✓/✗/!); paths shown relative via `sg_rel` (all auto-off when not a TTY / `NO_COLOR`). Long steps show a live progress line (`sg_run_quiet` for terraform init/apply, `run_parallel