Skip to content

tfchain-validator: unattended installs, safe re-runs, and secrets hygiene - #107

Open
PeterNashaat wants to merge 1 commit into
developmentfrom
validator-install-hardening
Open

tfchain-validator: unattended installs, safe re-runs, and secrets hygiene#107
PeterNashaat wants to merge 1 commit into
developmentfrom
validator-install-hardening

Conversation

@PeterNashaat

Copy link
Copy Markdown
Member

While moving a validator to new hardware and provisioning a few more from
Terraform, I ran into a handful of rough edges in tfchain-validator/. All of
these are things that bit me for real:

Secrets hygiene. tfchain-validator/ and tfchain-validator-snapshots/
are the only components without a .gitignore for .secrets.env — and they
are the ones whose .secrets.env holds the node key and the validator wallet
mnemonic. Following the readme leaves that file inside the clone, one
git add -A away from being committed. Added the same .gitignore the other
components already have.

Unattended installs. install-tfchain-validator.sh and
validator-init.sh block on read, so they can't be driven from CI,
Terraform or config management. They now accept -y/--yes (or
ASSUME_YES=1) and are unchanged when run by hand.

Re-running the installer could destroy a synced node. The snapshot restore
ran unconditionally and extracted over the existing database. It now skips the
restore when the chain database is already populated.

Snapshot download location. qanet, testnet and mainnet downloaded the
archive into the home directory, i.e. the root filesystem; devnet used /srv.
The mainnet snapshot is ~28G compressed, which is enough to fill a root disk.
All four now use /srv/grid_snapshots_tmp.

Smaller bits. validator-init.sh reminds the operator to clear MNEMONIC
afterwards, and prep-env-prereq.sh installs node_exporter 1.12.1 instead of
1.6.0.

Readme. Three additions:

  • Moving an existing validator to another machine — reusing the keystore and
    node key means no setKeys and no council motion, and the old node must be
    stopped first or you are equivocating. That path wasn't documented anywhere.
  • Container log rotation — uncapped container logs have taken validators
    down for us; log-opts only apply to containers created afterwards.
  • Running in a VM whose root filesystem cannot host container storage — on
    VM images with a virtiofs root, mknod is denied and image extraction fails
    on whiteout conversion. Recent Docker keeps images in the containerd image
    store, so moving data-root alone isn't enough; containerd's root has to
    move too.

No behaviour changes for anyone running the scripts interactively.

…iene

- .gitignore for .secrets.env under tfchain-validator/ and
  tfchain-validator-snapshots/. Every other component already had one; these
  two are the ones whose .secrets.env holds the node key and the validator
  wallet mnemonic, and without it a git add -A in a clone stages them.
- install-tfchain-validator.sh and validator-init.sh accept -y/--yes (or
  ASSUME_YES=1) so they can run from CI, Terraform or config management.
- the installer no longer restores the snapshot when the node already has a
  chain database, so re-running it cannot clobber a synced validator.
- snapshot download goes to /srv/grid_snapshots_tmp on every network; qanet,
  testnet and mainnet used the home directory, which can fill the root disk
  (the mainnet snapshot is ~28G compressed).
- validator-init.sh reminds the operator to clear MNEMONIC afterwards.
- prep-env-prereq.sh installs node_exporter 1.12.1 instead of 1.6.0.

readme: how to move an existing validator to new hardware (keys reused, no
council motion, stop the old node first), container log rotation, and the
container-storage caveat on VMs whose root filesystem disallows mknod.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant