[FEAT] Resolve Key Vault-Backend Environment References and Update .env_example - #2363
[FEAT] Resolve Key Vault-Backend Environment References and Update .env_example#2363Victor Valbuena (ValbuenaVC) wants to merge 43 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Azure Key Vault-backed environment bootstrapping with recursive reference resolution, precedence handling, warnings, and documentation.
Changes:
- Resolves
env:,kv:, aliases, and escaped literals. - Adds environment-source validation and AKV/local-file precedence.
- Expands tests and configuration documentation.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
pyrit/setup/initialization.py |
Implements AKV loading and reference resolution. |
tests/unit/setup/test_initialization.py |
Tests environment initialization behavior. |
doc/getting_started/pyrit_conf.md |
Documents loading precedence and AKV references. |
.pyrit_conf_example |
Updates example AKV configuration guidance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…nto env-refactor Merging latest changes from main.
Victor Valbuena (ValbuenaVC)
changed the title
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Suppressed comments (5)
tests/unit/setup/test_initialization.py:380
- This patch target is no longer called by
initialize_pyrit_async, leaving the output assertion dependent on any real default environment files. Patch_resolve_environment_filesinstead so unrelated local files cannot add output or trigger reference resolution.
@mock.patch("pyrit.setup.initialization._load_environment_files", return_value=True)
pyrit/setup/initialization.py:537
- Direct callers using the former list-shaped
env_akv_refreach.strip()here and getAttributeError, rather than the deliberateValueErrorused byConfigurationLoader. Validate the runtime type before calling string methods so this public API rejects legacy values consistently.
if not env_akv_ref.strip():
pyrit/setup/initialization.py:439
- This merge is case-sensitive even on Windows. For example, ambient
Path=oldplus a winningPATH=newleaves both keys, soenv:Pathreturns the ambient exact match and violates the documented merged-source precedence. Normalize keys on Windows while applyingvalueslast.
reference_environment = {**ambient_environment, **values}
tests/unit/setup/test_initialization.py:372
initialize_pyrit_asyncno longer calls_load_environment_files, so this patch is inert and the test can read real~/.pyritfiles (and even resolve their Key Vault references). Patch the resolver now used by initialization to keep the unit test isolated.
This issue also appears on line 380 of the same file.
@mock.patch("pyrit.setup.initialization._load_environment_files", return_value=True)
doc/getting_started/pyrit_conf.md:174
- This row contradicts both the implementation and the earlier AKV precedence section: when
env_filesis omitted, initialization loads both.envand.env.localafter the bootstrap. Remove the claim that only.env.localis loaded.
| Omitted or `null` | Load default `~/.pyrit/.env` and `~/.pyrit/.env.local`, or only `.env.local` after an AKV root |
…to env-refactor Merge in changes from main.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Suppressed comments (3)
.env_example:294
- This Azure OpenAI TTS base URL is missing the required
/openaisegment.OpenAITTSTarget._get_provider_examples()expectshttps://{resource}.openai.azure.com/openai/v1, so copying this example will fail endpoint validation or send requests to the wrong path.
AZURE_OPENAI_TTS_ENDPOINT2 = "<https://xxxxx.openai.azure.com/v1>"
pyrit/setup/initialization.py:208
- This still accepts malformed/nonexistent Azure vault hosts such as one-character names, leading/trailing hyphens, and consecutive hyphens. Azure vault names are 3–24 characters, start with a letter, end with an alphanumeric character, and cannot contain consecutive hyphens; without those checks, URLs that should fail preflight proceed to credential/client creation.
valid_vault_name = 1 <= len(vault_name) <= 63 and all(
char.isascii() and (char.isalnum() or char == "-") for char in vault_name
)
.env_example:55
- Dotenv preserves the angle brackets inside these quoted values, so this becomes the literal endpoint
<https://api.openai.com/v1>, which is not a valid URL. The same pattern now appears on every endpoint assignment in this file; remove the<and>from all actual URL values.
PLATFORM_OPENAI_CHAT_ENDPOINT="<https://api.openai.com/v1>"
|
There's a lot to read through, so apologies if this has been addressed. But I want to make sure we don't lose debuggability and that I can easily tell which targets we have. Right now, I use .env ~weekly to help debug myself and others. E.g. what is the default adversarial model? What is configured? Where is the default open ai target referencing? Where is X pointing at? Etc Right now I do a lot of that with .env. In theory we could download a .env and I could use it the same way and it could reference key vault secrets. I'm worried if all of .env is obstructed, I won't be able to see what's configured. e.g. what is the adversarial model? Or how do I configure for another target when not in the GUI? There might be answers to this. But if they're aren't, we may want to download a .env to help even see which targets are available (and have that be able to reference keyvault secrets) |
Fwiw, one of the goals of this PR is to avoid having users keep an I see what you mean though and I think we can fix this by adding a save to disk flag that saves the new |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.
Suppressed comments (5)
.env_example:191
- This primary TTS section still defines the old
OPENAI_TTS_*names, while both TTS target registrations now readAZURE_OPENAI_TTS_*(pyrit/setup/initializers/targets.py:360-371). Filling this section therefore does not configure either registered TTS target; reconcile these names and the duplicate Azure definitions later in the file.
OPENAI_TTS_ENDPOINT1 = "<https://xxxxx.openai.azure.com/openai/v1>"
OPENAI_TTS_MODEL1 = "tts"
OPENAI_TTS_UNDERLYING_MODEL1 = "tts"
OPENAI_TTS_ENDPOINT2 = "<https://xxxxx.openai.azure.com/v1>"
pyrit/setup/initialization.py:106
- The PR description promises that bootstrap documents remain in memory and are never written to disk, but this public option intentionally persists them, including any literal credentials in the bootstrap document. Either remove the write-to-disk feature or update the stated contract and scope so this security-sensitive behavior is explicitly reviewed.
env_akv_write_env (bool): If True, save fetched bootstrap documents with unresolved
child references to ``~/.pyrit/.env``. Defaults to False.
.env_example:32
- The angle brackets are literal dotenv value characters, not Markdown delimiters, so copying this example produces endpoints such as
<https://...>that URL clients will reject. This pattern occurs throughout the newly updated endpoint values; remove the<and>wrappers everywhere in this file.
AZURE_OPENAI_GPT4O_ENDPOINT="<https://xxxx.openai.azure.com/openai/v1>"
.env_example:140
- These aliases are interpolated before
PLATFORM_OPENAI_CHAT_ENDPOINTandPLATFORM_OPENAI_CHAT_MODELare assigned at lines 318–320. Because python-dotenv resolves in assignment order and does not revisit earlier values, both aliases become empty when this file is loaded. Move source definitions before their aliases (also for the response, realtime, image, and TTS forward references below) or move the aliases after the sources.
OPENAI_CHAT_ENDPOINT=${PLATFORM_OPENAI_CHAT_ENDPOINT}
OPENAI_CHAT_MODEL=${PLATFORM_OPENAI_CHAT_MODEL}
.env_example:172
- This primary image section still defines the old
OPENAI_IMAGE_*2names, whileTargetConfignow readsAZURE_OPENAI_IMAGE_*2(pyrit/setup/initializers/targets.py:349-352). A user following the file's instruction to fill only this section will not configureopenai_image_platform; reconcile these names and the duplicate Azure definitions later in the file.
This issue also appears on line 187 of the same file.
OPENAI_IMAGE_ENDPOINT2 = "<https://xxxxx.openai.azure.com/openai/v1>"
OPENAI_IMAGE_MODEL2 = "dall-e-3"
OPENAI_IMAGE_UNDERLYING_MODEL2 = "dall-e-3"
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 24 changed files in this pull request and generated no new comments.
Suppressed comments (9)
pyrit/setup/initializers/targets.py:371
- This preserves the
openai_tts_platformregistry name but changes all of its inputs to Azure-only variables. The former endpoint-2 example was the OpenAI platform endpoint, so users now get an Azure target under the platform registry name and there is noPLATFORM_OPENAI_TTS_*path anywhere in the repository. Keep a distinct platform variable set here, or rename this registration as a second Azure target and add the actual platform registration.
endpoint_var="AZURE_OPENAI_TTS_ENDPOINT2",
key_var="AZURE_OPENAI_TTS_KEY2",
model_var="AZURE_OPENAI_TTS_MODEL2",
underlying_model_var="AZURE_OPENAI_TTS_UNDERLYING_MODEL2",
.env_example:293
- These platform source assignments occur after
OPENAI_CHAT_*andOPENAI_RESPONSES_*interpolate them at lines 128-137. Since python-dotenv resolves in assignment order, copying this file into a clean environment leaves those generic endpoint/model/key aliases empty. Move the platform source block before its aliases (the same ordering contract is documented in this PR).
PLATFORM_OPENAI_CHAT_ENDPOINT="<https://api.openai.com/v1>"
PLATFORM_OPENAI_CHAT_KEY="sk-xxxxx"
PLATFORM_OPENAI_CHAT_MODEL="gpt-4o"
PLATFORM_OPENAI_RESPONSES_ENDPOINT="<https://api.openai.com/v1>"
PLATFORM_OPENAI_RESPONSES_KEY="sk-xxxxx"
.env_example:314
OPENAI_REALTIME_ENDPOINTandOPENAI_REALTIME_MODELinterpolate these names at lines 149-150, before these assignments are parsed. In a standalone copy of.env_example, both aliases therefore become empty. Define the platform realtime values before the generic aliases.
PLATFORM_OPENAI_REALTIME_ENDPOINT="wss://api.openai.com/v1"
PLATFORM_OPENAI_REALTIME_KEY="sk-xxxxx"
PLATFORM_OPENAI_REALTIME_MODEL="gpt-4o-realtime-preview"
.env_example:360
- The generic image aliases at lines 159-160 reference
AZURE_OPENAI_IMAGE_ENDPOINT2andAZURE_OPENAI_IMAGE_MODEL2before this block defines them. Python-dotenv does not resolve references retroactively, so those generic values are empty when users copy this example into a clean environment. Move these primary assignments before the alias block.
AZURE_OPENAI_IMAGE_ENDPOINT1 = "<https://xxxxx.openai.azure.com/openai/v1>"
AZURE_OPENAI_IMAGE_API_KEY1 = "xxxxxx"
AZURE_OPENAI_IMAGE_MODEL1 = "deployment-name"
AZURE_OPENAI_IMAGE_UNDERLYING_MODEL1 = "dall-e-3"
AZURE_OPENAI_IMAGE_ENDPOINT2 = "<https://xxxxx.openai.azure.com/openai/v1>"
.env_example:374
- The generic TTS aliases at lines 171-172 interpolate the endpoint-2/model-2 names before this source block is reached, leaving both values empty in a clean environment. Place the primary TTS assignments before those aliases so the documented assignment-order semantics produce usable values.
AZURE_OPENAI_TTS_ENDPOINT1 = "<https://xxxxx.openai.azure.com/openai/v1>"
AZURE_OPENAI_TTS_KEY1 = "xxxxxxx"
AZURE_OPENAI_TTS_MODEL1 = "tts"
AZURE_OPENAI_TTS_UNDERLYING_MODEL1 = "tts"
AZURE_OPENAI_TTS_ENDPOINT2 = "<https://xxxxx.openai.azure.com/v1>"
pyrit/setup/akv_initialization.py:496
write_textcreates/truncates the potentially secret-bearing file under the process umask before permissions are restricted. During that window it may be readable by other users, and ifchmodfails the code merely warns and leaves the exposed file in place. Create the file securely with mode0600before writing (and fail/clean up if permissions cannot be guaranteed).
env_file.write_text(content, encoding="utf-8")
try:
env_file.chmod(0o600)
except OSError:
logger.warning("Could not restrict permissions on written AKV environment file: %s", env_file)
pyrit/setup/initializers/targets.py:352
openai_image_platformnow reads the Azure deployment variables, while the updated example defines the actual platform values asPLATFORM_OPENAI_IMAGE_ENDPOINT/KEY/MODEL(.env_example:300-302). Consequently, configuring those documented platform values will not register this target, and the Azure endpoint-2 configuration is mislabeled as the platform target. Wire this entry to thePLATFORM_OPENAI_IMAGE_*variables and update the corresponding initializer test.
This issue also appears on line 368 of the same file.
endpoint_var="AZURE_OPENAI_IMAGE_ENDPOINT2",
key_var="AZURE_OPENAI_IMAGE_API_KEY2",
model_var="AZURE_OPENAI_IMAGE_MODEL2",
underlying_model_var="AZURE_OPENAI_IMAGE_UNDERLYING_MODEL2",
.env_example:32
- These angle brackets are stored literally by python-dotenv, so the copied endpoint becomes
<https://...>rather than a valid URL. The same Markdown-style wrapping appears in 52 URL assignments throughout this dotenv file; remove the angle brackets from all of them.
This issue also appears in the following locations of the same file:
- line 289
- line 312
- line 356
- line 370
AZURE_OPENAI_GPT4O_ENDPOINT="<https://xxxx.openai.azure.com/openai/v1>"
pyrit/setup/akv_initialization.py:422
- The PR description states that bootstrap documents remain in memory and are never written to disk, but this new public option writes them to
~/.pyrit/.env(including any literal secrets in the bootstrap). Either remove this disk-writing path to preserve the stated guarantee or update the PR's security contract and description explicitly.
env_akv_write_env: bool = False,
There was a problem hiding this comment.
Thanks Victor Valbuena (@ValbuenaVC) for working through all the earlier feedback on this. I pulled the latest changes, read through the existing review threads, and reran the focused setup/configuration tests locally. The source ordering is much clearer now, and keeping python-dotenv responsible for parsing was the right simplification.
I found a few cases that I think still need to be fixed before this merges. Most of them sit around the new debug-export path or configuration migration, so the current tests and green CI do not exercise them.
First, env_akv_strict and env_akv_write_env are annotated as booleans but are not validated at runtime. A config like this is accepted:
env_akv_write_env: "false"
env_akv_strict: "false"Both values remain strings, and because a non-empty string is treated as True, the first one actually enables writing plaintext secrets to ~/.pyrit/.env. I think these fields should reject anything that is not a real bool, both in ConfigurationLoader and at the direct initialization boundary. This one feels especially important because it can turn on a sensitive feature when the user explicitly wrote false.
There are also two data-integrity problems in the generated debug file:
-
_serialize_terminal_dotenv_value()does not escape backslashes before putting a secret in a single-quoted dotenv value. In a round-trip throughpython-dotenv, two consecutive backslashes become one and four become two. That means some passwords, tokens, paths, or other arbitrary secret values will be silently changed in the file. -
References created through interpolation are resolved at runtime but are not rewritten in the generated file. For example:
A=kv:https://vault.vault.azure.net/secrets/key B=${A} A=literal
At runtime,
Bbecomes the fetched secret. In the generated file,B=${A}is retained, so reloading the file givesBthe originalkv:URI instead. The renderer currently checks whether the raw assignment text looks like a reference, while resolution checks the interpolated value. I think both runtime application and debug rendering need to consume the same assignment-level resolution result.
The no-clobber guarantee for ~/.pyrit/.env also has a race. The existence check happens before the Key Vault network calls, but _write_akv_env_file() later publishes with unconditional os.replace(). I simulated another process creating .env during the fetch, and the newly created file was overwritten. A second exists() check would still race; the final publish needs an atomic create-if-absent operation. The early check can stay for fast feedback, but the writer needs to enforce the guarantee authoritatively.
I also found a non-strict fallback case that leaves a Key Vault URI in the environment instead of resolving it. If an ordinary file provides a valid kv: reference and a later .env.local overrides it with malformed kv:short, non-strict mode restores the earlier URI and then immediately continues. My probe ended with the literal earlier kv: URI in os.environ and made zero child-secret reads. Since the documented behavior is to skip the malformed assignment, I think the loader needs an ordered candidate chain per variable, rather than a single scalar fallback, so it can discard the malformed winner and fully resolve the next candidate.
The image/TTS environment-variable changes look breaking as well. On main, PyRIT accepts the documented OPENAI_IMAGE_*1/2 and OPENAI_TTS_*1/2 variables. This branch removes those inputs completely and replaces them with AZURE_OPENAI_*1/2, without aliases or a migration warning. Existing configurations will silently stop registering those targets.
There is a provider mismatch in the new names too: openai_image_platform and openai_tts_platform now read Azure-prefixed variables whose example values are Azure endpoints. The second TTS example is https://xxxxx.openai.azure.com/v1, which is also missing the /openai/v1 path that PyRIT recommends and passes through unchanged. I would preserve the old variables as deprecated aliases and keep the *_platform registry entries backed by actual platform OpenAI settings. A larger provider-explicit rename would be safer as a separate migration with compatibility tests.
Two smaller contract gaps are worth covering in the same pass:
- With
PYTHON_DOTENV_DISABLED=true,load_dotenv()returns false and the AKV loader exits before resolving child references, but debug mode can still write that raw document and call it fully resolved. Either dotenv-disabled mode should suppress this publication, the option combination should be rejected, or output resolution should be separated from environment mutation. test_env_example_names_are_referenced_in_repositoryis one-way: it begins with names already present in.env_exampleand looks for weak textual references elsewhere. It cannot catch a code-required name missing from the example, removed compatibility names, duplicate assignments, provider mismatches, or invalid endpoint families. Deriving required names fromTARGET_CONFIGS, comparing both directions with an explicit allowlist, and checking duplicates/provider URLs structurally would make this a real drift test.
I don't think the documented non-transactional updates, one-hop terminal references, sequential precedence, or lack of child-secret caching are problems by themselves. Those are clear choices now. The recurring issue is that precedence is represented by mutating os.environ plus one fallback value per variable, which loses information needed by fallback resolution and debug rendering. A small internal ordered assignment/candidate model would address both without changing the public API or replacing python-dotenv.
I also reproduced the boolean, backslash, interpolation, no-clobber, and fallback cases directly against the latest changes.
Description
This PR makes Azure Key Vault the canonical source for shared and deployed environment configuration while preserving process variables, explicit dotenv files, and a two-version migration path for legacy
~/.pyrit/.env. It also updates.env_exampleand adds an integration test to monitor drift between environment variables referenced in the repo and those mentioned in.env_example.Configuration
env_akv_strictraises on an unresolved or malformed secret when true, andenv_akv_write_envwrites the fully resolved.envwith secrets to disk for debugging purposes.Source Precedence
Sources load in this order:
env_akv_reforder.envor ordinary explicitenv_files.env.localProcess values are retained. Key Vault, legacy
.env, and ordinary explicit files fill only missing values. Only a file named.env.localoverrides existing values.When Key Vault is configured:
~/.pyrit/.envis ignored and emits a deprecation warning. In a later version of PyRIT, this will raise an error.~/.pyrit/.env.localstill loads as the supported local override.env_filesremain supported regardless of filename or location.Auto-discovered
.envremains available as a legacy source until PyRIT 1.3.0. Users should migrate shared configuration to Key Vault and use.env.localfor temporary plaintext overrides or cases where Azure is unavailable.Key Vault References
Bootstrap documents and local dotenv files support complete-value Key Vault references:
kv:is canonical.akv:,azure_key_vault:, andenv_akv_ref:are valid compatibility aliases.Remote bootstrap references must target the bootstrap document's vault. Local files may reference any validated supported Key Vault URL. References resolve one hop, so fetched child values are terminal (you can't chain secret references across key vaults).
Supported vault DNS suffixes:
.vault.azure.net.vault.azure.cn.vault.usgovcloudapi.netStrict mode (
env_akv_strict) rejects malformed bootstrap entries and malformed Key Vault references. Non-strict mode warns and skips malformed references while preserving the previous value. Authentication, authorization, transport, missing-secret, and empty-value failures always raiseKeyVaultInitializationException.Key Vault clients use asynchronous retries with exponential backoff.
Debug
.envOutputSetting
env_akv_write_env: truewrites a fully resolved~/.pyrit/.envfor debugging.The generated file:
.env.env.localvalues${NAME}textIf
~/.pyrit/.envalready exists, initialization fails before fetching Key Vault secrets and instructs the user to rename or remove it.The generated file contains plaintext secrets and should be removed after debugging.
.env_example.env_examplenow:${NAME}aliases< >URL wrappers because python-dotenv preserves them literallyPublic consistency tests verify repository references, URL formatting, comment formatting, and clean-environment alias resolution.
Integration-Test Scope
This PR removes the live
env-newKey Vault schema test and its Azure DevOps environment wiring from previous iterations of the PR.Key Vault behavior is covered through mocked unit tests. Public integration coverage is limited to
.env_exampleand PyRIT consistency. Cross-store drift and operational alerting belong in a separate internal pipeline.The remaining legacy
env-globalpipeline write is now performed atomically with restrictive directory and file permissions.Validation
.env_exampleconsistency tests: 4 passedtytype checking: passed