feat: Configure dvsim and FuseSoC from a config file outside the repository (or the command line)

[andreaskurth]

Motivation

This enables DVSim users to write configuration files outside the tree of the repository using DVSim (e.g., OpenTitan), in which they can define FuseSoC mappings, additional 'cores' search paths, and the project root to operate on. Example dvsim.hjson:

// dvsim configuration for this workspace.
//
// This is dvsim's own configuration, not a flow config: it is discovered by
// walking up from the working directory, so running dvsim from inside
// opentitan/ picks it up from here.  Relative paths below resolve against this
// file's directory.
{
  // The repository dvsim operates on.
  proj_root: "opentitan"

  fusesoc: {
    // Build against partner:prim_mytech instead of lowrisc:prim_generic.
    mapping: ["lowrisc:prim_generic:all:0.1=partner:prim_mytech:all:0.1"]

    // prim_mytech lives outside the OpenTitan tree, so FuseSoC needs to be
    // told where to find it.
    extra_cores_root: ["prim_mytech"]
  }
}

The reason this cannot be done with the flow configs today is that they hardcode the FuseSoC arguments they pass, including the --mapping that selects which technology library a design is built against. Building an existing config tree against a different library therefore meant editing those configs. The hjson overrides: key cannot do it either: a primary config loads its children before processing its own overrides, so an override written in a wrapper config never reaches them.

Setting proj_root in the same file completes the picture: with it, dvsim can be invoked from the workspace that holds the configuration, rather than only from inside the repository it builds.

Usage

With this PR, one can place a dvsim.hjson config file like the example above in a local workspace directory outside/above the repository using DVSim (e.g., OpenTitan), and then simply run

dvsim <path/to/cfg.hjson> [...]

either in the repository or in the workspace directory above it, to simulate against prim_mytech instead of prim_generic.

The config file is located as follows:

1. --dvsim-config FILE, if given;
2. otherwise the nearest dvsim.hjson, walking up from the working directory;
3. otherwise $XDG_CONFIG_HOME/lowRISC/dvsim/dvsim.hjson.

Walking upwards is what makes the workspace layout above work: dvsim is normally invoked from inside the repository it builds, while a file describing the workspace sits alongside that repository rather than inside it.

Paths are resolved so that both invocation styles mean the same thing. A relative proj_root or extra_cores_root in the config file is relative to the directory holding that file, while a relative --proj-root on the command line is relative to the working directory. Either may be given as an absolute path. proj_root is taken from --proj-root if present, otherwise from the config file, otherwise by searching for a git repository containing the working directory, as before.

The path of the config file in use is logged alongside proj_root, so it is clear which one was picked up:

[dvsim_config]: /home/me/workspace/dvsim.hjson
[proj_root]:    /home/me/workspace/opentitan

The FuseSoC settings are also available on the command line, which is useful for one-off runs:

dvsim <path/to/cfg.hjson> \
    --fusesoc-mapping lowrisc:prim_generic:all:0.1=partner:prim_mytech:all:0.1 \
    --fusesoc-extra-cores-root /path/to/prim_mytech

Both options are repeatable, and both apply to every config in a run, including all the blocks of a primary config. Values from the file are applied first, and command-line values are appended to them.

Implementation

Three commits, all feat:

feat: add a configuration file for dvsim itself adds dvsim.config, which locates and parses the file. Each feature owns a section of it, so the module itself knows nothing about FuseSoC. Unknown keys within a section are rejected, so a typo fails at the point it is made rather than silently doing nothing. Relative paths are resolved against the config file's own directory, which is what lets a checked-in file stay valid wherever dvsim is invoked from.

feat: select the FuseSoC mapping and cores-root from dvsim adds dvsim.fusesoc, the two --fusesoc-* options and the --dvsim-config option that selects the file. Three points are worth calling out for review:

• Where the rewrite happens. It runs at the end of FlowCfg._expand(). By then the option lists hold literal arguments rather than wildcards, so the --mapping and --cores-root tokens are visible whichever config key they came from. It also has to happen inside _expand() rather than after it, because subclasses call _create_objects() from their own _expand() and the build modes created there take a copy of these lists. Rewriting afterwards updates the config attribute but has no effect on the command that actually runs.
• Why command-line arguments rather than the overrides: key. Every child of a primary config is constructed with the same args object, so an args-driven rewrite reaches all of them, which is exactly what overrides: cannot do.
• Argument placement and de-duplication. Only lists whose command is FuseSoC are touched, so the options are inert for flows driven by something else. --cores-root is a FuseSoC global option and is inserted before the run subcommand, while --mapping belongs to run and is inserted after it. A mapping is given as [OLD=]NEW: the OLD= form replaces an existing --mapping=OLD, which is needed because configs that already pin a library would otherwise end up with two mappings covering the same source, which FuseSoC rejects. Repeated mappings are dropped for the same reason: configs build these lists by appending, so the same mapping can legitimately appear twice before rewriting.

feat: set proj_root from the dvsim config file adds the proj_root key and the resolution rules described under Usage, so that dvsim can be invoked from outside the repository. Two supporting changes come with it:

• The branch name is now read from the git repository at proj_root rather than from the working directory, which has no repository to read when dvsim is invoked from outside the project. Previously that printed fatal: not a git repository and fell back to a branch of default, silently dropping the branch from every scratch path.
• The config file is loaded once, in main(), rather than separately by each feature that reads a section of it. That is what makes it possible to log its path exactly once, and it is also where unknown top-level keys are rejected.

Resolving these paths to absolute ones is not cosmetic: the value is handed on to flows, and from there to tools such as FuseSoC, which run with a different working directory. A relative --proj-root previously reached FuseSoC unresolved and produced Failed to register library followed by an unresolvable core.

Testing

tests/test_config.py covers discovery order, the upward walk, relative-path resolution for both proj_root and section values, the rejection of malformed input, and the rejection of unknown top-level keys. tests/test_fusesoc.py covers mapping parsing, replacement, appending, argument placement, de-duplication, inertness for non-FuseSoC commands, config/CLI precedence, and the _expand() ordering described above. 392 tests pass, and ruff format --check, ruff check --config ruff-ci.toml and the license header check are clean.

Beyond the unit tests, this was exercised end to end against OpenTitan, in all three invocation styles: from inside the repository, from the workspace directory above it with proj_root taken from the config file, and with a relative --proj-root overriding the config file. A full 41-job block-level lint regression and a chip-level simulation build, both against an out-of-tree technology library, produced results identical to the in-tree baseline.

Checklist

• All commits are signed off (git commit -s), indicating acceptance of the CLA
• Commit messages follow the conventional commit format (<type>[(<scope>)][!]: <description>)
  • The commit type correctly reflects the semver impact of the change
  • Breaking changes are marked with ! or a BREAKING CHANGE: footer
• New behaviour is covered by tests

[gautschimi]

The way I understand the dvsim.hjson selection:

example setup:

folder_a/                     (no dvsim.hjson)
└── folder_b/
    ├── dvsim.hjson
    └── folder_c/      (no dvsim.hjson)

It depends which hjson is selected based on the invocation directory:

From folder_a → does not find anything, walks up towards the root and falls back to $XDG_CONFIG_HOME/lowRISC/dvsim/config.hjson
From folder_b → uses the dvsim.hjson from folder_b
From folder_c → no file here, walks up → finds folder_b's dvsim.hjson

Is this the indented behavior? I think it would be better if we walk down into a directory.

[andreaskurth]

Thanks @gautschimi, yes this is by design. It enables the following use case:

product_workspace/
├── dvsim.hjson
└── opentitan/      (no dvsim.hjson)

You can then invoke dvsim in the opentitan folder as you normally would, and the dvsim.hjson from the outside product_workspace applies. So you don't have to modify opentitan and can still work with product specific IPs and FuseSoC mappings.

[gautschimi]

Can we invoke dvsim from outside the project root directory? If yes, don't we also have to provide the proj_root. e.g. in this dvsim.hjson file

[gautschimi]

Thanks @gautschimi, yes this is by design. It enables the following use case:

product_workspace/
├── dvsim.hjson
└── opentitan/      (no dvsim.hjson)

You can then invoke dvsim in the opentitan folder as you normally would, and the dvsim.hjson from the outside product_workspace applies. So you don't have to modify opentitan and can still work with product specific IPs and FuseSoC mappings.

Ok but this would also be possible if you walked into the directory. In your case invoking dvsim from product_workspace and from opentitan would use the same dvsim.hjson. By walking into the directory, a dvsim invokation from the opentitan folder would fall back to the default.

[andreaskurth]

Ok but this would also be possible if you walked into the directory. In your case invoking dvsim from product_workspace and from opentitan would use the same dvsim.hjson. By walking into the directory, a dvsim invokation from the opentitan folder would fall back to the default.

I think that would unfortunately require bigger/breaking changes to DVSim, because the relative paths in the DVSim config files (e.g., in the OT repo) are relative to the directory in which you execute DVSim. You can reproduce that by trying to run dvsim opentitan/hw/top_earlgrey/dv/chip_sim_cfg.hjson -i smoke from a product_workspace directory outside opentitan. DVSim can't run that because it gets botched file paths.

I suggest this is something we consider for a next major version.

[andreaskurth]

@gautschimi As we just discussed offline, it would be even better if dvsim.hjson allowed setting proj_root, so that dvsim can be invoked directly in product_workspace. I'll look into that. The discovery flow for dvsim.hjson should be kept at "current directory or above".

[gautschimi]

Could we also add a log message that shows which dvsim.hjson was used? There is already a print for proj_root and scratch_path.

e.g.

[I 260825 09:32:47 run:990] [proj_root]: /home/mgautschi/opentitan
[I 260825 09:32:47 run:990] [dvsim_config]: /home/mgautschi/dvsim.hjson
[I 260825 09:32:47 flow:229] [scratch_path]: [chip] [/home/mgautschi/opentitan/scratch/rram_macro_virtual/chip_earlgrey_asic-sim-vcs]
[I 260825 09:35:41 report:489] [results]: [chip]

[andreaskurth]

@gautschimi I pushed a new commit implementing your two feature requests (being able to set proj_root in dvsim.hjson and printing dvsim_config in the log). Note that I chose to make relative path values for proj_root in dvsim.hjson relative to the directory containing dvsim.hjson.

[hcallahan-lowrisc]

This LGTM. It's slightly unfortunate we have to have fusesoc-specific CLI args, given that in theory this should be a run tool that is generic over the underlying utilities it can call, but I think it's acceptable to incorporate this change now and work on migrating back towards being fully generic as a future task.

I have two suggestions, neither of which should block merging as-is.

1. Along with @gautschimi's suggestion of logging when we have resolved options from a config file, it would be good to log at the INFO level when we are substituting values over the top of in-tree values. E.g.

   [I 260825 14:03:09 fusesoc:275] FuseSoC --mapping override in uart_lint build_opts [/proj/hw/ip/uart/dv/uart_sim_cfg.hjson]: lowrisc:prim_generic:all:0.1 -> lowrisc:prim_my_tech:all:0.1
   

   One way to view the external config file is as a source of build impurity, which is entirely the point of this change, but can also contribute to making it harder to understand what configuration changes have been applied. In my memory, commands added to .bazelrc has been the source of confusion in the past, forgetting they had been added for a previous debugging journey. Printing every time an override is applied makes it more difficult to forget this mechanism is being exercised.

2. IMO the user default config file in XDG_CONFIG_HOME should also be named dvsim.hjson for consistency with the out-of-tree file. A minor subjective nit.

In testing I made some local changes working towards 1) so I'll submit that for a future patch release.

[andreaskurth]

@hcallahan-lowrisc, thanks for your feeback! I'll implement it.

Just to ensure I understand you correctly: Would you be OK with $XDG_CONFIG_HOME/lowRISC/dvsim.hjson as the user default config file path?

[hcallahan-lowrisc]

@hcallahan-lowrisc, thanks for your feeback! I'll implement it.

Just to ensure I understand you correctly: Would you be OK with $XDG_CONFIG_HOME/lowRISC/dvsim.hjson as the user default config file path?

Great. Yes, that is fine, or $XDG_CONFIG_HOME/lowRISC/dvsim/dvsim.hjson would be fine as well.

[andreaskurth]

@hcallahan-lowrisc updated, PTAL

[hcallahan-lowrisc]

Looks great, just ran the tests and I can see the logs being emitted. Thanks!