Hermetic build actions
Fuchsia’s build system uses a tool to trace filesystem actions performed by build actions in order to detect that build actions correctly and fully state their inputs and outputs.
Continue reading this guide if you ran into an error that looks like this:
Unexpected file accesses building //some/target:label ...
(FileAccessType.READ /path/to/file/not/declared/as/input)
Or alternatively, if you’re looking at an action()
or action_foreach()
target that looks like this:
action("foo") {
...
hermetic_deps = false
}
Build graph correctness
The build is defined as a directed acyclic graph such that actions have their
inputs flowing into them and their outputs flowing from them. For instance, an
action that compiles a .cc
file into a .o
file will have the source file as
an input and an object file as an output. Any .h
headers used in compilation
are considered as inputs to the same action.
This graph representation ensures that the build system can correctly perform incremental builds. An incremental build is when a build was already performed, but then some of the actions’ inputs were changed, and now the build system is being asked to rebuild. In an incremental build, the build system will attempt to do the least amount of work needed, only rebuilding actions whose inputs have changed, whether due to modifications done by the user to sources or due to changes in the outputs of other actions that needed to be re-run.
For any action in the build graph, it’s required that all inputs and outputs be
stated in order for the build graph to be correct and for actions to be
hermetic. However, this is not validated by the underlying build system, Ninja.
Build actions run in the user’s local environment, with full access to the
entire filesystem, including all files in the source tree and in the out/
directory, so they’re not sandboxed and they can reach anywhere.
Failing to declare an input would result in failing to re-run an action (and everything downstream) when that input is updated. Failing to declare an output that is an input to another action produces a race condition between related actions, in which a single build invocation may miss a timestamp update, and manifest as a failure to converge in a single invocation (see Ninja no-op).
If you’re reading this, you’re probably dealing with a build action that did not fully state one or more of its inputs or outputs.
Extending the build with custom actions
Developers can use the GN metabuild system to define custom actions in their
BUILD.gn
files. This can be done with action
and
action_foreach
. Custom actions allow developers to invoke
custom tools at build time, and to hook them up to the dependency graph, such
that the tools can be invoked at build time and correctly re-invoked for
incremental builds when their inputs have changed.
Actions state their inputs using the following parameters:
script
: the tool to run. Often this is a Python script, but it can be any program that can be executed on the host.inputs
: files that are used as data inputs to the tool. For instance if the tool compresses a file, then the file to be compressed will be listed as an input.sources
: this is treated the same asinputs
. The difference is only semantic, assources
are typically used for additional files used by the tool’sscript
, e.g. dependent Python or script libraries.
Actions state their outputs using the following parameter:
outputs
: each action must produce at least one output file. Actions that don’t generate an output file, for instance actions that validate certain inputs for correctness, will typically generate a “stamp file”, which acts as an indicator that the action ran and can be empty.
Depfiles
If some of the inputs to an action are not known prior to running the action,
then additionally an action can specify a depfile
. Depfiles list
inputs to the action’s one or more outputs that were discovered at runtime. The
format of a depfile is one or more lines as follows:
[output_file1]: [input_file1] [input_file2...]
All paths in a depfile must be relative to root_build_dir
(which is set as the
current working directory for actions). See also:
prefer relative paths from rebase_path()
.
Tools like compilers should (and do) support emitting a trace of all of the files used in compilation in the form of a depfile.
Filesystem action tracing for detecting non-hermetic actions
The Fuchsia build system uses a filesystem action tracing tool to detect if
actions read or wrote files that were not listed as inputs or outputs, either
explicitly in the BUILD.gn
file or in a depfile, as shown above. This is done
in lieu of a sandbox for running actions, and as a runtime sanitizer of sorts.
If you are reading this page then you’re likely contending with an error from
this system. The error will have listed precisely which files were read or
written but were not specified as inputs/outputs in BUILD.gn
or in a depfile.
You should correct these omissions and attempt to rebuild until the error goes
away.
In order to reproduce this error in a local build, you will need to ensure that action tracing is enabled:
fx set what --args=build_should_trace_actions=true
or interactively, run fx args
, add a line build_should_trace_actions=true
,
save and exit.
Note that if your action is not defined hermetically, and you haven’t corrected it, then upon attempting to rebuild the action you may not be encountering an error. Because the action is not defined hermetically, it may not be correctly picked up in an incremental build (which is part of the problem that you’re trying to solve). To force all build actions to run, you’ll need to clean up your build’s output cache first:
fx clean
By default, CQ performs these hermeticity checks on all changes. It does so
using the build_should_trace_actions=true
argument mentioned above, so
developers can reproduce the exact same traced builds locally.
Suppressing hermetic action checks
Actions that are currently not hermetic have the following parameter set:
action("foo") {
...
# TODO(fxbug.dev/xxxxx): delete the line below and fix this
hermetic_deps = false
}
This suppresses the check that’s described above. If you spot an action that has this suppression, you should remove the suppression, attempt to reproduce the issue as outlined above, and fix it.
If instead of fixing it right away, you file a bug, title the bug with “[hermetic]” and include the output of tracing from the failed build action in the description. Comment about the access violation if you know where it is coming from.
Common issues and how to fix them
Missing inputs/outputs
Sometimes an input/output is well-known at build time but just isn’t specified, or it’s specified incorrectly. These are common and straightforward to fix. For instance:
Inputs not known until action runtime
As explained above, sometimes not all inputs are known at build time and so
cannot be specified in BUILD.gn
definitions. This is what depfiles
are for.
You can find an example for fixing a build action to generate a depfile here:
- 472565: [build] Generate depfile in generate_fidl_json.py
- 472657: [build] Fix hermeticity of hotsort_target_internal
- 473980: [build] Fix hermeticity of fidl-c-header
- 472658: [build] Make go_library build hermetically
- 472637: [build] Fix hermeticity of flatbuffer
Action arguments missing from inputs/outputs
Build actions are often scripts that take certain file paths as arguments.
action("foo") {
script = "concatenate.py"
outputs = [ "$target_out_dir/file1_file2.txt" ]
args = [
"--concat-from",
rebase_path("data/file1.txt", root_build_dir),
rebase_path("data/file2.txt", root_build_dir),
"--output",
] + outputs
}
In the above case you’ll get an action tracer error that concatenate.py
read from data/file1.txt
and data/file2.txt
. The mistake is easy to spot,
because you can see that these paths are passed as args to the script but are
not listed as inputs or outputs. While it’s technically possible to pass paths
as args and not actually have the script read/write to those paths, it’s very
unlikely.
The fix is as follows:
action("foo") {
script = "concatenate.py"
sources = [
"data/file1.txt",
"data/file2.txt",
]
outputs = [ "$target_out_dir/file1_file2.txt" ]
args = [
"--concat-from",
] + rebase_path(sources, root_build_dir) + [
"--output",
] + outputs
}
Expanding arguments from a file
There is a common pattern used especially in Python scripts to expand the
contents of a file as arguments. In BUILD.gn
you will find:
action("foo") {
script = "myaction.py"
args = [ "@" + rebase_path(args_file, root_build_dir) ]
...
}
Then in the associated Python file myaction.py
you will find:
def main():
parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
args = parser.parse_args()
...
The problem with the above is that args_file
is read at runtime by the Python
script, and should be specified as an input. To fix:
action("foo") {
script = "myaction.py"
inputs = [ args_file ]
args = [ "@" + rebase_path(args_file, root_build_dir) ]
...
}
See also: hermetic actions in open projects