Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

About

Engage is a process composer with ordering and parallelism based on directed acyclic graphs.

Given a file that configures a set of processes, including but not limited to dependencies between each process, the engage command line program can run all or a subset of the processes ordered by their dependencies, list available processes, or be used to generate a visualization of the dependency graph. Engage does not make use of any process isolation features and runs all processes as the current user, as its purpose is simply to automate running a set of commands in a specific order and in parallel where possible. This design puts relatively few moving parts between your intent and the actual behavior, which makes debugging simpler and reduces the odds of an intermediate part being misconfigured or broken.

Code of conduct

Community standards

Community members must have good vibes. Having bad vibes will result in a warning, temporary removal, or permanent removal from the community, depending on intensity. Behavior both inside and outside of community spaces is taken into consideration when evaluating vibes. Here are some examples of good and bad vibes:

Good vibes
Respecting others regardless of such characteristics as gender, sexual orientation, race, and neurodivergence.
Maintaining a good signal to noise ratio.
Understanding that there is no such thing as “apolitical”.
Bad vibes
Rules lawyering.
Lacking ethics.
Supporting fascism.

Generative AI

Those who interact with the project must acknowledge the following and act accordingly:

  • Collecting training data for generative AI models without consent is unethical.
  • Leveraging exploitative working conditions to label training data for generative AI models is unethical.
  • Using, promoting, or otherwise legitimizing generative AI models that were produced unethically or are operated unethically is unethical.
  • Generative AI models are unfit for purposes where correctness is a necessity.

Additionally, the contributors do not consent to the use of this project (including its source code, documentation, version control history, official discussion spaces, and so on), in part or in whole, as inference inputs to or as training data for generative AI models. There are many ways to describe those who violate the consent of others, none of which are pleasant.

Failure to abide by these precepts constitutes bad vibes, which will be handled as described in the Community standards section.

Tutorial

This section demonstrates common features of Engage. After understanding this section, the reference-style documentation (e.g. command line help messages and the File format chapter) should be sufficient for learning the other available features.

Choosing the Engage file

By default, Engage will search for a file called engage.toml in the current directory or all of its parent directories. Alternatively, a file can be specified via a command line option. Henceforth, the phrase “Engage file” means the file chosen by either of those strategies.

Adding processes

An Engage file is pointless without any processes, so let’s add a few:

[processes.create-foo]
command = ["touch", "foo"]

[processes.create-bar]
command = ["touch", "bar"]

Each [processes.<name>] table defines a process, where <name> is a placeholder for the actual name of the process. command is the only required field for each process.

Process names are useful for identifying which part of the Engage file is producing what output, visualizing the dependency graph, and selecting a subset of the processes to run at the command line.

When Engage runs these processes, the empty files foo and bar will be created (or their atime and mtime will be updated if they already exist) in the same directory as the Engage file, not in the working directory the engage command was run from (although these may be the same directory).

Adding processes with dependencies

Let’s say we want to clean up after ourselves by deleting these files before exiting:

[processes.delete-foo]
command = ["rm", "foo"]
after = ["create-foo"]

[processes.delete-bar]
command = ["rm", "bar"]
after = ["create-bar"]

Note the use of after for both of these processes; this is required to ensure that they run after, rather than the default of running in parallel with, the create-foo and create-bar processes.

Now let’s say we want to do something between creating and deleting these files; for example, we’ll just call stat on both of them:

[processes.stat-both]
command = ["stat", "foo", "bar"]
after = ["create-foo", "create-bar"]
before = ["delete-foo", "delete-bar"]

Note the use of after and before; this is what achieves the desired “between” semantics. The advantage of having and using both after and before rather than only one or the other is that this allows better organization of ordering constraints. For example, if we wanted to add or remove processes that run between creation and deletion and only had or used after, the deletion processes would have to be updated for each of those changes, whereas this way, only the processes being added or removed need to be modified.

Putting it all together

Incorporating all the changes from the previous three sections results in this complete Engage file:

[processes.create-foo]
command = ["touch", "foo"]

[processes.create-bar]
command = ["touch", "bar"]

[processes.delete-foo]
command = ["rm", "foo"]
after = ["create-foo"]

[processes.delete-bar]
command = ["rm", "bar"]
after = ["create-bar"]

[processes.stat-both]
command = ["stat", "foo", "bar"]
after = ["create-foo", "create-bar"]
before = ["delete-foo", "delete-bar"]

Visualizing process dependencies

The engage dot subcommand can be used to convert an Engage file into Graphviz DOT Language, which can then be processed by other tools. For example, it can be used to produce a graph like this from the above Engage file:

Graph of the tutorial Engage file

This illustrates the order in which Engage will run each process, what processes can be run in parallel with each other, and which of before and after created each dependency edge. This can be useful for debugging dependency cycles or unexpected ordering between processes in general.

In this graph, create-foo and create-bar will run in parallel, then stat-both will run by itself, and finally delete-foo and delete-bar will run in parallel. You may also notice that there are “redundant” ordering requirements (edges) in this graph, for example, create-foo -> delete-foo is redundant with create-foo -> stat-both -> delete-foo. While the presence of create-foo -> delete-foo has no effect on ordering, it is useful to retain it so that if stat-both (and thus its ordering requirements) are removed in the future, the intent of running delete-foo after create-foo is preserved.

Running processes

The engage command, when run with no subcommand, will run all processes in the selected Engage file. While doing so, Engage will forward stdout and stderr of the processes, alongside an indication of which process is generating the output, whether the output is coming from the process’ stdout or stderr (marked with an O or E, respectively), and some extra fluff to make it look pretty. At the end, Engage will print out whether the run succeeded or failed and exit with an appropriate status code:

Status codeMeaning
0All processes exited successfully.
1At least one process exited with an error status code.
2Other errors, such as issues with the Engage file.

Note that Engage’s own stdout and stderr output is not considered stable.

It’s also possible to use the engage just subcommand to run a subset of the processes in an Engage file.

If Engage receives the SIGINT signal (e.g. via ctrl+c) while running processes, it will prevent any more processes from spawning, send SIGINT to any running processes, and wait for them to exit before exiting.

File format

Engage files are in the TOML v1.1.0 format.

The following sections are named after the “path” to the key they document. Paths are formed by an initial . followed by zero or more key names separated by ., where subsequent keys belong to the table named by the previous key. The key name * is a placeholder which indicates that there may be zero or more keys in its place and that you are supposed to choose the names of those keys. Key names suffixed with [] indicate that the key’s value is an array of tables and the key name after the following ., if any, belongs to each table within the array.

The use of key names not documented here (aside from keys whose names you are supposed to choose) is forbidden and will cause Engage emit an error and exit with code 2.

.processes

Type
Table of tables.
Applicability
Optional.
Description
This table defines the set of processes. The keys in this table define the name of each process, which must match ^[a-z0-9-]+$ and cannot start with -. Each key’s value defines the configuration for that process.
Examples
Empty file
This file defines no processes.
Table without keys
[processes]
This file defines no processes.
One process
[processes.hello-world]
command = ["echo", "Hello, world!"]
This file defines one process named hello-world that prints Hello, world! to stdout and exits successfully.

.processes.*.after

Type
Array of strings.
Applicability
Optional.
Description
Names of processes that must exit successfully before this process can be spawned. Each process name may appear more than once, though this has no additional effect.
Examples
One dependency
[processes.first]
command = ["echo", "Hello"]

[processes.second]
command = ["echo", "Goodbye"]
after = ["first"]
This file defines the two processes first and second, where first is spawned first and second is only spawned after first exits successfully. Hello will be printed to stdout by first which will then exit successfully, followed by Goodbye being printed to stdout by second which will then exit successfully.

.processes.*.before

Type
Array of strings.
Applicability
Optional.
Description
Names of processes that must only be spawned after this process has exited successfully. Each process name may appear more than once, though this has no additional effect.
Examples
One dependency
[processes.first]
command = ["echo", "Hello"]
before = ["second"]

[processes.second]
command = ["echo", "Goodbye"]
This file defines the two processes first and second, where first is spawned first and second is only spawned after first exits successfully. Hello will be printed to stdout by first which will then exit successfully, followed by Goodbye being printed to stdout by second which will then exit successfully.

.processes.*.command

Type
Array of strings.
Applicability
Required.
Description
The command used to spawn this process after all of its dependencies have exited successfully. The array must have at least one value. The first value determines both the program to spawn a process for as well as the first argument to that process.
Examples
Program with no additional arguments
[processes.minimal]
command = ["true"]
This file defines one process named minimal that prints nothing and exits successfully.
Program with additional arguments
[processes.hello-world]
command = ["echo", "Hello, world!"]
This file defines one process named hello-world that prints Hello, world! to stdout and exits successfully.

.processes.*.environment

Type
Table of strings.
Applicability
Optional.
Description
Environment variables to set for this process. Each key-value pair in this table defines the name of an environment variable and its value respectively. Environment variables not defined in this table are left unset or are inherited normally.
Examples
Set one environment variable
[processes.hello-world]
command = ["bash", "-c", "echo Hello, $NAME\!"]
environment.NAME = "world"
This file defines one process named hello-world that prints Hello, world! to stdout by reading part of the output from the configured environment variable and exits successfully.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to SemVer.

Unreleased

Removed

  1. BREAKING: Remove the concept of groups. (!10)
  2. BREAKING: Remove the .task[].ignore key. It is seldom useful and can be implemented outside of Engage when it’s truly necessary. (!13)
  3. BREAKING: Remove the .interpreter key as it is no longer necessary. (!15)
  4. BREAKING: Remove support for non-Unix platforms. Non-Unix platforms may become supported in the future. (!29)
  5. BREAKING: Remove the -j/--jobs command line option. (!33)

Changed

  1. BREAKING: Replace the .task[] and .task[].name keys with the .processes and .processes.* keys respectively. “Tasks” are now called “processes”, because that’s what they ultimately are. (!11, !37)
  2. BREAKING: Rename the .processes.*.depends key to .processes.*.after. (!11)
  3. BREAKING: Replace the .processes.*.script key with the .processes.*.command key, which takes an array of strings rather than a string. (!15)
  4. BREAKING: Only permit ^[a-z0-9-]+$ in process names, and - cannot appear as the first character. (!18)
  5. BREAKING: Reject unknown keys in Engage files. (!19)
  6. Deduplicate elements in the array of the .processes.*.after key. Duplicates are not rejected, but they are ignored while constructing the dependency graph, so e.g. they will no longer show up in engage dot. (!11)
  7. Improve error messages when attempting to run processes. (!21)
  8. Improve the formatting of errors. The new format is much more likely to work well with screen readers, and can include more information than just the error message, such as suggestions for resolving the error. (!21)
  9. Spawn each process in its own process group. Primarily, this prevents them from receiving SIGINT directly from the shell when ctrl+c is pressed while running Engage, because shells typically send SIGINT to the entire process group rather than just the first process spawned. Engage already manages the forwarding and sending of signals to processes it spawns. (!33)
  10. Improve some error messages. (!33, !37)
  11. Stop canonicalizing the path to the Engage file. Paths involving symlinks will behave more predictably. (!33)
  12. Support the TOML specification version 1.1.0. (!36)
  13. Improve the “File format” chapter of the book by flattening headings, using definition lists, and adding examples. (!38)

Fixed

  1. No longer block the spawning of processes on other processes that they do not declare a direct or transitive dependency on. (#9, !24, !26)

Added

  1. Add the .processes.*.before key, which requires that the process in question exit successfully before spawning processes in the array of process names given to this key. Elements are deduplicated in the same way as the .processes.*.after key. (!12)
  2. Add the .processes.*.environment key, which allows configuring environment variables on a per-process basis. (!14, !16)
  3. Add the -l/--log-format command line option for choosing alternate log formats. (!27, !28)
  4. Add a ctrl+c/SIGINT handler which prevents new processes from spawning, signals running processes to exit, and waits for them to exit. (!29, !33)

v0.2.1 - 2025-09-08

Changed

  1. Greatly improve error messages in some cases, primarily task failures and dependency cycles. (!4)

Fixed

  1. Only load the Engage file when necessary. (2f394ad)

Added

  1. Add more thorough documentation in the form of a book. (Too many commits to link.)
  2. The long help CLI output links to a local copy of the book, if packaged to do so. (!2)

v0.2.0 - 2023-09-19

Changed

  1. BREAKING: Always exit with a status of 1 when a task fails instead of trying to exit with the same status code as the task. (c18357e)
  2. BREAKING: Exit with a status of 2 for errors other than tasks failing. (a991d12)
  3. BREAKING: Move subcommands from under self to the top level and remove the self subcommand. For example, you’d now use engage dot instead of engage self dot. (e537e9d)
  4. Remove all restrictions on group names. (0850054)

Fixed

  1. Report all failed tasks instead of just the first one. (b9fcdcd)

v0.1.3 - 2023-09-10

Removed

  1. Remove the Nix binary cache. (066e97c)

Fixed

  1. Wait for all started tasks to complete before terminating. (d98c6cc)

v0.1.2 - 2023-02-10

Changed

  1. Improve the behavior summary. (7be43e4)

Added

  1. Add a --jobs/-j option to limit parallelism. (da1cc61)

v0.1.1 - 2023-01-28

Added

  1. Add a subcommand to generate shell completions. (48c925b)

v0.1.0 - 2022-11-07

Changed

  1. BREAKING: Disallow certain group names for forward compatibility. (066b2e3)
  2. BREAKING: Make engage just <GROUP> [TASK] run all dependencies. (acda394)
  3. Allow engage self dot to take <GROUP> [TASK] arguments to show a subgraph for the given target. (acda394)
  4. Make error messages a little prettier. (257e49e)
  5. Allow engage self dot even if there are cycles. (f6ffbc2)
  6. Print out the status at the end to make it easier to see. (37a6b9f)
  7. Print out the group and name of the failing task, if any. (c3df8f6)
  8. Print errors to stderr instead of stdout. (9c5dfaf)
  9. Rework output while running the graph to be more functional and accessible. (8880003)

Fixed

  1. BREAKING: Prevent groups from depending on nonexistent groups. (69219ec)
  2. Fix a deadlock in an unusual situation. (acee5fe)
  3. Prevent a deadlock in an unusual situation. (295e3b6)
  4. Improve the graph in unusual self-loop situation. (a9f1a27)
  5. Improve UX when interpreter is given an empty list. (72e89a3)
  6. Improve UX of some error messages. (c663bb6)
  7. Greatly improve error messages for dependency cycle issues. (f6ffbc2)

Added

  1. Upload build artifacts to Computer Surgery Nix binary cache. (8773d4d)

v0.1.0-alpha.2 - 2022-10-09

Changed

  1. BREAKING: Rename task.cmd to task.script in engage.toml. (b403dbe)
  2. Change CLI UX due to a dependency update. (b0f7ac7)

Fixed

  1. Reset output style at the beginning of each line. (!1)

v0.1.0-alpha.1 - 2022-09-17

Initial release.

Contributing

  1. Install Lix, direnv, and nix-direnv.

  2. Enable Lix’s nix-command experimental feature.

  3. If using a graphical editor, ensure it has direnv support, e.g. by installing an extension. If no support is available, changing directories into the project and launching the editor from the terminal should cause it to inherit the environment; though the editor will likely need to be restarted to propagate any changes to the direnv setup to the editor if any such changes are made.