Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

About

Engage is a process composer with ordering and parallelism based on directed acyclic graphs.

In other words, given a collection of process definitions which can include ordering dependencies, Engage can run those processes in the order defined, while also running them in parallel to the extent permitted by the ordering. Engage supports both “task” processes, which allow dependents of a task to spawn after the task exits, and “service” processes, which allow dependents of a service to spawn after the service spawns but before it exits, and normally has the dependents exit before the service does.

Some related functionality is also provided, such as the ability to run a subset of the processes rather than all of them and the ability to generate a visualization of processes and the dependencies between them.

Notably, Engage does not make use of any process isolation features and it runs all processes as the current user. This deliberate design decision decreases moving parts between intented behavior and the collective actual behavior of Engage and the processes it runs. As a result, debugging is simpler and the odds of an intermediate part being misconfigured or broken are lower.

Code of conduct

Community standards

Community members must have good vibes. Having bad vibes will result in a warning, temporary removal, or permanent removal from the community, depending on intensity. Behavior both inside and outside of community spaces is taken into consideration when evaluating vibes. Here are some examples of good and bad vibes:

Good vibes
Respecting others regardless of such characteristics as gender, sexual orientation, race, and neurodivergence.
Maintaining a good signal to noise ratio.
Understanding that there is no such thing as “apolitical”.
Bad vibes
Rules lawyering.
Lacking ethics.
Supporting fascism.

Generative AI

Those who interact with the project must acknowledge the following and act accordingly:

  • Collecting training data for generative AI models without consent is unethical.
  • Leveraging exploitative working conditions to label training data for generative AI models is unethical.
  • Using, promoting, or otherwise legitimizing generative AI models that were produced unethically or are operated unethically is unethical.
  • Generative AI models are unfit for purposes where correctness is a necessity.

Additionally, the contributors do not consent to the use of this project (including its source code, documentation, version control history, official discussion spaces, and so on), in part or in whole, as inference inputs to or as training data for generative AI models. There are many ways to describe those who violate the consent of others, none of which are pleasant.

Failure to abide by these precepts constitutes bad vibes, which will be handled as described in the Community standards section.

Introduction

This page introduces the most important concepts in Engage, but does not cover everything; for the remainder, see the rest of the pages in the “For users” category and the command line help text.

Selecting the Engage file

Engage will search for a file named engage.toml in the current directory and all of its parent directories, and it will select the one closest to the current directory, or exit with an error if it can’t find one. Alternatively, the path to a file can be specified on the command line. The term “Engage file” refers to the file selected in either of these ways.

Process basics

Processes are Engage’s unit of work. An Engage file can contain zero or more process definitions, even though zero is an unhelpful number of processes. Each process defines a command that can be spawned into an OS process which in turn does the aforementioned work. When the OS process is spawned, its working directory will be set to the parent directory of the Engage file the process is defined in, rather than inheriting the working directory from Engage’s own OS process.

Process dependencies

Processes can depend on one or more other processes. The primary reason to define dependencies between processes is to control the order in which those processes spawn and exit. A process x can be made to depend on another process y either by configuring x to be ordered after y, by configuring y to be ordered before x, or both. Ordering, and thus dependencies, are configured per-process with the .processes.*.before and .processes.*.after keys.

It is safe, and often desirable, to have direct or indirect dependencies that appear to be “redundant”. A redundant dependency is one whose addition or removal would have no effect on the order in which processes spawn and exit. A major reason redundant dependencies are desirable is to reduce the likelihood of accidental ordering changes as processes are added, removed, and edited over time.

However, Engage will exit with an error before spawning any processes if it would have to spawn one or more processes that, directly or indirectly, depend on themselves (this is often called a “dependency cycle”).

Engage will run processes in parallel to the extent permitted by their dependencies. For example:

  • Three processes with no dependencies will be run in parallel with each other.
  • Given the processes x, y, and z where z depends on y, y will run before z, and x will run in parallel with y and z.
  • Given the processes x, y, and z where z depends on x and y, x and y will run in parallel, and z will run after x and y.

This can be a significant performance improvement over running all processes in serial.

Process readiness

Processes have a concept called “readiness” which refers to the point in time at which a process permits other processes that depend on it to be spawned. A process “becomes ready” when it reaches that point in time. The point in time is configured per-process with the .processes.*.ready-when key.

Based on this concept, processes are split into two kinds: “tasks” and “services”. A task is a process that can become ready by exiting successfully. A service is a process that can become ready after it spawns but before it exits. Another major difference between tasks and services is the order in which interdependent processes of each kind can spawn and exit. The behavior of each kind is explained in the following two sections.

Tasks

A task is a process that can become ready by exiting successfully.

Processes that depend on tasks will only be spawned after those tasks have exited successfully.

A task that fails to spawn or exits unsuccessfully will prevent any processes that depend on it from spawning.

For example, given the tasks x, y, and z where z depends on y and y depends on x, the following will happen when these processes are run:

  1. x spawns, because it has no dependencies.
  2. x exits successfully, thus becoming ready.
  3. y spawns, because its dependencies are ready.
  4. y exits successfully, thus becoming ready.
  5. z spawns, because its dependencies are ready.
  6. z exits successfully, thus becoming ready, but its readiness isn’t relevant here.

Services

A service is a process that can become ready after it spawns but before it exits.

Processes that depend on services will normally exit before those services exit; consequently, this means that interdependent services will normally exit in the reverse order from which they spawned. Abnormally, a service may exit before processes depending on it exit if the service decides to exit early of its own accord; for example, as a result of a misconfiguration or other runtime error.

A service that fails to spawn will prevent any processes that depend on it from spawning.

For example, given the services x and y and the task z where z depends on y and y depends on x, the following will happen when these processes are run:

  1. x spawns, because it has no dependencies, thus becoming ready.
  2. y spawns, because its dependencies are ready, thus becoming ready.
  3. z spawns, because its dependencies are ready.
  4. z exits successfully, thus becoming ready, but its readiness isn’t relevant here.
  5. y exits successfully.
  6. x exits successfully.

Beginning a run

The engage command (without any subcommand) will attempt to spawn all processes in the selected Engage file, starting with processes without dependencies and spawning subsequent processes as their dependencies become ready. If specified, the -p/--process option will cause Engage to only attempt to spawn the selected processes and their dependencies.

During a run

While running processes, Engage will forward stdout and stderr of those processes to Engage’s stdout, alongside an indication of which process emitted which output and whether the output came from the process’ stdout or stderr. In the default log format, whether a process’ output came from its stdout or stderr is indicated by an O or an E in Engage’s stdout, respectively.

Ending a run

If any of the following happen while Engage is running:

  • Engage receives SIGINT.
  • Any process fails to spawn or exits unsuccessfully.
  • All processes without dependents are tasks, and they have exited successfully.

Then Engage will send SIGINT once to each running process that either has no dependents or whose dependents have already exited, wait for those processes to exit, then repeat these steps until all processes have exited. For the avoidance of doubt, this does preserve the behavior of services normally exiting in the reverse order from which they spawned.

Notably, if any process without dependents is a service and all processes run successfully, Engage will continue running until it receives SIGINT.

When no more processes are running, Engage will print out whether the run ended in success or failure and then exit with an appropriate exit status.

Engage will also stop running under various other conditions, including but not limited to:

  • The laptop it’s running on depletes its battery.
  • The desktop it’s running on has a power outage due to inclement weather.
  • The gaming desktop it’s running on catches on fire due to 12VHPWR.
  • The datacenter it’s running on catches on fire due to a water leak.
  • The planet it’s running on gets enveloped by its star(s).
  • The inevitable heat death of the universe.

Exit statuses

The following definition list enumerates the exit statuses emitted by Engage and their meanings.

0
The command completed successfully.
1
The command attempted to run processes and at least one process’ exit status indicates an error.
2
The command encountered other errors, such as issues with the Engage file or invalid command line arguments.

API stability

The following sections enumerate which parts of Engage may and may not experience breaking changes between SemVer-compatible versions.

Stable

  • The command line arguments’ syntax, structure, and semantics.
  • The file format’s syntax, structure, and semantics.
  • The semantics of existing exit statuses.

Unstable

  • The format of output written to stdout and stderr.

File format

Engage files are in the TOML v1.1.0 format.

The following sections are named after the “path” to the key they document. Paths are formed by an initial . followed by zero or more key names separated by ., where subsequent keys belong to the table named by the previous key. The key name * is a placeholder which indicates that there may be zero or more keys in its place and that you are supposed to choose the names of those keys. Key names suffixed with [] indicate that the key’s value is an array of tables and the key name after the following ., if any, belongs to each table within the array.

The use of key names not documented here (aside from keys whose names you are supposed to choose) is forbidden and will cause Engage exit with an error.

.processes

Type
Table of tables.
Applicability
Optional.
Description
Defines the set of processes. The keys in this table define the name of each process, which must match the regex ^[a-z0-9][a-z0-9-]*$. Each key’s value defines the configuration for that process.
Examples
Empty file

This Engage file defines no processes.

Table without keys
[processes]

This Engage file defines no processes.

One process
[processes.hello-world]
command = ["echo", "Hello, world!"]
ready-when = "exited"

This Engage file defines one process. The process:

  • Is named hello-world.
  • Defines a command that prints Hello, world! to its stdout and exits successfully.
  • Becomes ready when it exits successfully.

When running all processes in this Engage file, the following will happen:

  1. The hello-world process spawns, because it has no dependencies.
  2. The hello-world process prints Hello, world! to its stdout.
  3. The hello-world process exits successfully, thus becoming ready.

.processes.*.after

Type
Array of strings.
Applicability
Optional.
Description
Defines that this process must be spawned after processes named in this array become ready. Each process name may appear more than once, though this has no additional effect.
Examples
One dependency
[processes.first]
command = ["echo", "Hello"]
ready-when = "exited"

[processes.second]
command = ["echo", "Goodbye"]
ready-when = "exited"
after = ["first"]

This Engage file defines two processes. One process:

  • Is named first.
  • Defines a command that prints Hello to its stdout and exits successfully.
  • Becomes ready when it exits successfully.

The other process:

  • Is named second.
  • Defines a command that prints Goodbye to its stdout and exits successfully.
  • Becomes ready when it exits successfully.
  • Spawns after first becomes ready.

When running all processes in this Engage file, the following will happen:

  1. The first process spawns, because it has no dependencies.
  2. The first process prints Hello to its stdout.
  3. The first process exits successfully, thus becoming ready.
  4. The second process spawns, because its dependencies are ready.
  5. The second process prints Goodbye to its stdout.
  6. The second process exits successfully, thus becoming ready.

.processes.*.before

Type
Array of strings.
Applicability
Optional.
Description
Defines that this process must become ready before processes named in this array can be spawned. Each process name may appear more than once, though this has no additional effect.
Examples
One dependency
[processes.first]
command = ["echo", "Hello"]
ready-when = "exited"
before = ["second"]

[processes.second]
command = ["echo", "Goodbye"]
ready-when = "exited"

This Engage file defines two processes. One process:

  • Is named first.
  • Defines a command that prints Hello to its stdout and exits successfully.
  • Becomes ready when it exits successfully.
  • Becomes ready before second is spawned.

The other process:

  • Is named second.
  • Defines a command that prints Goodbye to its stdout and exits successfully.
  • Becomes ready when it exits successfully.

When running all processes in this Engage file, the following will happen:

  1. The first process spawns, because it has no dependencies.
  2. The first process prints Hello to its stdout.
  3. The first process exits successfully, thus becoming ready.
  4. The second process spawns, because its dependencies are ready.
  5. The second process prints Goodbye to its stdout.
  6. The second process exits successfully, thus becoming ready.

.processes.*.command

Type
Array of strings.
Applicability
Required.
Description
Defines the command used to spawn this process after its dependencies become ready. The array must have at least one value. The first value determines both the program to spawn a process for as well as the first argument to that process.
Examples
Program with no additional arguments
[processes.minimal]
command = ["true"]
ready-when = "exited"

This Engage file defines one process. The process:

  • Is named minimal.
  • Defines a command that exits successfully.
  • Becomes ready when it exits successfully.

When running all processes in this Engage file, the following will happen:

  1. The minimal process spawns, because it has no dependencies.
  2. The minimal process exits successfully, thus becoming ready.
Program with additional arguments
[processes.hello-world]
command = ["echo", "Hello, world!"]
ready-when = "exited"

This Engage file defines one process. The process:

  • Is named hello-world.
  • Defines a command that prints Hello, world! to its stdout and exits successfully.
  • Becomes ready when it exits successfully.

When running all processes in this Engage file, the following will happen:

  1. The hello-world process spawns, because it has no dependencies.
  2. The hello-world process prints Hello, world! to its stdout.
  3. The hello-world process exits successfully, thus becoming ready.

.processes.*.environment

Type
Table of strings.
Applicability
Optional.
Description
Defines environment variables to set for this process. Each key-value pair in this table defines the name of an environment variable and its value respectively. Environment variables not defined in this table are left unset or are inherited normally.
Examples
Set one environment variable
[processes.hello-world]
environment.NAME = "world"
command = ["bash", "-c", "echo Hello, $NAME\!"]
ready-when = "exited"

This Engage file defines one process. The process:

  • Is named hello-world.
  • Sets an environment variable named NAME to world.
  • Defines a command that prints Hello, world! to its stdout and exits successfully.
  • Becomes ready when it exits successfully.

When running all processes in this Engage file, the following will happen:

  1. The hello-world process spawns, because it has no dependencies.
  2. The hello-world process prints Hello, world! to its stdout.
  3. The hello-world process exits successfully, thus becoming ready.

.processes.*.ready-when

Type

One of "exited" and "spawned".

Applicability

Required.

Description

Defines the point at which this process is considered ready. In turn, this defines when processes that depend on this one can be spawned, as well as the order in which this process and its dependents exit.

A value of "exited" causes dependent processes to be started after this process has exited successfully.

A value of "spawned" causes dependent processes to be started after this process has successfully spawned. It also causes this process to exit after its dependents have exited.

Examples
Two tasks
[processes.first]
command = ["echo", "Hello"]
ready-when = "exited"

[processes.second]
command = ["echo", "Goodbye"]
ready-when = "exited"
after = ["first"]

This Engage file defines two processes. One process:

  • Is named first.
  • Defines a command that prints Hello to its stdout and exits successfully.
  • Becomes ready when it exits successfully.

The other process:

  • Is named second.
  • Defines a command that prints Goodbye to its stdout and exits successfully.
  • Becomes ready when it exits successfully.
  • Spawns after first becomes ready.

When running all processes in this Engage file, the following will happen:

  1. The first process spawns, because it has no dependencies.
  2. The first process prints Hello to its stdout.
  3. The first process exits successfully, thus becoming ready.
  4. The second process spawns, because its dependencies are ready.
  5. The second process prints Goodbye to its stdout.
  6. The second process exits successfully, thus becoming ready.
One task and one service
[processes.service]
command = ["sleep", "infinity"]
ready-when = "spawned"

[processes.task]
command = ["echo", "Hello, world!"]
ready-when = "exited"
after = ["service"]

This Engage file defines two processes. One process:

  • Is named service.
  • Defines a command that sleeps forever.
  • Becomes ready when it spawns.

The other process:

  • Is named task.
  • Defines a command that prints Hello, world! to its stdout and exits successfully.
  • Becomes ready when it exits successfully.
  • Spawns after service becomes ready.

When running all processes in this Engage file, the following will happen:

  1. The service process spawns, because it has no dependencies.
  2. The service process sleeps forever, in parallel with steps 3 through 5.
  3. The task process spawns, because its dependencies are ready.
  4. The task process prints Hello, world! to its stdout.
  5. The task process exits successfully, thus becoming ready.
  6. Engage begins ending the run by sending SIGINT to the service process because all processes without dependents are tasks that have exited successfully.
  7. The service process exits because of the SIGINT it received from Engage.
  8. Engage exits with an error because service exited due to an unhandled signal.

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to SemVer.

Unreleased

Removed

  1. BREAKING: Remove the concept of groups. (!10)
  2. BREAKING: Remove the .task[].ignore key. It is seldom useful and can be implemented outside of Engage when it’s truly necessary. (!13)
  3. BREAKING: Remove the .interpreter key as it is no longer necessary. (!15)
  4. BREAKING: Remove support for non-Unix platforms. Non-Unix platforms may become supported in the future. (!29)
  5. BREAKING: Remove the -j/--jobs command line option. (!33)
  6. BREAKING: Remove the ability to specify the -f/--file option prior to any subcommand. Now it can only be specified after the subset of subcommands that can make use of it. (!33)

Changed

  1. BREAKING: Replace the .task[] and .task[].name keys with the .processes and .processes.* keys respectively. “Tasks” are now called “processes”, because that’s what they ultimately are. (!11, !37)
  2. BREAKING: Rename the .processes.*.depends key to .processes.*.after. (!11)
  3. BREAKING: Replace the .processes.*.script key with the .processes.*.command key, which takes an array of strings rather than a string. (!15)
  4. BREAKING: Only permit ^[a-z0-9-]+$ in process names, and - cannot appear as the first character. (!18)
  5. BREAKING: Reject unknown keys in Engage files. (!19)
  6. BREAKING: Replace the positional argument of engage dot with a -p/--process option. (!41)
  7. BREAKING: Replace the engage just subcommand with a -p/--process option for the engage command (without any subcommand). (!41)
  8. BREAKING: Make engage dot and engage list exit with an error without printing the usual output if there are invalid process dependencies. The new -r/--relaxed option for both commands can be used to approximate the old behavior. (!43, !46)
  9. Deduplicate elements in the array of the .processes.*.after key. Duplicates are not rejected, but they are ignored while constructing the dependency graph, so e.g. they will no longer show up in engage dot. (!11)
  10. Improve error messages when attempting to run processes. (!21)
  11. Improve the formatting of errors. The new format is much more likely to work well with screen readers, and can include more information than just the error message, such as suggestions for resolving the error. (!21, !43)
  12. Spawn each process in its own process group. Primarily, this prevents them from receiving SIGINT directly from the shell when ctrl+c is pressed while running Engage, because shells typically send SIGINT to the entire process group rather than just the first process spawned. Engage already manages the forwarding and sending of signals to processes it spawns. (!33)
  13. Improve some error messages. (!33, !37, !44)
  14. Stop canonicalizing the path to the Engage file. Paths involving symlinks will behave more predictably. (!33)
  15. Support the TOML specification version 1.1.0. (!36)
  16. Improve the “File format” page of the book by flattening headings, using definition lists, and adding examples. (!38)
  17. Replace the “Tutorial” page of the book with an “Introduction” page. (!35)

Fixed

  1. No longer block the spawning of processes on other processes that they do not declare a direct or transitive dependency on. (#9, !24, !26)

Added

  1. BREAKING: Add the .processes.*.ready-when key, which allows configuring processes to act like tasks (via the "exited" value) or services (via the "spawned" value), essentially allowing Engage to act as a service manager. (!35)
  2. Add the .processes.*.before key, which requires that the process in question exit successfully before spawning processes in the array of process names given to this key. Elements are deduplicated in the same way as the .processes.*.after key. (!12)
  3. Add the .processes.*.environment key, which allows configuring environment variables on a per-process basis. (!14, !16)
  4. Add the -l/--log-format command line option for choosing alternate log formats. (!27, !28, !41)
  5. Add a ctrl+c/SIGINT handler which prevents new processes from spawning, signals running processes to exit, and waits for them to exit. (!29, !33)
  6. Allow selecting multiple processes for the engage command and engage dot subcommand. (!41)
  7. Add the -r/--relaxed option to the engage dot subcommand which treats certain kinds of errors (such as dependency cycles) as warnings. (!43)

v0.2.1 - 2025-09-08

Changed

  1. Greatly improve error messages in some cases, primarily task failures and dependency cycles. (!4)

Fixed

  1. Only load the Engage file when necessary. (2f394ad)

Added

  1. Add more thorough documentation in the form of a book. (Too many commits to link.)
  2. The long help CLI output links to a local copy of the book, if packaged to do so. (!2)

v0.2.0 - 2023-09-19

Changed

  1. BREAKING: Always exit with a status of 1 when a task fails instead of trying to exit with the same status code as the task. (c18357e)
  2. BREAKING: Exit with a status of 2 for errors other than tasks failing. (a991d12)
  3. BREAKING: Move subcommands from under self to the top level and remove the self subcommand. For example, you’d now use engage dot instead of engage self dot. (e537e9d)
  4. Remove all restrictions on group names. (0850054)

Fixed

  1. Report all failed tasks instead of just the first one. (b9fcdcd)

v0.1.3 - 2023-09-10

Removed

  1. Remove the Nix binary cache. (066e97c)

Fixed

  1. Wait for all started tasks to complete before terminating. (d98c6cc)

v0.1.2 - 2023-02-10

Changed

  1. Improve the behavior summary. (7be43e4)

Added

  1. Add a --jobs/-j option to limit parallelism. (da1cc61)

v0.1.1 - 2023-01-28

Added

  1. Add a subcommand to generate shell completions. (48c925b)

v0.1.0 - 2022-11-07

Changed

  1. BREAKING: Disallow certain group names for forward compatibility. (066b2e3)
  2. BREAKING: Make engage just <GROUP> [TASK] run all dependencies. (acda394)
  3. Allow engage self dot to take <GROUP> [TASK] arguments to show a subgraph for the given target. (acda394)
  4. Make error messages a little prettier. (257e49e)
  5. Allow engage self dot even if there are cycles. (f6ffbc2)
  6. Print out the status at the end to make it easier to see. (37a6b9f)
  7. Print out the group and name of the failing task, if any. (c3df8f6)
  8. Print errors to stderr instead of stdout. (9c5dfaf)
  9. Rework output while running the graph to be more functional and accessible. (8880003)

Fixed

  1. BREAKING: Prevent groups from depending on nonexistent groups. (69219ec)
  2. Fix a deadlock in an unusual situation. (acee5fe)
  3. Prevent a deadlock in an unusual situation. (295e3b6)
  4. Improve the graph in unusual self-loop situation. (a9f1a27)
  5. Improve UX when interpreter is given an empty list. (72e89a3)
  6. Improve UX of some error messages. (c663bb6)
  7. Greatly improve error messages for dependency cycle issues. (f6ffbc2)

Added

  1. Upload build artifacts to Computer Surgery Nix binary cache. (8773d4d)

v0.1.0-alpha.2 - 2022-10-09

Changed

  1. BREAKING: Rename task.cmd to task.script in engage.toml. (b403dbe)
  2. Change CLI UX due to a dependency update. (b0f7ac7)

Fixed

  1. Reset output style at the beginning of each line. (!1)

v0.1.0-alpha.1 - 2022-09-17

Initial release.

Contributing

  1. Install Lix, direnv, and nix-direnv.

  2. Enable Lix’s nix-command experimental feature.

  3. If using a graphical editor, ensure it has direnv support, e.g. by installing an extension. If no support is available, changing directories into the project and launching the editor from the terminal should cause it to inherit the environment; though the editor will likely need to be restarted to propagate any changes to the direnv setup to the editor if any such changes are made.