Conductor Overview¶
Conductor (also know as command, especially as referenced in the codebase/database) is our system for managing our overnight batch jobs.
Engine¶
The engine is in charge of all state in Conductor, and is responsible for making sure runs are ready and available to be executed. It does not do any work by itself, that is instead by dedicated agents. (See cron agent which is dedicated to running cron commands)
See the engine overview for more information.
Agents¶
Agents in conductor terms just mean long running processes.
Currently, there are two agents:
| Type | Description |
|---|---|
| command-agent-engine | In charge of ensuring the state of the system as stored in the database, is up to date. See the engine overview for more information. |
| command-agent-cron | In charge of actually executing runs. In this case specifically executing Django management commands in separate processes. |
Database structure¶
The database structure is documented in its own section.
| Name | Description |
|---|---|
| CommandAccount | Simplified version of attrib_account, only contains accounts that are managed by Conductor |
| CommandDefinition | Definition of work to be executed (a command). Specifies how often it should be when, how it should retry etc |
| CommandDefinitionDependency | Represents parent/child dependencies between definitions |
| CommandSlot | Represents a time period for which each definition (potentially by account) should run |
| CommandRun | Represents a single attempt at running a definition for a particular time range determined by its slot |