Engine Overview¶
Engine interface¶
The engine is just a set of functions that run in sequence, in a continuous loop.
Every loop, each function is in charge of making sure that the current state of the database matches the desired state. e.g. if a CUBED account has been created, the ensure_accounts_exist method ensures that an equivalent Conductor account is created.
Due to this design, the engine is intended to be self healing. If a crash or similar happens, the engine will just try again on the next iteration of the loop.
Here is an overview of what each engine methods do:
| Method Name | Description | Creates/Removes |
|---|---|---|
| ensure_accounts_exist | Ensures that conductors list of accounts is up to date with the central CUBED list (adding or removing rows) | CommandAccount |
| ensure_cron_definitions_exist | Ensures that definitions created from update_agg_pct_batch are up to date |
CommandDefinition |
| ensure_cron_definitions_dependencies_exist | Ensures that dependencies listed in update_agg_pct_batch are up to date - based on the read_from/writes_to lists. |
CommandDefinitionDependency |
| ensure_slots_exist | Ensures that slots are up to date based on definitions and accounts created in previous steps | CommandSlot |
| ensure_runs_exist | Creates any runs that are ready to be created, based on their start policies | CommandRun |
| ensure_runs_timeout | Checks for any running runs that haven't sent a heartbeat beyond a threshold, and sets their status to timeout | CommandRun |
| ensure_retries_exist | Creates runs for any failed runs, based on their retry policies | CommandRun |
ensure*cron*[n] methods
In the future, we would want to allow definitions to be created not just via update_agg_pct_batch but via other methods, such as JSON/YAML files.
To do this, we would want to:
- Simplify the interface to
ensure_definitions_exist/ensure_definitions_dependencies_exist - Add a way of configuring pluggable sources of definitions, e.g.
UpdateAggPctBatchSource,JsonFileSource,YamlFileSourceetc which the engine would use to generate the list of definitions
Running the engine¶
See the documentation on the engine agent for running the engine both locally and in production.