Skip to content

Engine Steps

Ensure Accounts Exist

This method ensures that accounts from attrib_account are replicated in the Conductor system. This involves creating and removing accounts and as such when removed, any related slots/runs are also removed. (see cascading deletes)

Not all accounts that exist in attrib_account will exist in Conductor. See accounts for more information.

Ensure Cron Definitions Exist/Ensure Cron Definitions Dependencies Exist

These methods ensure that commands listed in update_agg_pct_batch exist in the Conductor system. This involves creating and removing definitions and as such when removed, any related slots/runs are also removed. (see cascading deletes)

For creating definitions dependencies, the reads_from and writes_to fields are used to figure out which command is dependant on another.

Ensure Slots Exist

Engine Steps

Slots are generated based on:

  • Their frequency. This can be Monthly, Weekly, Daily etc and affects the start/end date of each slot
  • Their matrix. With a matrix of Account, slots are duplicated once for each account.

Slots are generated starting from when the account was added, or when the definition was created, whichever is later.

Slots are generated in advance of when they are needed for ease of querying.

Tip

The amount of time in the future the engine will create slots is configured by the COMMAND_SLOTS_CREATE_EXTRA_DAYS setting.

Ensure Runs Exist

After Success Start Policy

Engine Steps

The engine will create a run for a slot if the follow conditions are met:

  • All the dependencies have finished successfully. These include the definition dependencies and also the implicit slot dependencies
  • The time the slot represents (i.e. the start/end date range) is now in the past

Warning

The start policy AfterSuccess is the only policy currently implemented

Ensure Timeouts Exist

Engine Steps

The engine will change the status of a run to Timeout if the following conditions are met:

  • It has been 10 minutes since the last heartbeat_at change.

See status transitions for more information

Tip

This duration can be configured using the COMMAND_TIMEOUT_DURATION_MINUTES setting.

Ensure Retries Exist

Standard Retry Policy

Currently only one retry policy is implemented.

Engine Steps

The engine will create a retry (another run for a slot) if the following conditions are met.

  • There has been less than 10 total failed runs in a row.
  • 10 minutes has elapsed since the last failure

Tip

Both the max failed attempts and the delay in between them can be configured using the COMMAND_RETRIES_STANDARD_MAX_ATTEMPTS and COMMAND_RETRIES_STANDARD_DELAY_MINUTES settings respectively