Debugging

Enabling an account in Conductor

Enabling accounts in Conductor is done using our "toggle" system.

Enabling

To enable an account, run these two procedures on the "cubed config" production database:

call sp_toggle_enable_aft('c-a-client-uk', 'cron', 'command') -- Turn on in new system 
call sp_toggle_disable_aft('c-a-client-uk', 'cron', 'control') -- Turn off in old system

Disabling

To disable an account, run these two procedures on the "cubed config" production database:

call sp_toggle_disable_aft('c-a-client-uk', 'cron', 'command') -- Turn off in new system 
call sp_toggle_enable_aft('c-a-client-uk', 'cron', 'control') -- Turn on in old system

Note

You may notice these are two separate steps. In theory you can enable the account to run in both systems. In practice you wouldn't want this to happen.

Running commands manually after enabling

When running update_agg_pct_batch, you may notice something similar to this output before the commands run:

Quote

[toggle] Running for c-a-client1, c-a-client2. Not running for c-a-client3. Use --ignore_toggles to override. Current tag is 'control'

By using the flag --ignore_toggles, you are still able to run any account manually, even if it is only enabled in the new system.

Rerunning dead slots

With the default retry policy, slots will be retried up to 10 times, after which they are treated as "dead" and will not be retried again.

Currently there isn't a great way of retrying these dead commands, however it can be done by executing some custom SQL.

Listing dead slots

Here is some example SQL you can use to find currently dead slots:

SELECT
    c.token,
    a.start_date,
    a.end_date,
    b.name
FROM command_slot a
JOIN command_definition b ON a.definition_id = b.id
JOIN command_account c ON a.account_id = c.id
WHERE b.retry_policy = 0
AND (
    SELECT SUM(x.has_failed)
    FROM (
        SELECT 
            CASE
                WHEN (y.status = 2) THEN 1
                WHEN (y.status = 3) THEN 1
                ELSE 0
            END AS has_failed
        FROM command_run y
        WHERE y.slot_id = a.id
        ORDER BY y.waiting_at DESC
        LIMIT 10
    ) AS x
) = 10

This can further be filtered by account, date, command definition name etc.

WHERE c.token = 'c-a-client1'
WHERE b.name = 'update_agg_sales'

Re-running dead commands

Re-running dead commands is done in the same way as the the engine creates retries, by inserting new runs.

INSERT INTO command_run 
(id, slot_id, status, waiting_at, running_at, timeout_at, failed_at, success_at, heartbeat_at)
SELECT
    (SELECT REPLACE(UUID(),'-','')) as id,
    a.id as slot_id,
    0 as status,
    NOW() as waiting_at,
    NULL as running_at,
    NULL as timeout_at,
    NULL as failed_at,
    NULL as success_at,
    NULL as heartbeat_at
FROM command_slot a
JOIN command_definition b ON a.definition_id = b.id
JOIN command_account c ON a.account_id = c.id
WHERE b.retry_policy = 0
AND (
    SELECT SUM(x.has_failed)
    FROM (
        SELECT 
            CASE
                WHEN (y.status = 2) THEN 1
                WHEN (y.status = 3) THEN 1
                ELSE 0
            END AS has_failed
        FROM command_run y
        WHERE y.slot_id = a.id
        ORDER BY y.waiting_at DESC
        LIMIT 10
    ) AS x
) = 10

No more retries?

Due to the way the retry policy is implemented in the engine, any runs created using this method will not have any further retries.