Skip to content

Deploying to Staging and Production

The development is the branch all new work is merged to. From time to time, we want to deploy this to staging, and subsequently production. This document outlines the steps required to do so.

Staging

  1. Pull the latest changes from the development branch.
  2. Create a new branch from development with the following template rc/<date> (e.g. rc/2001-01-01) where rc stands for release candidate.
  3. Push the new branch to the remote repository.
  4. In bitbucket, run a pipeline build for the new branch for both control-staging and dash-staging pipelines. Any developer can do this.
  5. Once the builds are successful, you want to ssh into CUBED_STGE_CONTROL, the command is ssh cubed@<cubed-stge-control-host> -i <path-to-ppk>. If you have not been supplied with a ppk file, please contact your team lead.
  6. Once in the box, go to the backend directory cd /srv/attrib-backend/backend/.
  7. Instantiate a screen session via screen -S <screen-session-name>. Documentation on linux screens can be found here and it is strongly encouraged that you familiarise yourself with screens. If the session is already running, you can attach to it via screen -r <screen-session-name>. Additionally, to detach from a screen session via Ctrl + A + D.
  8. Run migrations via sudo python3 manage.py migrate_client --loaddata --noinput. Also we may need to run migrations for other apps, Seopt etc. Check this by running sudo python3 manage.py migrate --list and see if there are any unapplied migrations.

Before running migrations

Whenever a developer is running migrations, they should always be in a screen session. This is to ensure that the migrations are not interrupted if the connection is lost.

The final command will run migrations against a few staging accounts, this allows us to "test" the migrations before running them on production. As a final precaution, go to staging.withcubed.com and make sure everything is working as expected on dash.

Running migrations on different apps

An exhaustive list of migration commands to run for a given app are as follows:

  • Base sudo python3 manage.py migrate base
  • Client sudo python3 manage.py migrate_client --loaddata --noinput
  • Seopt sudo python3 manage.py migrate seopt --database=seopt
  • Command/Conductor sudo python3 manage.py migrate command --database=command

No accounts showing in dash?

If no accounts are shown when logging into dash, it is likely you have an account configuration issue. Contact your team lead for assistance.

Production

  1. Run a pipeline build for branch rc/<date> for both control-prod and dash-prod pipelines. Here, we usually run control-prod before running dash-prod. This is because unapplied migrations on dash will cause the frontend to crash in certain scenarios.
  2. After the builds are successful, we now want to ssh into CUBED_CONTROL_A box via ssh cubed@<cubed-prod-control-host> -i <path-to-ppk>. If you have not been supplied with a ppk file, please contact your team lead.
  3. Again go to the backend directory cd /srv/attrib-backend/backend/.
  4. Instantiate a screen session via screen -S <screen-session-name>. Documentation on linux screens can be found here and it is strongly encouraged that you familiarise yourself with screens. If the session is already running, you can attach to it via screen -r <screen-session-name>. For example, to detach from a screen session via Ctrl + A + D.
  5. Run migrations via sudo python3 manage.py migrate_client --loaddata --noinput.Also we may need to run migrations for other apps, Seopt etc. Check this by running sudo python3 manage.py migrate --list and see if there are any unapplied migrations.
  6. You can check on Grafana which clients are currently in maintenance mode - here. The retry buffer should slowly creep up during this time as incoming hits are queued up.

Perhaps obviously, unlike staging, these migrations are run on all of our clients in production. As a final step, go to dash.withcubed.com and make sure everything is working as expected on production.

Wrapping up

  1. Once we have confirmed that everything is working as expected, create a PR from rc/<date> to master. Once the PR has been approved by the team, the release candidate branch can be merged into master.
  2. Go to monday.com and mark all tasks under Pending Deploy as Done.
  3. Go to the Sprint Review board and review the sprint progress.
  4. As a final clean up, delete screens on both control and production boxes via screen -X -S <screen-session-name> quit.

Info

Points 2 and 3 are only applicable if you are the sprint lead. If you are not, please liaise with your sprint lead to do so to ensure that the sprint is closed off properly.