Deploying to Staging and Production¶
The development
is the branch all new work is merged to. From time to time, we want to deploy this to staging, and subsequently production. This document outlines the steps required to do so.
Staging¶
- Pull the latest changes from the
development
branch. - Create a new branch from
development
with the following templaterc/<date>
(e.g.rc/2001-01-01
) where rc stands for release candidate. - Push the new branch to the remote repository.
- In bitbucket, run a pipeline build for the new branch for both
control-staging
anddash-staging
pipelines. Any developer can do this. - Once the builds are successful, you want to ssh into CUBED_STGE_CONTROL, the command is
ssh cubed@<cubed-stge-control-host> -i <path-to-ppk>
. If you have not been supplied with a ppk file, please contact your team lead. - Once in the box, go to the backend directory
cd /srv/attrib-backend/backend/
. - Instantiate a screen session via
screen -S <screen-session-name>
. Documentation on linux screens can be found here and it is strongly encouraged that you familiarise yourself with screens. If the session is already running, you can attach to it viascreen -r <screen-session-name>
. Additionally, to detach from a screen session viaCtrl + A + D
. - Run migrations via
sudo python3 manage.py migrate_client --loaddata --noinput
. Also we may need to run migrations for other apps, Seopt etc. Check this by runningsudo python3 manage.py migrate --list
and see if there are any unapplied migrations.
Before running migrations
Whenever a developer is running migrations, they should always be in a screen session. This is to ensure that the migrations are not interrupted if the connection is lost.
The final command will run migrations against a few staging accounts, this allows us to "test" the migrations before running them on production. As a final precaution, go to staging.withcubed.com and make sure everything is working as expected on dash.
Running migrations on different apps
An exhaustive list of migration commands to run for a given app are as follows:
- Base
sudo python3 manage.py migrate base
- Client
sudo python3 manage.py migrate_client --loaddata --noinput
- Seopt
sudo python3 manage.py migrate seopt --database=seopt
- Command/Conductor
sudo python3 manage.py migrate command --database=command
No accounts showing in dash?
If no accounts are shown when logging into dash, it is likely you have an account configuration issue. Contact your team lead for assistance.
Production¶
- Run a pipeline build for branch
rc/<date>
for bothcontrol-prod
anddash-prod
pipelines. Here, we usually runcontrol-prod
before runningdash-prod
. This is because unapplied migrations on dash will cause the frontend to crash in certain scenarios. - After the builds are successful, we now want to ssh into CUBED_CONTROL_A box via
ssh cubed@<cubed-prod-control-host> -i <path-to-ppk>
. If you have not been supplied with a ppk file, please contact your team lead. - Again go to the backend directory
cd /srv/attrib-backend/backend/
. - Instantiate a screen session via
screen -S <screen-session-name>
. Documentation on linux screens can be found here and it is strongly encouraged that you familiarise yourself with screens. If the session is already running, you can attach to it viascreen -r <screen-session-name>
. For example, to detach from a screen session viaCtrl + A + D
. - Run migrations via
sudo python3 manage.py migrate_client --loaddata --noinput
.Also we may need to run migrations for other apps, Seopt etc. Check this by runningsudo python3 manage.py migrate --list
and see if there are any unapplied migrations. - You can check on Grafana which clients are currently in maintenance mode - here. The retry buffer should slowly creep up during this time as incoming hits are queued up.
Perhaps obviously, unlike staging, these migrations are run on all of our clients in production. As a final step, go to dash.withcubed.com and make sure everything is working as expected on production.
Wrapping up¶
- Once we have confirmed that everything is working as expected, create a PR from
rc/<date>
tomaster
. Once the PR has been approved by the team, the release candidate branch can be merged intomaster
. - Go to
monday.com
and mark all tasks underPending Deploy
asDone
. - Go to the
Sprint Review
board and review the sprint progress. - As a final clean up, delete screens on both control and production boxes via
screen -X -S <screen-session-name> quit
.
Info
Points 2 and 3 are only applicable if you are the sprint lead. If you are not, please liaise with your sprint lead to do so to ensure that the sprint is closed off properly.