Backend Pipeline¶

This pipeline is run during the pull request process in the Backend steps in Bitbucket, which can be verified in the bitbucket-pipeline.yml file. The steps are divided into three parts: pre_test, test, and post_test. The pre_test and post_test steps are part of a CI role, defined in Ansible.

Pre Test:¶

During this step, the check, database, backend, and validate roles are run.

Check¶

This role ensures that the MySQL service is properly configured.

Database¶

This role ensures that the databases exist.

Backend¶

This role sets up the pipeline with the necessary dependencies and environment variables.

Validate¶

This role ensures that the migration files are in a state that can provision the test database.

pre_test task of CI role:¶

The pre_test tag in the CI role is run at the end of the pre-test phase. In this step, a database snapshot is pulled from S3 and loaded using the MySQL command. To pull the snapshot from S3 and load it into the database, certain packages are required and are therefore installed. The AWS CLI is configured, and the snapshot is pulled. The migration of the current branch is compared with the development branch. If the current branch contains all the migrations that the development branch has, then the snapshot is loaded into the databases and the test is run. This process saves around 20 minutes of time that would otherwise be spent creating a new database. There are some tables that have constant values; these are also loaded if we are loading the snapshot into the test database.

Following image shows the interaction with S3 snapshot bucket when the backend test pipeline runs.

Snapshot Create and Update

Running the test¶

Tests are always run with the keepdb flag after setting up the required dependencies and databases. They are run as a shell script, providing us with the log of the entire test run.

Post Test¶

After running the test, the test databases are flushed because Django could leave some traces of data undeleted in the database during the teardown process. The databases are then restored with constant tables. A snapshot of the databases is created and saved to the S3 bucket. The bucket cubed-development-migration-snapshot contains all the snapshots. When a PR is merged into the Development branch, its snapshot replaces the development_snapshot in the S3 bucket.