Utility Server

The utility server (available as cubed-util) is a special server dedicated to managing the deployment pipelines.

It houses both all incoming changes pushed by the pipelines and any deployment scripts (bash and ansible) needed by each project.

See servers for connecting to cubed-util.

Deployment scripts

Deployment scripts are simple bash scripts that are used . These are what are directly called by the pipelines. These scripts are stored in /home/deployments/ and are separated by project. (DCS, dash etc)

cubed@cubed-util:~$ ls /home/deployments/
prod_command  prod_dps_segmenter  prod_seopt    stge_control  stge_dps_segmenter  stge_seopt
prod_control  prod_dps_visscore   prod_web      stge_dcs      stge_dps_visscore   stge_web
prod_dcs      prod_rsrv           stge_command  stge_dps      stge_rsrv

Incoming directory

Inside of each deployment directory, is an incoming directory. This is where the pipelines will copy any new code before running the deployment scripts.

cubed@cubed-util:~$ ls /home/deployments/prod_dcs/
backups  deploy.sh  incoming  revert.sh

Ansible playbooks

The playbooks are usually called via the deployment scripts. These playbooks are in charge of actually deploying to each individual server.

The playbooks are stored under /home/ansible/:

cubed@cubed-util:~$ ls /home/ansible/
files                              playbook_deploy_seopt.yml              playbook_mysql_uninstall.yml    playbook_seed_server.yml
inventory                          playbook_install_promtail_command.yml  playbook_mysql_users.yml        playbook_setup_keys.yml
playbook_deploy_command.yml        playbook_install_promtail.yml          playbook_null_pydcs_logs.yml    playbook_setup_ml.yml
playbook_deploy_control.yml        playbook_mysql_check_user.yml          playbook_prediction_enable.yml  playbook_setup_telegraf.yml
playbook_deploy_dash_backend.yml   playbook_mysql_create_import_user.yml  playbook_prediction_start.yml   playbook_setup_users.yml
playbook_deploy_dcs.yml            playbook_mysql_create_users.yml        playbook_restart_promtail.yml   playbook_updated_predict_ips.yml
playbook_deploy_dps_segmenter.yml  playbook_mysql_defaults.yml            playbook_restart_pydcs.yml      playbook_update_geoip.yml
playbook_deploy_dps_visscore.yml   playbook_mysql_fix_conf.yml            playbook_restart_pydps.yml      repo_ansible
playbook_deploy_rsrv.yml           playbook_mysql_standby.yml             playbook_restart_telegraf.yml   roles

Production and staging deployments share the same playbooks, separated only be the inventory. You can view the inventories in /home/ansible/inventory/[production|staging]/.

NFS Mounts

The code used by each server is not actually stored on their local drives, but is instead stored on a separate shared server (cubed-nfs-a/b, or cubed-stge-nfs for the staging equivalent) via NFS.

The utility server has mounts used by every production system. You can view what is currently mounted using this command:

cubed@cubed-util:~$ ls /srv/
cubed_prod_command  cubed_prod_dps            cubed_prod_seopt    cubed_stge_dcs            cubed_stge_playwright
cubed_prod_control  cubed_prod_dps_segmenter  cubed_stge_command  cubed_stge_dps            cubed_stge_rsrv
cubed_prod_dash     cubed_prod_intweb         cubed_stge_control  cubed_stge_dps_segmenter  cubed_stge_seopt
cubed_prod_dcs      cubed_prod_rsrv           cubed_stge_dash     cubed_stge_intweb         cubed_stge_storybook

Viewing mounts

To see what it is currently mounted via NFS on a particular server. You can use the follow command:

cat /proc/mounts

Updating mounts

You can edit the NFS mounts by using the following command:

nano /etc/fstab

Once, you've finished your changes, you must reload the mounts by using this command:

mount -av

Be warned, you may break production systems or pipelines by changing these mounts.

Gaining root

The cubed user has limited permissions on this server. If you need to make changes, it is possible to become root by first logging in as the deployment user, followed by logging in as root.

For this you need the password for the deployment user, which is stored in AWS Secrets Manager.

su - deployments # Login as "deployments", password is required
sudo -i # Login as root

Once you are root, you can also login as other users, such as ansible:

su - ansible

Info

The ansible user is useful for connecting to servers which do not have the cubed key on them. See the inventory files for which servers you can connect to.

Danger

Be warned when making changes as root, as doing so can break the deployment pipelines.

SSH Hopping

The utility server can also be used to hop to other servers. This is useful when you need to connect to a server that is not directly accessible from your local machine. Some examples are the cubed-prediction-<client>-a/b servers. The ansible user is one user whose permissions are allowed to hop to other servers.

To hop to another server you can simply ssh to the utility server and then ssh to the server you want to connect to. To find an up to date list of servers you can connect to you can cat out the /home/ansible/inventory/dnsmasq/dnsmasq.leases file.

Warning

To gain access to the ansible user you will need to change to the deployments user, then change to the ansible user. The password for the deployments user is stored in AWS Secrets Manager under Prod/Login/Positive/Deployments. If unable to change to the ansible user from the deployments user, you may need to change to root as a intermediary step.