Welcome¶
Welcome to Cubed's Internal Documentation. Please use the side bar to navigate through the different sections of our system.
System Overview¶
Cubed collects Visit data via our tag.js, this is then processed by our Data Collection Servers (DCS) before being inserted into an account specific Database on our Database Server.
Our frontend and backend are both powered by Django - making development easy across the two.
We use our backend Django to pull data from our Database and External Sources (such as Adwords, Adobe etc..) before processing and inserting back into the Database.
Development Process¶
In order to develop parts of Cubed, pending on the area youre working on, you will need to pull that relative repo and then consult the area of the Docs here.
Ecosystem Layout¶
Connection¶
We connect to servers both locally and online using either the built in Operating System's SSH client or the excellent PuTTY.
Service Breakdown¶
Let's quickly break down each section of our service and view the main technologies that sit behind them.
Cubed Tag¶
The Cubed tag is built with vanilla JavaScript - meaning we try not to rely on any libraries (such as jQuery). Our tag is minified before deploy and is cached on Amazon's Cloudfront meaning it should download onto client's websites almost instantly.
Langauges/Tools used:¶
- JavaScript
Data Collection Servers¶
Our DCS is built on Flask, using NGiNX. The server accepts requests from our tag, puts them into a buffer and then, during a constant loop, will take each one and attempt to insert them into the Database.
During the insert process our server will also make calls to our Data Processing Servers - pending on the type of request it's dealing with.
Langauges/Tools used:¶
- Ubuntu 14
- Python
- Flask
- WSGI
- Nginx
Database Servers¶
Our databases are hosted by Positive Internet. We use the free open source client HeidiSQL to connect to our Databases.
Langauges/Tools used:¶
- MySQL
- HeidiSQL
Control¶
This is the main Cubed backend server, and runs on Django. Control is where all Cubed commands run for all accounts. This usually happens in the early hours and processes the previous day's data. Control talks to the database - pulling and inserting data, calling custom commands on the Data Processing Servers, and pulls from external sources.
Langauges/Tools used:¶
- Ubuntu 18
- Python
- Django
- Nginx
Dashboard¶
We have multiple dashboard servers to deal with the frontend traffic of the Cubed dashboard site. Here users can log in and view their processed data, and configure their account. The frontend is built primary on Django, using Tastypie to interact with Django/DB, and Knockout for the frontend framework.
Langauges/Tools used:¶
- Ubuntu 18
- Django 2.0
- Tastypie
- Python
- Knockout
External Data Sources¶
This can include any kind of non Cubed data sets. Our Cubed system talks to many different external services, below is some of the services we talk to:
- Bing
- Magento
- Zendesk
- TikTok
- Klaviyo
- SEMRush
- Oxylabs
Overall Tech Stack¶
-
Backend Development:
- Python: Main programming language.
- Django: Web framework for building web applications.
- TastyPie: REST API framework for Django.
- Uwsgi: Application server for serving Python apps.
- MySQL: Database management system.
-
Frontend Development:
- JavaScript/TypeScript: Programming languages used for our dashboard.
- Knockout: JavaScript library for responsive and dynamic UIs. Now only used for simulate page.
-
Infrastructure & Deployment:
- Nginx: Web server for serving web pages and proxy services.
- Ansible: Automation tool for software provisioning and application deployment.
- Terraform: Infrastructure as Code software tool for building, changing, and versioning infrastructure.
-
Data Analysis:
- R: Programming language and environment for statistical computing and graphics. In our case, used for data analysis and machine learning.