Infrastructure¶

The Cubed APIs are based on REST principles. Data resources are accessed via standard HTTPS requests in UTF-8 format to an API endpoint. We serve this feature via Amazon's API Gateway and AWS Lambda. The gateway sits between the client and the Lambda function, routing requests to the appropriate function. The Lambda function processes the request and returns a response to the gateway, which then returns it to the client, and deploy using Serverless.

API Gateway¶

The Cubed APIs are served via Amazon's API Gateway. The API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. The API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, authorization and access control, monitoring, and API version management.

There is a single API Gateway endpoint that serves the Cubed APIs:

api.withcubed.com

These can be seen inside API Gateway > Custom Domain Names. Each domain name is associated with a stage, which is a reference to a deployment of the API Gateway. The stages are used to manage the lifecycle of the API Gateway, and each stage has its own configuration settings.

We use AWS Route 53 to manage the DNS records for the withcubed.com domain, and the API Gateway is configured to use the custom domain name api.withcubed.com. This is an A record that points to the API Gateway endpoint, and the traffic is routed to the appropriate api gateway invoke URL. This can be seen inside Route 53 > Hosted Zones and searching for api to see the A record.

API Keys¶

The api.withcubed.com/subscription endpoint requires an API key to authenticate requests. The API key is generated by AWS and is used to authenticate requests to the API Gateway. The API keys can be found in the AWS console under the API Gateway service > API Keys.

Each API key is associated with a usage plan, which is a collection of one or more API stages. The usage plan specifies the rate limits and quotas for the API key, and the API key is used to authenticate requests to the API Gateway. Furthermore, each usage plan is associated with an API stage.

Warning

In the future it would be worth creating an API key per client, as this would allow for more granular rate limiting and monitoring of usage. As of '2024-07-01' only one client is interested in the subscription API, so there is one API key. In the future - if more clients use the API - it would be worth creating an API key per client.

Models¶

The API Gateway uses models to define the structure of the request and response payloads. The models are defined in the API Gateway console under API Gateway > Models. The models are used to validate the request and response payloads and ensure that they conform to the expected structure. For example, we have restricted the data object (see later) to have a maximum of 30 objects. This stops clients from sending too much data in a single request and helps to prevent abuse of the API.

Gateway Responses¶

The API Gateway uses gateway responses to define the structure of the response payloads. The gateway responses are defined in the API Gateway console under API Gateway > Gateway Responses. You can modify the response body to include custom error messages or additional information. This is useful for providing more information to the client about why a request failed, and can help to debug issues with the API.

Lambda¶

In our infrastructure, the Lambda functions are the compute resources that run the code to process the requests. The Lambda functions are written in Python 3.8 and are deployed using the Serverless framework. The Serverless framework is an open-source framework that allows you to build and deploy serverless applications using AWS Lambda, Azure Functions, Google Cloud Functions, and more.

Each Lambda function has it's own configuration settings, such as memory size, timeout, and environment variables. These configuration settings can be found in the serverless.yml file, or in the AWS console under Lambda > Functions > Configuration.

Runtime settings¶

The Lambda functions are set up with the following runtime settings:

Memory Size: 128 MB
Runtime: Python 3.8
Handler: handler.<function_name>

Handlers and Functions

The handler is the name of the Python file that contains the Lambda entrypoint function, and the <function_name> is the name of the function inside the Python file that is the entry point for the Lambda function (A tongue twister, I know).

In the case of the visscores and segments services function_name="get" and for the subscription service function_name="post". Therefore, the handler for the visscores and segments services is handler.get and the handler for the subscription service is handler.post.

Finally I strongly encourage the developer to cross reference the serverless.yml file with the AWS Console configuration before deployment, to be sure that any niche configurations in the console aren't overwritten by the serverless.yml file.

Environment Variables¶

In our case the lambda functions utilise environment variables to store database connection variables and are different for the staging and production Lambdas.

Making Changes via AWS Console¶

When making changes via the AWS console, it is important to remember that the changes made in the console will not be reflected in the serverless.yml file. This is because the serverless.yml file is mutually exclusive from the console, and any changes made in the console will not be reflected in the serverless.yml file. Therefore, it is important to make changes in the serverless.yml file and deploy the changes using the Serverless framework.

Warning

As of '2024-07-01' I (Jordan) struggled to deploy via serverless. Because of time constraints I was forced to make changes via the AWS console for the Subscription API. I have created a serverless template and added all relevant code inside the DPS repository for the API but the serverless deploy has not been tested.

For a future developer working on this, it seemed to be a problem with the AWS Secret-Key and Access-Key. The serverless deploy command was not able to authenticate with AWS even though the keys were correct.

If you do make changes via the console you must ensure that you redeploy the API Gateway and/or Lambda functions. To do this in the console, you can go to the API Gateway service and click on the API Gateway that you want to redeploy. Then click on the Actions dropdown and select Deploy API. This will redeploy the API Gateway with the changes that you made in the console.

For the Lambda functions, you can go to the Lambda service and click on the Lambda function that you want to redeploy. Then click on the Code tab and click on the Deploy button. This will redeploy the Lambda function with the changes that you made in the console.

Tip

You don't need to redeploy the Lambda function if you only make changes to the API Gateway and vice versa. If you make changes to both the Lambda function and the API Gateway, you will need to redeploy both.

api.withcubed.com/{visscores|segments}¶

The api.withcubed.com domain serves the Visscores and Segments services. These services allow you to retrieve visitor scores and segment data.

Supported Endpoints¶

Service	Endpoint	Method	Supported	Auth Type
Visscores	`https://api.withcubed.com/visscores`	GET	Yes	Cubed Cask Token
Segments	`https://api.withcubed.com/visscores/segments`	GET	Yes	Cubed Cask Token

Parameters¶

In requests to the Cubed API endpoints /visscore and /segments and responses from it, you will encounter the following parameters:

Parameter	Description	Example
vid	The `vid` is the token provided by data.withcubed.com to describe the visitor	`5f64ff6a840a7b62205b3f7e7cbecf24`
aid	The `aid` is the account token for the client	`c-a-<client_name>-<location>`
cask	The `cask` or Cubed Attribution Signing Key, is a token returned in the response from firing a hit to data.withcubed.com, it provided access to data belonging to either the `vid` or the `sid` for up to 24 hours after creation.	`gAAAAABfortbEq4BNqIJRXU6hXA8I1iS6ux44T-duSLNM7MSAaL8RXaV1B_AiIBLd1JNidWZGIapvJxtHq3FcOEDuKj_jAwlpBwsyi1b8CMq_0dptI1S6GX9g2RW87sCM3szBP_j6M0zI7eCTIhSqjRbvOT8hltWDY8wTvAvTnuDsbRWmfEV7w=`

Note

Information on getting the parameters provided above can be found here for the vid and sid.

Rate Limiting¶

Rate Limiting enables our Cubed API to share access bandwidth to its resources equally across all users.

Rate limiting is applied as per application based on VID. Currently, there is no rate limiting set up (for /visscores and /segments). When this API is used by more clients, it's strongly suggested rate limiting is set up, this can be done here.

Authorization¶

The /visscores and /segments endpoints require a Cubed Cask Token to authenticate requests. The Cubed Cask Token is generated by Cubed and is used to authenticate requests to the API Gateway.

This authorization is a query string parameter in the URL. The cask parameter is returned by the Cubed tag and is valid for 24 hours for a given visit. The authorization token is passed to a secondary Lambda function - visscore-auth-[prod|stge] that validates the token.

If the token is valid, the Lambda generates a policy that allows the user to access the resource. If the token is invalid, the Lambda generates a policy that denies access to the resource. The policy is then returned to the API Gateway, which uses it to authorize the request.

api.withcubed.com/subscription¶

The api.withcubed.com domain also handles subscription data for the /subscription endpoint. This service allows clients to post subscription data to the /subscription endpoint, which is then processed and stored in the client database under attrib_external_subscription_event.

Costs¶

The AWS services used are API Gateway and Lambda. Based on the following configurations:

API Gateway: 1,000,000 requests per month are free, then $3.50 per million requests. Let's say we have 2,000,000 requests per month, the cost would be $7.00 per month.
Lambda: 1,000,000 free requests per month, then $0.20 per 1 million requests. Let's say we have 2,000,000 requests per month. With a compute time of 10s and memory of 128MB, the cost would be $14.17 per month.
CloudWatch: 1,000,000 lambda invocations seems to be $3.00 per month.

The total cost would be $21.17 per month at 2,000,000 requests per month.

Warning

The above calculations are based on the AWS pricing as of '2024-07-01'. The costs may vary depending on the usage and the AWS pricing at the time of usage. Please refer to the AWS Pricing page for the most up-to-date pricing information.

Supported Endpoint¶

Service	Endpoint	Method	Supported	Auth Type
Subscription	`https://api.withcubed.com/subscription`	POST	Yes	API Key

Staging Endpoint

The staging endpoint doesn't have an associated domain name. This is to protect it from being hit by external sources. However, the staging endpoint can be accessed via its invoke URL, which can be found in the API Gateway console under API Gateway > Stages. As of '2024-07-01' the staging endpoint is https://jusiydg8mi.execute-api.eu-west-1.amazonaws.com/staging/.

Rate Limiting¶

Rate Limiting is applied to the /subscription endpoint at a rate of 10 requests per second and a burst of 1 requests. If you exceed this limit, you will receive a 429 status code. Additionally, there is a monthly limit of 1,000,000 requests per month. AWS generously provides a free tier of 1,000,000 requests per month, so this limit should not be exceeded without consideration.

Requests¶

Headers¶

The /subscription endpoint requires the following headers to be included in all requests:

Header Name	Description
`Content-Type`	The `Content-Type` header specifies the media type of the request body data.
`x-api-key`	The `x-api-key` header is generated by AWS and is used to authenticate the request. The API keys can be found in the AWS console under the API Gateway service.

Request Body¶

The /subscription endpoint requires the following parameters to be included in the request body:

Parameter	Description	Primitive	Example	Required	Limits
`aid`	`str`	The `aid` is the Cubed account token for the client, please ask for yours.	`c-a-<client_name>-<location>`	Yes	N/A
`data`	`array`	The `data` parameter is a JSON object that contains the subscription data to be stored in the client database.	`[{ "tid": "<str>", "subid": "<str>", "cid": "<str>", "evt": "<str>", "rvn": "<int>" }]`	Yes	30 objects
`simulate`	`bool`	The `simulate` parameter is a boolean value that determines whether the request is a simulation or not. i.e. whether the request is actually stored in the database or not.	`true`	No	N/A

The data parameter is an array of JSON objects that contain the subscription data to be stored in the client database. Each JSON object in the array can contain the following parameters:

Parameter	Description	Primitive	Example	Required
tid	`str`	The `tid` is the transaction ID for the subscription event. It is used to keep sales unique in our system. Cubed also uses this to join a Subscription sale that started on a web page, but was approved later.	`transaction_id_one`	Yes
subid	`str`	This is the unique Id from your internal system so we can track a Subscription repeating. This can be the same or different from the Transaction Id. If you don't tag your subscriptions with a unique Id, then simply repeat the transaction Id in this field.	`subscription_id_one`	Yes
cid	`str`	We can use Customer Ids in the Cubed system to help link external and internal data, similar to using Transaction Id. It is not needed as we usually join on Transaction Id, but does give us more data for the Subscription hits we receive.	`customer_id_one`	No
evt	`str`	The `evt` is the event type for the subscription event, this could be a newsletter or any other type of subscription event.	`newsletter`	Yes
rvn	`int`	Revenue is collected as British pence in Integer format. See our Tag for further information on this. If left blank, we will default the revenue to zero.	`100`	No

Example Request¶

Notice in the example below, the aid parameter is required, and the data parameter is an array of JSON objects that contain the subscription data to be stored in the client database. The simulate parameter is optional and is a boolean value that determines whether the request is a simulation or not.

Content-Type: application/json

{
  "aid": "c-a-<client_name>-<location>",
  "data": [
    {
      "tid": "12345",
      "subid": "12345",
      "cid": "12345",
      "evt": "newsletter",
      "rvn": 100
    },
    {
      "tid": "12346",
      "subid": "12346",
      "cid": "12346",
      "evt": "newsletter",
      "rvn": 100
    }
  ],
  "simulate": true
}

POST /subscription

import requests
import json

# The URL of the Lambda function endpoint
url = "https://api.withcubed.com/subscription"

data = {
    "simulate": True,  # OPTIONAL. If set to True, the request will not be stored in the database.
    "aid": "example_account_id",
    "data": [
        {
            "tid": "example_transaction_id",
            "subid": "example_subscription_id",
            "cid": "example_customer_id",
            "evt": "example_event",
            "rvn": 10
        }
    ],
}

# Convert the Python dictionary to a JSON string
data_json = json.dumps(data)

# Set the appropriate headers for a JSON request
headers = {
    "Content-Type": "application/json",
    "X-API-Key": "<your-api-key>",
}

response = requests.post(url, data=data_json, headers=headers)

Responses¶

The /subscription endpoint returns responses in JSON format and is not designed to return HTML or other formats. The API uses standard HTTP status codes to indicate the success or failure of a request, unlike the Cubed API, which uses custom status codes to indicate the success or failure of a request. This API is not intended to be used by a browser, but rather by a server-side application.

Warning

If clients want to use the /subscription endpoint in a browser, we would need to allow CORS (Cross-Origin Resource Sharing) in the API Gateway. This would allow the API to be accessed by a browser from a different domain. This is not currently set up, but can be done in the API Gateway console under API Gateway > Custom Domain Names.

Some of the common status codes returned by the API are:

Status Code	Description	Response Body
`200`	The request was successfully simulated.	{"message": "Subscription simulated successfully.", "validationErrors": null}
`201`	The request was successfully stored in the database.	{"message": "Subscription created successfully.", "validationErrors": null}
`400`	The request was malformed.	{"message": "Malformed JSON.", "validationErrors": "<exception message>"}
`400`	The request was missing required parameters.	{"message": "Error inserting subscription.", "validationErrors": "<exception message>"}
`400`	The request had an invalid account token.	{"message": "<exception message>", "validationErrors": "Invalid account".}
`403`	The request was not authenticated.	{"message": "Forbidden", "validationErrors": null}
`429`	The request was rate limited.	{"message": "Rate limit exceeded.", "validationErrors": null}
`500`	The request was not successful for some server-side reason.	{"message": "Internal server error.", "validationErrors": "<exception message>"}