worker.py

init()

This function sets up various Worker configurations of which name (string passed in to identify worker), server (server instance), input (main_buffer or retry_buffer) and output (retry_buffer) queue are required. Once it has setup it's configurations it will create an instance of AccountTotals and then call reset().

def __init__(self, name, server, input_queue, output_queue, done_sleep=0.05, empty_sleep=1, empty_timeout=1, connection_timeout=5):
        self.name = name
        self.server = server

        self.input_queue = input_queue
        """Description of input queue"""

        self.output_queue = output_queue
        """Description of output queue"""

        ...

        self.totals = AccountTotals()   
        self.reset()
        super(Worker, self).__init__()

reset()

Simply sets many of the properties that are set after insert_visit() is called to insert data into the database.

loop()

This function will try to get an input from the input_queue and then call insert_visit. If the input queue is empty the worker will go to sleep for a certain amount of time before retrying to get a visit. After any visit is inserted or exception raised all Worker variables are reset and the worker sleeps for an amount of time.

insert_visit()

Insert visit takes an argument of visit which is passed in from loop which is taken from the input queue.

1. check if the visit has expired
2. check if account is active & is not in maintanence
3. set totals to a new instance of AccountTotals to record local totals for this insert
4. check if ip is within a range of blocked ups set by account
5. check if visit passes validation has passed
6. handle an impression visit
7. match the visits referrer to a pattern

8. start a transaction, inserting:

   1. page
   2. events
   3. labels
   4. redirects
   5. endpoints
   6. then perform sync and finally commit the transaction

9. Once that has completed update_totals is called to keep track of the metrics