Update KWR Semrush

As part of the AKWR pipeline we would like to tighten the quality of the keyword pool by removing lesser quality keywords, we can do this by utilising the SEMRush API to get keyword data such as search volumes, cost per click, competition level and trends.

Before we pass up keywords to the API we first we first apply configuration filters to the keywords, as to temper the amount of keywords we pass up to the API. The configuration filters are found in kwr_config and are market specific.

We iterate over any active configuration rows by market in kwr_config and apply the filters. The filters used in update_kwr_semrush are:

Filter Name	Description
market_id	The market to apply the filter to.
topic_id	Is the configuration topic specific, or site-wide.
brand_name	This allows us to strip out any branded keywords as they are not relevant to AKWR. This is done using the Levenshtein Distance
reduction_impressions	This allows us to strip out any keywords which have a low number of impressions.

Beyond these filters, we also ensure that we exclude keywords - which originate from Google Search Console - that have both a low impression count and we haven't seen them in the last 30 days. Finally, we also exclude any blacklisted keywords which can be found in kwr_blacklist_keyword.

Once we have applied the filters we then pass the keywords up to the SEMRush API. The API will return the following data: search volumes, cost per click, competition level and trends. We then store this data in kwr_semrush_volume and kwr_semrush_trends.