Research and prep for API outage drill

[lbeaufort]

• Research and categorize past post-mortems, define types of incidents that occur 🔒 https://docs.google.com/document/d/1hF8Dqea2AKymxx_-2VYGp3WzIg9o--nkU95QAvNVXjg/edit 🔒
• Find example queries from August 7
• Generate queries to be used in Locust to replicate heavy load

[lbeaufort]

WIP PR with queries and additional locust testing: https://github.com/fecgov/openFEC/pull/4031/files

    """Extract API query from Kibana log file"""
    import csv
    import json

    with open("/Users/lbeaufort/Documents/FEC-LB-big-queries.csv", "r") as file:
        reader = csv.reader(file, delimiter=',')
        # Endpoint/uery lookup
        queries = {}
        for row in reader:
            # Column 4 has the queries. Throw out some bad data
            if 'GET' in row[3] and 'maxRedirects' not in row[3]:
                # Get the endpoint- everything after the first '/' to the first '?'
                endpoint = row[3].partition("/")[2].partition("?")[0]
                # Get the query param string - everything between the ? and the ' '
                query_parameters = row[3].partition("?")[2].partition(" ")[0]
                # Split each query pair out into a list
                parameter_groups = query_parameters.split("&")
                # Make a dictionary of the parameters (this is how locust needs them)
                query_dict = {}
                for result in parameter_groups:
                    # Split the parameters from the values
                    parameters_split = result.split("=")
                    query_dict[parameters_split[0]] = parameters_split[1]
                # Add to the endpoint/query lookup
                if not queries.get(endpoint):
                    queries[endpoint] = [query_dict]
                else:
                    queries[endpoint].append(query_dict)

        print(json.dumps(queries, indent=1))

[lbeaufort]

I tested these queries in stage with Locust and was able to create an outage.

We will want to look at the API umbrella logs to see what kind of load we're trying to replicate

August 7: 3,030,072 hits
Peak hour: 277,015 hits = 4616/minute = 77/second

[lbeaufort]

query_dict[parameters_split[0]] = parameters_split[1] this didn't properly handle multiples, IE multiple contributor names. It would have overwritten any previous value.