Analyzing your Google Search History with Rakam

Google allows you to export your Google searches that you did since you created your Google account but doesn’t have any dashboards for analyzing your historical data. I thought that it would be an interesting case if I could download my search history from Google and analyze the data and create a dashboard for my Google search history in Rakam.

First of all, you will need to export the raw events from Google using the Takeout service. Depending on the data volume, it may take some time but if you don’t have to wait for a few hours, uncheck all the checkboxes and only select Google Search because we will only use the searches in this tutorial.

It usually takes a few minutes for Google to process your historical data and create an archive for you. Download the archive when it’s ready, extract the contents and you will see the directories in the below:

Each JSON file is an array or elements for your searches:

{  
   "event":[  
      {  
         "query":{  
            "id":[  
               {  
                  "timestamp_usec":"1183032424150752"
               }
            ],
            "query_text":"my search term"
         }
      },
      ...
   ]
}

In order to be able to send the events to Rakam, we need to transform the data to a proper format but before that, if you noticed that Google only includes the search term and the timestamp as part of the event and these two fields are not enough for us to analyze the data efficiently.

I looked at the API services for categorizing the search terms but couldn’t find an efficient one that supports multiple languages. Then, I remembered that Google Trends has an autocomplete feature that also shows the category of the search term. After some debugging the browser network activity, I realized that the service doesn’t use a public API but doesn’t actually require a token so it seemed to be a good fit for my use-case.

I wrote a simple event mapper for Rakam in order to enrich my events with Google Trends API and attaches category, title and mid (Google Trends ID) attributes the events. I attached the script at the end but here is how it looks like:

The full code is in google_seach_event_mapper.js file.

Login Rakam UI app, select your project and visit this link. (Settings (left corner) -> Integrations -> Custom event mappers), click add new custom event mapper, choose “Custom Event Mapper” in the list, copy the full code in google_seach_event_mapper.js file and press create button.

Now we have our custom event mapper, we should send our search history events to our Rakam API. The collect_events.py file is a simple Python script that reads all the files in your Takeout/Searches directory, transforms and sends it to your Rakam API. Do not forget to pass search_dir, rakam-write-key and rakam-api-url parameters to the script.

The script will send all our search history to Rakam (mine is 9 years of history!) and when it’s done (usually done within a few minutes) you can analyze your data on Rakam UI.

Register Rakam, import the recipe for Google Search history using the recipe provided by us and that’s it! Now, you built an analytics service for your personal history.

rakam-recipes/google-seach-history.yaml at master · rakam-io/rakam-recipes · GitHub
rakam-recipes - A collection of Rakam recipesgithub.com