Introducing metriql: Open-source metrics store
Meet rakam’s newest product, metriql: The first open-source metrics store where companies can define their metrics centrally as code on top of their dbt projects and then sync their data models to multiple BI or data tools at once.
metriql was born as a spin-off project from rakam as we decided to open-source rakam’s modeling layer and integrate it with other BI and data tools so companies using rakam for product analytics can also make use of their data models inside other BI and data tools for further analysis.
Today, most data-led/driven companies successfully establish the data warehouse as the single source of truth for their data and collect all their data into their data warehouse through ETL jobs. However, ETL is only one part of the whole process. Companies want to collect and own their data because they want to consume it in various ways to grow their business whether it might be building analytical dashboards for stakeholders or building an ML application. To consume the data via BI or data tools, companies need to model their data, define business metrics and KPIs so stakeholders can have an agreement in the definition of the data. Establishing the single source of truth for metadata is very difficult because tools and even teams consume the data in different methods than each other. This topic has also been addressed by many data folks recently: Ben has written The missing piece of the modern data stack and Basecase came up with the Headless BI concept, Airbnb introduced Minerva.
We have been also giving careful thought to this particular problem for a while. The way we think about metrics is not just about the definitions, instead, better integration between data tools and data warehouses. Most companies already use a modern data warehouse and have adopted the data warehouse as the single source of truth. They ingest data in real-time, run ELT jobs directly in their data warehouse, run ad-hoc queries and get the results in low-latency, and even run ML jobs on them.
If you’re using a data transformation tool such as dbt, you can model your data and expose it to your data tools but that’s not often enough for the data tools, we often found ourselves doing extra data modeling in data tools for metric definitions. This is primarily because most of the data tools introduce different ways to define metrics and you need to adopt it to be able to analyze the data reliably using these tools.
There are two major issues that most data teams run into when defining metrics in BI/data tools:
- You need to learn yet another expression language that’s never flexible enough compared to SQL. (It must be compiled to SQL anyway)
- Since you usually need to use a GUI to define the metrics, it’s not easy to automate, test, or share them with other data tools. (I’m not even talking about continuous integration and deployment)
Please visit the metriql blog to read the original introduction post: https://metriql.com/blog/introducing-metriql-open-source-metrics-store
