DoK #51 Promscale: Using Prometheus + Promscale + PostgreSQL to go from Observation to Understanding // Matvey Arye

Data on Kubernetes Community

Abstract of the talk…

Often when I talk about putting observability data into PostgreSQL people ask me: are you crazy? And yet this somewhat heretical view has the potential to unlock a lot of the power and promise of observability. Thanks to TimescaleDB (an extension to PostgreSQL), storing time series, metric data inside of a relational database is now efficient, fast, and scalable. This is thanks to its unique partitioning, compression, and horizontal-scalability features. But even if this is possible to do, why would you? The answer to that lies in the power of a flexible data model, joins, and SQL (which Promscale supports in addition to PromQL). A flexible data model allows you to combine metric data with various other data - from machine information such as the number of cores, memory, etc. to location information using GPS coordinates. This allows you to enrich your metrics with supplemental information using joins and performing much more sophisticated analysis using SQL for capacity analysis, BI, and more. A flexible data model brings us to our second heretical idea: combining multiple modalities of observation in a single database. Combining metrics, logs, traces, event data, etc. in one DB has two major advantages: the first being a similar analytical advantage to what is described above: the ability to join and cross-correlate various types of signals together. The second major advantage is operational simplicity. As we all know, databases are the hardest things in our infrastructure to maintain and operationalize because of that pesky thing called state. So why maintain multiple different types of database systems if you could maintain just one? While these ideas about observability data on Kubernetes may seem unusual and counter-intuitive, I hope they will generate interest and start a good conversation.

Bio…

Mat has been working on data infrastructure in both academia and industry for the past decade. Currently, he is leading the Promscale team, to make it easy for people to store and analyze their Prometheus data in both PromQL and SQL. Previously, he completed his Ph.D. at Princeton and then worked as one of TimescaleDB's core architects where he concentrated on performance, scalability, and query power.