OpenTelemetry: starting series
Posted on December 2023 in Web DevelopmentBrought to you by Observability
According to official OpenTelemetry docs:
OpenTelemetry is an Observability framework and toolkit designed to create and manage telemetry data such as traces, metrics, and logs.
First time I ran into this is when it was still called OpenTracing. Well technically speaking OpenTracing and OpenCensus merged into OpenTelemetry. At that time I was leading a team of developers working on a distributed system and looking into how to trace requests through the system. That's when I found about honeycomb.io and observability as such. Do not mix it with washed out term now calling everything "observability". You allow to run some queries on collected logs? Market it as an observability tool (which might actually be quite close to truth sometimes IMO). Allow to query in low cardinality metrics? Observability! And in the last few years term observability started to mean much less than it did before. At least to me. According to Charity Majors (honeycomb.io CTO at the time of writing and one of observability preachers) in Observability: A Manifesto:
Monitoring is about known-unknowns and actionable alerts, observability is about unknown-unknowns and empowering you to ask arbitrary new questions and explore where the cookie crumbs take you.
So observability from monitoring differs mainly in that you don't need to ship new code to get new insights into your system. You can have high cardinality data, structured logs, some other automated magic and have fully observable system. That idea resonated with me and I really wanted to try it out. And now opportunity presented itself to me to do just that. After such long introduction let's get to the point.
OpenTelemetry
What is OTel and why it's important
Observability should start from somewhere. And that somewhere is usually data collection. In days of distant past (give or take a decade), when systems were more simple and mostly monolithic (or at least that's the systems I worked on) you could get away with some logs, add some metrics, server performance monitoring and you were good to go. Now we live in the world of microservices, distributed systems, cloud computing, and other buzzword you can fit in. And now simple logs and metrics are not enough. We need to see more, trace requests throughout the system, see where we're bottlenecking. Logs, metrics, and debugger just doesn't cut it.
And that's where OpenTelemetry (also called OTel) comes in. It is not some concrete tool, but rather a set of specifications and libraries that define how to collect telemetry data (logs, traces, and metrics), how it should be shaped, how it can be exported. And when we have all that data (that is not vendor locked) we can then use other tools to analyse it. I always dreamed to try out honeycomb.io, but my current workplace hsa something else setup, so I'll play with that for now. And there lies the beauty of OpenTelemetry - you can use it with any backend of your choice. The difference is only in how capable your backend is.
How to take advantage of it
For the most common languages and environments there are already libraries and tools available. For some of them (like k8s) it is possible to just enable it and it will start collecting data. For others there might be some additional configuration needed. I'm mostly concerned with PHP and Python, and looking into official docs it seems that I'm covered there. How to use it in practice I'll explore in the next post. From what little I know about it, it seems that the biggest difference from previous tooling is logging, because now it has to be collected by OTel.
This is part 1 of the OpenTelemetry and Observability series.
Tags: OpenTelemetry, tracing, observability, monitoring, distributed systems, logging, metrics