Preface

This post will be short. I think. Because there's not much to say about setting up a Symfony app. I'll just use tech stack I'm using in my day job as a reference and because I want to play with it for my actual code instrumentation project there: Symfony framework, monolog bundle for logs, messenger component for async messaging, and doctrine for database queries. Plan is to have these 4 things done:

1. create Symfony app accessible from outside of the cluster
2. have that app use messaging to send async messages (via redis) to another Symfony app inside the cluster
3. setup database and have the app query it
4. have grafana setup with prometheus for metrics, loki for logs, and tempo for traces

So let's get started.

Creating Symfony app

This is one pretty straightforward. For that I just need some k8s deployments to host PHP app behind nginx proxy. Then wrap it as a service and expose it to the outside via ingress. Not to mention updating /etc/hosts file to resolve the domain name to the cluster IP. Then it's just a matter of running some commands to create Symfony app, composer require some packages, and configure the app to use them. While I'm at it I can play a bit on how simple API could be implemented without much dependencies - no API Platform, no other in my opinion bloatware. Just plain Symfony and a bit of boilerplate code.

Next thing is to have redis and database setup. And that also is done by installing some of composer packages and playing with config. Not to forget to have redis and MySQL services up and running in the cluster.

Observability backend

Initially I thought that this part will be pain in the ass. But it turned out to be quite simple and I would say easy thing to achieve. Who would have guessed that microk8s has already everything prepared. And by prepared I mean it has microk8s enable observability. Only thing I had to do is to define ingress controller to expose grafana dashboard via pretty URL (again, not to forget to add that to /etc/hosts).

Now I need to send logs to loki, and maybe install OTel collector since I'm playing with k8s configs. To collect logs seems like promtail is the agent of choice when working with Grafana and loki. And it also seems like having single promtail pod running results in collecting logs from all pods (from their stdout). Just need to tweak monolog and nginx configs to spit logs to stdout and I'm covered with logs.

Onto OTel agent. Trying config found in official OpenTelemetry docs and otel-collector service is created. I'm going to tweak it to send metrics and traces to endpoints conveniently defined in Grafana admin page. And the assumption is that it will just work had to be thrown out of the window. I wasted couple of evenings trying to figure out how to send metrics to prometheus from otel-collector, and boy was I in for a ride. To start it off, I had to figure out how to enable endpoint scraping in prometheus. 4 evenings and half of a weekend later I still have no idea how I managed to achieve it. But finally managed to curl metrics from application to otel-collector and it seemed to be working. Thankfully, I had much more luck with sending traces - worked like a charm from the start.

Results

So after all this work and brain melting I have a Symfony app that uses messenger, database, and calls other service, which covers basically all my use cases to instrument. Then I have Grafana running with loki, prometheus, and tempo as observability backends. And I have OTel collector running as a k8s service to connect all that. So I think I'm good to start actual playing with code instrumentation helped by (and inspired by) OpenTelemetry tools.