Get started with OTel Logs for Python apps on K8s and learn how to auto-instrument logs via auto-instrumentation with the OTel Operator

Adri Villela

·

Follow

Published in

Cloud Native Daily

·

6 min read

·

Aug 18, 2023

--

Listen

Share

Do you want to get started with OpenTelemetry (OTel) Logs in a Python app running on Kubernetes, but aren’t sure how to go about it? What if I told you that it’s not as scary as you think and that you can leverage the OTel Operator for OTel Logs auto-instrumentation?

Then you, my friend, have come to the right place! In this post, you will:

• Learn how to quickly get started with OTel Logs in Python
• Auto-instrument logs via auto-instrumentation with the OTel Operator

Python Logs Auto-Instrumentation

If you’re looking for all the source files in this example, you can find them here. It also includes the OTel Collector Custom Resource (CR) config.

Assumptions

Before we move on, I am assuming that you have a basic understanding of:

• Kubernetes. You can check out my Kubernetes blog posts here and here for a wee refresher if you need them.
• Python. This is a Python-centric example, after all. 🙃
• The OTel Operator. If not, check out my blog posts here and here. The OTel docs are also a great resource.

⚠️ NOTE: This is not a full-fledged tutorial on auto-instrumentation with the OTel Operator. I’ll be including code, but won’t be including end-to-end instructions on deploying the various pieces. I’m assuming that you know your way around Python and Kubernetes here. 🙃

The Code

Let’s start with the Python code:

If you’ve followed my writings on OTel with Python before, the example above may be familiar to you. It’s a simple Python Flask server app that rolls a virtual die and outputs a number when the user requests the /rolldice endpoint at http://localhost:8082/rolldice.

The code emits Traces (lines 4, 25–28, and 62), Metrics (lines 4, 32, and 65–66), and Logs (lines 5–13, 30, 37–57, and 69).

If you look more closely at the Logs code, you’ll notice that it’s not using any OTel Python logging libraries. It’s actually using the Python logging library. Say wut?

Unlike Traces and Metrics, there is no equivalent Logs API. There is only an SDK. 🙃 I know, right? I was pretty surprised too. The idea is that you can use whatever logger you want for your language (for Python, it’s the logger library), and then the OTel SDK attaches an OTLP handler to the root logger, basically turning your log library’s logs into OTLP logs. 🪄✨

⚠️ NOTE: There is a logs bridge API; however, it is different from the Traces and Metrics API, because it’s not used by application developers to create logs. Instead, they would use this bridge API to setup log appenders in the standard language-specific logging libraries. More information can be found here.

But wait. There’s no OTel Logs SDK in this code either. We’re just using the Python logging library. We’re not specifically attaching the OTLP handler to the root logger, so like…did we give up on OTel logs?

Nope! See, this is where auto-instrumentation comes in! Yes, there is auto-instrumentation support for Python logs! Yaaaay!

So now you might be wondering…where does that happen, because it’s definitely not happening anywhere in the above code, is it?

Well, it turns out that there’s a special OTel Python auto-instrumentation attribute called OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED, and if we set it to true, the logging magic described above happens.

Awesome, awesome…but where do we set it to true? Well, since we’re doing auto-instrumentation via the OTel Operator, the config goes in the Operator’s Instrumentation CR. Like this:

Check out lines 19–20 above:

- name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
  value: "true"

As you may recall from my last piece on Auto-Instrumentation with the OTel Operator, there’s a special section for including language-specific configs just like this. For Python, it’s in the spec.python.env section of the Instrumentation CR.

But wait…there’s more! You may recall that Python auto-instrumentation only works with HTTP, and not gRPC, so we also need to set another attribute, called OTEL_LOGS_EXPORTER, to make sure that we’re using HTTP, and not gRPC for Logs, otherwise, it no workie:

- name: OTEL_LOGS_EXPORTER
  value: otlp_proto_http

And finally, we need to make sure that our Deployment YAML includes the special auto-instrumentation annotation, so that the Operator can inject auto-instrumentation into the Pod:

This is accomplished by lines 20 and 21:

annotations:
    instrumentation.opentelemetry.io/inject-python: "true"

Remember that this annotation must go to go under spec.template.metadata, and NOT under the main metadata section. If you try to put it in the main metadata section, it no workie. Believe me, I’ve made that mistake a zillion times, and it’s annoying AF when you’re scratching your head wondering why the annotation is “there”, but auto-instrumentation ain’t working.

When you deploy python-server.yaml (includes Deployment and Service definitions), otel-collector.yaml, and instrumentation.yaml to Kubernetes, you should something along these lines in your OTel Collector’s output:

ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope opentelemetry.sdk._logs._internal 
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-08-17 16:29:23.971459584 +0000 UTC
SeverityText: ERROR
SeverityNumber: Error(17)
Body: Str(Derp! Toronto, we have a major problem.)
Attributes:
     -> otelSpanID: Str(73f9eb4977391ab2)
     -> otelTraceID: Str(885e9ce5faf1009fdc37eaee8ff7e660)
     -> otelTraceSampled: Bool(true)
     -> otelServiceName: Str(py-otel-server)

Logs auto-instrumentation sans Operator

Now, what if you wanted to do Python auto-instrumentation without the Operator? Like, if you were doing dev on your desktop. What would that look like? The Pythonopentelemetry-instrument agent is used for Python auto-instrumentation. The agent configuration would look something like this:

# Set the logs auto-instrumentation flag
export OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true

# Start the agent
opentelemetry-instrument \
    --traces_exporter otlp \
    --metrics_exporter otlp \
    --logs_exporter otlp \
    --service_name server-py \
    python server.py

Notice how we set the OTel_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED environment variable, which tells the agent to go ahead and use Python logs auto-instrumentation (it’s disabled by default).

⚠️ NOTE: The above config setup assumes that telemetry signals are being sent to a Collector running on http://localhost:4317 (gRPC port) — the default. For more on the auto-instrumentation, check out this blog post.

Final thoughts

OTel Logging in Python may seem intimidating a first glance, but it’s not quite so scary once you understand what’s up. And you’ve got to admit that the Logs auto-instrumentation bit is pretty damn cool and simplifies things a TON! Simple is always a win in my book!

I hope y’all learned something new and cool with this! There’s obviously a LOT more to dig into on this topic, but hopefully this gives you enough of a starting point for Python Logs auto-instrumentation with the OTel Operator. If you’d like to learn more about the OTel Operator, you should check out #otel-operator channel in the CNCF Slack. The folks on there are super helpful and responsive.

Massive thanks to my awesome colleague, Alex Boten, for teaching me the ways of Python Logs auto-instrumentation.

In the interest of making sure that this information makes it into the actual OTel docs, I also took the liberty of creating a PR with this info, which has been merged. 🎉

Now, please enjoy this rare photo of my rat Mookie, who is usually moving too fast to snap a good pic of her!

Until next time, peace, love, and code! ✌️💜👩‍💻

Want to learn more about OpenTelemetry? Check out my other OTel content here:

Adri Villela

OpenTelemetry

View list

19 stories

Further Reading: