TL;DR: right now OTel Logs are a little harder to use than span events, but otherwise they are superior. Today I wanted to send OTLP Logs and see them correlated with my traces. First problem: logs aren’t available in the OTel JavaScript SDK. They’re still in development. (It is the end of 2023.) There is … Read moreSending OTLP Logs from JavaScript
When I make a website, even a toy one, I want to get telemetry. Has anyone visited it? Did my JavaScript crash? Little things. I want to send this with OpenTelemetry, because it’s the industry standard and all. Currently (November 2023), that requires a whole JavaScript build chain to package up the libraries etc etc. … Read moreOpenTelemetry in a script tag
TL;DR: skip to this readme for the easiest path When you’re running applications in Kubernetes, and you’re using OpenTelemetry for observability, then one recommended practice is to run the OpenTelemetry Collector on each node in your cluster. Then each of your applications can send telemetry data (traces, metrics, logs) to that local collector. To run … Read moreTesting an OpenTelemetry Collector deployed as a Daemonset in Kubernetes
Today we wrote some code, told it to send a trace to Honeycomb, and it didn’t. No traces, no error message, why??? This happened to us on AWS Lambda, and it could happen other places too. These debugging techniques can be helpful anywhere you’re struggling with OpenTelemetry. Here I’ll demonstrate them in a tiny Node … Read moreWhy no traces? Traceflags 00 edition
My objective is to troubleshoot 502 errors in my load balancer. The first step is to set up access logs. That says to create an S3 bucket, create a policy for it, and then configure the load balancer. I refuse to do this in the console; I’m using terraform to create resources and kubernetes to … Read moreSetting up ALB Access Logs in EKS using Terraform
Why is this so hard? To send gRPC messages in Rust, one uses the tonic library. If you’re sending to an https endpoint, you’ll need to provide a ClientTlsConfig. By default, you’ll get a very distrustful one. Here’s the error message: the grpc server returns error (The service is currently unavailable): , detailed error message: … Read moreMake https work on gRPC in Rust: load a root certificate into the TLS config
TL;DR: Run the OpenTelemetry Collector helm chart with –values=values.yaml pointing to the values.yaml in this gist. Use kubectl get services to find the collector’s URL. The rest of this blog post walks through the whole process of setting this up. This will teach you the why and how of each step, so that you can adjust it to your needs. There’s also … Read moreRun the OpenTelemetry Collector in Kubernetes for Front-End Tracing
This is an experience report for my future reference (and yours). The OpenTelemetry Collector is useful for receiving trace data in whatever format and exporting it to the back-end of your choice for storage and querying. For instance, I wanted to receive traces over HTTP/JSON from this sneaky browser extensionLINK and send them to Honeycomb. … Read moreRun an OpenTelemetry Collector locally in Docker