Long running spans are one of my biggest “we don’t really actually have a good standard solution for this” issues in opentelemetry. They’re something I’ve run into before, weirdly frequently, and especially so when attempting to try and instrument front-end or mobile facing systems.It turns out, though, that the issues here with long running...
When going on an observability journey, there tends to be a few concrete phases that every company goes through. Understanding how those unfold and take shape as you mature your observability practices can help you identify when you’ll run into certain types of challenges, and when you’ll start really wanting certain tools and practices to help...
Caching is hard. So hard. But also, we are so fucking bad at it. Every time I have to use a public wifi setup I have a joker moment. Does absolutely nobody test shit on anything less than wired symmetric gigabit anymore?Web SPA apps are some of the worst for this. Motherfucker, you have the same fucking iconography for three years, why does it...
I like to home roll abstractions, but commoditize implementations.What I mean by that is a fairly simple rule that has a very powerful effect, but can be tricky to find the right balance.Home rolling the abstraction, to me, means deeply exploring and fleshing out out an abstraction from whole cloth, whether it be an interface, or a mental model,...
I thought today about why I do what I do.When I was a child, I yearned very deeply to be seen and recognized for who I truly was. I didn’t quite realize the cause of this, or reason. I couldn’t have articulated it had I tried. Nevertheless, it was there.However, though I tried desperately, nobody ever saw me or recognized me, and so I grew up...
One of the hardest things for people to understand with distributed systems is that eventual consistency is the same thing as eventual inconsistency. The very same pattern that lets you non atomically deal with things also ensures that eventually you’ll have a system that doesn’t match your understanding. Resources will go stale, things will go...
I am normally nuanced and full of empathy for differing perspectives; I try very hard to present things in a balanced way and to steer away from inflamatory nonsense that doesn’t actually result in making things better. However, this is a rant, because I am having feelings, and those feelings are going to be feel’d.*snorts line of code* HAVE YOU...
I miss the forums. I miss the forums so much it hurts. I miss when research was about discovery and learning and sharing.I miss when humanity felt like it had hope, when human interaction was plentiful, when genuine connection wasn’t rarer than gold.I miss the days before our souls were destroyed for the sake of the market, before our knowledge...
If there’s one thing I wish I could burn entirely to the ground and wipe away all traces and remnants of, its the misplaced notion that the productivity of Knowledge Work can be managed, measured, analyzed, and optimized as if all one needed to do was drip feed heroin up the arse of their hapless workers.What is Knowledge Work™, you ask? There’s...
Observability is a bit of a hot topic, and while it’s increasingly been playing a larger role in engineering strategy, I think the way it’s presented can often cause a lot of leaders to miss the value or to over-index on the wrong things. I’m going to present the current definitions of observability that are widely used in engineering and other...
Something that I find missing in almost every software company is this thing that I’m not sure I’ve seen explicitly called out anywhere, but I’m going to call it an Engineering Language. This Engineering Language is something that I’m going to attempt to describe, motivate, outline, and then illustrate with an example.Engineering LanguageThe...
I read this post from John Cutler and Tom Kerwin recently on how leaders navigate uncertainty and ambiguity and it intrigued me. I decided to give my shot at answering these as a writing exercise and as an opportunity for self reflection. The past few quarters have seen a lot of change for me, and haven’t taken the time I need to reflect as much...
I am normally nuanced and full of empathy for differing perspectives; I try very hard to present things in a balanced way and to steer away from inflamatory nonsense that doesn’t actually result in making things better. However, this is a rant, because I am having feelings, and those feelings are going to be feel’d.Infrastructure as code is great...
I am normally nuanced and full of empathy for differing perspectives; I try very hard to present things in a balanced way and to steer away from inflamatory nonsense that doesn’t actually result in making things better. However, this is a rant, because I am having feelings, and those feelings are going to be feel’d.“If your service can’t be run...
I find it fascinating that we talk about overengineering as if it’s a thing in software engineering. It’s not, cut it out, stop talking about that.If I’m a construction contractor and I overengineer something, guess what? It works for that use case and all the future ones. The only thing I wasted was money and time. Did you make a 10 ton capacity...
There’s this thing about group work which is really interesting and somewhat hard to quantify, but it’s something I’ve been spending a lot of effort fixing in my team lately.It’s the fact that there’s this curious thing that happens to a teams output when they get too overloaded: it goes to zero. You might think they could continue to make some...
When starting a new job as a software engineer, it’s natural to feel the pressure of delivering immediate value and meeting the expectations of your role. However, there’s a unique opportunity during this initial period that often goes unnoticed: nobody expects you to actually do useful work right away. So not only can you can feel free to...
I see you want to hire a developer to work on internal developer tooling, developer experience, and the generally intangible but admirable goal of “making life better for devs”. That’s awesome; you’ve got one hell of a challenge ahead of you. This role is extremely difficult to hire for. In my opinion, and in my experience, it’s been the most...
So the big thing that makes everything so difficult for browsers is that opentelemetry has a concept of a lifecycle for telemetry that doesn’t map very well to how you ergonomically propagate context and correlate traces together. Opentelemetry works super super well in cases where you have a very linear callstack that’s fully synchronous in...
I was asked recently for my thoughts on a wonderful article about software correctness, human convenience, and flossing, and I ended up dumping out an entire blog post worth of thoughts. So, this blog post serves as both a reminder to myself to write more, and also a sincere apology to my wonderfully patient friend, Kelly, who graciously puts up...
Y’all ready for a story about one of the wildest fuckups production outages I ever took part in? Buckle up; we’re going for a ride far, far away from any security cameras.Setting the SceneAt a previous job we had some fairly intense mismanagement. No tech debt was ever allowed to be handled. No good deed was ever unpunished. No non-white-male...
This blog post will be kept up to date as I find out more information and publish my findings. It’s currently organized in no particular order, as a collection of several fragmented thoughts.NginxNginx config for object storageThe nginx config used to proxy to an object storage with a cacheyou will have to tune nginx by increasing its...