Running Dask in Databricks

I think I’ve heard people say data is the new gold at least twenty times. Having worked in data for a while, I’m pretty sure what they mean by that is that extracting and processing is a laborious and often violent process. It’s possible that my take is inspired by working as a data engineer, and therefore being the victim of other people’s data...

I’ve been working hard on a new project called Wimsey lately. I’ll save writing about the project itself for another day, but it got me thinking about github actions (or really any CICD workflow tool). And crucially, how my process for setting up a new CICD flow always winds up being: Write a nice clean yaml config Feel like everything is very...

I don’t keep it much of a secret- I love functional programming. Or maybe I’m just burned form spending hours of my life chasing back inheritance to see where an object variable was defined. Either was, when I saw that hy lang v1.0 was release the other day, I was pretty kean to try it out! One of the downsides of new or more experimental...

Especially in the dark realm of data engineering, there’s a huge range of neat low-code/no-code UI tools. I don’t want to complain about those today, but I do want to talk about why libraries (as opposed to low-code UI) are really awesome. Low code is good code! One thing that I think get’s missed out, is that low-code can still be code. Plotly...

I don’t know where abouts we sit on the wave of yaml-domain-specific-languages. I really hope it’s the peak, and that things will simmer down. I like yaml a lot as a configuration language, but every time I was to work in a domain specific language pretending to be configuration, it sends chills down my spine. A big part of it is that github...