I've been thinking a lot about this in preparation for the next I've been thinking a lot about this in preparation for the next HYTRADBOIHYTRADBOI.. My experience of online conferences has mostly been underwhelming. They typically borrow the form and structure of an in-person conference without considering whether those still make sense online, and whether the goals of an online conference should even be the same as an in-person conference.My experience of online conferences has mostly been underwhelming. They typically borrow the form and structure of an in-person conference without considering whether those still make sense online, and whether the goals of an online conference should even be the same as an in-person conference. The most important function of...The most important function of...
It's easy to think of being serious and having fun as opposite sides of a spectrum. The problem is that 'being serious' has many unrelated meanings, for example:It's easy to think of being serious and having fun as opposite sides of a spectrum. The problem is that 'being serious' has many unrelated meanings, for example: Serious as in somber or solemn. "This is a serious event, stop playing around."Serious as in somber or solemn. "This is a serious event, stop playing around." Serious as in actually trying to attain your goals, as opposed to just going through the motions. "This is a serious effort."Serious as in actually trying to attain your goals, as opposed to just going through the motions. "This is a serious effort." Only the first meaning is actually opposed to fun. Fun/playful vs...Only the first meaning is actually opposed to fun. Fun/playful vs...
hytradboi 2025
HYTRADBOI is coming back in 2025, this time with a programming languages track.
I have 14 speakers confirmed so far, but there is still plenty of room. Let me know who you want to see!
(This is part of a series on the design of a language. See the list of posts here.) Many 'scripting' languages use a hashmap for their default associative data-structure (javascript objects, python dicts, etc). Hashtables have a lot of annoying properties: Vulnerable to hash flooding....
2025 Feb 28. Put it in your calendar. It's been three years since HYTRADBOI. Long enough that I've mostly forgotten how stressful it was to run a conference and it's starting to seem like a good idea again. talks The format will stay the same. 10 minute, pre-recorded, heavily-edited talks. Asynchronous chat. Join from a different time-zone, watch talks on your lunch break, answer questions...
zest progress I've started working on the runtime. Many of the features of zest are going to be implemented by the runtime rather than by the compiler, but the runtime is itself written in zest. I'm slowly unpicking the dependency graph of features to make that work, so the last month saw a lot of tiny changes: Added a != operator. I somehow forgot it earlier. Added support for strings and string...
I finally got the zest compiler architecture and codegen more or less settled. Lot's of details in baby's second wasm compiler. I'm gonna take a two week break and then move on to writing the self-hosted runtime. zig honggfuzz Zig doesn't have support for coverage-guided...
(This is part of a series on the design of a language. See the list of posts here.) The zest compiler today is ~4500 loc with no dependencies except the zig standard library. It generates wasm of similar quality to an llvm at -O0, although I have some ideas to try later that I hope will push the output closer towards -O1. There are still some low-hanging...
(This is part of a series on the design of a language. See the list of posts here.) I have two goals for zest that are in tension: Be a reasonably efficient imperative language (eg go, julia). Treat values as data (eg erlang, clojure, see the shape of data). In...
Minimal writing this month: Zest syntax But maximum coding! zest progress Last month I mentioned that I had a good chunk of the language working in a single-pass compiler in destination-passing style, but it was becoming increasingly...
(This is part of a series on the design of a language. See the list of posts here.) Popular advice for designing a language is to focus on semantics and worry about syntax later. So it might seem ill-advised to write about syntax before writing about semantics. But a) I think syntax design is underrated - it has a huge impact on the subjective feel of working with a language, and b) it's hard...
I wrote a lot this month.
More unexplanations:
Relational algebra is math
Sql is syntactic sugar for relational algebra
Some musings about small software businesses:
(This is part of a series on the design of a language. See the list of posts here.) zig (Based on reading code.) Zig has a bunch of IRs: Ast. Basic parse tree. Potentially contains parse errors.
a better sql As much as I complain about sql, it's historically been too entrenched to be worth trying to compete. Building a database is really hard, and adopting a new database is really risky, so both vendors and customers have strong incentives to be risk-averse. ORMs, BI tools, database vizualizers etc all add to the incredibly strong network effects. The
Suppose that all your expertise is in making software for use by other programmers, and you want to run a small business. How do you make money? The weird thing about being a system engineer is that it's quite easy to make something that is valuable and widely used. But quite a lot harder to get paid for it. It's easy to wake up as this brick:
This idea is particularly sticky because it was more or less true 50 years ago, and it's a passable mental model to use when learning sql. But it's an inadequate mental model for building new sql frontends, designing new query languages, or writing tools likes ORMs that abstract over sql. Before we get into that, we first have to figure out what 'syntactic sugar' means. Wikipedia...
I see this claim appear in various forms: relational algebra is math, is based on math, comes from math, has strong mathematical foundations. It's a statement that I struggle to assign any precise meaning to, but typically the intent seems to be something like: Relational algebra contains some sort of fundamental truth - discovered rather than invented. Relational algebra is powerful because of it's special math sauce. Relational databases...
(This is part of a series on the design of a language. See the list of posts here.) I now have much of the core language sketched out and (poorly) implemented, including mutable value semantics, control-flow-capturing closures and a tentative solution for error-handling, but I'll cover those another day. Today I want to talk about the three dialects I'm...
zest progress I'm continuing to chip away at zest. Some new posts: Notation and representation Dialects and metaprogramming I'm really...
Here is a simple sql query. select users.id, ( select sum(posts.likes) from posts where posts.user_id =...
(This is part of a series on the design of a language. See the list of posts here.) I want to be able to describe all data with a consistent notation when writing code, printing values or in graphical inspectors (see the shape of data). On top of this notation, I have two goals that are in...
Kind of a fragmented year. tigerbeetle I left in May. I learned a lot and I'm proud of the work I did. But I don't like the culture that surrounds startups, and that discomfort turns to burnout over time. 7am meetings probably didn't help either, putting me well out of sync with my wife who works evening shifts. I spent very little of that time writing about what I learned, other than scattered
Here's my yearly log entry for 2023. So you wanna de-bog yourself A list of ways to get stuck. It's nice to have labels for them, so they're easier to recognize and talk about. Terrible situations, once exited, often become funny...
consulting lessons I ended my consulting gig not long after starting, when it became obvious that we had very different ideas about how to write high-performance software. I guess I can add this to my list of things learned about consulting. It's not enough to be clear on what I'm being hired to produce - we also need to make sure we're on the same page as to what producing that output will actually look like. I'll have to think about how...
(This is part of a series on the design of a language. See the list of posts here.) Long ago Steve Yegge wrote about software that feels alive - emacs, smalltalk, lisp machines etc - and lamented that the industry prefers to create dead things. Puppets on strings, not real boys. I'm...
I'm currently doing some consulting, adding a vectorized query engine to a project written in go. I wrote a thing - columnar kernels in go - about some prep work I did before starting, figuring out how to translate familiar idioms into this foreign (to me) language. go grouches So far I'm extremely unimpressed with go. It disallows using...
Over the winter I'm going to be adding a columnar query engine to an existing system written in go. I'm not at all familiar with go, and it's also not ideally suited to this kind of problem, so I started out with a little toy problem that has similar challenges to help puzzle out the best strategy for the kernels. The toy problem is a simple columnar compression library. (It was intended to be a reimplementation of
I published a shallow survey of OLAP and HTAP query engines. The last 2/3rds or so of this post contains all the supporting notes. Also a lot of papers on strategies for low-latency compilation. lobster A surprisingly neat little language, exploring a lot of ideas...
I had to put this on a different page. It felt too surreal for it to be mixed in with a bunch of technical notes. Genocide requires coordination. You can't whip people up into a frenzy if you can't reach them. Thanks to heavy data subsidies most people in Myanamr access the internet exclusively through facebook, which has replaced radio and printed news. So the mass genocide has been organized mostly through facebook. As the violence in Mandalay...
Focused mostly on data layout and query execution. Query planning seems more or less the same as OLTP systems, and I'm ignoring distribution and transactions for now. Also see my full notes here. It was hard to figure out what systems are even worth studying. There is so much money in this space. Search results are polluted with barely concealed advertising (eg "How to choose between FooDB and...
I wrote up some of the compiler stuff I've been noodling on for the last month or so: Implementing interactive languages Baby's first wasm compiler I also fleshed out my
(This is part of a series on the design of a language. See the list of posts here.) I made a compiler from a toy language to wasm. The code quality is very much plan-to-throw-one-away. I'm just trying to get a feel for the amount of effort involved in a non-optimizing compiler. The toy language is mostly unsurprising. It has number, strings, maps (hashtables) and first-class functions,...
(This is part of a series on the design of a language. See the list of posts here.) Suppose I want to implement an interactive language - one where code is often run immediately after writing. Think scientific computing, database queries, system shells etc. So we care about both compile-time and run-time performance because we'll usually experience their sum. A traditional switch-based...
I'm putting aside around four months this year for consulting. Likely late October to end of March, with a gap in December. Here are some ideas for projects that work well for short engagements: Isolated or research-heavy features. Eg adding decorrelation to materialize took ~6 weeks and didn't require changing much existing...
cheap compilation? The most popular options for implementing programming languages and query languages are: Write a bytecode interpreter and try to amortize the runtime overhead through vectorized operations (eg python calling into c libraries or duckdb's vector operations). Use LLVM and suffer the slow compilation (and the...
dynamic mutable value semantics I worked through a simple implementation of mutable value semantics here (tree-walking interpreter, no optimizations). The main difference from swiftlet/val is that it's a dynamic language - demonstrating that nothing about MVS requires static typing. The swiftlet paper goes into some...
Taking it easy this month. typescript I'm trying to clean up the mess of bash/python/clojure/julia/rust scripts/apps/glue that I've accumulated. Javascript seems like the most obvious candidate to consolidate on, with fast startup, big ecosystem and gradual typing. I went through most of the javascript and typescript courses on Execute Program. The interface is slick and the...
I left tigerbeetle this week. They didn't do anything wrong, far from it. I'm just not well suited to long-term employment. I should know that by now, but I convinced myself that this time would be different. At the beginning I was talking about a 3-month trial contract, knowing the risk, and then I just got overexcited and dove in head-first. Before taking this job I was approaching critical mass - between hytradboi, github sponsors, a grant from emergent ventures...
tigerbeetle perf handover compaction unchained crash harder sketching the query engine focus catchup android update policies legopunk reading a world without email nobody cares about our concurrency control research segcache bloomRF existential consistency parameter-aware io management for ssds fantastic ssd...
February was really broken up by Systems Distributed so not as much coding or papers to talk about as I'd like. The talks were all recorded but I don't know when they'll be published yet. tigerbeetle table sizing bench harder wasm first steps sycl vancouver reading breathing for warriors move your dna the molecule of...
at tigerbeetle undroppable tombstones forest fuzzer manifest race hash_log musing zig coercions zig pointer hops zig object notation domain knowledge reading built from broken database internals papers... I restarted my habit of reading a paper every...
(See other years). Office hours From January to March I ran office hours every friday morning. I wrote at the time: I'm trying to virtually recreate the cafe/salon culture I loved before moving to a tech backwater during an epidemic. I ended up talking to 35 people. Most of the time it was awkward and exhausting, much more so than...
I gave up on twitter, so this log is now only posted via atom or email. I wrote a retrospective for 2022. December things: systems distributed at
Things I've been doing in November: handmade seattle! at tigerbeetle hunting performance bugs in the lsm tree adding tracing evacuating preimp musing colorblind concurrency reading database...
I'll be in San Francisco Nov 6-11, and in Seattle for Handmade Nov 15-19. tigerbeetle In October I joined the database team at TigerBeetle. Like most of my major life decisions, this was made somewhat impulsively. There wasn't one big reason, rather many nudges that all happened to peak at the same time: I'm...
HYTRADBOI jam
The jam results are now up at hytradboi.com/jam. I haven't had a chance to read through all of them yet but it looks like there were some neat projects.
sqllogictest in a week
For my own jam project, I tried to make a sql frontend that could pass
I keep seeing discussions that equate zig's level of memory safety with c (or occasionally with rust!). Neither is particularly accurate. This is an attempt at a more detailed breakdown. This article is limited to memory safety. See Assorted thoughts on zig and rust for a more general comparison. I'm concerned mostly with security. In practice, it doesn't...
Published 2022-08-31 A minor update that can't wait until the end of the month: Have you tried rubbing a database on it: The Jam An upcoming coding jam with the same theme as the HYTRADBOI conference: Turning a data-centric lens onto familiar problems to yield...
I got back to work Aug 8. The rest of the month has been blessedly uneventful. preimp I got as far as I could with the clj version. The main obstacle is provenance. The main feedback from the essay was that it would feel much more natural with...
break Nobody could come to our covid-era wedding, so instead we have a wave of family and friends visiting for our first anniversary. Which means that between July 16 - Aug 7 I'm barely computering at all. Progress will resume Aug 8. preimp I wrote a draft of an essay explaining preimp: the program is the database is...
Preimp I've made a lot of progress on preimp. Persistence, server/client sync and collaborative editing are all working. Values are nicely rendered as tables. Functions are rendered as forms, which you can fill out to call the function. Functions can call edit! to change the value of data cells. Metadata can be used to tweak the rendering of values. Together these features...
All the HYTRADBOI videos are now up at hytradboi.com (click on the talk titles). I also wrote a post-mortem. I'm still not sure whether I want to do another HYTRADBOI, since it took a huge chunk of energy and attention out of my year. But one option I'm considering is to aim for dramatically less polish:...
Genesis One morning I drank too much coffee, got over-excited and on a whim proposed a conference - Have you tried rubbing a database on it?. Would you watch a "have you tried rubbing a database on it" conference?Thinking short demo videos of weird and non-traditional uses of database ideas eg writing a dvcs in sql (
I made a HYTRADBOI teaser trailer. As I write we're at 293 attendees. All the infrastructure is wired up and my todo list is mercifully clear for these last few days. I sat down briefly with Marco Munizaga to look at using dida in
(This is part of a series on the design of a language. See the list of posts here.) Some assorted musings on how data is represented in programming systems, with no clear thesis. Mostly focused on how application-level data is represented, manipulated, viewed, stored and transmitted, as opposed to how new types and data-structures are implemented within a single language. Also...
Continuing with the 'the program is the database' theme from last month, I made a clojurescript dialect (called preimp) where the only source of mutable state is the source code itself. It works well enough for spreadsheet-sized datasets: I have much more interesting interactions planned. Hopefully for next month. Doing this in clojurescript...
HYTRADBOI I just published the schedule and you can now buy tickets! Imp In imp v2, the database was accessible as a value. You could write code that described a transaction and then use a keyboard shortcut in the editor to apply...
HYTRADBOI Many new talks have been added to the HYTRADBOI schedule. There are another 15 or so in the pipeline. It's starting to look pretty exciting. Milestones I've been thinking lately about the psychological need for a sense of completion, or at least milestones. In industrial jobs my work has typically been organized around discrete...
I'm going to make this newsletter public. Reasons: As best I can tell, most of my sponsors don't much care about the newsletter itself and just want to support writing/coding. Some don't even know it exists because github kinda buries the checkbox. I'm trying to orient my own consumption towards deliberate explorations of specific subjects rather than the froth of daily news. It seems hypocritical to then be offering infotainment as an...
Writing: I finished the reflections series: Coding Testing Writing Why...
This is a part of a series - start at the beginning. Unlike sql, the denotational semantics for imp are carefully written to allow a set to contain rows of different lengths. The reason for that is that we can use such sets to model structs, tagged unions, first-class modules etc. // defining a module my_module: | "inclinations",...
(See other years). I felt like I didn't do much this year, but now that I write it all down... Streaming/incremental streaming-consistency: 94 commits, 2952 insertions(+), 36 deletions(-) I wanted to understand this whole space better. The goal was to produce the high-level view that appears in
This post is part of a series, starting at Reflections on a decade of coding. This is going to be much more vague than the other parts of the series because this is the actual work. Good judgement is learned from experience, not from blog posts. So I think the most useful thing I can convey is what kinds of things I think about when coding, rather than what...
Somehow I am organizing a conference called Have you tried rubbing a database on it? This isn't what I planned to do this year. How did this happen? Inception The answer is that I drank too much coffee before breakfast and ended up on twitter, where I saw someone joking that the solution to thinking that everything is a compiler problem is to learn about databases - now you have two...
HYTRADBOI now has 10 confirmed speakers and a couple more maybes. The submissions will stay open until 2022 Feb 28 - if there is someone you would like to see speak, send them to hytradboi.com to submit a talk. Also, if anyone can intro me to someone involved with
This post is part of a series, starting at Reflections on a decade of coding. This will be a short entry because I agree with almost everything written here. Read that first. I think about tests in terms of
This post is part of a series, starting at Reflections on a decade of coding. I have a file called 'ideas' where I write down potential projects or thoughts that might be worth writing about. Entries grow over time as I add more thoughts. The entry that eventually became Against...
New stuff:
In dida:
I added a global validation routine that runs on every debug event. So far it detects any unintended aliasing. I'll test other invariants as needed.
I wrote a debugger (video,
This post is part of a series, starting at Reflections on a decade of coding. If you can focus hard on something for at least a few hours per day for several years, you will get good at it. The main obstacle to doing so is being a poorly-designed meat...
This post is part of a series, starting at Reflections on a decade of coding. I don't think I'm very fast in an absolute sense, but I'm much much faster than I was 5 years ago. These are the things that I think made the most impact. Care The main thing that helped is actually wanting to be faster. Early on I...
This post is part of a series, starting at Reflections on a decade of coding. I think that one of the most important things to focus on improving is how fast you can work. I wrote strucjure in 2012 and
9 months of being paid to make things by strangers on the internet! Here is what I did in the last quarter: In imp: Added a live repl (demo video here) Redesigned the syntax to be much more concise (before and...
New stuff: I drastically changed the internal representations in imp. This was setup work for decorrelation of higher-order functions and fix/reduce, but as a side effect it also made closures much smaller and fixed some nasty edge cases in type inference...
In every explanation of parsing that I've seen, operator precedence is handled by assigning a precedence number to each operator. Operators with higher precedence numbers bind tighter eg a + b * c would parse as a + (b * c) because * has a higher precedence than +. We want this because it reduces the number of parentheses needed, and we can get away with it because the precedence rules of arithmetic are familiar to most...
This post is part of a series, starting at Reflections on a decade of coding.
When I don't know where I want to go, I usually don't get there.
Setting explicit goals for each project is essential for:
Measuring progress
This post is part of a series, starting at Reflections on a decade of coding.
This is one of my favorite questions to ask people: what are some things that you used to strongly believe but have now changed your mind about?
I want to focus especially on ideas that I wasted a lot of time on, or that got in the way of success.
I've been programming professionally for about 12 years. Here are some of the things I worked on in the last 2 years. In a small team (averaging ~3 people, I think?) in a little under a year, we built a SQL query planner that passed the >6 million tests in the sqlite logic test suite. I figured out how to
This post is part of a series, starting at Reflections on a decade of coding. Like many programmers, I'm largely self-taught. I've rarely worked with anyone more experienced than myself, especially early in my career where I spent a lot of time working with other 20-something-year-olds who also had only a few years of experience. So we all learned about how...
I spent some time thinking about how to benchmark streaming systems. Though I didn't go through with the project, the notes might still be useful for others. What is the point of benchmarking? Decision making, capacity planning. I have X queries with Y data and Z slo - what software do I use and what hardware do I buy? Focus engineering effort on bottlenecks. Inform the design of future systems by understanding design...
So far the editor has to do these things with text: iterate forwards/backwards insert/remove strings lookup the point corresponding to a given line find out the line at a given point The simplest thing we could possibly do is just store the entire text in a contiguous string. How would this perform for a 2mb text on my 2017 xeon laptop? A string is the most efficient choice for iterating - we get a simple loop...
First order of business is to put things on the screen. I chose to render to opengl rather than make a terminal app because a) I rarely need to run an editor over ssh b) it's hard to do smooth scrolling in a terminal app c) terminal protocols are insane. SDL is a library that abstracts over the details of various different windowing and input systems and makes it easy to eg
Over the winter of 2020, during the time where I had planned to be working on a port of differential dataflow, I instead wrote a text editor which I now use for all my writing and coding. This was perhaps not the most productive use of my time, but it wasn't wasted either: It's a huge project to write a text editor that people will actually use, but it turns out to be fairly easy to write a text editor that only one person will use. Far from being a toy...
I wrote a live repl for imp and made a demo video. I think this is one of those things where a quantitive change in speed makes a qualitative difference - in theory this is just a slightly faster way to type things into a repl, but in practice it feels totally different. Pointing at a thing to query it feels very natural. To make the live repl experience work better I made a lot of...
Published 2021-09-04 This is a part of a series - start at the beginning. I wrote a live repl for imp. It's hard to explain interaction in text, so I made a video. Turn...
New stuff: I got married! Great for me, sucks for anyone hoping for more code / blog posts this month :) At some point this month my income from sponsorships passed minimum wage. I'm about 2/3rds of the way towards meeting all my expenses and being able to do this indefinitely. Six months ago that seemed like an impossibly unrealistic goal. Dida updates: The core now has actual memory management. All the tests are running with leak and...
I tried implementing a text crdt in sql and in imp. Doing this naively isn't particularly hard. The challenge here is to implement it as if writing a batch query, but in such a way that an incremental system like materialize or
https://www.goodreads.com/book/show/44770129-ultralearning Core idea is combining everything we know about effective (as opposed to traditional) education to learn skills dramatically faster than is generally believed to be possible. The better one gets, the more one recognizes how much better one could become. Don't need talent to be...
TLDR The relational model is great: A shared universal data model allows cooperation between programs written in many different languages, running on different machines and with different lifespans. Normalization allows updating data without worrying about forgetting to update derived data. Physical data independence allows changing data-structures and query plans without having to change all of your queries. Declarative...
We're coming towards the end of my second quarter on github sponsors. Thanks to everyone who gave money to a stranger on the internet so that they could type more fun things and less boring things. Here is what I did: Released dida, a library for streaming, incremental, iterative, internally-consistent computation on time-varying collections, based on
New stuff: I wrote a detailed breakdown of things that are wrong with sql and how to do better in a new query language. Dida updates: Added a js<->wasm api (example here, sugar...
I made the dida repo public. I still have a lot of work to do but: It works! I came up with a slightly different way of handling progress tracking that I think simplifies the implementation. DD's algorithm encapsulates subgraphs so that they look like single nodes in the parent graph. This requires nodes to be able to have multiple outputs and report complex transformations on timestamps over each...
I like to make live repls that evaluate as you type. Here is an example for a silly little stack language with singleton sets (0 and 1), set union (|), set product (*) and dup (^, makes an extra copy of the top value on the stack). Limit memory usage to 2^ bytes Am I broken? Try changing the numbers and seeing how that affects the...
The main thing I've been working on is a port of differential dataflow. I have most of the basics sketched out with naive implementations and I'm currently working on implementing progress tracking correctly, at which point simple examples like graph reachability should start working. Github doesn't have any way of saying "give all sponsors read-only access to this private repo" short of making a brand new github organization and manually spamming everyone with invites. I...
Many groups are working on running sql queries in incremental/streaming systems. Query planning in this context is not a well understood problem. This post is a quick pointer towards the many open problems. It's not intended to be complete or authoritative, because I haven't engaged with the problem much beyond noticing that it's hard and deciding not...
Hello to everyone who joined recently!
I made a short summary of what I've done so far in the first quarter of 2021 and some ideas for what I'll do next (blog version, twitter version). If any of them are exciting, let me know.
Published 2021-04-22 Today marks the end of my first quarter on github sponsors. Thanks to everyone who made this possible by giving money to some rando on the internet. Here is what I did so far (some already public, some sponsors-only):
Disclaimer: I used to work for Materialize which is one of the systems being compared in this post. This is a space of systems for which we don't have clear terminology yet. The core idea uniting them is this: We have some function that turns an input into an output. The input changes slightly over time. We want to calculate the new output in a way that performs better (lower latency, higher throughput, lower resource usage etc)...
Disclaimer: I used to work for Materialize which is one of the systems being compared in this post. This post covers the internally consistent/inconsistent branch in my map of streaming systems. The core message of this post is that intuitions from distributed key-value databases don't carry over well to systems which do complex...
It's alive! Internal consistency in streaming systems. Tell your friends. Major changes since the last update: A bigger dataset with better delay distribution. A second dataset which keeps the values in balance between -1 and 1. This removes some noise and makes the strange dynamics in flink much...
Yet more internal consistency! I settled on a simple example that stresses most of the failure modes I'm aware of. The final view, total, should always contain a single row with the number 0. This makes it super easy to spot consistency violations. I have
https://www.goodreads.com/book/show/22245334-the-utopia-of-rules I found 'Utopia of Rules' less convincing than his other books, but helps provide context to his other arguments Iron Law of Liberalism - any attempt to make market more free ends up creating more red tape The rise of modern corporations was seen as a matter of applying modern bureaucratic techniques...
https://www.goodreads.com/book/show/6617037-debt 'Debt' is about the origins of money and debt. I found it fascinating, but I'm not sure where to trust it, especially given the clear agenda around tying debt to violence. Barter economies have never been seen (see Caroline Humphrey for more detail) A communalist setting is one in which people share by default It...
https://www.goodreads.com/book/show/38217638-bullshit-jobs Interesting ideas. Probably overstated, but worth reading. Outline of Bullshit Jobs: medieval model - wage labour as a path to adulthood industrial revolution - wage labour forever 70s - scientific management stole productivity increases Medieval model of...
As expected, I didn't have much access to electricity in the first two weeks of march. Things are getting back on track now. I put a bunch more work into the internal consistency article. Rewrote the description of internal consistency and much of the conclusion. After a back and forth with one of the noria authors, I realised...
Kevin Lynagh and I spent some time playing around with zig on nrf52 boards. He's written about the experience here. I wanted to additionally highlight the api we used for memory-mapped IO...
New stuff: The first draft of internal consistency in streaming systems. I still have a few more systems to study, repros to write and authors to badger for fact-checking. Memory-mapped IO registers in zig - a little self-contained case study of api design in...
https://mitpress.mit.edu/books/small-matter-programming An excellent book on end-user programming. A strong counter-argument to the widespread assumption that all we need to do is turn existing language models into pretty point-and-click diagrams. End-user programming is needed because it's impossible to predict in advance what the user will need, and expensive to cover all the...
Subqueries are a SQL feature that allow writing queries nested inside a scalar expression in an outer query. Using subqueries is often the most natural way to express a given problem, but their use is discouraged because most databases struggle to execute them efficiently. This post gives a rough map of existing approaches to optimizing...
http://carlos.bueno.org/optimization/ 'The Mature Optimization Handbook' is about monitoring and profiling continuous systems. Reasonable ideas but not particularly dense. The performance problem definition must be falsifiable Use performance measurements to try to falsify theory, not just confirm it A measurement is a number obtained during some profiling event Metadata are...
New: An opinionated map of incremental and streaming systems. I wanted to write about query planning for streaming systems, but it turned out to be hard to do that without first writing about streaming systems in general. First draft of
https://www.goodreads.com/book/show/40672036-digital-minimalism I found Digital Minimalism compelling, and much better thought out than most similar texts given eg the focus on having a high-quality leisure life. Outline of Digital Minimalism: digital media is advertising-driven makes money by stealing your solitude which you need for...
Published 2021-02-06 New stuff: The dive into optimizing correlated subqueries is finished. Notes on Digital Minimalism. See also older notes on...
Welcome to everyone who signed up since the last update. Thank you for giving me money. I will spend it on sensible things like beans and running water. New posts: The intro section for the post on correlated subqueries. I'm still working on the remaining sections....
https://nayafia.substack.com/p/22-working-in-public Working in Public is a book about how open source projects work. It reminds me strongly of Governing the Commons - rather than rely on ambient mythology, Eghbal actually went and looked at how open source projects work and found that...
So one weekend I drank too much coffee and impulsively decided to start a newsletter. My biggest worry at the time was that it would launch to the sound of crickets. I've now learned that what I should have been afraid of was waking up on Monday morning to a sea of expectant faces and realizing that maybe I should have written some of this stuff in advance... Luckily, I procrastinated all winter by writing a text editor from scratch so I have ready-made filler material....
Differential dataflow is a library that lets you write simple dataflow programs and a) then runs them in parallel and b) efficiently updates the outputs when new inputs arrive. Compared to competition like spark and kafka streams, it can handle more complex...
Continuing the tradition from last year. I moved to Canada in March this year, so these numbers are only for April-December for the sake of my accounting sanity. Numbers are actual/budgeted Canadian dollars per month. total = 2420/2397 basics = 1642/1571 physical bills = 1150/1150 digital bills = 82/121 phone = ?/17 fastmail =...
I'm trying to find a better debugging experience for my current projects (zig/c, nixos). Usually I just use vanilla gdb. Here is a debugging session I ran recently on this linux x64 binary. 1. Open and run until crash. [nix-shell:~/focus]$ gdb ./test ... (gdb) run ... Test [1/1]...
Continuing my search for a better debugging experience for my current projects (zig/c, nixos). The first post resulted in a ton of suggestions and I tried them all. gdb -tui (again) taviso explained how to use the tui properly. C-l fixes the display and using...
Wesley Aptekar-Cassels' recent posts on getting a gold card and moving to Taiwan reminded me that I meant to write about my experience with Canada. I immigrated to Vancouver, BC in March 2020. I was given permanent residency before setting foot in Canada and without a job offer, under the
I've been using zig for ~4 months worth of side projects, including a toy text editor and an interpreter for a relational language. I've written ~10kloc. That's not nearly enough time to form a coherent informed opinion. So instead here is an incoherent assortment of thoughts and experiences, in...
I frequently see debates about whether it's better to be a cog at a giant semi-monopoly, or to take investment money in the hopes of one day growing to be head cog at a giant semi-monopoly. Role models matter. So I made a list of small companies that I admire. Neither giants nor startups - just people making a living writing software on their own terms. sqlite Sqlite is an...
This is a part of a series - start at the beginning. It's hard to be Turing-complete without some form of iteration or recursion. There is a tricky design problem here - I'm trying to steer between two extremes: Programming languages usually allow arbitrary user-defined iteration via loops or recursion, but this introduces false data dependencies that inhibit parallelization and...
This is a part of a series - start at the beginning. Previous posts had fancy interactive examples. Keeping these in sync with constant language changes was taking a lot of time, so I'm abandoning them for now. They'll return when the language has settled enough to write docs. This is the thorniest part of imp. So far we've been dealing with first-order relations - sets of...
This is a part of a series - start at the beginning. Here is some imp code that does a reverse lookup on a table: This is kind of verbose - worse even than the equivalent sql: select name from departments where department = "marketing" This is a problem, because one of the design goals of imp is to make it easy to...
The Zig stdlib often uses open single dispatch eg: // in stdlib pub fn serialize(self: *Serializer, value: var) !void { const T =...
After anki trashed my history, uploaded the trashed history to the sync server, and then repeatedly re-synced the trashed history every time I tried to restore from backup, I wrote my own spaced repetition app. It reads input from a markdown file, asks questions in the terminal and stores state in a json file. I wrote those few hundred lines of code in an afternoon and I've been using it ever since. I wanted to run it on my android phone too, but after several days of...
Working on a postgres-compatible query compiler has taught me many things. Things I was better off not knowing. Let's begin: jamie=# create table nums(a int primary key, b text); CREATE TABLE jamie=# insert into nums(a,b) values (0, 'foo'), (1, 'foo'), (2, 'foo'), (3, 'bar'); INSERT 0 4 jamie=# select a+1 from nums group by a+1; ...
This is a part of a series - start at the beginning. In SQL, when a subquery references a variable from the surrounding query, this is called a correlated subquery. If you were feeling a bit strange, you could use a correlated subquery to express a join. These two queries are identical (as long as bar.b is a primary key): select...
In the last year I started to pay much more attention to how I spend money. I'll write much more about the reasons at some point, but the short version is that I care a lot about autonomy and agency, and spending less is a critical part of that. I set myself a target of £2000/month for 2019. The median take-home salary in London is £2250/month, so this is hardly a stretch, but it's a start. Today I went through my accounts for the year to see how that worked out on...
This is a part of a series - start at the beginning. imp requires javascript! The denotational semantics define what an imp expression means, but it defines the meaning in terms of infinite sets. How can we evaluate those semantics? The little interactive...
This is a part of a series - start at the beginning. Before looking at the interpreter, I'm going to introduce a very simple type system. The type system has two jobs. It rules out sets that contain tuples of varied lengths. The denotational semantics allow sets like (1 x 2) | 3, but it's confusing in practice and banning it allows us to make...
This is a part of a series - start at the beginning. This is just going to be a big pile of \( \LaTeX \). If you're not into that, check out the previous post where I explain the same things with pretty interactive repls instead. \[ \gdef\D#1#2{\llbracket #1...
This is a part of a series - start at the beginning. What do I want out of the core language? simple denotational semantics no evaluation order logical data independence amenable to parallel evaluation amenable to incremental maintenance The datalog family of languages fits the bill, but they struggle to express and compress many...
Most people have a mental model of budgeting which is roughly linear. If you spend half as much money, your money will last twice as long. As you approach zero spending, your runway goes up to infinity. In this model, the space of options looks like this: [I am an interactive graph made of javascript!] Return on investment = This model is wrong. It's wrong because your savings grow over time. If you change the...
While working with RelationalAI I wrote Blobs.jl, a library for zero-copy deserialization in Julia. Not super exciting in itself, but it nicely demonstrates the kinds of zero-overhead abstractions that are possible in Julia. The problem Folks at RelationalAI want to build various complex on-disk data-structures, with...
Published 2018-08-16 I gave a talk at JuliaCon 2018 on my experience using Julia to implement various declarative languages, including a datalog variant, a
(EDIT: Much more accurate measurements are available eg for native software and for web software. The former says that most native psych libraries can get to frame-level accuracy with the appropriate hardware. The latter has results for web software similar to what I found here and...
Last week I was writing an interpreter for a query language. On arithmetic-heavy queries the interpreter overhead was >10x compared to a compiled baseline. I tried staging the interpreter to move the overhead out of the inner loops. In the end the results weren't worth the complexity compared to just writing a compiler so I didn't end up finishing it. But I think it's a neat idea anyway so I wrote a much simpler example to demonstrate.
(It's essentially a
I found contrast codes really confusing on first contact. In hindsight, this is because they are typically presented as being part of the model, but it seems much more ergonomic to me to consider them part of the inference algorithm, as I'll explain here. If you haven't encountered contrast codes before - good. Stay there. You are not missing out. If you have encountered contrast codes...
TLDR: I'm working on a relational programming language intended for rapid GUI dev. Having tried a couple of different approaches to describing GUIs, I've settled on a React-like library that binds relational data to HTML templates.
I want to start getting up in the morning, rather then in the afternoon. It seems like a pretty clear choice: Sleep in all morning and be grumpy Get up early and be happy So afternoon-me makes a plan to get up early every morning, sets an alarm and calls it a day. And that would be the end of it, except that when morning rolls around morning-me discovers a third option. Sleep in all morning and be grumpy Get up...
Suppose I want to track how I spend my day. There are plenty of existing time-tracking apps, but none of them were invented here. I could build an app from scratch. I would just need to write code for displaying entries, adding new entries, editing old entries, summarizing data, storing data, tracking changes... Or I could just dump it in a spreadsheet, add a few formulas and call it a day. This is the real magic of spreadsheets - there is a large class of...
Inspired by Ji Yi's homework assignment, I decided to skim all 360-odd papers in the Psychology of Programming Interest Group archives. Yi reports his students taking ~10 hours to skim 100 papers over a single weekend. His students are clearly way more disciplined than I am. I dragged it out for months. It...
I bought a HTC Vive during the summer and made some simple toys. These are all a few months old, but I didn't get around to writing anything down until today. This is the first time I've found a VR experience convincing. I'm very susceptible to motion sickness in general, and every previous VR system I've tried has made me unhappy, sometimes for hours afterwards. The Vive is mostly fine, with the exception of a few...
Imp needed a relational database that is simple enough to experiment with but fast enough to power real applications. Relational databases are usually complicated beasts. Even SQLite, a relatively lightweight database, is 116,000 lines of code. Its btree...
I notice a tendency to make individual engineering decisions by maximising 'goodness'. Patch X makes the code more complex, but it adds a new feature or increases performance or makes debugging easier. We add up the goodness points, subtract the badness points and if the result is more than zero it's a good patch. Unfortunately, complexity does not add up linearly. The total cost of a set of features is not just the sum of the cost of each feature. Complexity limits how...
I work on Eve, a functional-relational programming language and environment. Since the Eve editor has to run in a browser we built the first few versions entirely in javascript. This has been pretty painful, so a little over three months ago we started looking at other options. The only hard requirements for the runtime are a) we need control over memory layout and b) we need to safely execute untrusted Eve...
The programming world is obsessed with scaling up. How many million lines of code can we maintain? How many petabytes of data can we process? How deeply can I customise this editor? More code, more data, more people, more machines. Nobody talks about scaling down. The vast majority of programs are never written. Ideas die stillborn because the startup cost is too high to bear. When we focus entirely on the asymptotic cost of developing large systems we neglect the...
People regularly tell me that imperative programming is the natural form of programming because 'people think imperatively'. I can see where they are coming from. Why, just the other day I found myself saying, "Hey Chris, I'm hungry. I need you to walk into the kitchen, open the cupboard, take out a bag of bread, open the bag, remove a slice of bread, place it on a plate..." Unfortunately, I hadn't specified where to find the plate so at this point Chris threw a null pointer...
Much of the pain in programming is taken for granted. After years of repetition it fades into the background and is forgotten. The first step in making programming easier is to be concious of what makes it hard. So let's put ourselves in the shoes of a smart but inexperienced end user trying to build, test and maintain a simple application. Anon the intern is charged with managing lunch orders and quickly realises that their job could be done by a computer: Every day at...
Picture a traditional webapp. We have a bunch of stateless workers connected to a stateful, relational database. This is a setup with a number of excellent properties: All state can be queried using a uniform api - SQL. This enables flexible ad-hoc exploration of the application state as well as generic UIs like django admin Every item of state has a unique and...
Last week David Nolen (the author of core.logic) was visiting Hacker School so I decided to poke around inside core.logic. I made a PR that adds fair conjunction, user-configurable search and a parallel solver. First, a little background. From a high-level point of view, a constraint solver does three things: specifies a search space in the form of a set of...
I feel that the readme for strucjure does a reasonable job of explaining how to use the library but not of explaining why you would want to. I want to do that here. I'm going to focus on the motivation behind strucjure and the use cases for it rather than the internals, so try not to worry too much about how this all works and just focus on the ideas (the implementation itself is
Causal ordering is a vital tool for thinking about distributed systems. Once you understand it, many other concepts become much simpler. We'll start with the fundamental property of distributed systems: Messages sent between machines may arrive zero or more times at any point after they are sent This is the sole reason that building distributed systems is hard. For example, because of this property it is impossible for two...
Texsearch is a search engine for LaTeX formulae. It forms part of the backend for latexsearch.com which indexes the entire Springer corpus. Texsearch has only a minimal understanding of LaTeX and no understanding of the structure of the formulae it searches in, but unlike it's competitors (eg
Published 2010-09-04 My MSc dissertation 'Design and Analysis of a Gossip Algorithm', in which I present an algorithm for forming a dynamic, unstructured overlay in which each node can generate a stream of independent, uniformly distributed samples of the overlay membership. Such peer sampling services form the basis for a number of gossip algorithms...
Scampy is a bot for engaging 419 scammers in pointless conversation and consuming time that could have been spent on real victims. It was originally intended to be a smart bot. I had visions of data mining conversations and inventing dsl's for chat scripts. This all takes time however, so in order to get up and running quickly the prototype just selects responses at random from a prewritten list. This turns out to be...