React server components. Streaming server side rendering. Edge compute. Web app architecture advancements are making classic SPAs appear long in the tooth. Time to see what I’ve been missing, and try to keep up with the Joneses (negative connotation fully implied). I might suffer from Perpetual Reengineering Syndrome as this is the umpteenth time...
I’m moving more code into Cloudflare workers everyday as they are cheap, fast, and powerful. But it hasn’t been without struggle. This is the story of one such struggle with a troublesome endpoint. The endpoint in question verifies the OpenID authentication signature directly with Steam. In short, this code runs after the user authenticates with...
Did you know that the files originating from an file input don’t have all their data buffered into memory? Seems intuitive that this is how JS would work, otherwise web sites operating over files would be terribly memory inefficient. Don’t let this efficient File deceive you. If you create one yourself, you’ll soon find that you need to buffer...
In an app that has S3 files and a Postgres database storing file metadata, we might have a DELETE route that looks something like: Select file from db, and if it does not exist throw a 404. If the file’s owner isn’t our session’s user, check the db if the session’s user is an admin Delete file from the db Delete file from S3 The above steps...
I came across some high usage under my Vercel account. Vercel usage donut charts. Cropped for brevity What is “Fast Origin Transfer”? The docs are vague, and whatever it is, seems like I’m about at my limit. Vercel used to have straightforward pricing, but they recently updated their model. Instead of...
Last month I found myself in the unenviable position of having just executed the most dangerous command (dd) on the boot drive of the NAS of my homelab. I was trying to wipe stale ZFS partition information from a data drive after it became briefly detached, but on reboot, the drive labels had been reassigned and I didn’t double check if I was...
Let’s write a function that takes in a file extension and returns a file type: function fileType(fileExtension: string) { switch (fileExtension) { case ".mp4": return "video"; // ... } } Our function assumes the file extension has a leading dot, but should it? The Python standard library includes the dot in file extensions. Rust takes...
An image sprite sheet is a collection of images combined into one, so that browsers don’t need to request each image separately. How does one create a sprite sheet? And how can we measure any potential benefit? What problems can appear and how do we work around them? This post features screenshots from pdx.tools. Game assets shown are from EU4...
Tanstack Query, (née React query for the purposes of this post) is a staple in async state and data fetching management. In addition to saving me from reimplementing similar caching mechanisms and fine grained updates, it was also the first library that solidified the concept of client vs server state, and how they are fundamentally different....
While React animation libraries have seductive landing pages and can be powerful, consider whether you truly need one. These libraries can enhance user experience, but sometimes the simplest solution is the best one. Many animation needs are met with just CSS, so come along for a tour at ways one can add effects to their web application using...
Repeatedly adding orthogonal properties to a React component causes unwieldy bloat. Transitioning to a compound component will invert control and allow clients the flexibility they need. It comes at a cost, though. What are these costs, can we mitigate them, and what are alternative solutions? Let’s build a motivating example of a component’s...
In his 2024 programming predictions video, Theo downplayed WebAssembly (Wasm): I don’t think [Wasm is] going anywhere next year. […] When I see all of these new ways of building Rust based web applications, I cringe a tiny bit […], the size of the binaries are still pretty bad, the performance wins aren’t there, and the flexibility of a language...
I felt clever. After spending a week developing an incremental lexer that would allow pdx.tools to deserialize decompressed saves at a constant memory usage, I was excited to integrate it. I had worked very hard to craft a serde deserializer that was optimized to pull from a lending, incremental lexer without the performance caveats seen in...
I was migrating a Next.js application (pdx.tools) from a self hosted instance to run serverless and edge functions over on Vercel. One serverless endpoint accepts an uploaded EU4 save file, parses it via a microservice, uploads to S3, and sticks the parsed data in Postgres. Conceptually, the endpoint can be written: export function POST(req:...
I was reading an article by PlanetScale, a hosted MySQL platform, called Why we chose NanoIDs for PlanetScale’s API. A decent enough article about Nano IDs and storing them in databases. While there are no mistakes in the article, I think there is enough nuance omitted to warrant an addendum, otherwise others run the risk of misapplying the...
The modern web is so complex, there came a time when updating to either the latest Ant Design 4 (antd), or latest Next.js 13.4 version would cause PDX Tools to completely break with a nonsensical error. Something about a tooltip. After a couple hours searching the web and tinkering, no solution was in sight. A frustrating and humbling experience...
Most should be in agreement that the following signature is in bad taste: type MyObject = { price: number; } Is the price in dollars, cents, or is it even some other currency? What’s protecting another developer (including the future self) from footguns? // Mixing up dollars and cents const discountDollars = 2; const productCents = 10000; const...
There’s one endpoint in PDX Tools that has been a bit of a thorn in my side. It’s a critical, resource intensive endpoint that accesses proprietary embedded assets, but it’s executed very infrequently. For a site that recently broke a million monthly requests, only about 100 of these are to this endpoint. The endpoint in question digests an...
In pdx.tools, users have the option to upload their EU4 save files. Behind their .eu4 extension, these files are typically zips; zips that aren’t compressed very well. Extracting and recompressing the files nets around a 20% reduction in size. As the person who pays for file storage, storing poorly compressed zips isn’t appetizing. Though I can...
When new browser APIs get announced or are implemented, like the recent announcement from the Chrome team about WebGPU, there is an inevitable push back: [Is] this yet another information leak anti-feature that we need to disable? [source] Nobody needs new fancy features, what people really need is more reliable fingerprinting [source] The...
I, unironically, wrote a todo app: CycleList. This was only after trying and failing to find an app or site where I could have a sequence of unscheduled tasks where completing a task would have it fall to the bottom of the list with an updated completed time. Tasks completed least recently then bubble up to the top, creating a sort of cycle....
Designing responsive web sites is a tale as old as CSS media queries, but with the proliferation of JS component libraries, like React, there is a trap that is starting to become a thorn in the side of rising popularity of server-side rendered (SSR) React. The trap is that conditionally rendering above the fold components based on values from...
Interested in running the compression benchmarks yourself? It’s hosted at bench.nickb.dev Lack of transparent compression IndexedDB, the Cache API, and network requests will not transparently compress data, and I think this is a problem. Testing was done on Chrome and behavior may vary by browser. Let’s say we want a function to persist a...
Web workers are crucial for responsive client side compute, but they have a notorious ergonomic hurdle, so large that I’ve previously debated whose responsibility it is to offload compute on the web: the library or the application developer. Not to spoil that article, but the decision is nuanced, essentially boiling down to: it is the library...
My apartment building recently switched ISPs. After they finished installing, all the sites I host on my home server were no longer accessible. Cue slight panicking, as I host everything from code to documents to analytics to a dozen other use cases. I thought my dynamic DNS client, dness, would save the day, but it had already detected,...
Rust lacks a runtime, enabling small .wasm sizes because there is no extra bloat included like a garbage collector. You only pay (in code size) for the functions you actually use. The Rust Wasm Book, “Why Rust and WebAssembly?” We can see small Wasm sizes in action with a contrived example: use wasm_bindgen::prelude::wasm_bindgen; #[wasm_bindgen]...
At PDX Tools, one has the option to upload an EU4 save file and the app will deduce what achievements were earned and how many ingame days have elapsed in the save. With this information we can create a leaderboard for individual achievements. Here’s the Postgres database schema to start us off, where we store the save id, the time it was...
The Rust crate for handling ZIP files, zip-rs is quite flexible. The zip crate is compiled with builtin features to support deflated data (among other compression algorithms). This makes it incredibly easy to hit the ground running for reading and writing zips. We can disable these builtin features. This may sound undesirable, but in fact, it is...
With React continuing to march towards a strong streaming server side rendering story, the React working group published a guide for how dynamic CSS-in-JS libraries can adapt. At the end of the guide, it gives a lukewarm sendoff to dynamic CSS-in-JS libraries: While this technique for generating CSS is popular today, we’ve found that it has a...
Banner octopus graphic from the Compose repo Docker Compose has a nice feature where several Compose files can be seamlessly fashioned together and allow for configuration reuse across environments. There is, however, counterintuitive behavior that can lead one to accidentally overwrite remote container images. Since I lost several hours to this,...
For the uninitiated, JavaScript environments like node.js and the browser have a main thread that runs basically everything. Exhausting the main thread can have consequences, as MDN puts it: “long-running JavaScript functions can block the thread, leading to an unresponsive page and a bad user experience”. In the context of this article,...
Interested in seeing a library employing recommendations that will be laid out in this article? Check out jomini. The highwayhasher library takes this one step further by juggling an implementation that uses native code too! Don’t like reading a life story on recipes? Skip to the next section. With Wasm support in nearly all JS environments, I’ve...
A year and a half ago I wrote that My Bet on Rust had been Vindicated, as an incredible amount of code could be shared between the server and web client, and I could easily spin out C libraries and executables. No use case was out of reach. But based on the title you know that the backend is no longer leveraging a Rust web server (warp in this...
The AWS S3 Storage API is ubiquitous and has been picked up by other 3rd party storage vendors like Backblaze B2, Minio, Wasabi, Storj, and IDrive. This is excellent for developers and sysadmins as it facilitates integration testing and experimentation with cloud storage providers. There is an AWS SDK available for 10 languages so chances are you...
Chrome, Firefox, and Node LTS have all stabilized the SIMD extension to Wasm in the last few months (Safari is lagging at the time of writing. See the updated roadmap for changes). Additionally, Rust has stabilized Wasm SIMD intrinsics recently too. All the pieces are set and now is the time to start authoring libraries that take advantage of the...
The day I’ve known for a while has come: a drive in my ZFS array has become degraded. It’s my first drive failure in over four years, not bad. Here’s the status report: pool: tank state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to...
This example from a Rust issue does not compile: struct X { a: u8, b: u16, c: u32, d: i8 } struct Y { a: u8, b: u16, c: u32, d: i8 } let x = X { a: 1, b: 2, c: 3, d: 4 }; let y = Y { a: 5, ..x }; The code does not compile as the struct update syntax, which is the ..x part from above, is contingent on the target struct being a subtype of the...
With native Rust support recently announced for Cloudflare Workers, I wanted to take a moment and write about the possibilities, but also the obstacles as a sort of reality check for myself. I’m extremely bullish about Cloudflare, Wasm, Rust, edge computing, and the recently announced native Rust support. If I’m not careful, my enthusiasm could...
The age old question for EU4 map staring developers (or at least me) is asking how the game transforms a relatively low resolution map where each province is coded with unique color: 1:1 section of game map focused on the eastern baltic sea region into a much richer experience once in game (the image...
You’ve been working on a development branch over a couple days, testing things out and exploring solutions. You aren’t comfortable losing work, so you create superficial commits. They’re only for you so that you can recover prior work or failed experiments. Finally you arrive at a suitable solution and painstakingly create a commit with a message...
Deploying code via Wasm is convenient as one can run the same code in the browser, nodejs, or via one of the several Wasm runtimes. What, if any, performance is sacrificed for this convenience? I demonstrate the tradeoffs in a real world example where I can take Rust code and benchmark the performance difference between running it as a node...
Over the past week I’ve tried to teach myself WebGL, so that I could leverage the GPU for a performance boost. I emphasized “tried” as a week later I don’t have anything to show for it, as WebGL (and my assumption: most GPU coding in general) has a notoriously steep learning curve. I’ve decided to momentarily shelve these ambitions as the stress...
In Rust, workspaces help organize a project into smaller packages. The downside is if there are shared dependencies that list different features, then one will most likely see unexpected behavior when compiling a binary from the workspace level. This post shows an example of when this can occur and what solutions exist. For me, workspaces have...
I recently published highwayhasher as an npm package. Highwayhasher exposes Google’s HighwayHash to the JS world through the pure Rust reimplementation. There will be a lot to unpack, so here’s a rundown of the library’s features that we’ll cover: ✔ Isomorphic: Run on node and in the browser with the same API ✔ Fast: Generate hashes at over 700...
This post will walk one through a real world migration on how to apply the principle of minimal privilege to a systemd service. This is accomplished by extracting sensitive configuration fields into an environment file, templating the config, running the service as a dynamic user, and sandboxing the application with systemd primitives. I’ve...
Sometimes it makes sense to ship an application with the necessary data already inside. No need to fetch it from an external source and deal with failure when everything is bundled together. How the data is bundled inside the application is what we’ll be looking at today, specifically in the context of Wasm apps, though lessons learned here...
The EU4 achievement leaderboard and save file analyzer, Rakaly, has several requirements when it comes to maintaining the leaderboard: There are hundreds of achievements in EU4 and each one should have its own leaderboard (eg: forming Italy is much quicker than owning the world). Every EU4 patch can introduce different gameplay mechanics that can...
I recently completed a major overhaul of a Javascript library of mine. This overhaul replaced a pure JS parser (generated by jison) with one that delegated parsing to an implementation written in Rust compiled to Webassembly (Wasm). I want to share my journey, as overall it’s been positive and I expect that other JS libraries could benefit from...
The site you’re reading this on had been rendered by Jekyll since September 2012. That’s 8 years of content, totalling 160k words. I create a post about every month, so this is not a small blog anymore. Recently I’ve noticed a slowdown in how prolific I am, and while it would be foolish to put the blame squarely on Jekyll, it certainly deserves...
When cost optimizing, consider using Backblaze B2 over Github’s git LFS. B2 has a much more generous free tier compared to Github’s: 10x more storage (10GB vs 1GB) and 30x more bandwidth (30GB vs 1GB). Even after exceeding the free tier, Github’s git LFS still commands a 12x price premium. Before abandoning git lfs, there are several tips to keep...
As a side project, I’m writing an Europa Universalis IV (EU4) leaderboard and in-browser save file analyzer called Rakaly. No need to be familiar with the game or the app, but feel free to check Rakaly out if you want, we’re just getting started. I’m writing this post as whenever Rust is posted on the internet, it tends to elicit polar opposite...
Gatsby is a site generation framework. I recently was using it for one of my side projects, sff.life, a site dedicated to small form factor computing (eg: small computers). I decided to migrate away from Gatsby, as it is not a zero cost abstraction for static sites for two reasons: Needless bloat via javascript and JSON bundles Way too many...
I have a toy site that parses Rocket League replays using the rust crate boxcars. Parsing replays is done synchronously and shouldn’t block the browser’s UI thread, so parsing is offloaded to a web worker. Getting this side project to a point where it works + is maintainable (minimum configuration) has been an exercise in doggedness as I spent...
2020-08-12: This article described a performant method to parse data formats and how to aggregate serde fields with the input buffered. While the serde demonstration is still valid, I opted to create a derive macro that will aggregate fields that isn’t susceptible to edge cases. There’s a format that I need to parse in Rust. It’s analogous to...
I have several sites deployed on VPSs like DigitalOcean that have been dockerized and are reverse proxied by traefik so they don’t have to worry about Let’s Encrypt, https redirection, etc. Until recently I had very little insight into these sites and infrastructure. I couldn’t answer basic questions like: How many requests is each site handling...
Timescale v1.3 was released recently and the major feature is continuous aggregations. The main use case for these aggregations is downsampling data: which has been brought up several times before, so it’s great that these issues have been addressed. Downsampling data allows queries spanning a long time interval to complete in a reasonable time...
For a small, personal project I use sqlite as a time series database and make queries like the following: SELECT referer, COUNT(*) AS views FROM logs WHERE host = 'comments.nbsoftsolutions.com' AND epoch >= 1551630947 AND epoch 1551632947 GROUP BY referer ORDER BY views DESC Over time, I’ve noticed that this query has...
EDIT: 2019-03-07: The bug described in the edit has now been fixed, but I’m leaving the edit in for posterity. EDIT: 2019-03-01: (this section describes a bug that has been fixed, so one can skip to “END EDIT”) Currently there is a critical bug in creating cross platform jobs. Adapting the sample code for cross platform jobs to: strategy: ...
This post covers: Benchmark reports for contributors Benchmark reports for users Profiling with valgrind / kcachegrind Reproducible benchmarks and graphics Tips for benchmark behavior and benchmarking other languages Lots of libraries advertise how performant they are with phrases like “blazingly fast”, “lightning fast”, “10x faster than y” –...
Someone new to investing asked for some advice of how they should allocate their SIMPLE IRA funds in their Vanguard account. There are so many funds that if you didn’t know what to look for, it’s easy to get lost or worse – make bad decisions. Here’s my philosophy: Be average: I don’t have the insight into companies to outmaneuver hedge funds and...
Part of the Wireguard series: Wireguard VPN Routing Select Docker Containers through Wireguard VPN Viewing WireGuard Traffic with Tcpdump Leaning on Algo to route Docker traffic through Wireguard (most recent and consolidates the previous articels) I write about Wireguard often. It’s been a bit over a year since my initial article and a lot has...
Update: 2018-09-26: This article has been updated to clarify what I meant when I described Timescale as bloated, and a correction to the collectd’s batch insert claim (it does support a form of batch inserting) There’s a new time series database on the block, TimescaleDB, which is an extension to Postgres. I decided to test it to see how well it...
This post has been adapted from the documentation I wrote for Farmhash.Sharp. I think there is misleading information out there when it comes to string hashing, so I wanted shed some light on how Farmhash.Sharp does it efficiently. Farmhash.Sharp It is very easy to hash a string with Farmhash.Sharp with performance equivalent to hashing bytes...
Quick rant about the roadblocks that IPv6 only networking has caused: Gist of it: launchpad.net is ipv4, so adding Ubuntu apps via apt-add-repository or by an apt list (that reference launchpad.net) will fail. This severely hinders acquiring software that isn’t distributed by Ubuntu. keyserver.ubuntu.com is ipv4 so one can’t use the default...
A short and to the point article. I have: A NAS configured with 5 disks in raidz2 with auto snapshots Docker applications like gitea running on the NAS Users who access the NAS over a CIFs share via their client of choice (Mac or Windows) Users who also have documents stored in cloud providers like Microsoft’s Onedrive This setup is ok, but there...
I previously wrote an article back in November 2017: Replacing Elasticsearch with Rust and SQLite. In it, I needed to create a few HTTP endpoints that ingested JSON, perform a database lookup, and return JSON. Very simple. No query / path parameters, authentication, authorization, H2, or TLS. Any v0.1 microframework would do (Project repo). I...
Part of the Wireguard series: Wireguard VPN Routing Select Docker Containers through Wireguard VPN Viewing WireGuard Traffic with Tcpdump Leaning on Algo to route Docker traffic through Wireguard (most recent and consolidates the previous articels) On the article, WireGuard VPN Walkthrough, Glen posted the tantalizing question: How would you...
I have a self-hosted Nextcloud for cloud storage installed on a ZFS Raid-6 array. I use rclone to keep my laptop in sync with my cloud. I was setting up a new computer wanted a local copy of the cloud, so I executed rclone sync . nextcloud:. This ended up deleting a good chunk of my cloud files. The correct command was rclone sync nextcloud: .....
Part of the Wireguard series: Wireguard VPN Routing Select Docker Containers through Wireguard VPN Viewing WireGuard Traffic with Tcpdump Leaning on Algo to route Docker traffic through Wireguard (most recent and consolidates the previous articels) Scenario: You have a host running many Docker containers. Several sets of these containers need to...
Kotlin is a newish language that runs on the JVM and integrates seamlessly with existing Java code. Kotlin is concise, ergonomic, and thoughtful. We’ll look at low cost and frictionless ways of integrating Kotlin into an existing Java codebase. You can add in Kotlin piecemeal so there is very little time lost to writing Kotlin today. For purposes...
In Dropwizard 1.3, you could see a 10x improvement in latency reduction with services that deal with large payloads (either in the response or request). There’s a lot that went behind this effort, so let’s break it down. See the example code on dropwizard-tls-benchmark repo. Reference Solution Get the benefits by modifying your pom.xml ...
Banner generated through trianglify Update: February 5th, 2018: all of the issues written about in this post have been resolved. The beauty of open source is the community and the transparency. I wrote an article about a month ago titled Replacing Elasticsearch with Rust and SQLite that can be seen in the rrinlog project. It’s a simple web app...
Caveat: Elasticsearch is a multi-purpose, distributed, battle-tested, schema-free, log and document storage engine. We’ll be replacing it with a single-purpose, relational schema, me-tested, and custom nginx access log storage engine. Knowing that, let’s begin. Find this project on Github After building my home server, I started hosting many...
Introduction A year and a half ago, I wrote Federal Register Data Exploration with R, and I had written it mainly for myself, as a way of exhibiting a technique I called “flexml” (a portmanteau of flexible and xml). The post is light on details about how to transform the Federal Register data (natively stored in xml) into a more queryable form...
This article was last updated: March 29th 2018 Part of the Wireguard series: Wireguard VPN Routing Select Docker Containers through Wireguard VPN Viewing WireGuard Traffic with Tcpdump Leaning on Algo to route Docker traffic through Wireguard (most recent and consolidates the previous articels) With the rise of privacy consciousness, people are...
I’m building a home server and I’m using Pi-hole to blackhole ad server domains in a test environment. It’s not perfect, but works well enough. The Pi-hole admin interface shows a dashboard like the following to everyone (admin users get an even greater breakdown, but we’ll focus on just the depicted dashboard) ...
I wrote this article not because I’ve built a home server but because I’m on the verge of doing it and I’d like to justify (to myself) why building one is reasonable and why I chose the parts I did. I want to build a home server because I realize that I have content (pictures, home movies, etc). This content used to be stored on an external hard...
Update: 2017-08-01: I posted this question on Stackoverflow, which was subsequently answered. Check it out! Previously, I wrote a Rust TLS termination proxy where I tested it with a Node.js echo server and noted the terrible performance. This post will be a deeper dive into this problem. See this post on stackoverflow. The weird thing is I don’t...
I feel like I don’t write about the Rust language that often, but it turns out today is my 2nd anniversary of dabbling in Rust. Exactly two years ago, I wrote a post about processing arbitrary amount of data in Python where I benchmarked a Python csv solution against Rust (spoiler, Rust is faster). Since then, Rust has only been mentioned in...
Update: September 4th, 2018: I decided to remove AMP from this site as it wasn’t a quick and easy win. I didn’t want to style tables in amp or figure how to import gists without errors. The amp jekyll plugin is ok, but can print errors and is slow to be released. Thus, I only recommend jekyll readers continue if they are have relatively...
To start, Journald’s minimalist webpage describes itself as: [as a] service that collects and stores logging data. It creates and maintains structured, indexed journals. With one of the sources of logging data being: Structured log messages Before diving into journald, what’s structured logging? Structured Logging Imagine the following...
Farmhash.Sharp is a pet project of mine where I port Google’s Farmhash algorithm to the .NET platform, and I recently migrated it to use the new MSBuild architecture. Previously the project was structured based on the ProjectScaffold, but was found to be deficient. While the migration was not without some difficulties, in the end, I’ve found it...
May 2nd 2017 Update The author of HikariCP posted a really informative comment here on several shortfalls and that the test wasn’t really a fair comparison between HikariCP and Tomcat. This article is provides context for the comment. Once read I also urge readers to follow the conversation between Brett and I over on the HikariCP mailing list. I...
I wasn’t lying when I said I was getting rid of my old Windows phone and grabbing myself a Google Pixel. I’ve had a wide range of emotions as I’ve been exploring the phone and ecosystem. It’s been less than a week, and things have been great, though there are still some rocky areas. This post mostly serves my needs for documenting my journey to a...
People are always surprised when I say this, but I have a Windows phone. I’ve had one for the past four or five years, and there were only a couple regrets: the app store and a modern browser. I only found these to be minor annoyances and if Verizon hadn’t dropped Windows phone, I’d probably be getting another Windows phone. I know it sounds...
Update: February 5th, 2018: For a new solution see Dropwizard 1.3 Upcoming TLS Improvements When someone says that SSL/TLS is slow, I used to chuckle and point people to Is TLS Fast Yet?. There a couple of headliner quotes on the main page from engineers at Google and Facebook, which I’ll copy here for convenience. On our production frontend...
Update: 2017-11-01: I’ve opensourced this utility as OhmGraphite, It can be ran as a CLI app or Windows service. The code showcased here demonstrates how to get a minimal example running. See the repo for a more production worthy approach! Hot on the heels of Monitoring Window’s system metrics with Grafana are two fixes for the dashboard. Graphs...
Update: 2017-10-30: With new preferred graphite docker image Update: 2017-11-01: I’ve opensourced a Windows utility that will export Open Hardware sensor data into Graphite. It adds GPU, power, temperature, and more, which won’t be covered in this post. Check it out! [The follow up post]({% post_url...
I know nothing about car accidents in Michigan, but together, we’ll be digging into some data. Depending how this post goes, this could be a pretty cool series as we dig repeatedly deeper. Keep the data dictionary close at hand, 60 columns is a lot to keep straight. Similar posts: Visualizing Ann Arbor Incidents and Federal Register Data...
Let’s track current progress: 2012: move from homegrown static site solution to jekyll 2013: incorporate LESS CSS, RequireJS, minification using Gruntfile.js. I can laugh at it now as the title is “The Perfect Blogging Workflow”. Heh. 2014: migrate from GoDaddy/IIS/WCF/C#/MySQL to DigitalOcean/Apache/Flask/Python/SQLite. 2015: I didn’t write...
I want to like F#. It’s like what C# should have been 7 years ago. F# has beautifully terse, yet readable syntax. With algebraic data type, duck typing, and computation expressions, the language can lend itself to pretty powerful code. But I’ve been having some doubts. I’ve slowed down in using the language. My enthusiasm wanes. I’ve written a...
As of writing this, the free no gimmicks SSL certificate service, Let’s Encrypt, has issued nearly 15 million certificates. I’ve written about Let’s Encrypt before – it’s taking over the world and that’s a good thing. This time I want to talk about Let’s Encrypt in the context of a web framework that I’ve gotten quite familiar with over the...
The other day I had the requirement where I had to code an endpoint such that a client could submit two bodies, one containing JSON, and the other one containing XML. One could get clever and embed the XML in JSON, vice versa, or even make two requests, but the HTTP spec already has this issue solved: multipart messages. If you took a peek at the...
Boxcars, also stylized as boxca-rs, is an example of a Rocket League replay parser written in Rust using nom for parsing and serde for serialization. As stated in the title, this is an example, as this library in no way competes with the other feature complete parsers such as Octane and RocketLeagueReplayParser. Rather, let boxcars be a good...
Dealing with concurrent and parallel code is hard, so I’ll be using concrete examples with Dropwizard that demostrate an asynchronous server and client implementations. For our tests, we’ll have 100 requests sent to a server that waits for a second before returning hello world. All java metrics were recorded using Java VisualVM From the Jersey...
Follow Jetty’s instructions on creating a certificate (either self signed or generating a CSR). Tip: when securing a key with a password via the PKCS12 mechanism, ensure that the keystore and PKCS12 file share the same password. Stop reading when you hit the Configuring the Jetty SslContextFactory You should now have a keystore that you can play...
Introducing Pfarah This is post will be a break from the norm. Pfarah is a .NET library written in F# that parses files that are ingested by the Clausewitz engine made by Paradox Interactive. I wrote a tutorial for the library, and I was so proud that I decided to post it my blog, for additional publicity but also as a time capsule. This way, I...
Introduction Given a date format that supports arbitrary precision, the question of how best to represent the date in a language of choice is generally tough to answer. In situations like these, I like to look at current standards and see how implementations work. As an example I will use ISO 8601, which is a standard way to represent a datetime...
Not to waste any time at all, I’ve compiled a list of incidents reported by the University of Michigan’s police and plotted their latitude and longitude on a map. Ann Arbor Reports Mapped by Location Ann Arbor Reports Mapped by Location and Type Draw what...
Introduction This will be a detailed, though not exhaustive, quickstart into using IMAP. Initially this was also going to highlight the python library, imaplib, but the post became too long! Maybe next time. The hope is that this’ll contain enough information about querying email servers that additional questions would most likely be redirected...
Introduction No one can deny that Javascript has significantly changed in the last few years. If you were a Javascript master in 2012 and lived under a rock for the last few years, you’d be shocked to see what the community had evolved into. Since I’ve been at least dabbling in Javascript since 2012, I thought I’d give my view on this topic. To...
Banner generated through trianglify. I think I’m addicted In most languages, exceptions should be exceptional. I’ll exclude Python idioms and possibly Erlang. Newer languages, such as Rust and Go are eschewing exceptions in favor of other mechanisms. Rust exposes the possibility of a function encountering an error through the Result type. While...
Banner generated through trianglify Introduction The Dropwizard web framework, simplifies java web development significantly. What I’m interested in, is how much overhead are we accepting compared to a more raw solution because if you look at the TechEmpower Web Framework Benchmarks Round 11 Json Serialization, you’ll notice Dropwizard is a fifth...
Federal Register Data Every year the Federal Register publishes an XML dataset, which contains daily breakdown of rules and notices that were published. In this post, I will be using the 2015 dataset, where each xml document represents a day. Here’s a synopsis of what’s in the federal register (from their site): Notices This category contains...
Banner Generated through trianglify The free and open certificate authority, Let’s Encrypt has been out in public beta for the last few months and v0.3.0 of their client was recently released, so I decided to give it a try on one of my side projects: eu4stats.com. Additionally, Nginx, the web server behind eu4stats.com, received an update in...
In F#, there is a feature called active patterns that can greatly simplify and beautify code. The only problem is that ‘active patterns incur a performance overhead’ as stated in the book, Expert F# 3.0. We’ll explore how much by comparing a string parsing implementation using active patterns and Nullables, which are structs that “represents...
I’m interested in learning Electron so that I can practice writing a cross platform GUI app (something that I have never done). One thing that I was interested in was what libraries others were using to structure their app. In lieu of a recommendation in the Electron’s documentation, I checked out awesome electron, which is a repo keeping tabs on...
Introduction This post delves deep into the world of bulk loading data with PostgresSQL. I’ve written about this subject before in Optimizing Innodb for Bulk Insert for MySQL-esque databases. These posts differ not merely by which database they target, but also in their target audience. The Innodb post is more a write oriented setup, where there...
Introduction Through the xml2 module, Postgres can apply XSL stylesheets to arbitrary texts. There are two big caveats with this approach: The xslt functionality is through libxslt, which conforms to XSLT 1.0 The xml2 module is deprecated, slated for removal, and Postgres doesn’t implement the functionality in the core server, either. All is not...
Every application should have a series of continuously updated metrics. This allows for realtime insights and alerts into a service. The Java web framework, Dropwizard, which I’ve written about before, comes with the venerable Metrics library and about 75 already defined metrics baked in. All you need to do is persist and interpret these metrics,...
I’ve recently written about data processing in F#, and I thought I’d keep up the trend, but this time showcase a bit of python. A naive Pythonista can easily exhaust all available memory when working with large datasets, as the default Python data structure, which are easy to use, eagerly evaluate elements causing a million elements to be...
The Dropwizard Getting Started guide is a great introduction to the framework, but it in regards to testing the created applications it leaves some wanting. This post will walk one through adding logical and practical tests to the application created in guide and in the end, we will achieve with 100% test coverage. Baby Steps Add the...
As a programmer, I’m constantly on the look out to do less work when it comes to coding, both in terms of maintenance and the initial code through. A great example of this is my quest to write a perfect library for dealing with Paradox Interactive’s custom format (briefly highlighted here). I’ve been trying to work with this format for eight...
This article assumes some precursory knowledge of Dropwizard, a framework that has been invaluable when creating JSON oriented RESTful web services. Dropwizard is incredibly fast (I should insert a citation here, but metrics provided with the framework suggest there is neglible overhead due to itself). But maybe the most important aspect of the...
Update: October 2nd, 2015: This article contains an important omission, as it does not consider LOAD DATA INFILE command. If the data that needs to be loaded can be loaded as is, or can be loaded with minimal tampering, I’d recommend using LOAD DATA INFILE first, and then apply the MySql server variable changes I recommended as needed....
Let me prefix this by apologizing because this post will be a mix of content for technical people as well as content for those that just want to see statistics for their save game! Introduction As briefly alluded to – EU4 Stats is a web application that when given a save game, will generate a series of statistics that you can share with friends....
Continuous integration and deployment have become all the rage with services like Travis, Appveyor, and Codeship it has become almost trivial for everyone to set up. I believe everyone should employ these two concepts as they make developers much more efficient. These services start building and testing the code after the developer syncs it...
In all my experiences, I have had no need to run C# code on linux. This has recently changed as the demographics for my side project, Pdoxcl2Sharp has expanded to include linux. Being the ambitious developer that I am, I decided that not only can people use the code on linux, but also develop the code without any problems. Turns out there are a...
What if you had one single threaded writer updating a variable while there were many concurrent threads reading it, what kind of threading issues would you run into? The answer is none if you don’t mind the reader threads returning millisecond stale data. Scenario: When a file changes, a class reads the contents of the new file and updates a...
Await async is the hottest thing in C# (natively supported in .NET 4.5 and supported in .NET 4.0 through the use of Microsoft Async package). It makes asynchronous programming simple, while promising great performance improvements. No more blocking a single threaded program waiting for a download or dealing with BeginXXX and EndXXX functions....
Check out and play our game here Introduction Our team is composed of Tyler Stokes, Nick Babcock, Ian Propper, Tommy Seidel, and Mark Mevorah. We were tasked with creating an original game that would be publicly showcased at the end of the semester. Initial Phase and Brainstorming Our team of five formed soon after the final week of individual...
In a span of a few days I have moved this site off of GoDaddy/IIS/WCF/C#/MySQL and onto DigitalOcean/Apache/Flask/Python/SQLite to gain better performance, consolidate all my websites, and to give a good refactoring. Reasons This site is essentially a collection of static pages, and so should be blazingly fast under all circumstances. However,...
Screenshot from our zelda project It turns out for our first project we also have to complete a postmortem document, which is fitting because I was going to write a post regardless! Introduction Our team is composed of Tyler Stokes and Nick Babcock, we decided to implement The Legend of Zelda’s first dungeon along with an original level of our...
It’s the second week into this course, and my feelings are still mixed. We went through a Unity tutorial in class, and I found out that Unity utilizes C# to control the logic, and C# is my forte. I may have an advantage over some of my other classmates because C# is not taught at the University, and taking a Java class is somewhat frowned upon,...
Today marks the first lecture for EECS 494 Winter 2014 - Computer Game Design and Development. Today also marks the first of a series of posts detailing my trials and tribulations in the class. The main reason I’m doing this is that at Michigan, Computer Game Design and Development is considered a major design experience, also known by the...
This article was last updated: March 23th 2016 This is a walkthrough of setting up virtual Linux machine hosted by Hyper-V and being able to share files between the host OS and the Linux machine. For those curious on what this post will accomplish, here is a breakdown of what will follow: Setup a internal virtual switch in Hyper-V Allow internet...
Before the holidays started, I had to write a report for class on a programming project I had been working on for nearly a month. I was unsure how I should write this report. Should I use Latex, markdown, or Microsoft Word? Considering this report wasn’t going to be published in some scientific journal, and I don’t know Latex, it was decided that...
Compared to others, I haven’t had that many interviews: one for my freshman year internship, three sophomore year, and none in the foreseeable future, as I’ve accepted an offer to go back as an intern to Citadel, which I’ve written about a couple of times before. Despite the small number, I routinely find people entertained when I retell my...
After finishing a large application for a couple clients and having it thoroughly tested on development machines, the hardest part was yet to come. We had no control over their environment. We tested with Python 2.7 but they could have 2.4, which some of our dependencies couldn’t handle. It isn’t right to force clients to upgrade or overwrite...
No two users have the same installation and in order for an application to work the user may need to edit and maintain configuration files. Ideally this shouldn’t be necessary, but configuration is an easy means to an end. This raises the question, how are these files written? As far as I am aware of there are these options: XML, JSON, TOML,...
Disclaimer: I am far from a résumé expert; these are just some guidelines that I believe in. Single Page Résumés on more than one page is a big pet peeve of mine. More than one page speaks of wastefulness and verbosity. There are people that are more famous on Wikipedia that if printed out will be on a single page. Now if Wikipedia can...
Question to be answered: How to handle an object that has its data in an invariant language, but also supports a subset of its strings converted into a specified language. Introduction This is not a trivial problem as localization is a crosscutting concern. In short, adding this aspect to the code base will result in either code duplication or...
I didn’t realize anyone but me visited this site, so imagine my surprise when I connected to the MySQL server powering the comments and saw that not only were there comments, but there was a highly constructive comment left by a Patrick Cash. He pointed out some serious flaws with the article, so I’m going to attempt to construct a comprehensive...
I’ve finished my first week at my new job at Citadel. It is quite the contrast going from an Ann Arbor startup to one of the world’s largest hedge-funds, if not the largest (I forget what they told all the interns at orientation.) I feel like the only thing that is similar is that my job entails me typing on a keyboard, all the rest seems...
I periodically talk about this site and how it is ran. Well after nine months my baby has gotten an upgrade. I’ll try and detail the programs that I use to render this site thorough enough to so that they can be followed and not be too tedious. Before getting started, it should be noted that I have a test web server and machine where the web...
I’m starting a new job soon, and with any new programming job one should expect to learn new languages and technologies. What are these? Perl and Sybase. Out of these two, Perl has a much more vibrant community, so I have decided to learn Perl. How does a programmer learn a new language? Pick up a book. What book? Modern Perl. Reading...
Today is my last day at Hiperlogic/TotalCAE, I’ll probably be back in the fall, but these next ten weeks will be the first time I will consider myself an employee of a different company, and that company is Citadel LLC. Citadel is a hedge-fund company and is absorbing me into their Citadel Technology (CTECH) Professional Services. Don’t ask me...
This is the story of the founding of NBSoftSolutions. Queue the nostalgic music. It was the beginning of 2008 and I was 15 and obsessed with a game called Europa Universalis III. I had recently started to get more into programming and needed a project to test my new found skills. The perfect idea struck me. Whenever the player saved their...
In what format should a REST API return and accept timestamps? The two most popular ways are Unix time (or a slight variation thereof) or ISO-8601. Both have their strengths and weaknesses and both are equally popular as we shall see. A sample of 20 APIs yielded nearly a 50/50 split. Therefore, no matter if this holds any persuasion or not,...
Working on a personal project that will never be entirely satisfied can bring moments of extreme brilliance and dread. This is called a forever project. The main driving force behind the project is yourself and the itch that you’re scratching. This itch isn’t constant. After scratching for a day or a week, the itch is satisfied for a week, a...
Look at the following images and think about how a user could toggle this change back and forth in an application. Before and after border detection The task is simple, create borders between countries. A border is drawn whenever there are two differing pixels that are adjacent to each other. This...
Bottle is a great micro web framework that can be as minimalist or feature rich as one wants. Bottle is great for rapid development and for debugging. However, Bottle is not recommended to be deployed in production as it lacks security and speed. The developers of Bottle know this and so made Bottle easily extendible. A common want in web...
Recently in an EECS course we were asked to solve the Traveling Salesman problem using Manhattan Distances. I will give a general overview to how I solved this problem, so a basic understanding of the concept is necessary. Approximation Before computing the actual Hamiltonian circuit that will satisfy the problem, it is best to approximate how...
Here at the University of Michigan it is the time of year again for registration for classes for next semester. One of the classes that piques the inner perfectionist is called EECS 381 - Object-Oriented and Advanced Programming. Just in the name, every word rings true to me and how I program. However, as it should be, this class is difficult. ...
Thanksgiving morning I wake up to this in my email. After careful inspection, we found that some students did not follow the project 2 spec correctly. In particular when the self flag is specified, many students still used the stl libraries and vice versa. If this was the case, we have deducted 50% from your grade. This should not affect the...
Update 2/13/2014 Since this article is still getting a significant number of hits, please see this newer post of mine that walks one through, in my opinion, the most optimal way to set up a Linux VM on Client Hyper-V. First and foremost, I would like to state that I extensively use the Hyper-V feature included in Windows 8. I enjoy it despite...
In the first lecture in an programming class at the University of Michigan, it is taught that it is highly improbable for one to write code that is simple, fast, and elegant. Instead we should strive for one of the categories or two if we’re ambitious. Simple: few lines of code using basic constructs Code can be digested in a short amount of...
int[] arr = { 1, 2, 3, 4 }; for (int i = 0; i Console.WriteLine(arr[i]); foreach (int i in arr) Console.WriteLine(i); for (int i = 0; i Console.WriteLine(arr.ElementAt(i)); What are the differences between these loops? They look the same and have the same result. The truth is that they are extremely different and serve different...
There is no one lazier than the programmer, and I am included in this group. When I was constructing this site everything was dynamic and repetitive. The home page was only a placeholder for content that was retrieved via a server- side call, and all the content pages needed the same duplicated HTML so that they would appear correctly. A...
At the University of Michigan, all first year engineers have to take an intro to a computer and programming class, normally ENGR101, which involves C++ and MATLAB. However, for those who have previous experience or have an expressed interest in programming then one can opt for ENGR151 to take its place, which is an accelerated introduction. I...
Update January 24th, 2015 This article has been deprecated in favor of a more recent article. Sorry for the clickbait title, the gist of the article is don’t be surprised when switching to unsafe code doesn’t yield the performance benefits predicted. Always benchmark. Also the article contains misinformation regarding character encoding (as...
The other day a friend messaged me via Facebook I’ve been reading a few things on Windows 8 lately and most of what I’ve read has been giving it really bad reviews. Can I get your opinion on it? Just curious because some articles have said Linux might start getting momentum. Good question. I’ve read similar negative reviews and I have to agree...
Donald Knuth has been famously quoted as saying: We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up on our opportunities in that critical 3% (pp. 268). Knuth, D. E. (1974). Structured Programming with go to Statements. ACM Computing Surveys, 6, 261–301....
There have been quite a few posts on the internet about experiences about programming at a young age. Some recount their days of hacking on computers in middle and high school, whereas others write their first experience is with a college programming class. Those starting in college are often self-conscious about their skills as they know that...
Let’s create a file input and chunk into 4MB chunks input type="file" onchange="processFile(...arguments)" /> script> const CHUNK_SIZE = 4 * 1024 * 1024; function processFile(e) { const file = e.currentTarget.files[0]; const start = performance.now(); const fileSize = file.size; for (let i = 0; i fileSize; i += CHUNK_SIZE)...