Jaseem Abid

Big Endian's Guide to SQLite Storage

Sun, 24 Nov 2024 00:00:00 GMT

I wanted to learn how databases like SQLite store data under the hood, so I decided to write some code to inspect the database file. SQLite famously stores the entire database in a single file, and the file format is very well documented. Here is one diagram¹ to get started instead of the roughly 13,848 words in that document.

While not impossible, it is somewhat tedious to read through the file format documentation and figure out what exactly you need to get started. The above diagram is that minimal starting point.

🌱 Quick Demo

It’s easier to demonstrate this with a quick example. We create a tiny test database and dump all its internal state with hexdump and a small program I wrote called rsqlite.

$ sqlite3 planets.db < planets.sql
$ cat planets.sql

CREATE TABLE planets (
  id INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  type TEXT NOT NULL,
  diameter INTEGER NOT NULL,
  distance INTEGER NOT NULL,
  moons INTEGER NOT NULL
);

INSERT INTO planets (id, name, type, diameter, distance, moons) VALUES
  (1, 'Mercury', 'Terrestrial', 4879, 57910000, 0),
  (2, 'Venus', 'Terrestrial', 12104, 108200000, 0),
  (3, 'Earth', 'Terrestrial', 12742, 149600000, 1),
  (4, 'Mars', 'Terrestrial', 6779, 227900000, 2),
  (5, 'Jupiter', 'Gas Giant', 139820, 778500000, 79),
  (6, 'Saturn', 'Gas Giant', 116460, 1433000000, 83),
  (7, 'Uranus', 'Ice Giant', 50724, 2871000000, 27),
  (8, 'Neptune', 'Ice Giant', 49244, 4495000000, 14);

This creates a tiny, mostly empty 8KB file we can play with. Hexdump is smart enough to avoid duplicate null rows with a single * and make the output readable, so we can share the whole contents inline right here. No hidden magic, this is it!

Side note: If you aren’t familiar with hexdump output, the first column is the offset from the beginning of the file. The 16 columns in the middle are raw values in hexadecimal and the last section is the ascii representation of the input data. Anything outside printable range becomes a dot.

$ hexdump -C planets.db

00000000  53 51 4c 69 74 65 20 66  6f 72 6d 61 74 20 33 00  |SQLite format 3.|
00000010  10 00 01 01 00 40 20 20  00 00 00 02 00 00 00 02  |.....@  ........|
00000020  00 00 00 00 00 00 00 00  00 00 00 01 00 00 00 04  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 01 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 02  |................|
00000060  00 2e 7e 58 0d 00 00 00  01 0f 25 00 0f 25 00 00  |..~X......%..%..|
00000070  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000f20  00 00 00 00 00 81 58 01  07 17 1b 1b 01 83 07 74  |......X........t|
00000f30  61 62 6c 65 70 6c 61 6e  65 74 73 70 6c 61 6e 65  |ableplanetsplane|
00000f40  74 73 02 43 52 45 41 54  45 20 54 41 42 4c 45 20  |ts.CREATE TABLE |
00000f50  70 6c 61 6e 65 74 73 20  28 0a 20 20 20 20 69 64  |planets (.    id|
00000f60  20 49 4e 54 45 47 45 52  20 50 52 49 4d 41 52 59  | INTEGER PRIMARY|
00000f70  20 4b 45 59 2c 0a 20 20  20 20 6e 61 6d 65 20 54  | KEY,.    name T|
00000f80  45 58 54 20 4e 4f 54 20  4e 55 4c 4c 2c 0a 20 20  |EXT NOT NULL,.  |
00000f90  20 20 74 79 70 65 20 54  45 58 54 20 4e 4f 54 20  |  type TEXT NOT |
00000fa0  4e 55 4c 4c 2c 0a 20 20  20 20 64 69 61 6d 65 74  |NULL,.    diamet|
00000fb0  65 72 20 49 4e 54 45 47  45 52 20 4e 4f 54 20 4e  |er INTEGER NOT N|
00000fc0  55 4c 4c 2c 0a 20 20 20  20 64 69 73 74 61 6e 63  |ULL,.    distanc|
00000fd0  65 20 49 4e 54 45 47 45  52 20 4e 4f 54 20 4e 55  |e INTEGER NOT NU|
00000fe0  4c 4c 2c 0a 20 20 20 20  6d 6f 6f 6e 73 20 49 4e  |LL,.    moons IN|
00000ff0  54 45 47 45 52 20 4e 4f  54 20 4e 55 4c 4c 0a 29  |TEGER NOT NULL.)|
00001000  0d 00 00 00 08 0e fc 00  0f df 0f c0 0f a1 0f 82  |................|
00001010  0f 61 0f 41 0f 1f 0e fc  00 00 00 00 00 00 00 00  |.a.A............|
00001020  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00001ef0  00 00 00 00 00 00 00 00  00 00 00 00 21 08 07 00  |............!...|
00001f00  1b 1f 03 05 01 4e 65 70  74 75 6e 65 49 63 65 20  |.....NeptuneIce |
00001f10  47 69 61 6e 74 00 c0 5c  00 01 0b ec 41 c0 0e 20  |Giant..\....A.. |
00001f20  07 07 00 19 1f 03 05 01  55 72 61 6e 75 73 49 63  |........UranusIc|
00001f30  65 20 47 69 61 6e 74 00  c6 24 00 00 ab 1f fb c0  |e Giant..$......|
00001f40  1b 1e 06 07 00 19 1f 03  04 01 53 61 74 75 72 6e  |..........Saturn|
00001f50  47 61 73 20 47 69 61 6e  74 01 c6 ec 55 69 d8 40  |Gas Giant...Ui.@|
00001f60  53 1f 05 07 00 1b 1f 03  04 01 4a 75 70 69 74 65  |S.........Jupite|
00001f70  72 47 61 73 20 47 69 61  6e 74 02 22 2c 2e 66 f7  |rGas Giant.",.f.|
00001f80  a0 4f 1d 04 07 00 15 23  02 04 01 4d 61 72 73 54  |.O.....#...MarsT|
00001f90  65 72 72 65 73 74 72 69  61 6c 1a 7b 0d 95 7a 60  |errestrial.{..z`|
00001fa0  02 1d 03 07 00 17 23 02  04 09 45 61 72 74 68 54  |......#...EarthT|
00001fb0  65 72 72 65 73 74 72 69  61 6c 31 c6 08 ea b7 00  |errestrial1.....|
00001fc0  1d 02 07 00 17 23 02 04  08 56 65 6e 75 73 54 65  |.....#...VenusTe|
00001fd0  72 72 65 73 74 72 69 61  6c 2f 48 06 73 00 40 1f  |rrestrial/H.s.@.|
00001fe0  01 07 00 1b 23 02 04 08  4d 65 72 63 75 72 79 54  |....#...MercuryT|
00001ff0  65 72 72 65 73 74 72 69  61 6c 13 0f 03 73 a2 f0  |errestrial...s..|
00002000

Some of the internal structure is already visible, but rsqlite makes it a lot more clear and explicit.

$ cargo run -q ./data/planets.db .dump

SQLite Database Header
  database page size:  4096
  write format:        1
  read format:         1
  file change counter: 2
  database page count: 2
  freelist page count: 0
  freelist page count: 0
  schema cookie:       1
  schema format:       4
  default cache size:  0
  autovacuum top root: 0
  incremental vacuum:  0
  text encoding:       1 (utf8)
  user version:        0
  application id:      0
  software version:    3047000
  number of tables:    ?
  number of indexes:   ?
  number of triggers:  ?
  number of views:     ?
  schema size:         ?
  data version:        ?

Page 0
  Page Header:
    Type:                    LeafTable
    First freeblock:         0
    Number of cells:         1
    Cell content start:      3877
    Fragmented free bytes:   0
  Cell Pointers:             [3877]
  First 3 Cell Types

    [String(5), String(7), String(7), I8, String(189)]

  Cells

    │ Size   │ Row ID │ Col 0  │ Col 1   │ Col 2   │ Col 3 │ Col 4           │
    ├────────┼────────┼────────┼─────────┼─────────┼───────┼─────────────────┼
    │    216 │      1 │ table  │ planets │ planets │ 2     │ CREATE TABLE... │

Page 1
  Page Header:
    Type:                    LeafTable
    First freeblock:         0
    Number of cells:         8
    Cell content start:      3836
    Fragmented free bytes:   0
  Cell Pointers:             [4063, 4032, 4001, 3970, 3937, 3905, 3871, 3836]
  First 3 Cell Types

    [Null, String(7), String(11), I16, I32, Zero]
    [Null, String(5), String(11), I16, I32, Zero]
    [Null, String(5), String(11), I16, I32, One]

  Cells

    │ Size │ RowID │ Col 0 │ Col 1   │ Col 2      │ Col 3      │ Col 4      │ Col 5 │
    ├──────┼───────┼───────┼─────────┼────────────┼────────────┼────────────┼───────┼
    │   31 │     1 │ NULL  │ Mercury │ Terrest... │ 4879       │ 57910000   │ 0     │
    │   29 │     2 │ NULL  │ Venus   │ Terrest... │ 12104      │ 108200000  │ 0     │
    │   29 │     3 │ NULL  │ Earth   │ Terrest... │ 12742      │ 149600000  │ 1     │
    │   29 │     4 │ NULL  │ Mars    │ Terrest... │ 6779       │ 227900000  │ 2     │
    │   31 │     5 │ NULL  │ Jupiter │ Gas Giant  │ 139820     │ 778500000  │ 79    │
    │   30 │     6 │ NULL  │ Saturn  │ Gas Giant  │ 116460     │ 1433000000 │ 83    │
    │   32 │     7 │ NULL  │ Uranus  │ Ice Giant  │ 50724      │ 2871000000 │ 27    │
    │   33 │     8 │ NULL  │ Neptune │ Ice Giant  │ 49244      │ 4495000000 │ 14    │

👀 The similarity between the input data, raw hexdump and rsqlite is pretty clear if you squint² hard enough.

To really understand how the data is stored, we need to take one more step. The cell pointers at the start of the page point to a set of Leaf Cells containing a Record each at the end of the page. Every record stores the size, types and values of a single database row in a compact block.

That’s pretty much all the information you need to get started.

🦋 Motivating Example 1: Database Updates

It’s really useful to have an exploratory tool of your own to deeply understand the internals. For example, let’s see how updates are handled.

$ sqlite3 planets.db "UPDATE planets SET name='🌍' WHERE id=3;"

We can make an educated guess from the rather self explanatory diff³ of the states.

--- Initial Version
+++ Inline Update
@@ -2,7 +2,7 @@
   database page size:  4096
   write format:        1
   read format:         1
-  file change counter: 2
+  file change counter: 3
   database page count: 2
   freelist page count: 0
   schema cookie:       1
@@ -45,13 +45,13 @@
     First freeblock:         0
     Number of cells:         8
     Cell content start:      3836
-    Fragmented free bytes:   0
+    Fragmented free bytes:   1
   Cell Pointers:             [4063, 4032, 4001, 3970, 3937, 3905, 3871, 3836]
   First 3 Cell Types

     [Null, String(7), String(11), I16, I32, Zero]
     [Null, String(5), String(11), I16, I32, Zero]
-    [Null, String(5), String(11), I16, I32, One]
+    [Null, String(4), String(11), I16, I32, One]

   Cells

@@ -59,7 +59,7 @@
     ├─────┼─────┼───────┼─────────┼────────────┼────────┼────────────┼────┼
     │  31 │   1 │ NULL  │ Mercury │ Terrest... │ 4879   │ 57910000   │ 0  │
     │  29 │   2 │ NULL  │ Venus   │ Terrest... │ 12104  │ 108200000  │ 0  │
-    │  29 │   3 │ NULL  │ Earth   │ Terrest... │ 12742  │ 149600000  │ 1  │
+    │  28 │   3 │ NULL  │ 🌍      │ Terrest... │ 12742  │ 149600000  │ 1  │
     │  29 │   4 │ NULL  │ Mars    │ Terrest... │ 6779   │ 227900000  │ 2  │
     │  31 │   5 │ NULL  │ Jupiter │ Gas Giant  │ 139820 │ 778500000  │ 79 │
     │  30 │   6 │ NULL  │ Saturn  │ Gas Giant  │ 116460 │ 1433000000 │ 83 │

It’s a different story when the data cannot be updated in place. The new string here is longer than the preallocated space, so sqlite does something more interesting.

$ sqlite3 planets.db "UPDATE planets SET name='The pale blue dot' WHERE id=3;"

--- Inline Update
+++ Full Update
@@ -2,7 +2,7 @@
   database page size:  4096
   write format:        1
   read format:         1
-  file change counter: 3
+  file change counter: 4
   database page count: 2
   freelist page count: 0
   schema cookie:       1
@@ -42,16 +42,16 @@
 Page 1
   Page Header:
     Type:                    LeafTable
-    First freeblock:         0
+    First freeblock:         4001
     Number of cells:         8
-    Cell content start:      3836
+    Cell content start:      3793
     Fragmented free bytes:   1
-  Cell Pointers:             [4063, 4032, 4001, 3970, 3937, 3905, 3871, 3836]
+  Cell Pointers:             [4063, 4032, 3793, 3970, 3937, 3905, 3871, 3836]
   First 3 Cell Types

     [Null, String(7), String(11), I16, I32, Zero]
     [Null, String(5), String(11), I16, I32, Zero]
-    [Null, String(4), String(11), I16, I32, One]
+    [Null, String(17), String(11), I16, I32, One]

   Cells

@@ -59,7 +59,7 @@
     ├────┼───┼───────┼────────────┼────────────┼────────┼────────────┼────┼
     │ 31 │ 1 │ NULL  │ Mercury    │ Terrest... │ 4879   │ 57910000   │ 0  │
     │ 29 │ 2 │ NULL  │ Venus      │ Terrest... │ 12104  │ 108200000  │ 0  │
-    │ 28 │ 3 │ NULL  │ 🌍         │ Terrest... │ 12742  │ 149600000  │ 1  │
+    │ 41 │ 3 │ NULL  │ The pal... │ Terrest... │ 12742  │ 149600000  │ 1  │
     │ 29 │ 4 │ NULL  │ Mars       │ Terrest... │ 6779   │ 227900000  │ 2  │
     │ 31 │ 5 │ NULL  │ Jupiter    │ Gas Giant  │ 139820 │ 778500000  │ 79 │
     │ 30 │ 6 │ NULL  │ Saturn     │ Gas Giant  │ 116460 │ 1433000000 │ 83 │

The whole row is written to a new location (at offset 3973) and the pointers are updated. Now why is the new pointer offset smaller than the previous one?

Docs got you covered.

Cell content is stored in the cell content region of the b-tree page. SQLite strives to place cells as far toward the end of the b-tree page as it can, in order to leave space for future growth of the cell pointer array.

By inserting cell pointers at the start of the page from left to right and data at the end of the page right to left, SQLite is aiming to not waste any disk space pre allocating any blocks or sections and keep the database as small as possible. The first cell pointer entry 4063 with the largest offset is the earliest record, written to the end of the page. The rest of the cell pointers are monotonically decreasing [4032, 4001, ...] as expected until an update breaks the order and the cells have to be rearranged.

The first_freeblock is presumably tracked for the next insertion small enough to fit there.

💡 This is the category of things I would have never learned if I didn’t write this from scratch or paid enough attention.

💨 🧹 Example 2: Vacuum

What really is vacuum?

The VACUUM command rebuilds the database file, repacking it into a minimal amount of disk space.

Let’s just run it and see what happens.

$ sqlite3 planets.db "VACUUM;"

And we have this diff again.

--- Full Update
+++ Vacuum
@@ -2,10 +2,10 @@
   database page size:  4096
   write format:        1
   read format:         1
-  file change counter: 4
+  file change counter: 5
   database page count: 2
   freelist page count: 0
-  schema cookie:       1
+  schema cookie:       2
   schema format:       4
   default cache size:  0
   autovacuum top root: 0
@@ -42,11 +42,11 @@
 Page 1
   Page Header:
     Type:                    LeafTable
-    First freeblock:         4001
+    First freeblock:         0
     Number of cells:         8
-    Cell content start:      3793
-    Fragmented free bytes:   1
-  Cell Pointers:             [4063, 4032, 3793, 3970, 3937, 3905, 3871, 3836]
+    Cell content start:      3824
+    Fragmented free bytes:   0
+  Cell Pointers:             [4063, 4032, 3989, 3958, 3925, 3893, 3859, 3824]
   First 3 Cell Types

     [Null, String(7), String(11), I16, I32, Zero]

We can make the following observations.

File change counter went up by 1 as expected. Schema cookie also changed, but I have no idea why.
The free_block is gone, SQLite successfully reclaimed the space in the middle 🧼.
The rows are ordered once again! I can imagine how running vacuum occasionally can reduce fragmentation and improve query performance, especially if you don’t have any indexes.
The cell_content_start used to be 3836 in the first version, moved left to 3793 after second update due to fragmentation and now back right to 3824. All tidy.
Compared to the first version, the cell pointers of rows 3-8 changed, only 1,2 remained where it used to be. This is hinting that on large tables with a lot of changes, vacuum could be potentially very expensive.

🧑🏼‍🎓 Miscellaneous lessons learned

Other miscellaneous things I learned this week while working on this.

🦀 Rust is very good at this kind of programs

The code to parse the core data structures at schema.rs is only about ~130 LOC excluding docs, tests and the pretty printer. It took me a moment to understand⁴ the amazing binrw library I used to read/write the binary data, but it was remarkably simpler than I expected. Would definitely recommend.

⚖️ SQLite loves variable length encoded numbers

A substantial fraction of schema.rs is dedicated to SQLite’s variable length encoded numbers called varints.

From the docs,

A variable-length integer or “varint” is a static Huffman encoding of 64-bit twos-complement integers that uses less space for small positive values. A varint is between 1 and 9 bytes in length. The varint consists of either zero or more bytes which have the high-order bit set followed by a single byte with the high-order bit clear, or nine bytes, whichever is shorter. The lower seven bits of each of the first eight bytes and all 8 bits of the ninth byte are used to reconstruct the 64-bit twos-complement integer. Varints are big-endian: bits taken from the earlier byte of the varint are more significant than bits taken from the later bytes.

SQLite source includes a standalone tool/varint.c which can be compiled into a small executable to convert numbers varint ↔ decimals. Alternatively see varint.rs for a naive readable implementation in Rust.

$ cc tool/varint.c -o varint
$ ./varint 1992
1992 = 8f 48

Numbers 1-128 would be represented in a single byte, but 128 for example would need 2 bytes [0x81, 0x00] and 1992 would be [0x8f, 0x48].

I would naively assume that varints are substantially slower than fixed size numbers but SQLite must have good reasons to make this tradeoff. Maybe real world datasets contain a disproportionate amount of small numbers and the size reduction must be worth the additional complexity. There might even be a perf gain due to reduced disk IO. Please let me know if you have more background context on this.

🎪 The native data types are a very small set

Every datatype understood by SQLite at a storage later is documented at § Record Format and it’s a pretty small set. Notably there is a dedicated type for numbers 0 and 1 and none for booleans or native timestamps. SQLite is not well typed by design and that is easily my least favorite part of the whole project.

🚀 Questions I want to ask later

I found it really useful to have this project as a starting point to ask more deeper questions I’ve always had about databases. Recently for an unrelated work I was looking deeper into DuckDB and how row storage differs from columnar storage, but you get a much better understanding by dipping your toes into the code a bit beyond only reading docs. It’s also a vehicle for experimentation for me to learn about databases in general, which I’m really looking forward to.

A few questions I’d like to figure out:

I have only explored the tip of the iceberg here implementing 1 of the 9 different page types. The current version would fail for most non trivial tables spanning multiple pages for example.
Figure out how indexes work. How do they maintain the sorted order?
Are there any tricks to handle really wide rows with 100s of columns?
Parsing the file for metadata is a non trivial amount of work even with all the information available in the headers. How much of this state is maintained internally and reused between queries?
Types are stored inside each and every record and that feels incredibly wasteful for properly well typed databases or newer STRICT tables. Figure out if there are ways to make this more efficient.
Transactions, Journals, Atomic Commits, MVCC, Vacuum, WAL.
Explore projects in the wider ecosystem like litefs or libsql.

🔗 Other useful references

SQLite Internals: Pages & B-trees by fly.io has a pretty good intro on varints and record structure. The blog briefly covers the various types of btree pages in the “Growing a tree” section, but my favorite is when Ben asks “OK, But Why?” ⁵ and follows up with some simple answers.
Julia Evans’s How does SQLite work? takes a very different approach to the exploratory problem and the blog is nicely complimentary.
Build your own SQLite by Geoffrey Copin covers a whole lot more than I did, but our implementations differs a lot.
Let’s Build a Simple Database has some useful info, but the project is now unmaintained. The spiritual successor is CodeCrafter’s Build your own SQLite project, but I personally won’t recommend it. The progression b/w subsequent steps are non linear and it’s not easy to follow through.
SQLite File Format might help if you are starting from scratch again.
Definitely SQLite source. See the DB Header description here for a really good example.

That’s it. FIN 👋🏼

It took me more time to generate these diagrams than write the entire code. I used an LLM to generate the basic SVG outline and edited the rest by hand. I found popular GUI SVG editors incomprehensible and I really couldn’t make it do the basic things I wanted. ↩
I’d love to build a UI tool that can show the bidirectional link between the raw hexdump and parsed data structures like Godbolt shows the relationship b/w the high level programming language and the generated assembly. ↩
$ diff -u -d --label 'Initial Version' state_init.txt --label 'Inline Update' state_inline_update.txt ↩
The library makes very heavy use of proc_macros, which I still don’t understand very well. It’s me, not them. ↩
An earlier version of this blog did had the same section but I replaced it with with 2 motivating examples. Sometimes knowing little details can make a big difference. I could have massively reduced the size of some time series datasets I worked with previously if I stored small integers (as low as 1 byte) instead of floats with decimal points (always 8 bytes). Lookup Gorilla paper for more info. ↩

Copy URLs from Safari or Chrome with a keyboard shortcut

Wed, 25 Sep 2024 00:00:00 GMT

I really love the Arc Browser, but after last week’s major security incident, I wanted to look at some alternatives. They did a hundred little things to make it the most polished web browser I have ever used and I will definitely miss it.

Anyway, I went back to Safari and its many paper cuts. I got used to most of the quirks, but I would end up pressing ⌘ ⇧ C a dozen times daily to copy the URL, only to be presented with a very silly element selection feature in dev tools, which I never need.

I figured it would be easier to fix the browser than retrain my muscle memory, and thankfully, there are enough knobs built into macOS to make this not so painful.

Step 1: Write some Apple Script.

I found the rough template on reddit, which I didn’t have to tweak much except disabling the conflicting keybinding.

Create a new service in Automation.app
Select Safari.app, with “workflow receives” set to “no input”.
Use the following AppleScript¹ code, and save it with a name like “Copy Safari URL”.

on run {input, parameters}
    tell application "Safari"
        set theURL to URL of front document
        set the clipboard to theURL
    end tell
end run

Step 2: Enable the keybinding in keyboard settings.

Settings > Keyboard > Keyboard Shortcuts > Services > General

Step 3: Disable the conflicting command

Found the basic idea of disabling inbuilt shortcuts from this StackOverflow post. You find the exact name of the command you want to disable (‘Start Element Selection’), and rebind it to something completely different.

Settings > Keyboard > Keyboard Shortcuts > App Shortcuts > Safari

🎉 📋 Happy Copy Pasting

Update 1: Ditched Safari, went back to Chrome

Safari’s paper cuts add up, and it’s time to give Chrome another try.

Safari does not respect “Always open tabs instead of windows” consistently, it’s infuriating.
Safari’s vertical tabs implementation is still an alpha product.
Tab groups are also a good idea, but the implementation is still pretty rough.
The tab sync across devices (iPad & iPhone) is pretty neat though! Will miss it.

Update 2: Ditched Chrome, went back to Arc again

My daily driver is a LG 27“ 5K monitor and no website really needs more than ~1200px. That’s about half of the available horizontal width. Vertical tabs are so essential for me on desktop, and Arc has the best implementation that I know of.
I tried to update the shortcut on Chrome as well, but couldn’t disable the conflicting inbuilt command. The shortcut is documented here, but the exact command string isn’t.

End of the day, Arc is easily the most polished desktop browser experience on macOS. They seem to be trying to resolve the incident from last week as best as they can, so I’m willing to give them another chance.

🌈 Back to Arc again! Here is my 🎟️ invite code if you want to give it a try.

AppleScript must be the most bizarre and quirky programming language ever invented. I can write code in over a dozen programming languages but AppleScript is the most incomprehensible of them all. ↩

This blog doesn't track you anymore!

Sat, 29 Feb 2020 00:00:00 GMT

I avoid 3rd party tracking on the web as much as I can with several ad blockers but then forcing it on the readers of this blog felt very hypocritical.

So I’ve removed all Google Analytics from this blog as of now!

Fonts, CSS and some icons are loaded from 3rd party CDNs, which I hope to inline soon. There is a link to Recurse Center below and it contains a unique token for this blog, but it should not track you.

Enjoy your privacy and slightly faster loading times 🏎

What I talk about when I talk about Programming

Fri, 25 Oct 2019 00:00:00 GMT

Broadly speaking there are programmers who think of programming as a means to an end and as an end in itself. The instrumental and intrinsic value appeal very differently to different people.

I’m usually far more excited about how the TCP packets go from one end to the other rather than what’s in them. Cat pictures, banking ledgers or irrelevant spam - they are all the same to me at an abstraction below.

I must have spent 10000 hours tinkering with programming languages and compilers in my free time. I’m reasonably proficient at working with static types (like Rust), functional programming (like Erlang) and the beautiful intersection between them (like Haskell) even though in the last 5 years, I’ve barely used any of it at work.

When I look back at all the side projects I’ve built over the years, none of them had any implicit value except the fun I had learning about seemingly pointless and obscure things. I happen to know a lot about arcane x86 calling conventions now even though I’ll probably never need them in my career. A few summers ago I was micro optimizing LLVM IR trying to save single-digit bytes of memory while I’m probably going to be spinning up r4.8xlarge machines on AWS for the rest of my career. Sometimes I genuinely wish I didn’t know about any of this so I’d be less opinionated and hence less frustrated about broken software.

It’s difficult to find your space in a world where 99.99% of programming is product engineering where you pick whatever tools necessary to get the job done and mechanical sympathy is seen as an undesirable trait.

Title inspired by 📚 What I Talk About When I Talk About Running

My favorite rust function

Fri, 11 Oct 2019 00:00:00 GMT

My favorite rust function is std::mem::drop which is used to free or deallocate a value, similar to free() in C.

Quoting the docs from stdlib, This function is not magic; it is literally defined as

pub fn drop<T>(_x: T) { }

How could a function definition with an empty body ever be useful?

Ownership and lifetimes are Rust’s most unique features and it gives you the predictability and performance of static memory management without any of its safety problems.

The ownership rules are pretty simple.

Each value in Rust has a variable that’s called its owner.
There can only be one owner at a time.
When the owner goes out of scope, the value will be dropped.

Calling a function moves the ownership of the arguments and if the called function decides to do nothing with it, the arguments will be dropped - which is exactly what we wanted!

Now this might seem like a hack, but it really is not. Most languages would either ask the programmers to explicitly call free() or implicitly call a magic runtime.deallocate() within a complex garbage collector.

The beauty of programming language design is not building the most complex edifice like Scala or making the language unacceptably restricted like Go - but giving the programmer the ability to represent complex ideas elegantly and safely. Rust really shines in that regard.

A language that doesn’t affect the way you think about programming is not worth knowing. ― Alan J. Perlis

This post was edited after publishing, see the full history on github.

An Incremental Approach to Compiler Construction 🎤

Tue, 17 Sep 2019 00:00:00 GMT

I spoke at Papers We Love, London about my toy compiler and the video is up on YouTube!

Wealth of Nations

Wed, 29 May 2019 00:00:00 GMT

I grew up and spent the first 25 years of my life in India. I lived a short while in NYC to attend the recurse center and now I live in the UK. I saw a bit more of the tiny blue marble in the meanwhile.

Hans Rosling says in Factfulness that you cannot divide the world into broad “First” and “Third” world categories anymore and the divide is closing; but as someone who got to see a fair bit of both sides of the spectrum, it is my opinion that the divide is unspeakably huge and is much worse than he paints it to be. I earn almost 2x per day today than my first monthly salary out of college doing pretty much the same job. Life can change a lot when you get paid 70x.

India is unspeakably poor. Purchasing power is around 120th among the 180 countries tracked by the world bank. Literacy rate is around the same bracket. What the average British could take for granted a 100 years ago won’t happen in India for another century. Looking at the current state of the nation, it is only getting much worse.

Not a day go by without me thinking over and over again how fundamentally different the rich first world and the rest of it is. Yes I've read Factfullness and I think he missed a party of elephants in the room. The more I see, read and learn the more depressing it gets.
— Jaseem Abid (@jaseemabid) May 19, 2019

I’ve been thinking a lot about what makes some countries wealthy and some poor. I enjoy learning macro economics and I hope to spend time in the next few months learning more; hopefully updating this post with details.

What's the modern equivalent of Adam Smith's Wealth of Nations? I normally enjoy reading #economics and I'd love to read more about how nations build their economies and the factors that make them wealthy or poor. I'd like to get over my own biases. #book #suggestions #goodreads
— Jaseem Abid (@jaseemabid) May 26, 2019

Here are some of the book suggestions from the twitter thread.

1. The Wealth of Nations by Adam Smith

I’m honestly not sure if much of this is relevant anymore since he spends maybe 30% of the book talking about wheat prices in London. Its over 1200 pages long and a bit too dry to keep reading.

2. Why Nations Fail: The Origins of Power, Prosperity, and Poverty

I’ve a paperback already, so this might be the first in the list.

Why nations fail | James Robinson | TEDxAcademy

3 Debt: The First 5,000 Years by David Graeber

DEBT: The First 5,000 Years | David Graeber | Talks at Google

4. Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty

5. Development as Freedom by Amartya Sen

6. The Rise and Fall of Nations: Forces of Change in the Post-Crisis World by Ruchir Sharma

7. The wealth of Networks by Yochai Benkler (pdf)

Let me know your thoughts. You can reach me over email or ~~@jaseemabid~~

Amsterdam, A history of the world's most liberal city 📚

Fri, 10 May 2019 00:00:00 GMT

I finished reading “Amsterdam, A history of the world’s most liberal city” last week and absolutely loved it. 5/5 🌟

The book is equally about a place, the people and an ideology it stands for - liberalism. Gedogen in Dutch means “illegal but tolerated” and Amsterdam seems to have developed this curious and unique attitude to so many social issues. There aren’t so many places in the world where human rights are equally respected, where you can smoke weed legally, sit by the canals appreciating architectural marvels, bike around almost anywhere without being run over by cars and appreciate some Rembrandt and Van Gogh in the most majestic art museums. The vibe is just magical and unreal. If you are curious about why things are the way they are, it’s impossible to leave the city without getting a few history books and that’s exactly what I did at the American Book Store.

The narration is beautiful. This is definitely a history book - the author describes how they built a city by draining the sea and building a dam to prevent the floods and how the Dutch East India Company made them immensely rich with trade and colonialism. We also meet real people who start over their lives leaving behind everything and moving to unknown lands to become early settlers. We read about true pioneers like Aletta Jacobs who made remarkable progress fighting for women’s rights and suffrage and Benno Premsela for gay rights. We meet celebrities like Anne Frank, Rembrandt and Spinoza. History books can’t just describe a sequence of events - this one is an avalanche of emotions, people and stories you cannot forget.

New and Old Amsterdams are 2 of my favourite cities in the world and this book made me realize again how connected they really are. Manhattan’s grid layout impressed me more than every other human achievement combined - sending humans to Mars looks so simple in comparison. This could be because I spent the 5 years before moving to NYC in one of the world’s most congested cities were going from one end to the other was an unspeakable nightmare. Every single day I commuted the length of the island from 125 St to Canal St without it being a ceremony, I’ve asked myself how NYC could do in 1811 what India cannot do now 200 years later. How could some parts of the world be centuries ahead of others?

The book describes an incident in which the English had to find a Dutch translator more than 100 years after the Dutch left to negotiate a trade deal with the native Indian population because it was the only European language known to them. Somehow Manhattan’s grid is so obvious after Amsterdam’s canals. It might not seem like the most astonishing human achievement to you, but I had such a great moment of epiphany connecting the dots.

NYC and Amsterdam both evolved to becoming the most liberal and cosmopolitan cities in the world. Both were at some point world’s wealthiest cities, largest ports and after opening doors to flocks of immigrants during days of prosecution and crisis, the most diverse as well.

To understand the extent of the VOC trade empire at the height of the Dutch Golden age, we could look at this would map from 1630.

Just 100 years after the first European reached India via the sea, we have a map that looks remarkably similar to the precise GPS versions we have now. This is before most of the science as we know of was invented. The company that dominated world trade also lead to one of the first stock exchanges where the average citizen invested some of their savings to lay the foundation of modern capitalism.

To be honest, even though I just said all that, during the short time I spent there I could barely find any brown people in the cafes and pubs I spent hours reading. The Dutch were liberal, but they definitely made life in Indonesia and other colonies hell and they waged wars all over the world including one quite close to where I was born and brought up. More Jews died proportionately in Amsterdam than almost anywhere else in Europe during WW2. There was definitely a time when the Catholic priests and nuns were stripped and killed and churches desecrated. A prime minister was killed on the streets and decapitated. Dutch Immigration laws aren’t great compared to Germany. I took me 46 days to get a 45 day Schengen visa.

There is a whole lot more. A brief glimpse into the Dutch monarchy and we learn how closely tied every European monarchy really is. The dutch took over the English crown once without even a war. For a while 30% of all the books published in world were from Amsterdam. Roots of religious tolerance and how Protestantism replaced Roman Catholicism. There is a lot said about how NYC influenced the rest of America and what it stands for. Obviously a lot about the canals and the tulip mania. This book made me think hard and I loved it ❤️

Loneliness

Sat, 04 May 2019 00:00:00 GMT

The difficult part about moving halfway across the world is the loneliness that comes with it. No matter what you do and who you meet, often it’s just impossible to replace those loved ones.

Small lessons I learned at Monzo 🚀

Sun, 13 Jan 2019 00:00:00 GMT

I’m a Platform Engineer at Monzo since August 2018 and a recent conversation made me think of all the things I’ve learned over the last 4 short months. I wanted to share about how we work and what I’ve learned here.

Briefly, the Monzo banking platform is roughly 820 micro services on Kubernetes. Some of them run a handful of replicas while others run into over 100. As of writing, we run over 6200 pods in about 350 nodes. Except for a small set of physical servers required to connect to various payment networks, everything else runs on AWS. Almost all data is stored in Cassandra with a little bit of Elastic Search. Kafka and NSQ act as message brokers. Prometheus and Grafana offer extensive monitoring. See the blog Building a Modern Bank Backend for more details.

The platform team is split into 2 squads, one that concentrates on the infrastructure and reliability while the other takes care of engineering effectiveness. I’m part of the former and that in practice means we provision, scale and monitor the infrastructure required to run the bank and continuously make improvements to support a rapidly evolving product and the roughly 100 engineers working on it.

Monzo is the largest ~~company~~ project I’ve ever worked on in terms of users, engineers, lines of code and scale. A ton of the software we use was new to me and I had lots of fun (and a fair share of anxiety) learning them all as quickly as possible during the first 2 or 3 months. I’ve learned a lot about how some specific software works like Kubernetes, Cassandra and Wireguard but the soft skills or life lessons were far more valuable. When I look back, the relief that not all software engineering jobs in the world suck makes me so much happier 🙏.

1. Culture can make an enormous difference 🏦

We have a diverse workforce building something together that they (and their moms) would love to use and would be proud of. I worked on developer tools at a previous job and every single engineer on the project I knew of passionately hated the project and never used any of it. It is impossible to be a productive engineer and look ahead while you are soaked in so much toxicity. I see employees excited about what we can already do and the mountain of possibilities ahead of us. Conversations change from bitching about management to knowledge sharing and cool feature updates over lunch. You go back to your desk happy. It is a very powerful feedback cycle.

We acquired a million customers in just about 3 years while maintaining a NPS score over +80. We had an incredible crowdfunding round raising 20M£ in just under 3hrs which shows how much customers believe in the product. I think behind every milestone, there are a lot of happy employees loving what they do. This manifests as proactively sending customers new cards even before they realise that British Airways leaked their information or the whole company joining hands to answer support queries when the wait times got high.

2. Scaling teams and code 👩‍💻

Almost all of the application code is in Go and each micro service is a top level folder in a monorepo built with a few shared libraries and tools and works incredibly well for us. New engineers can start with a service without having to learn about the whole platform. Experiments and old features and get cleaned up when it’s not needed anymore because its usually just deleting a service and a few callers into it. Service deployments are independent of each other and usually, there are more than 50 deployments every day.

I learned how valuable good tooling is. A small engineering effectiveness team can make the lives of so many engineers so much better when the tools to deploy, monitor, check logs etc “just work”. If you can deploy and rollback safely with just one single command, it encourages engineers to deploy small and often making the whole process a lot safer. This improves confidence in doing things and helps engineers iterate so much faster. I’ve wasted days in the past building completely unrelated downstream projects after changing shared Jenkins libraries. Never again.

💡 Invest in tooling.

3. A good orientation matters 🧭

Please don’t link your next new engineer to a git repo and README.md and expect them to find everything on their own. Every company have unique ways of doing things, the context that is not written down anywhere and broken processes that everyone is not aware of.

A “buddy” 👫 for the first week or so just to show you how everything works can be a huge bonus 💪. I started at Monzo looking into how we run our database cluster and mistakes there would have been costly. I cannot overstate how much having someone to ask dumb questions helped. Come back @spikelindsey!

4. Scaling infrastructure 💾

I’ve never been in a platform/infrastructure/devops role before and I had to start at level 0. The most complicated thing I had done with AWS before Monzo was to setup a VM for a tiny web application. It took me quite some time to get used to the nuances of Terraform to manage a large production fleet. It took a while to clean up some of the cruft left behind, but we are very close to the state of the entire infrastructure described in Terraform and Puppet and “Infrastructure as code” is taken for granted at Monzo. Being able to raise a Terraform PR and discuss it instead of clicking random buttons in the EC2 UI 🕵️‍♀️ made it so much easier for a newbie like me to get work done.

Not rocket science 🚀 but you should switch to a tool that lets you describe infrastructure as code asap.

5. Technical debt 🕸

I learned that at the heart of it, technical debt is a cultural problem than a technical problem.

When I started, the puppet code we used for configuring infrastructure at Monzo was really old and crufty, but now we are actually spending time and effort to clean it up. I was mad about not version controlling our Grafana dashboards and not syncing them across prod and staging environments - I fixed it last week. Some of our critical infrastructure diverged from the version controlled config, we fixed it too. We needed centralised logging and distributed tracing, both got shipped in the last 3 months. The operations around Cassandra improved massively over the last 6 or 7 months. And several other projects that I’m not aware of are getting fixed right now.

There is cruft and mess in every large software out there but they differ in how people deal with it. Unless you have a culture of fixing things up once in a while, eventually it compounds so much to make it hell. Under some conditions, technical debt is a useful tool but you need to be careful about it. It’s a thin line. The next time I come across something badly broken at Monzo, I know I can eventually get it fixed and won’t have to live with it forever. Building that culture is hard, but would pay off massively in the long run.

6. Monitoring 📈

Good infrastructure starts with good monitoring and alerts. I’ve never worked with any sort of serious monitoring before and I do not want to go back. I learned Prometheus and Grafana at Monzo and they are both really good at what they do and I developed some sort of fascination for pretty informative charts. Each graph is a story. Here are some random ones I liked:

We changed the way Kubernetes pods connect to the database and you can see in the first graph how one policy changes to the other cleanly and in the second graph how the cluster picks DB nodes more intelligently considering several factors rather than a simple round robin. This would be impossible to understand from just logs.

![](./round to hostpool.png)

This graph on open file descriptors helped us track down a leak in the Prometheus JMX exporter and fix it.

This one shows how Kafka acts as a nice message buffering queue and lets us restart databases for a short time with no visible downtime.

And a few more.

We have a strong feedback culture and people do it frequently. It helps you learn about the problems early and act on it.
I went to an actual data centre and learned a little bit about how the physical infrastructure works. Fascinating.
Making processes inclusive for remote workers forces us to write down a lot more documentation which is awesome.
Cassandra is a really really complicated piece of software to understand, use and operate. I’m having a love hate relationship with it right now. More on it later.
Go works really well for Monzo but I’m not a fan of it. Almost everyone else seems to love it. I’m extremely biased when it comes to programming languages because I spent most of my free time thinking about Haskell.

It a been a long one with lessons, rants and stories. Let me know what you think. Thanks a lot to @indradhanush92, @aishpant, @vimalk78 and @rusrushal13 for proof reading early versions of this draft.