I wrote an agent that’s so good I can’t beat it. Here’s how it works.

First, think about the simplest possible agent. When it’s their turn, they see what actions they have available. These actions are each of the columns they could drop the chip into. There are 7 different choices (unless some of the columns are full, then there are fewer). The simple agent just randomly picks an available column. It doesn’t know anything about Connect4 or what move is best, it just picks one. This agent isn’t very good.

Another approach would be to try and write a smart agent that understands Connect4. Maybe there’s an optimal strategy, it could detect twos and threes in a row to build on or protect against, or maybe it’ll use heuristics for tricking the other player.

Or maybe we could train a machine learning model to analyze past games and figure out which game states are good and which actions lead to them. We could train it on professional Connect4 Twitch streams.

I didn’t do any of that.

The way my agent works is like this. Instead of using any sort of logic about the current state of the game, the agent runs simulations to figure out which move is best.

It needs to figure out which move has the highest probability of winning. It does this by trying every move. It copies the game state for each possible move and takes it. These are all the possible next states. To see which one is the best it simulates 10,000 “random” games that start from each of these states. In the simulated games it just picks random moves for each player. After these 10,000 games, it scores the state.

wins - losses / 10,000

Whichever action led to the state with the highest score is the action it takes in the real game.

So on each turn, the agent plays up to 70,000 games of Connect4 before picking a move.

This algorithm is called Monte-Carlo Tree Search. You can think of the game states as a tree where every action you could take is an edge leading to the next game state. The tree expands out to the leaf nodes which are the endgame states, where a player wins or it’s a tie.

Tree of Game States

This tree is huge since most nodes branch into 7 more. We can’t just traverse the whole tree so we do a bunch of depth-first random samples (the simulated games) to try and find a path more likely to lead to us winning.

This is cool because we don’t need to “understand” Connect4 to write the agent. There’s no logical evaluation of how “good” a move or game state is. We just need to simulate matches. The logic for turn selection is not much different than the random agent, we just pick a lot of random actions until we get a good vibe.

I’ve been doing a lot of research lately on game-playing agents. There are lots of different algorithms for different kinds of games. Connect4 has a lot of game states so it’s hard to just fully compute the probabilities of winning for each state and action. (You could do this in a simpler game like tic-tac-toe and can just calculate every possible move for every possible game.) Connect4 is easy to simulate though so this works well.

The agent is written in rust. It’s not really optimized, just runs all the simulations on a single thread, but it still goes fast enough (on my laptop) to play against.

The code for the agent (which you can play against) is here. https://github.com/saolsen/connect4 Good luck trying to beat it!

There were a few hard problems I ran into back then that stopped me from shipping it.

1. Consuming Twitter data is really annoying.

2. Telling which tweets are about products is hard.

3. I barely even knew how to program.

I still think it's a pretty good idea so I did a little hacking on it over Christmas break

All of these are still problems today, but I think I can work around them. Twitter data is still annoying to consume so I decided to use Reddit data instead, which is slightly easier to scrape. To start, I decided to scrape book-related subreddits. I can use some off-the-shelf machine learning model to answer, "Is this post about a book and does the person like the book or not like the book?". And as far as being bad at programming goes, I just don't let that stop me anymore.

Reddit has some APIs to get data but I don't feel like doing the work to obey rate limits. I just found a tool that already does it, the Bulk Downloaded for Reddit (bdfr). I can point it at a subreddit and it will pull a bunch of posts and comments. Perfect.

After running bdfr I get a directory full of JSON files, one for each post. I wrote a rust program to parse the file and store the posts and comments in a DuckDB database.

This is where I hit a problem. Some of the posts with lots and lots of comments (each having replies that are also comments) created some extremely nested JSON objects. They were so nested that serde (the library to parse JSON in rust) was erroring out, hitting recursion limits. Also, I was using the DuckDB crate and it compiled all of DuckDB into my program, which takes forever when done from scratch.

Because of these errors, I did something I never thought I'd ever do. I rewrote my rust program in python for performance and maintainability.

The last step in collecting the data is just to run the program often. bdfr can only scrape the "new" feed, the "top" feed, or the "hot" feed so I need to check regularly to get all the posts and comments. I spun up a GitHub actions workflow to do it for me. I think this will work for at least a week or so until the data set is too large to even use git lfs. Then I guess I'll have to put them in s3 or something.

I mostly work on macOS. As far as debuggers go lldb is really the only one you can use. It's what all the "codes" use (Xcode and VScode) and any other debuggers use. It's actually almost impossible to use anything else because the way ptrace works on macOS (the syscall that lets you do debugger stuff like pause execution) doesn't let you read or write registers. You would also have to use private APIs so you couldn't distribute the debugger without dealing with some weird codesigning stuff and some system preferences stuff people would have to enable and it's all just kinda annoying. This is probably why everybody just builds on top of lldb.

I did discover something in my research though. lldb has two parts. It has a server, the actual process that makes the ptrace calls and inspects registers and whatever else, and it has a client, the lldb cli that you type commands into. The server process is called debugserver and it ships with macOS as part of the Xcode command line tools. It's possible to write your own client that talks to debugserver directly, which means you can build a debugger that can do all the same stuff lldb can do, but without having to use any system APIs yourself. This seems promising and fun.

debugserver and lldb talk over the gdb remote protocol. It's a low-level way for lldb to say stuff like "set this breakpoint" or "continue execution" and debugserver to reply. You can see all the packets sent between lldb and debugserver by starting an lldb session and turning on gdb-remote logging with log enable gdb-remote all -f logfile.

I compiled a simple program, hello_world.c.

#include <stdio.h>

int main()
{
    printf("Hello, World!");
}

Then I ran it with lldb and turned on logging. I just ran the program, no breakpoints or anything.

❯ /usr/bin/lldb test/a.out
(lldb) target create "test/a.out"
Current executable set to '/Users/steve/dev/debugger/test/a.out' (arm64).
(lldb) log enable gdb-remote all -f lldb.log
(lldb) r
Process 48876 launched: '/Users/steve/dev/debugger/test/a.out' (arm64)
Hello, World!Process 48876 exited with status = 0 (0x00000000) 
(lldb) 

The log file for this is HUGE. lldb does a lot of work before even starting the process. It sends commands to look up information about the processor like its registers. It sets a few different breakpoints in startup code and runs to them so it can look up things like what dynamic libraries are loaded and other things I can't even really tell yet. Finally, at the end, it sends a continue command and gets back a process exited reply.

<   5> send packet: $c#63
<   7> read packet: $W00#00

Cool, can we just do that without all the extra stuff lldb does? Time to write a debugger. First, launch the program under debugserver. It's put in a very specific place when you install the Xcode command line tools so it'll probably be here for everyone, if it isn't try xcode-select --install.

❯ /Library/Developer/CommandLineTools/Library/PrivateFrameworks/LLDB.framework/Resources/debugserver 0.0.0.0:1234 test/a.out
debugserver-@(#)PROGRAM:LLDB  PROJECT:lldb-1400.0.38.13
 for arm64.
Listening to port 1234 for a connection from 0.0.0.0...

Then we can connect to that, I'm using telnet (you can get it with brew install telnet).

❯ telnet 0.0.0.0 1234
Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.

Then we can try sending just the continue command. Commands are a $ followed by the command, followed by a checksum. The gdb server protocol works over TCP but it also works over serial and there's a lot of stuff in it about acking responses and requesting re-sends when things mess up.

$c#63
+$O48656c6c6f2c20576f726c6421#c2

That's not the $W00#00 I expected but it's even better. $O is console output and the rest of the message is hex-encoded ASCII. Decoding it gives Hello, World!.

We can send + to say we received the message and then it gives us what we were looking for. The whole session looks like this.

Trying 0.0.0.0...
Connected to 0.0.0.0.
Escape character is '^]'.
$c#63  
+$O48656c6c6f2c20576f726c6421#c2
+
$W00#b7
+
Connection closed by foreign host.

And there you have it, you don't have to use lldb anymore, now you can debug your programs with debugserver and telnet!

Since next week is the first of the year we'll call that the first actual post. Here's a little dump of some things I've done or discovered this year to prepare for that.

Last Year

I really learned a lot about "modern" front-end stuff this year. There are some really exciting things going on. I knew a bit about nextjs and react because we use them at work. There are a lot of other cool things in that ecosystem like https://trpc.io/ that make the frontend/backend divide a lot easier. I played around with Deno and Astro and Solid and even Leptos. These are all worth checking out and could be whole posts of their own in the future.

I also played around with a bunch of new products that I like a lot. Supabase and Planetscale and Clerk were all really nice finds and I like them all for different reasons.

One thing I started actually building is a programming language for random data generation. I was calling it "AmpleSamp" and I learned a lot building a small prototype. I built a little virtual machine which was really fun, I learned some new tricks from Crafting Interpreters while I did it. I never got to the real features of the language which are "constrained random value generation" and I also sort of think a custom language for this isn't a great idea anymore. I'm going to try this again in the future just as a library or even just a configurable model thing. You can check out the prototype and the code if you want. I build the front-end in Astro and learned a lot of CSS in the process.

This Year

I'm not sure what I want to get into in the future. I have some ideas.

• A new kind of graphical debugger/VM hybrid that shows you exactly what the computer is doing.

• A bot that plays some sort of game. Previously I've tried to write bots for the binding of Isaac but getting past the "recognize things on the screen" part was always too hard. Maybe a card game or a board game or something where I can play with some fun algorithms like Monte Carlo tree search rather than get into deep learning or something like that.

• Make a game?

• Start some simple products? IDK

• Write a book?

• Some sort of procedural generation art or game?

• A new blog? (jk hopefully)

Who knows what I'll do? If you wanna find out you can learn all about it here.

Footnotes

1. Was introduced to the "weeknotes" format by Justin, the guy that runs Buttondown. It's a really cool newsletter where he writes about building Buttondown.

Check it out!

https://www.osohq.com/post/cross-platform-rust-libraries

That isn’t a joke, it’s kind of crazy how much this book fits into its ~200 pages. First, it teaches you AWK, the whole language is covered in about 60 pages. Then it shows you how to build some stuff. These projects include.

• Data Processing (sure, classic AWK)

• Generating reports. (ok, makes sense)

• A word processor. (uh)

• A parser, compiler, assembler, and multiple programming languages. (what?)

• A bunch of CS basics, sorting, searching, etc. (sure, why not)

Yeah, all in AWK, all in 200 pages.

The simplicity of AWK is a big reason this book works. You can sit down and learn it in a few hours and the rest of the book can rely on that to explore a bunch of awesome things you can program with it. I wish more books were like this.

Go read it, you can just google the title and find the PDF. Make sure you do it before December so you can do all your Advent of Code1 problems in AWK.

Pretend you’re a computer for a sec

Sombody has given you a program to execute. It is a list of instruction that somebody smart (a programmer) using some smart tools (a programming language and a compiler) translated into something a dummy like you (a computer) can do. Lets start with a super simple model of how computers work and see if we can answer our question.

Computers have a few places they can keep numbers called registers. They can do some math on the numbers in these registers (like ADD the values in two registers together and put the result in another register). They also have a much larger place to store data called memory. Memory isn’t super close to the computer so they can’t do any operations on data stored there, all they can do is LOAD a pice of data from memory to a register and STORE a piece of data from a register back to Memory.

Lets assume our list is in memory and we’re going to execute a program that adds up all the elements. An array is a contiguous block of numbers in memory. All you need to know to handle an array is the address of the first element and how many elements there are.

An array in memory that starts at address 5 might look like this.

...
5: 1
6: 2
7: 3
8: 4
9: 5
...

A program to add the elements in an array could look like this.

p_array = 5 // address the array starts at
length = 5  // length of the array
i = 0       // index of the element we're on
result = 0  // sum of all the elements
// Loop over the array
loop:
  p_element = ADD(p_array, i)      // address of element
  element   = LOAD(p_element)      // load the element
  result    = ADD(result, element) // add element to result
  i         = ADD(i, 1)            // increment index
  if i < length GOTO loop          // loop

In a linked list the elements don’t need to be in order in memory. Each element also contains a pointer to the next element which is the address the next element is at. So each node of the list takes two slots in memory, one for the number and one for the pointer. A liked list in memory that starts at address 5 might look like this.

...
5:   1
6:   9
7:   5
8:   0
9:   2
10:  11
11:  3
12:  950
...
950: 4
951: 7
...

A program to add the element in a linked list could look like this.

p_element = 5
loop:
  element   = LOAD(p_element)      // load the element
  result    = ADD(element, result) // add element to result
  p_next    = ADD(p_element, 1)    // address of next pointer
  p_element = LOAD(p_next)         // load next element address
  if p_element != 0 GOTO loop      // loop

Can we tell which one is better yet?

By instruction counts alone they are the same. Each loop has 51 instructions per iteration. Does that mean they are equivalent?

The array program has 3 ADD instructions and 1 LOAD, the linked list program has 2 ADD instructions and 2 LOAD instructions. So if ADDs are faster than LOADs it seems like the array version would be faster.

This is true, adding two numbers together is faster than loading a number from memory. It’s actually WAY WAY WAY faster. Memory is really far away in computer terms.

Modern computers have a way to deal with this problem. Since loading from memory is so slow they don’t want to just wait for data all day. They do this in two ways. They have caches for data from memory that are closer to the cpu. There’s small fast ones close to the processor that tie into larger and slower ones chaining out to main memory. When you LOAD an address it first looks in the cache, if it’s not there it checks the next cache, etc...

If you LOAD address 7 it’s slow because you go all the way out to main memory but if you LOAD it again later it’s fast because it’s in the cache.2

That seems like it would help but in our program we’re only ever LOADing a single address once. This is where the other trick comes in. The caches don’t LOAD elements from memory one at a time but they LOAD them in big blocks called cache lines. If you ask to LOAD address 0 it’ll actually LOAD addresses 0-64 into the cache so any further LOADs of those addresses will be fast. This means that for our array example we’ll LOAD the whole array into the cache at once and all the LOADs will be fast. The linked list will work similarly but OH NO in our example we have one of the nodes out all the way at 950 which won’t be in the cache. That means we’ll have at least one more slow LOAD in that program than the array version so clearly arrays are faster than linked lists!

Hold on a sec though, what if all the nodes in the linked list were in the same cache line? Both programs would only be doing fast cache loads and adds so they would be equally fast. Is this the answer? That the data structure doesn’t matter, all that matters is the data locality? Sort of. This is one of the biggest factors in performance. Make sure all the data you’re using is packed together.

What if the lists were very long? In our model if a cache line is 64 addresses, that would store an array with 64 elements, or a linked list with 32 element. Since each slot in a linked list also has the address of the next element you can fit twice as much data in the array than the linked list. This can be a huge factor when the lists are large. Another point for the array!

There’s a big piece of this that our model doesn’t factor in yet. In our model each instruction is done one at a time. Modern computers don’t actually work this way.3 They try to execute many instructions at the same time. You don’t tell them which ones to do in parallel4, They figure it out based on the data dependencies of the individual instructions. If we have two instructions.

a = ADD(b,c)
d = ADD(f,g)

Those two adds could be done at the same time because they don’t depend on each other. If instead you have.

a = ADD(b,c), d = ADD(a, e)

They can be done at the same time.

Lets look at our examples. If they instructions had to execute one at a time a trace of the full loop would look like this in the array case.

00: p=ADD(a,i),
01:            e=LOAD(p),
02:                      r=ADD(r,e),
03:                                 i=ADD(i,1)
04: p=ADD(a,i),
05:            e=LOAD(p),
06:                      r=ADD(r,e),
07:                                 i=ADD(i,1)
08: p=ADD(a,i),
09:            e=LOAD(p),
10:                      r=ADD(r,e),
11:                                 i=ADD(i,1)
12: p=ADD(a,i),
13:            e=LOAD(p),
14:                      r=ADD(r,e),
15:                                 i=ADD(i,1)
16: p=ADD(a,i),
17:            e=LOAD(p),
18:                      r=ADD(r,e),
19:                                 i=ADD(i,1)

And this for the linked list.

00: e=LOAD(p),
01:           r=ADD(e,r),
02:                      n=ADD(p,1),
03:                                 p=LOAD(n)
04: e=LOAD(p),
05:           r=ADD(e,r),
06:                      n=ADD(p,1),
07:                                 p=LOAD(n)
08: e=LOAD(p),
09:           r=ADD(e,r),
10:                      n=ADD(p,1),
11:                                 p=LOAD(n)
12: e=LOAD(p),
13:           r=ADD(e,r),
14:                      n=ADD(p,1),
15:                                 p=LOAD(n)
16: e=LOAD(p),
17:           r=ADD(e,r),
18:                      n=ADD(p,1),
19:                                 p=LOAD(n)

Pretty much the same.

But in the array we can actually do all the steps at the same time if we look across loops. We can do the math and load the next element while we’re adding up the last one. A full trace would look like this.

00: p1=ADD(a,i0),                        i1=ADD(i0,1)
01: p2=ADD(a,i1),e1=LOAD(p1),            i2=ADD(i1,1)
02: p3=ADD(a,i2),e2=LOAD(p2),r=ADD(r,e1),i3=ADD(i2,1)
03: p4=ADD(a,i3),e3=LOAD(p3),r=ADD(r,e2),i4=ADD(i3,1)
04: p5=ADD(a,i4),e4=LOAD(p4),r=ADD(r,e3),i5=ADD(i4,1)
05:              e5=LOAD(p5),r=ADD(r,e4),
06:                          r=ADD(r,e5),

For the linked list we can’t do quite as well. We can’t load the next element until we’ve loaded the pointer so we can’t get multiple steps ahead like we can with arrays. A full trace would look like this.

00: e1=LOAD(p0),
01:             r=ADD(e1,r),n1=ADD(p0,1),
02:                                      p1=LOAD(n1)
03: e2=LOAD(p1),
04:             r=ADD(e2,r),n2=ADD(p1,1),
05:                                      p2=LOAD(n2)
06: e3=LOAD(p2),
07:             r=ADD(e3,r),n3=ADD(p2,1),
08:                                      p3=LOAD(n3)
09: e4=LOAD(p3),
10:             r=ADD(e4,r),n4=ADD(p3,1),
11:                                      p4=LOAD(n4)
12: e5=LOAD(p4),
13:             r=ADD(e5,r),n5=ADD(p4,1),
15:                                      p5=LOAD(n5)

Seems like arrays are the clear winner!5

I’ve only recently started to understand a computer this way. Some of the places I’ve learned from include.

• This Mike Acton talk.

• This blog by Fabian “ryg” Giesen. especially the post in the footnotes.

• Handmade Hero, especially this episode.

I really want to see these things in action. Does anybody know if there are ways to see which instructions are executed each cycle or when loads miss the cache or anything like this on any mainstream computers / operating systems?

After applying the rules once “FX+YF+”.

After applying them again“FX+YF++-FX-YF+”.

After applying them a third time “FX+YF++-FX-YF++-FX+YF+--FX-YF+”.

A little program could look like this.

#include <stdio.h>
#include <string.h>

typedef struct {
  char val;
  char *replacement;
  size_t replacement_len;
} Rule;

#define RULE(c, rep)               \
  (Rule) {                         \
    .val = c, .replacement = rep,  \
    .replacement_len = strlen(rep) \
  }

#define BUF_MAX 1024

int main() {
  char input[BUF_MAX] = "FX";
  size_t input_len = strlen(input);
  Rule rules[] = {
      RULE('X', "X+YF+"), RULE('Y', "-FX-Y"), {0, 0, 0}};

  char output[BUF_MAX] = {};
  size_t output_len = 0;

  for (int char_i = 0; char_i < input_len; char_i++) {
    char c = input[char_i];
    Rule *rule = NULL;
    for (Rule *r = rules; r->val != 0; r++) {
      if (c == r->val) {
        rule = r;
        break;
      }
    }
    if (rule) {
      for (int i = 0; i < rule->replacement_len; i++) {
        output[output_len++] = rule->replacement[i];
      }
    } else {
      output[output_len++] = c;
    }
  }
  output[output_len++] = 0;
  printf("%s\n", output);
  return 0;
}

Iterate over the input string. For each character check if it matches a rule. If so output it’s production. Otherwise output the original character.

To evaluate it more than once, use the output of one pass as the input to the next.

  int num_iterations = 2;
  
  int iteration = 0;
  do {
    for (int char_i = 0; char_i < input_len; char_i++) {
      char c = input[char_i];
      Rule *rule = NULL;
      for (Rule *r = rules; r->val != 0; r++) {
        if (c == r->val) {
          rule = r;
          break;
        }
      }
      if (rule) {
        for (int i = 0; i < rule->replacement_len; i++) {
          output[output_len++] = rule->replacement[i];
        }
      } else {
        output[output_len++] = c;
      }
    }
    output[output_len++] = 0;
    memcpy(input, output, output_len * sizeof(output[0]));
    input_len = output_len;
    output_len = 0;
  } while (++iteration < num_iterations);
  printf("%s\n", output); 

Each evaluation depends on the evaluation before it. However each letter of each evaluation comes from a specific letter of the previous one. All the letters and where they came from form a tree.

Instead of evaluating this tree a whole layer at a time, we could instead walk the tree to output the lowest level directly.

Using a stack to keep track of the parent nodes we can do a depth first traversal of the tree.

  typedef struct {
    int level;
    char *prod;
    size_t prod_len;
    size_t prod_pos;
  } Frame;

  Frame stack[26];  // At least as large as num_iterations
  int sp = 0;

  Frame frame;
  frame.level = 0;
  frame.prod = &input[0];
  frame.prod_len = input_len;
  frame.prod_pos = 0;

  for (;;) {
    char c = frame.prod[frame.prod_pos++];

    // Check for a rule unless we're in the last level.
    Rule *rule = NULL;
    if (frame.level < num_iterations) {
      for (Rule *r = rules; r->val != 0; r++) {
        if (c == r->val) {
          rule = r;
          break;
        }
      }
    }

    if (rule) {
      // drop down into rule.
      stack[sp++] = frame;
      frame.level += 1;
      frame.prod = rule->replacement;
      frame.prod_len = rule->replacement_len;
      frame.prod_pos = 0;
    } else {
      output[output_len++] = c;
      while (frame.prod_pos == frame.prod_len) {
        if (sp == 0) {
          goto done;
        } else {
          frame = stack[--sp];
        }
      }
    }
  }
done:
  printf("%s\n", output);

It’s really easy to just thread the output of one algorithm as the input to the next. You can wrap the algorithm in a loop or a recursion or a fold or any other way to repeatedly evaluate the same thing. Bu thinking through the structure of the problem and the exact result you want can lead to better 2 solutions.3

A little note on LSystems

LSystems can be used to model biological systems. When you combine the output strings with some graphical interpretations they can create really cool fractal and recursive structures. I’ve played a bit with them myself. To learn more there is an amazing book you can check out called The Algorithmic Beauty of Plants.

Recently we’ve been shipping a ton of things at work2 so my side project time has been much less about making things and more just about learning things. I think it would be nice to share some of those things I’m learning with others so going to try this out and see how it goes.

I hope you enjoy.