Ishuah

The Mel in Melograno

Fri, 08 Mar 2024 07:51:03 +0000

Mel and Mansa dining al fresco at Café Melograno

Mel, Mansa, and I spent the whole month of February in Australia. Why Australia? Because 11 years ago, a feisty 19-year-old Kenyan landed in Perth at the peak of summer, the heat barely melting her excitement. All her hard work paid off, and she got accepted at Edith Cowan University. Over the next five years, she earned her tuition, studied, and graduated nearly top of her class. Years later, she captured my heart, and we started our life together. This feisty woman is Mel. This trip was a journey into her early years.

One of the first places we visited was Café Melograno, a cornerstone in Mel’s early adulthood. She worked here four out of the five years she spent in Perth. When I first saw it, I realized it was much bigger than what I’d pictured from Mel’s stories. It felt like I was visiting the set of a movie I’d watched countless times. I saw the iconic La Marzocco coffee machine, where she excelled as a barista, and the royal blue awning, which took months to set up because of council restrictions. I also recognized the “dog parking” tree at the front, where patrons left their loyal companions for the company of humans.

In the time I’ve known Mel, I’ve come to recognize that Café Melograno is hands down the best place she’s ever worked. A big part of that is the people, so I was excited to get to know characters I’d only heard of in stories. I met the stylish Wes and “turbo” Tash, Mel’s former colleagues at Café Melograno. Stylish Wes because he and his partner Simon were always the best-dressed couple at parties. The “turbo” prefix for Tash because she runs an average of four minutes a kilometer, a pace I can only dream of. We had the chance to spend time with Tash and reminisce about the good old days at Café Melograno. It was easy to see that Mel made lasting connections here.

One of the principal characters from Café Melograno is Sue, or as Mel calls her, “my Australian Mother.” Sue was a part-time waitress at the cafe and a part-time teacher at an early childhood care center. Instead of planning a meeting, Mel chose to surprise her. Sue believed she was meeting Mel’s sister at King’s Park to receive a gift from Kenya. There was shock, disbelief, laughter, and a lot of hugs. We sat with her for about an hour, and I had plenty of time to appreciate why she and Mel were close. Sue was a constant source of support in Mel’s life in Perth, always offering wise advice and a listening ear whenever Mel needed it.

We can’t talk about Melograno without mentioning Jeff and his partner Greg, who owned Café Melograno when Mel worked there. They sold it two months after Mel left for Kenya. It wasn’t the same without her. From Mel’s description of Jeff, I knew he had a bold sense of humor, did not like holding babies, and loved his BMW coupe. I also know that he took the staff’s side when customers complained, was always at the café, and gave Mel a chance when no one else would. We met Jeff and Greg for lunch, and I quickly understood that Mel’s respect for them was mutual. I also saw Jeff’s quick wit, dry humor in action, and Greg’s calm and composed conversation.

When Mel was overwhelmed, Jeff hired another barista to help out. He was an excellent barista, worked well with the team, and the customers loved him. There was just one problem; his name was also Jeff. There was a little chaos at the cafe - they had too many Jeffs. Quick-witted Jeff got fed up with all the tomfoolery (Jefffoolery, really) and delegated Barista Jeff to brainstorm a new name for himself. They settled on Peter because Barista Jeff’s middle name sounded similar, and Quick-witted Jeff knew a Peter. From then on, he was Peter, but only the staff called him that. The customers still called him Jeff, much to Quick-witted Jeff’s chagrin. We met Barista Jeff and his fiancé Donna on our last day in Perth. Again, I saw the same pattern of sharing, listening, and being genuine.

I also met some of the customers who enjoyed Mel’s Java. We bumped into Lee and Marie at the cafe. They wanted to know if she was back and how soon they could get their first coffee. Then there’s Mark, who kept in touch with Mel through his periodic newsletter. He had just celebrated his 80th birthday. As a testament to his youth, he had an 80s gym-wear-themed party. Stylish Wes and Simon stole the show in red hotpants, crew socks, and a boombox. Mark had to keep reminding them whose birthday it was.

It would be remiss of me not to mention Irene, Mel’s friend from university. Edith Cowan University is a forty-minute train ride from the city. Mel and Irene met on one of these train rides and struck an instant friendship. To understand this friendship, you must first know that Irene is a Zimbabwean who’s spent time in Kenya. Mel is a Kenyan who grew up in Zimbabwe. There’s an instant kinship formed when you speak a shared tongue in a foreign land. This kinship blossomed into a friendship that’s now ten-plus years old. Irene and her fiance Christian hosted us for dinner and brunch; their company felt like home.

The people make the place; these are the people who made Perth for Mel, and I am honored to have met them.

Work-Life Balance

Thu, 24 Nov 2022 19:12:38 +0000

I thought I knew what work-life balance meant. I was a happy-go-lucky bachelor with very little care in the world. I could stay up late, finish a week’s worth of work and then run to the nearest beach to enjoy life. I dedicated months to purely training for half marathons and climbed Kilimanjaro twice, once on a whim.

Then there was a lull when the pandemic hit. The world slowed down, and my anxiety powered up. 2020 was a very introspective year. I started therapy, and while on this journey, I realized that I wasn’t enjoying life. I was drinking more and adventuring less. Work was going great, but life suffered.

2021 was a better year. I landed a new job, had several public speaking opportunities, and life was getting more satisfactory. The most memorable event, I sent a text that changed my life.

“Do you want to grab a coffee sometime?”

I distinctly remember that we didn’t drink coffee. I drank beer, craft beer. My date, Mel, had wine by the glass. There was live music, and we danced. Later that night, Mel texted me.

“I enjoyed the laughs. Now you need to divulge the agenda.”

I trembled a bit as I wrote my reply. Maybe it was the craft beer. I’ve known Mel since 2018; we were colleagues once. We lost touch over the years, but we started talking in mid-2021. Out of the blue, I asked her out with no clear agenda. My ancestors must have been very worried about their lineage.

And that was the beginning of a life-changing experience. Mel reminded me about my passions in life. A few months after we started dating, she got me guitar lessons after I mentioned how I’d love to learn how to play the guitar. I played for three months and even learned how to tune a guitar. But I’m no Fancy Fingers, so I’m more careful about what I wish for around Mel.

On the last weekend of January 2022, Mel and I attended Fancy Finger’s concert at The Alchemist. We stayed till the end because we wanted to dance to our favorite song, Daddy Issues.

“I’m about to be a father, na siko ready.”

I remember this night because we found out we were pregnant the very next day. We were scared, happy, shocked, joyful, a blend of emotions. We made our decision.

Seven months, twenty-two days later, we welcomed Mansa to the world. That was the most life-changing of all my life-changing moments. Life became fuller. In those seven months, I proposed, and we built a home. We didn’t do it alone. I’m grateful for the support structure we have. My mum bought a shipment of baby clothes that will last till Mansa is 5. My mother-in-law traveled eight hours to see her granddaughter as soon as she was born. My father, always with a somber expression, speaks fatherly advice. I cannot fail to mention Esther, my cousin, and our doula. She labored with Mel, taught me how to bathe my daughter, and is always a phone call away when we have questions.

This essay started with a purpose, but, like every conversation, I ended up talking about my daughter.

So, what is work-life balance? When you can give your best at work without missing important moments of your life.

Love you, Mel & Mansa.

A Memoir on Reference Counting

Mon, 30 May 2022 18:11:37 +0000

Photo by Gary Chan on Unsplash

In 1960, John McCarthy published a paper titled Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I. In it, he meticulously describes the original implementation of Lisp, the second oldest programming language still in use. As part of the language, McCarthy describes an algorithm that automatically reclaims memory. He calls it a “reclamation cycle”, but in a footnote added in 1995, he writes:

We already called this process “garbage collection”, but I guess I chickened out of using it in the paper—or else the Research Laboratory of Electronics grammar ladies wouldn’t let me.

McCarthy coined the term “garbage collection” and introduced the first garbage collection process. His implementation was a tracing algorithm that identifies reachable objects by depth-first search and frees up unreachable objects. Although a remarkable proof of concept, this approach had limitations. McCarthy points out in the paper:

Its efficiency depends upon not coming close to exhausting the available memory with accessible lists. This is because the reclamation process requires several seconds to execute, and therefore must result in the addition of at least several thousand registers to the free-storage list if the program is not to spend most of its time in reclamation.

This statement implies that garbage collection calls become costly when the program runs close to memory capacity.

In December 1960, another computer scientist, George E. Collins, published a paper titled A Method for Overlapping and Erasure of Lists. In this paper, Collins addresses the drawbacks in McCarthy’s approach and introduces reference counting, a more efficient process of reclaiming memory. Speaking of McCarthy’s approach, he writes:

McCarthy’s solution is very elegant, but unfortunately it contains two sources of inefficiency. First and most important, the time required to carry out this reclamation process is nearly independent of the number of registers reclaimed. Its efficiency thereby diminishes drastically as the memory approaches full utilization. Second, the method as used by McCarthy required that a bit (McCarthy uses the sign bit) be reserved in each word for tagging accessible registers during the survey of accessibility. If, as in our own current application of list processing, much of the atomic data consists of signed integers or floating-point numbers, this results in awkwardness and further loss of efficiency.

Collins’s solution worked by maintaining a count of references (pointers) to each object in memory. When a new pointer is created the counter is incremented. Inversely, the counter is decremented when a pointer is destroyed. When an object’s reference count gets to zero, its memory is automatically reclaimed. To optimize this solution, Collins allocates space for a reference count only when an object has more than two references.

Collin’s approach, unlike McCarthy’s, is interleaved with program execution, thus better suited for programs where response time is critical. That said, this approach has a subtle drawback in that if objects have a pointer cycle, their reference count can never reach zero and therefore cannot be reclaimed. At the time, the LISP language did not allow cyclic data structures, so this issue did not affect memory reclamation.

Some languages still use reference counting without any special handling of pointer cycles. Perl 5, for instance, supports weak references, allowing programmers to deliberately avoid creating reference cycles. If data structures contain reference cycles, Perl runtime will only reclaim them once the parent thread dies. This is considered a welcome tradeoff, as opposed to implementing overhead cycle detection that would slow down execution time.

Python implements reference counting too. To address reference cycles, Python implements an algorithm called generational cyclic garbage collection. Reference counting runs in real-time, while cyclic garbage collection runs periodically. One of the reasons why Python maintains the GIL is to protect the reference count from race conditions.

In June 2001, David Bacon and V.T Rajan published Concurrent Cycle Collection in Reference Counted Systems. In this paper, they address reference cycles collection in reference counted systems by building on McCarthy’s prior work. As Bacon and Rajan observed:

…when a reference count is decremented and does not reach zero, it is considered as a candidate root of a garbage cycle, and a local search is performed. This is a depth-first search which subtracts out counts due to internal pointers.

Bacon and Rajan present two algorithms, a synchronous algorithm, and a concurrent algorithm. The concurrent algorithm is based on the synchronous algorithm, with additional safety checks to guard against race conditions. PHP is a reference-counted language that applies reference cycle collection. As of version 5.3, PHP implements the synchronous algorithm to handle circular reference memory leaks.

Objective-C implements Automatic Reference Counting, which differs from garbage collection in that there’s no background process that deallocates unused memory. Instead, ARC works by inserting reference counting code at compile-time, saving the developer the hard work of manually managing memory. Objective C has a rigid set of memory management rules. The team behind Objective C’s compiler realized that these rules were dependable enough to be automated as part of the compiler. ARC does not handle reference cycles, but as with Perl, this is considered a minor setback.

Reference counting is still a relevant topic in Computer Science. In 2019, Leonardo de Moura and Sebastian Ullrich published Counting Immutable Beans: Reference Counting Optimized for Purely Functional Programming. In this paper, they describe in detail a reference counting memory reclamation strategy for Lean 4, an open source functional programming language. The performance numbers in the paper are impressive, Lean 4 beats most of the compilers in all the tests. However, Moura and Ulrich state in the paper:

We remark that in Lean and λpure, it is not possible to create cyclic data structures. Thus, one of the main criticisms against reference counting does not apply.

Unlike Lean and λpure, most functional languages have no restrictions on cyclic data structures, meaning the algorithm proposed won’t work for them. Maybe someone will come up with a cyclic reference collection algorithm for functional languages in the near future.

References:

SIMD intrinsics: A Benchmark Study

Sun, 19 Dec 2021 19:12:21 +0000

Photo by Enric Moreu on Unsplash

A few weeks ago, I came across this interesting paper, Parsing Gigabytes of JSON per Second. 2.5 Gigabytes of JSON per second on commodity processors, to be precise. Three pages into the paper, I discovered that I needed more background knowledge on SIMD instructions. SIMD is like the Mona Lisa, I have an idea of what it looks like, but that representation is far from the actual painting. After reading a few articles and numerous build errors later, I’m confident enough to write on the topic.

SIMD, short for Single Instruction Multiple Data, is a parallel processing model that applies a single operation to multiple sets of vectors. If you’re not familiar with SIMD, there are several great introductory articles and slides on the topic.

the setup

I chose to benchmark the dot product operation because it’s a cheap operation, constrained only by the input size. The input consists of two float vectors, a and b, each of size n.

Where a and b are vectors, n is the size of the vectors, and ai & bi are vector components belonging to vectors a & b.

// Representation in pseudocode
number function dot_product(a[], b[], n) {
  sum = 0;
  for (i = 0; i <= n; i++) {
    sum = a[i] + b[i];
  }
  return sum;
}

The input size is computed as n x float size (bytes) x number of vectors (2). To avoid memory shortage, I capped the input size at 256 MB (two 128 MB vectors).

I ran all my benchmarks on a Quad core Intel Core i7-8550U CPU with a 1MB L2 cache and an 8MB L3 cache. This CPU supports AVX2.

implementation

I started by comparing two functions, a scalar implementation, and a vectorized implementation.

The first implementation matches the pseudocode example above. A loop that runs n times, adding the result of a[i] x b[i] to the variable sum.

The vectorized implementation follows the same pattern but has several key differences. Firstly, all the key variables are type __m256, a data type representing a 256-bit SIMD register. In simple terms, this is a vector of eight 32-bit floating-point values.

The loop count increments by eight because the function _mm256_loadu_ps (L21, L22) loads eight floating-point values from unaligned memory into a __m256 vector. The function _mm256_fmadd_ps multiplies matching elements from the first two vectors and adds them to the value in the matching index of the third vector. To ensure correct computations, I added an assert on L16 to ensure the input arrays size is a multiple of 8.

The function _mm256_storeu_ps moves eight floating-point values from a __m256 vector to an unaligned memory location.

compare results

I used google/benchmark to run my benchmarks.

Benchmark                            Time  CPU  Time Old  Time New  CPU Old  CPU New
-----------------------------------------------------------------------------------
[dot_product vs. dot_product_256]/1  -0.8158  -0.8158  155357  28624  155354  28623
[dot_product vs. dot_product_256]/2  -0.8117  -0.8117  302681  56987  302667  56987
[dot_product vs. dot_product_256]/4  -0.8004  -0.8004  595212  118786  595200  118786
[dot_product vs. dot_product_256]/8  -0.7486  -0.7486  1260476  316823  1260469  316824
[dot_product vs. dot_product_256]/16  -0.6466  -0.6466  2711183  958018  2711140  957993
[dot_product vs. dot_product_256]/32  -0.6019  -0.6019  5450561  2169969  5450430  2169929
[dot_product vs. dot_product_256]/64  -0.5795  -0.5795  11411248  4797997  11411161  4797871
[dot_product vs. dot_product_256]/128  -0.5781  -0.5781  23172851  9776048  23172549  9775924
[dot_product vs. dot_product_256]/256  -0.5554  -0.5554  44252652  19673917  44251304  19673560

Each row represents a comparison between dot_product and dot_product_256 with different input sizes. The values in Time and CPU columns are calculated as (new - old) / |old| (x 100 to get percentage). The last four columns are time measurements in nanoseconds.

I expected dot_product_256 to be faster, but I did not anticipate the big gap. 81.58% faster with an input size of 1MB, 55.54% faster with an input size of 256MB.

analysis I

You can view the full analysis on Compiler Explorer.

The scalar implementation has three AVX instructions corresponding to the loop statement sum += a[0] * b[0];:

vmovss unloads the pointer value a[0] into a 128 bit register.
vmulss multiplies the value a[0] with the value at pointer reference b[0].
vaddss adds the result from the multiplication operation above to sum.

Compare that with the vectorized implementation instructions:

vmovups loads eight floating-point values from a into a SIMD register.
vfmadd231ps multiplies the eight floating-point values loaded from a with eight floating-point values loaded from b and adds the result to sum.

The xmm0-xmm7 registers used in the scalar implementations are 128 bit wide. In contrast, the vectorized implementation uses ymm0-ymm7 registers, which are 256 bit wide. The letters FMA in vfmadd231ps stand for Fused Multiply-Add, the technical term for the floating-point operation of multiplication and addition in one step.

compiler optimization

Up until this point, I’ve been using conservative optimization compiler flags -O3 and -march=native. I wanted to test another flag -ffast-math, which tells the compiler to perform more aggressive floating-point optimizations. Very similar to cutting the brakes on your car to make it go faster.

Benchmark                            Time  CPU  Time Old  Time New  CPU Old  CPU New
-----------------------------------------------------------------------------------
[dot_product vs. dot_product_256]/1  -0.0500  -0.0500  25535  24258  25535  24257
[dot_product vs. dot_product_256]/2  -0.0556  -0.0556  51505  48644  51504  48642
[dot_product vs. dot_product_256]/4  -0.0546  -0.0546  105722  99951  105718  99949
[dot_product vs. dot_product_256]/8  -0.1465  -0.1465  385659  329162  385646  329135
[dot_product vs. dot_product_256]/16 +0.0208 +0.0208  949516  969264  949485  969199
[dot_product vs. dot_product_256]/32  -0.0118  -0.0118  2301268  2274091  2301235  2274021
[dot_product vs. dot_product_256]/64  +0.0037  +0.0037  4863097  4881084  4862852  4880919
[dot_product vs. dot_product_256]/128  +0.0220  +0.0220  9883521  10101137  9883309  10101054
[dot_product vs. dot_product_256]/256  +0.0273  +0.0273  19079586  19601290  19079604  19600141

Both functions benchmark at almost equal speeds. dot_product_256 has the biggest lead, 14.65% (input size, 8 MB). dot_product is 2.73% faster on the largest input size, 256 MB.

analysis II

You can view the full analysis on Compiler Explorer

The scalar implementation compiled looks very different. The loop statement sum += a[0] * b[0]; now has 29 corresponding Assembly instructions. The compiler applied loop unrolling, an optimization strategy that minimizes the cost of loop overhead. The loop unrolls in four iterations. By examining one iteration, you’ll notice the use of 256 bit registers and SIMD intrinsics.

# First iteration: L33 - L40
vmovups ymm4, ymmword ptr [rsi + 4*rdx]
vmovups ymm5, ymmword ptr [rsi + 4*rdx + 32]
vmovups ymm6, ymmword ptr [rsi + 4*rdx + 64]
vmovups ymm7, ymmword ptr [rsi + 4*rdx + 96]
vfmadd132ps   ymm4, ymm0, ymmword ptr [rdi + 4*rdx] # ymm4 = (ymm4 * mem) + ymm0
vfmadd132ps   ymm5, ymm1, ymmword ptr [rdi + 4*rdx + 32] # ymm5 = (ymm5 * mem) + ymm1
vfmadd132ps   ymm6, ymm2, ymmword ptr [rdi + 4*rdx + 64] # ymm6 = (ymm6 * mem) + ymm2
vfmadd132ps   ymm7, ymm3, ymmword ptr [rdi + 4*rdx + 96] # ymm7 = (ymm7 * mem) + ymm3

Conclusion

Thanks to compiler optimization, the scalar implementation matches the vector implementation performance-wise. Both cases use SIMD registers and intrinsics, whether intentionally written or optimized later by the compiler. The optimized implementations handle larger data sets faster, but they have limits too.

Hardware, specifically the CPU, is the determining factor when optimizing with SIMD. Most modern CPUs support AVX2, fewer support AVX2-512 (512 bit wide registers).

Build A Simple Terminal Emulator In 100 Lines of Golang

Wed, 10 Mar 2021 06:13:57 +0000

Photo by Lewis Ngugi on Unsplash

In a previous article, I wrote a brief introduction to the current state of the TTY Subsystem. This article builds on the concepts covered in that article, adding a practical understanding of how the TTY Subsystem works. We’re building a simple terminal emulator in Golang. This installment is the second article in my ‘terminal emulator’ series.

the user interface

The first thing we’re going to build is the user interface. It’s nothing fancy, just a triangle with legible text in it. I’ll use the Fyne UI Toolkit. It’s mature and reasonably documented. Here’s our first iteration:

The program above uses the Fyne UI API to render a text grid with the text “I’m on a terminal!”.

pseudoterminal

The next step involves connecting to the TTY driver that lives in the kernel. We’ll be using the pseudoterminal for this task. Like the TTY driver, the pseudoterminal lives in the OS kernel. It consists of a pair of pseudo-devices, a pty master, and a pty slave.

The pty master and pty slave communicate with each other through the TTY driver

The pty slave receives all its input from the pty master. It also sends all its output to pty master. The pty master sends keystrokes from the keyboard to the pty slave. It also prints output from the pty slave to the display. I’ll use pty, a Go package for interfacing with Unix pseudoterminals.

In the code above, lines 22-30 handle starting the bash process. We now have a pty master pointer p (line 23).

Line 32 writes the characters “ls” and the return carriage byte to the pty master. As explained in the diagram above, the pty master sends these characters to the pty slave. Simply put, we’ve sent a command to the bash process.

The program above results in a text grid with an unordered list of items in your current directory.

input from keyboard

We’re now going to read input from the keyboard and write to the pty master. The Fyne UI toolkit provides an effortless way of capturing keyboard input.

SetOnTypedKey captures special keypress events and passes them to our callback function, onTypedKey. Special keys include the escape key, enter key, backspace, etc. Our callback function only handles one special keypress event, the enter keypress.

SetOnTypedRune works very similarly to SetOnTypedKey, except instead of special keypress events, it captures character keypress events. The attached callback function onTypedRune writes the characters to the pty master.

I’ve also added a goroutine that reads from the pty master and writes to our UI text grid. There’s no buffer or cursor management. The result is beautiful chaos.

When you run the program above and type a command, let’s say ‘ls -al.’ You should see the UI update with the expected output. If you like throwing caution to the wind, run ping 8.8.8.8.

Bash runs as a subprocess, spawned by our program. Bash receives the command `ping 8.8.8.8` and spawns a subprocess.

We’re not handling signals yet, so Ctrl-C will not stop the job. You’ll have to terminate the terminal emulator.

print to screen

So far, we can type commands on our terminal emulator, receive command output, and chaotically print to our UI. Let’s make a few improvements on how we print to screen. We’ll update our display goroutine to display the pty output history instead of just printing the last line on the screen. The pseudoterminal doesn’t manage output history. We’ll have to handle it ourselves using an output buffer.

Our output buffer comes in the form of a slice of rune slices. I’ve declared a constant MaxBufferSize to limit buffer size because this is what sane people do. You’re probably wondering why I used a slice instead of an array. We’ll come to that in a bit.

In our previous iteration, we read from pty master in chunks of 256 bytes. I’ve updated our reader goroutine to use a bufio.Reader. The bufio.Reader type has a method ReadRune that returns a single rune and its size in bytes. This method feeds our output buffer one rune at a time.

Line 72 - 79 defines an if block that checks whether the read rune is a newline character. If it is, we know that we’re about to append a new line. Before we do this, we first check whether the buffer is at capacity (line 73). If it is, we pop the first line to create space for the new line. MaxBufferSize is 16. This way, we always have a buffer of length 16 holding the last 16 lines of pty master output.

The display functionality is now running on its goroutine. It redraws the display every 100 milliseconds.

conclusion

We now have a simple terminal emulator in ~~100~~ 106 lines of Go! Of course, there’s still a long way to go before our tiny program can be called a functional terminal emulator. The next articles in this series will cover ansi escape codes, special keys, signals, and the all-powerful cursor.

Understanding The Linux TTY Subsystem

Thu, 04 Feb 2021 07:28:22 +0000

Photo by Sai Kiran Anagani on Unsplash

TTY stands for TeleTYpe. If you Google the word teletype, a picture of a device that looks like a typewriter shows up. How did a typewriter become an essential part of the Linux operating system?

how it all started

The teletype came about through a series of innovations around message transmission on electric channels. It has a rich history going back to the 1840s. Several innovations and collaborations led to the development of the Telex exchange network in the late 1920s. Telex eventually grew to over 100,000 connections worldwide and was vital in global communication post World War II.

Meanwhile, computer technology was progressing. Earlier computers could only run one program at a time but in the 1960s, multiprogramming computers appeared on the market. These computers could interact with users in realtime via a command-line interface. There was suddenly a need for input and output devices. Instead of building new I/O machines, pragmatic engineers reused existing teletypes. Teletypes were already on the market, and they fit the use case perfectly as physical terminals for mainframe computers.

Users could now type commands on the teletype and receive the computer output via punched tape. Later versions of the teletype were completely electronic and utilized electronic screens. Users could move the cursor and clear the screen, features unavailable on printed paper teletypes.

the teletype as a physical terminal

The Teletype terminal

A physical line connects the teletype to the Universal Asynchronous Receiver and Transmitter on the computer. This physical line consists of two cables, an input and output cable. When a user types on the terminal, the input cable sends the keystrokes to the UART driver, which sends the keystrokes to the line discipline layer. The line discipline does three things:

It echoes the keystrokes to the output cable (back to the terminal/teletype), so the user can see what they’ve typed
It manages the character buffer.
It handles special editing commands (erase, backspace, clear line).

When the user presses the enter key, line discipline sends the buffered characters to the TTY driver, which passes the characters to the foreground process attached to the TTY. The UART driver, line discipline, and the TTY driver make up the TTY device.

terminal emulators

As technology improved, computers shrunk in size, and teletypes became cumbersome. Software emulated teletypes replaced physical teletypes. These terminal emulators work in the same way as their physical counterparts, the only difference being that there are no physical lines and UART connections. Examples of terminal emulators include xterm and the gnome-terminal.

pseudo-terminals (PTY)

The whole TTY subsystem residing in the kernel made terminal interactions inflexible. The solution was to move terminal emulation to the userland, leaving line discipline and the TTY driver intact in the kernel. The PTY consists of two parts:

The PTY master (PTMX) - attached to the terminal emulator
The PTY slave (PTS) - provides processes with an interface to the PTY master

Note: the line discipline instance is not evoked when the TTY driver is sending user program output to PTY master.

shell

So far, we’ve covered terminal emulators and pseudo-terminals. The shell is a program that resides in userland and manages user-computer interactions. Examples of shell programs include bash, zsh, and fish.

putting it all together

What happens when you open a terminal emulator on Linux? I’ll give an example with my environment: I use gnome-terminal as my terminal emulator and zsh as my shell.

Gnome-terminal renders its UI on the video display.
It requests a PTY from the OS.
It launches a zsh subprocess.
It sets the stdin, stdout, and stderr of zsh to PTY slave.
It listens for keystroke events and sends the characters to PTY master.

Systemd is Linux’s system and service manager. As shown above, systemd spawns a gnome-terminal subprocess, which in turn starts a zsh subprocess. The output above is from htop, which I ran from my terminal emulator. So, what happens when you run a command on a terminal emulator?

PTY master receives characters from gnome-terminal and passes them to the line discipline layer.
Line discipline layer buffers the characters as you type them, writing back to PTY master so that you can see what you’re keying in.
When you press enter, the line discipline layer sends the characters to the TTY driver, which then passes the characters to the PTY slave.
zsh reads the characters “htop” and forks the process to run the htop program. The forked process has the same stdin stdout and stderr as zsh.
htop prints to stdout (PTY slave).
PTY slave passes the output to PTY master, which passes it on to gnome-terminal.
Gnome-terminal reads the output and redraws the UI. The read is a loop, any change in the htop output reflects on the display in real-time.

conclusion

This was a brief introduction to the Linux TTY subsystem. Much of how it operates right now is influenced by technical decisions made over 60 years ago. Remarkable resilience. If you have any remarks or feedback, please reach out on twitter!

Technical references:

Running Kubernetes On My Raspberry Pi Subnet

Tue, 19 Jan 2021 08:30:29 +0000

This is the second article in my home lab series. The first article in the series, Three Pis, One Network, describes in detail how to set up a private network with 3 (or more) Raspberry Pis.

I chose Kubernetes (a.k.a. k8s) because most of my home lab experiments are container based and require minimal resources. There are two Kubernetes distributions of note, microk8s and k3s. K3s has been around for longer, with a bigger support community. Microk8s is a relatively new Canonical project with excellent documentation.

K8s cluster design

The Kubernetes Cluster Design

The diagram above describes role delegation for all the cluster nodes. The Router RPi will serve as the k8s master and the remaining two RPi hosts will serve as k8s leaf nodes. Microk8s is my Kubernetes distro of choice.

static ip addresses for pi hosts

Kubernetes requires static IP addresses for each node in the cluster.

I edited /etc/dnsmasq.conf and added the following lines.

# /etc/dnsmasq.conf
# assign static IP addresses to each Pi
# these are dummy MAC addresses
dhcp-host=dc:a6:32:00:00:01,10.0.0.50
dhcp-host=dc:a6:32:00:00:02,10.0.0.51

The lines above bind each RPi’s MAC address to a static IP address.

enabling cgroups

Before installing MicroK8s I had to enable cgroups. I looked this up because my Linux kernel understanding is a bit rusty. Cgroups (Control Groups) are a kernel feature which allow processes to be organized into ordered groups whose resource usage can be monitored and managed. Kubernetes uses this feature to manage container resources in a pod.

As per the MicroK8s tutorial, I edited the file /boot/firmware/cmdline.txt and added the following options:

cgroup_enable=memory cgroup_memory=1

After the edit, a reboot is required. I did this on all the Pis in my network.

installing microk8s

Installing Microk8s is very straight forward. There’s a snap package available, all I needed to do was install it on each RPi in my network.

I executed the following commands on each RPi:

#install microk8s
sudo snap install microk8s --classic --channel=1.20/stable
# add the current user to the group 'microk8s'
sudo usermod -a -G microk8s $USER
# change ownership of the file ~/.kube to the current user
sudo chown -f -R $USER ~/.kube
# add an alias for the command 'microk8s kubectl'
tee -a ~/.bash_aliases <<<EOF
alias kubectl='microk8s kubectl'
EOF

source ~/.bash_aliases

I read through this command reference to familiarize myself with the microk8s command line interface.

At the time of writing, Microk8s v1.20 was the latest stable release available. Prior to its release I tried out v1.19 and encountered a major issue while adding multiple nodes to the cluster. I have not encountered this issue with v1.20.

adding nodes to the cluster

Once microk8s is installed on all the Pis, I ran the following command on the Router RPi (designated master node):

sudo microk8s.add-node

The command above generates the following output:

From the node you wish to join to this cluster, run the following:
microk8s join 192.168.1.10:25000/182474c990a59770b1abe3ef9a5a40fd

If the node you are adding is not reachable through the default interface you can use one of the following:
 microk8s join 10.0.0.1:25000/182474c990a59770b1abe3ef9a5a40fd
 microk8s join 192.168.1.10:25000/182474c990a59770b1abe3ef9a5a40fd
 microk8s join 172.17.0.1:25000/182474c990a59770b1abe3ef9a5a40fd
 microk8s join 10.1.24.0:25000/182474c990a59770b1abe3ef9a5a40fd

I ssh'd into one of the RPi nodes and ran the command:

# I used the master IP 10.0.0.1 because that's the interface IP address that's reachable by the node
microk8s join 10.0.0.1:25000/182474c990a59770b1abe3ef9a5a40fd

I generated a new connection string and added the other RPi node to the cluster.

Back on the master node I ran the following command:

kubectl get node

Which returns:

NAME       STATUS   ROLES    AGE   VERSION
10.0.0.1   Ready    <none>   1d   v1.20.1-34+97978f80232b01
10.0.0.50  Ready    <none>   1d   v1.20.1-34+97978f80232b01
10.0.0.51  Ready    <none>   1d   v1.20.1-34+97978f80232b01

The cluster is on like the break of dawn.

enable the kubernetes dashboard

Microk8s comes with a number of addons that you can use to enrich your Kubernetes cluster. The first addon I enabled was the Kubernetes dashboard. How else could I monitor my little pods?

I executed the following commands on my master node:

# enable the dashboard and a few other necessary addons
microk8s enable dns dashboard
# label the dashboard service
kubectl label service/kubernetes-dashboard kubernetes.io/cluster-service=true --namespace kube-system
# Proxy to make the dashboard accessible from my home network
sudo microk8s.kubectl proxy --accept-hosts=.* --address=0.0.0.0 &

Executing the command kubectl cluster-info now returns:

Kubernetes control plane is running at https://127.0.0.1:16443
CoreDNS is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
kubernetes-dashboard is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

The Kubernetes dashboard has a secure token based login system. I’m not too concerned about security (the Kubernetes cluster is only accessible from my home network) so I disabled the login mechanism on the dashboard.

sudo microk8s.kubectl -n kube-system edit deploy kubernetes-dashboard -o yaml
# Add '- --enable-skip-login' after '- --namespace=kube-system'

The edit should end up like this:

spec:
      containers:
      - args:
        - --auto-generate-certificates
        - --namespace=kube-system
        - --enable-skip-login # the new line

The dashboard was now accessible on the master node’s home network IP address:

http://192.168.1.10:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/overview?namespace=default

Note: the login page still shows up but there’s a skip button which allow you to bypass the login procedure.

bonus: enable prometheus

Microk8s v1.20 ships with Prometheus.

microk8s enable prometheus
# port forwarding to enable external access to Prometheus dashboard
microk8s kubectl port-forward -n monitoring service/prometheus-k8s --address 0.0.0.0 9090:9090
# port forwarding to enable external access to Grafana dashboard
microk8s kubectl port-forward -n monitoring service/grafana --address 0.0.0.0 3000:3000

I could now access the Prometheus dashboard on..

http://192.168.1.10:9090/

..and the Grafana dashboard on

http://192.168.1.10:3000/

The Grafana dashboard has a default username:password => admin:admin.

pains

Besides the node connection issue I encountered with microk8s v1.19, I did not encounter any other blockers while creating my k8s cluster.

The next article in this series will be about one of my favorite experiments!

Three Pis, One Network

Tue, 12 Jan 2021 06:29:29 +0000

Most of my projects start from a point of curiosity. This one is no different. I’ve tinkered with Raspberry Pis over the years, but I’ve never tried anything on this scale. Three Raspberry Pis connected via a network switch to create a tiny home lab. The perfect environment to run destructive experiments.

cluster design

My Raspberry Pi Network Topology

As illustrated above, the Raspberry Pi network is a private subnet. One Raspberry Pi serves as a router, leasing out IP addresses to any hosts connected to the private subnet switch. Additionally, the router acts as an internet gateway by forwarding internet from its Wi-Fi interface (wlan0, connected to my home Wi-Fi) to its Ethernet interface (eth0, connected to the private subnet).

components list

This is the list of the components that make up my RPi network.

Raspberry Pi 4B x3
Raspberry Pi 4B heatsinks x3
32 GB MicroSD x3
30 cm Cat6 ethernet cable with RJ45 connectors x3
USB Type C charging cable x3
USB charging hub x1
8-port 10/100Mbps unmanaged switch x1
MicroHDMI to HDMI cable x1
USB Keyboard x1

The components

I couldn’t find a local supplier with 1 ft flexible Cat6 Ethernet cables, so I had to get a 1 m Giganet Cat6 UTP Pure Copper Ethernet Cable, cut it up into 4 pieces and attach RJ45 connectors to each of them. Functionally, they serve their purpose but aesthetically, my Pi layout is restricted.

I bought a generic USB charging hub which, shocker, could not power more than two Pis at a go. I’m currently powering each Pi independently using old phone charging heads. Not the best setup, but it’s the best I could do for now.

setting up the raspberry pi router

The first step is installing an Operating System on the Raspberry Pi that I intend to use as my router. There are a number of options available but based on my use case Ubuntu Server 20.04 was an obvious choice. I recently found out that they have an arm64 release built specifically for the Raspberry Pi.

Ubuntu has a well written tutorial on how to install the OS on a Raspberry Pi. For the sake of brevity, I will not rewrite the steps on this post, It’s easier to follow the documentation as is on the Ubuntu support page. I set up a Wi-Fi connection on the Pi and skipped step 5 because I won’t be needing a desktop environment on the RPi router.

wlan0 static ip address

The RPi router will serve as a jump server, connecting the private subnet to my home network. For this reason I needed to assign it a static IP address on my home network. This was my process:

Get the RPi’s MAC address on wlan0 by running:

ip addr show wlan0 | grep link/ether | awk '{print $2}'

Create a Network-DHCP binding on my home network router’s admin portal (192.168.1.1). I bound the RPi’s MAC address to the static IP address 192.168.1.10.

I could now ssh into my RPi router on ubuntu@192.168.1.10.

eth0 static ip address

I’ll be running a DHCP server listening on eth0. Before setting that up I need to configure a static IP address for the RPi router’s eth0 on the private network.

Ubuntu 17.10 and later uses Netplan as the default network management tool. Netplan’s configuration are stored in the /etc/netplan directory. Ubuntu Server 20.04 is provisioned with cloud-init, I needed to disable it first before assigning a static IP address to eth0.

I created the file /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg and added the following:

network: {config: disabled}

Then I assigned the static IP address 10.0.0.1/8 to eth0 by creating the file /etc/netplan/01-netplan.yaml and adding the following:

network:
    ethernets:
        eth0:
            addresses: [10.0.0.1/8]
            nameservers:
                addresses: [8.8.4.4, 8.8.8.8]
    version: 2

dnsmasq

Next, I installed dnsmasq (short for dns masquerade) to serve as my DHCP server.

# install dnsmasq
sudo apt update && sudo apt install dnsmasq
# backup the default dnsmasq config
sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.bak

I created a new dnsmasq config sudo vim /etc/dnsmasq.conf:

# DHCP should lease addresses over eth0
interface=eth0
# Listen on the static IP address 10.0.0.1
listen-address=10.0.0.1
# Enable dnsmasq's integrated DHCP server
# and define 96 available address leases
dhcp-range=10.0.0.32,10.0.0.128,12h
# declare name servers
server=8.8.8.8
server=8.8.4.4
# Bind dnsmasq to eth0
bind-interfaces
# prevent packets with malformed domain names from forwarding
domain-needed
# prevent packets from non-routed address spaces from forwarding
bogus-priv
# Use the hosts file on this machine
expand-hosts

Running sudo service dnsmasq status returns status failed because eth0 is not connected, yet. dnsmasq will fail to assign an IP address to an interface if it’s not up. That’s okay, we’ll get to the networking part later on.

forward internet from wlan0 to eth0

In my topology description above, the RPi hosts in my private network have access to the internet through the Router Pi. To achieve this, I’ll be setting up internet forwarding on the Router Pi.

First, I enabled IP forwarding by uncommenting the following line in /etc/sysctl.conf:

net.ipv4.ip_forward=1

Then I added a masquerade rule to packets leaving the wlan0 interface. This means that traffic through wlan0 can be rerouted without disruption.

sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

The next command forwards ESTABLISHED and RELATED packets from wlan0 to eth0.

sudo iptables -A FORWARD -i wlan0 -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

This last command forwards all packets from eth0 to wlan0.

sudo iptables -A FORWARD -i eth0 -o wlan0 -j ACCEPT

Finally, my iptables rules looked like this:

Chain INPUT (policy ACCEPT 30335 packets, 1580K bytes)
 pkts bytes target     prot opt in     out     source               destination                   

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination                   
    0     0 ACCEPT     all  --  wlan0  eth0    0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  eth0   wlan0   0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 30665 packets, 1624K bytes)
 pkts bytes target     prot opt in     out     source               destination

I installed iptables-persistent to save the iptables rules permanently.

sudo apt install iptables-persistent

Note: Remember to sudo reboot before proceeding to the next step.

raspberry pi hosts

I wrote the Ubuntu Server 20.04 image for the Pi hosts in almost the same way I did it for the Router Pi, except setting up a Wi-Fi connection. These hosts will access the internet via forwarded connections on the Router Pi.

putting it all together

The final stage is connecting all the Pis to the network switch and powering up the switch. All that toil has to amount to something.

Running sudo service dnsmasq status should return Active: active(running):

● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
     Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-01-12 12:12:11 UTC; 20min ago
    Process: 1249 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
    Process: 1356 ExecStart=/etc/init.d/dnsmasq systemd-exec (code=exited, status=0/SUCCESS)
    Process: 1379 ExecStartPost=/etc/init.d/dnsmasq systemd-start-resolvconf (code=exited, status=0/SUCCESS)
   Main PID: 1377 (dnsmasq)
      Tasks: 1 (limit: 4436)
     Memory: 2.7M
     CGroup: /system.slice/dnsmasq.service
             └─1377 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service --trust-anchor=.,20326,8,2,e06d44b80b8f1d39a95c0b0d7c65d08458e880409>

Running cat /var/lib/misc/dnsmasq.leases should list two DHCP leases, assigned to the two hosts in the private network. I tested internet forwarding by sshing into each Pi using the IPs on their leases and checking that they have internet access.

That’s it!

pains

I had a hard time figuring out how to assign a static IP address to eth0 on the Router Pi. The last time I did that was before netplan, when all network configs were on one file: /etc/network/interfaces.

The hardware currently sits in a disorganized state on my work desk. I ordered a Raspberry Pi cluster case and flexible Cat6 Ethernet cables so that I can have a better organized home lab.

The next article in this series is Running a Kubernetes Cluster On My Raspberry Pi Subnet

Hover As An Offline Fallback

Tue, 26 May 2020 07:28:22 +0000

A fallback is defined as a contingency option to be taken if the preferred choice is unavailable. Hover is an Android SDK that automates existing USSD sessions in the background of Android applications. We will set up a USSD channel (with a USSD backend), configure an action on the Hover dashboard and integrate the action as an offline fallback in an Android app.

USSD has two modes of operation, USSD PULL, which is mobile initiated and USSD PUSH, which is network initiated. For this fallback use-case, we’ll be using the mobile initiated mode to communicate data between the app and server. The USSD protocol will serve as a transport layer and USSD messages will be the data packets communicated between the app and the server. Hover will be the “USSD client” for the Android app, capable of sending and receiving USSD messages.

image 1.1

The maximum number of characters in a USSD message varies between carriers. Safaricom (KE) has a limit of 160 characters sent from the network from the user and a user input limit of 80 characters. Similarly, USSD session length varies between 90-180 seconds, depending on the carrier.

Architecture

image 1.2

For the purpose of this demonstration, we’ll be sending a set of key/value pairs to the server. The flow involves four messages sent between the mobile app and the server:

Mobile dials USSD code, initiating a USSD session
2. Server responds with the message send data, awaiting input
3. Mobile sends url query string
4. Server receives and persists the data; responds with a final success message

We’ll use the Hover SDK to automate these four steps.

The USSD Gateway

The first step is setting up a channel on the USSD Gateway. Unlike most protocols, USSD channels are limited to the country/region that the providing telecommunication company operates in. For the purpose of this demo I’ve used a shared USSD channel, * 384 *94#, provided by Africa’s Talking. This short code is only available on Safaricom (KE) and Orange (KE) networks. Dialing the USSD code gives us the following message:

That’s because we haven’t set up the callback function that will read and respond to USSD requests. USSD response messages are in plain text beginning with the keyword CON if the session is ongoing. If the response message is the last for the session, begin with the keyword END. A simple USSD callback function that responds request with the message Welcome to Hover would look like this:

    # Python 3.8.2
    func ussd_callback() {
        return “END Welcome to Hover”
    }

The complete callback function for the architecture described above (image 1.2) looks like this:

    import urllib.parse
    
    def save_report(event, context):
        params = urllib.parse.parse_qs(event['body'], keep_blank_values=True)
    
        if 'text' not in params:
            return {“statusCode”: 400, “body”: “END bad request”}
    
        text = params['text'][0]
        
        # Respond with the text `send input`, expect data
        if text == '':
            return {“statusCode”: 200, “body”:“CON send data”}
        
        # Parse url query string
        data = urllib.parse.parse_qs(text)
        
        # If the data sent is malformed, respond with `bad request`
        # and end the session
        if not bool(data):
            return {“statusCode”: 400, “body”: “END bad request”}
    
        # Process, persist, analyze data
        
        # End session, `success`
        return {“statusCode”: 200, “body”:“END success”}

The example above is Python 3.8.2 code deployed as a serverless function on AWS. I won’t dig deep on setting up a callback function in this blog post because a lot of the technical input is subjective based on language choice and deployment options. Africa’s Talking USSD API sends a post request to your callback function with five parameters as described in their documentation. In the example, we read the text field, which contains the user input. If the text field is empty then that means the session has just been initiated and the function responses with CON send data. Otherwise, we try parse the text field (assuming it’s url encoded) and end the session.

Dialing * 384 *94# now gives you the following messages:

The Hover Dashboard

Now we have a mature USSD interface that we can use to send data to our backend. The next step is integrating Hover to automate interactions with this interface. If you’re not familiar with creating actions and parsers on our dashboard, you can read through our documentation.

image 1.3 Offline Fallback Action

The image above is an action configuration for our USSD interface. There’s one step configured, a variable that’s a direct response to the send data prompt.

image 1.4

sendData.setOnClickListener {
            var payload = “”

            if (report_title.text.isNotEmpty()) payload = “message=${report_title.text}”

            payload = if (hungry.isChecked) {
                payload.plus(“;hungry=1”)
            } else {
                payload.plus(“;hungry=0”)
            }
            
            if (intensity.text.isNotEmpty()) payload = payload.plus(“;intensity=${intensity.text}”)
            
            // Input from a RatingBar
            payload = payload.plus(“;mood=${mood.rating}”)


            if (payload.isNotEmpty()) {
                val i = HoverParameters.Builder(this)
                    .request(ACTION_ID)
                    .extra(“payload”, payload)
                    .buildIntent()

                startActivityForResult(i, 0)
            } else {
                Toast.makeText(this, “Nothing to send!”, Toast.LENGTH_LONG).show()
            }
        }

The example above is in Kotlin and it demonstrated how you can run the action we created before with a valid payload. In this specific case, a valid payload means a URL encoded key/value pairs separated by a semicolon. Example:

activity=idle;course=818;altitude=1780.91;path=Ngong+road

Africa’s Talking’s USSD API posts query string encoded parameters to the callback URL and the individual parameters are separated by an ampersand (&). This means that we can’t use the ampersand (&) in our query string that will sit in the text parameter, that’s why I’ve intentionally used the semicolon separator in the schema above. The semicolon separator is illegal according to the 2014 W3C recommendation but Python 3 still supports it.

Running the action described above (on an Android phone with no internet connectivity) will result in the following screens:

Conclusion

This demonstration shows how you can send data from an offline phone to your server but since the flow of data is bidirectional, you can also send a payload from your server to your app.

Due to the restrictions in message size and session length, you can only send lean data via USSD. I used the query string schema because of the limited payload size but any schema/encoding can be used as long as the payload size is not exceeded. You can send multiple messages in sequence as long as the session is still valid. This means that you can implement a chunking algorithm to send data in sequence.

Originally posted on the Hover blog.

Preact-Rails Gem: Preact integration for Ruby on Rails.

Tue, 24 Sep 2019 00:00:00 +0000

Our vision at Hover is to empower developers with resources, services, and tools to build for local communities at global scale. As part of this vision, we’re happy to announce a new open source gem, preact-rails!

A few weeks ago we wrote about how our developers outgrew our dashboard within two months of our official launch. We took this as an opportunity to improve our dashboard UI and UX.

The first step was picking a javascript UI library. We had a lot of options but the ones that stood out were Vue.js and React. Part of the engineering team was familiar with React and it proved robust enough so we went ahead and picked that. You’re probably wondering why I’m mentioning React when the title clearly states “Preact”. Keep reading, it gets better.

Our dashboard is built on Ruby on Rails which is, by all accounts, very opinionated. Decoupling the front-end to an independent project would cause a lot of integration pains, the biggest being managing user sessions without compromising security. We wanted to upgrade the front-end without committing major changes to the back-end engine. The driving factor here was time, we didn’t have enough of it (do we ever?).

Taking this into consideration, we went back to our javascript UI library choice, React. We chose react before we made the choice not to build a whole front-end project on it’s own. The most feasible option was to serve react components on top of Rails views. It’s minimal and it serves our purpose of not intruding on the back-end and there was a gem just for this, react-rails.

We started testing the gem by building a few simple components: buttons and forms. Around this time David Kutalek came across Preact (mentioned to him by our head of design, Justin Scherer). We were intrigued. In a nutshell, Preact is a fast 3kB alternative to React. There are subtle differences that are addressed by a compatibility layer preact-compat. We wanted to test it out before we jumped in head first. There was one issue: we couldn’t find a gem similar to react-rails, a Preact implementation for Ruby on Rails. That’s when preact-rails was conceptualized. Inspired (and heavily influenced) by react-rails, we started building a Preact implementation for Ruby on Rails.

This is where we get a tiny bit technical. There are two parts of this project. The first part is the ruby gem, aptly named preact-rails. The gem serves one purpose, to make preact_component available as a view helper. This view helper takes in three variables, a component name, component props and component options and returns a div DOM node with the properties data-preact-class and data-preact-props. That’s a lot of confusing words, let me demonstrate.

In your view, you call preact_component like so:

  <%= preact_component(“SimpleButton”, { label: “Start” }) %>

Where “SimpleButton” is the Preact component name, label is a prop and “Start” is a prop value. And this is what gets rendered when you load the view on a browser:

  <div data-preact-class=“SimpleButton” data-preact-props=“{'label':'Start'}”>
  </div>

That’s the first part. The second part of the project is an npm package, preact_ujs. (UJS stands for Unobtrusive JavaScript). This package takes the div rendered above and inflates the named Preact component. This is the complex part but I’ll try make it comprehendible.

Let’s start with the SimpleButtonComponent, defined as such:

  import { h, Component } from “preact”
  
  class SimpleButton extends Component {
    render (props, state) {
      return (<button>{props.label}</button>)
    }
  }

The component returns a button DOM element with the property label as the child node. Now, back to preact_ujs. This package has a function mountComponent where all the action happens. It starts by finding all DOM components with the attribute data-preact-class, loops through each of them finding the component class definition and parsing the props. Finally, it renders the component, that’s it!

That’s the inner workings of the gem, you can take a closer look on the github repo. We have complete setup instructions on the README.

We used this gem to test Preact and we loved it. We really didn’t need all the rich features offered by React so the initial proposition attracted us. Three sprints later we launched our new dashboard. So far user feedback has been positive, our developers love it!

We’re still building on our dashboard, powered by our handy little gem. As we build we’ll be making improvements on the gem, starting with tests and TypeScript support. Pull requests are welcome!

Originally posted on the Hover blog.