The release of Gary Bernhardt’s screencast Functional Core, Imperative Shell mentioned at the end of his talk Boundaries was a transformational moment for me. Up until that point in my career, I hadn’t really thought too much about “architecture” as it relates to software, well at least not in the way I had heard software architects speak about it, which was, to my eager-but-naive ears, filled with a bunch of buzzwords and acronyms that I did not understand.

I have returned to both the screencast and talk many times because the concepts are packed with wisdom that continues to shape my thinking; it is evergreen content, you should watch both.

Gary’s insight was that separating the pure from the effectful lets you test what matters and push complexity to the edges. I think we’re at a similar inflection point now — but the “shell” has become something far more unpredictable than imperative I/O. The shell is now an LLM.

Where I Learned State Machines

From ~2008-2011 I worked at my first startup, Vendasta Technologies. They’re still around, and doing well, and dare I say had some of the best startup marketing of the 2000’s. Seriously just watch this 40 second spot riffing on how we named the company and tell me you didn’t laugh out loud …

The beauty of startups is twofold in my mind: you get to work with great people and you are forced to wear many hats. A few of those great people are folks who I deeply respect and admire, who have all gone onto greater things:

• Kevin Pierce was the person who got me into software development as a career in the first place and graciously spent time mentoring me at coffee-shops in Saskatoon, SK, answering all of my questions and helping me connect the dots when all I really knew was web frontend tech.
• Shawn Rusaw introduced me to the idea of expressing workflows and logic via finite state machines (FSMs) and did deep research into whitepapers on FSM architecture that shaped how our team built systems.
• Jason Collins was the CTO who led all of our teams and made all of the experimental ideas come to life knowing exactly how to put all the pieces together.

One of the highlights of working at Vendasta was the chance to get exposure to a ton of ideas; state machines resonated with me in particular because of how elegantly they enabled complex workflows to be built on top of simple concepts.

An early project the team built called MashedIn — which mashed your accounts from Twitter, LinkedIn, and Facebook to surface common connections — was where I began to connect the dots on these concepts.

MashedIn needed async workflows — fan-out across multiple social media APIs, reconciliation, deduplication — and the team built an async state-machine workflow engine in Python called Fantasm to power them. I recall eagerly watching demos and listening to Shawn speak about how it all fit together.

It was the first thing we open-sourced as a company, and Shawn, Kevin, and Jason were instrumental in making it happen on the beta version of AppEngine (what would eventually become a part of GCP).

It got so popular at the time that it was featured on the AppEngine blog. You can see how it has evolved on GitHub.

I want to underscore this was back in 2011!

Foundations of State Machines

Finite state machines trace back to the 1950s and 60s — Mealy (1955) and Moore (1956) formalized the two canonical FSM models, and they became foundational in compiler design, protocol specification, and hardware engineering. Shawn and the team at Vendasta took those ideas and made them practical for web-scale async workflows.

Fantasm’s FSM implementation was inspired by a later paper, published in 1999 by Jilles van Gurp and Jan Bosch: On the Implementation of Finite State Machines, and a talk by Brett Slatkin at Google I/O 2010 titled Building high-throughput data pipelines with Google AppEngine.

I re-read the paper this morning, and what struck me was how much it anticipates what came later. Their assessment section walks through how easy it is to add a new state — you just add a line of XML and retarget some transitions.

Their future work section even calls out conditional transitions (“transitions that only occur if the trigger event occurs and the condition holds true”) and Statechart support (“normal FSMs + nesting + orthogonality + broadcasting events”) — which is literally what XState implements today with guards, nested states, parallel states, and the sendTo/raise event system.

They also call out the need for “tracing and debugging tools” to keep track of what’s happening in the context — which is pretty much where XState’s Stately Inspector and Stately Studio have landed, giving you live statechart visualizations and sequence diagrams, but I digress.

The point I really wanted to make was, since my time at Vendasta, I have repeatedly returned to the concepts of finite state machines, and often found myself migrating business logic that was in the form of a poorly-specified-hidden-half-baked state machine into an actual state machine with transitions, guards, and easily unit-tested logic.

If there’s a golden hammer I think is actually worth swinging at every codebase, it’s definitely state machines.

(And if this wasn’t enough to summon David K. Piano, I don’t know what is). 😛

The Pattern Repeats

Five years ago I was working at SurveyMonkey as part of the Analyze team. The survey product was composed of a few pieces that fit together in a workflow:

Design -> Preview -> Collect -> Analyze -> Present

At the time, my team was tasked with coming up with a vision for how we could improve the Analyze experience for users; it had been built with a custom JavaScript framework that was difficult to follow, frequently violated Locality of Behaviour due to sprawl, had very fuzzy architectural boundaries, and made heavy use of event-driven architecture that was opaque.

Events fired randomly, state changed, and debugging was an exercise in frustration, requiring a detailed specification of “the framework” open and a lot of trial and error to trace code execution when debugging. I ended up building a proof-of-concept using XState v4.26.1 during one of our hackathons, and I recall thinking:

this entire workflow, end to end, feels like it should be a state machine, what would it look like if we had survey creators use a GUI to produce a state machine definition that we could serialize in our db as the source of truth, and then when survey takers go to take the survey, we reinflate that definition and have them run it live in their browser

I was pretty convinced this could work, and the POC provided enough direction to give me confidence.

The scenario I tested with was a school allergy survey. Three questions, starting with one about chocolate bars. The idea was that a survey author could configure conditional branching logic in the GUI — if a respondent picked a chocolate bar that didn’t contain nuts then the follow-up question about nut allergies wouldn’t apply and should be skipped.

Simple enough to explain, but it maps perfectly to a state machine: each question is a state, each answer is an event, and branching logic is a guard.

Because I like code, here’s the core of the POC (syntax is XState v4, so it’s old and out of date compared to v5, but illustrative enough to get the gist):

// xstate v4 syntax from ~5 years ago
import { Machine } from 'xstate'

const surveyTakingMachine = Machine({
  id: 'survey',
  initial: 'q1',
  context: {
    answerOptions: {
      q1: ['Snickers', 'Big Turk', 'Oh Henry'],
      q3: ['$0.50 USD', '$0.75 USD', '$1.00 USD'],
    },
  },
  states: {
    q1: { // What is your favorite chocolate bar?
      on: {
        RECEIVE_ANSWER: [
          {
            target: 'q3',
            cond: (context, event) => {
              // no nuts → skip the nut allergy question
              return event.answer === context.answerOptions['q1'][1]
            },
          },
          { target: 'q2' },
        ],
      },
    },
    q2: { // What is your schools policy on peanut allergies?
      on: { RECEIVE_ANSWER: [{ target: 'q3' }] },
    },
    q3: { // What is a fair price for a chocolate bar?
      on: { RECEIVE_ANSWER: [{ target: 'completed' }] },
    },
    completed: { type: 'final' },
  },
})

The builder (“config”) side dynamically generated these machine configs from user input. You’d add questions in a UI, define answer options, and input branching logic as a text expression like Q1 = C2 => SKIP TO P3 (if Question 1, Choice 2 is selected, skip to Page 3).

Obviously this was a POC — a production version would need a more robust expression language — but for the purposes of proving the concept it was sufficient. Every change regenerated the machine config in real-time — you could see the JSON update as you built. The config was then serialized to storage.

The survey taker doesn’t know there’s a state machine. They just see questions, one at a time, with branching that “just works”.

My design notes from the time are interesting to revisit in the age of AI:

SurveyTakingOneAtATimeMachine
SurveyTakingClassicMachine
SurveyTakingConversationMachine

A conversational survey (SurveyTakingConversationMachine) is, in hindsight, just a voice agent running a state machine. I just didn’t have the voice agent yet.

The POC didn’t seem to get much traction internally, which had me scratching my head — the approach seemed like such a natural fit to me. Maybe the idea was too ahead of its time, maybe XState wasn’t mature enough for the team, or maybe I just wasn’t good at pitching. 😅

A more likely reason is that my POC hadn’t fully solved the serialization/runtime challenges; I had conceptualized how it would work — a machine definition as a JSON blob serialized in your database and inflated at runtime — but I didn’t show how that could happen aside from a basic demo using localStorage.

XState v4 had not yet formalized use of the Actor model, which sits beautifully on top of a state machine; in v5, XState made this explicit — every running machine is an actor, and you get getPersistedSnapshot() to serialize the full actor state and restore it later with createActor(machine, { snapshot: restored }).

In 1999, van Gurp & Bosch were solving the same problem with XML and Java serialization:

FSMAction components are instantiated, configured and saved to a file using serialization. The saved files are referred to from the XML file as .ser files. When the framework is configured the .ser files are deserialized and plugged into the FSM framework.

The round-trip from database to runtime and back that they were building with XML and .ser files is the same round-trip XState v5 gives you with JSON and getPersistedSnapshot().

Deterministic Core, Agentic Shell

I keep coming back to this idea of core/shell as I repeatedly see the pattern of legacy applications that could be simplified by using a state machine in the core.

Which brings me back to where I started: Gary defined the functional core and imperative shell as a pattern that informed the type of code and approach to testing that I have seen work extremely well in practice. I think we are now entering a time where we can apply a similar architectural lens to apps that leverage LLMs.

Because I lack originality, and I really like Gary, I’m calling it deterministic core, agentic shell.

The trick as I see it is, all these companies want to build “agentic” features and rely heavily on the LLM, which frequently (and rightfully) violate every software engineer’s sensibilities around determinism. Evals, LLM-as-judge, vibes-based-testing, and all of the “rigor” around how to “test” prompts and use of agents is all well and good, but for my money nothing beats the fast feedback loop and confidence of a sufficiently isolated unit test with pure functions.

So, similar to how functional core was Gary’s answer to testability in a world full of side effects, my assertion is that state machines are the answer to determinism in the era of AI agents. I have seen time and again that if we draw a larger box around that core and try as hard as possible to shove all the things that are important into it, and into a state machine, that the layers above (both imperative and agentic) become minimized, reducing risk, and making it much easier to verify correctness in the core of the system.

A Pragmatic Reference Architecture

I’ve been spiking on a project with Jon Girard and Michael Timko and applying these ideas — the heritage from van Gurp & Bosch, the configuration-driven approach from Fantasm, the machine-as-source-of-truth thinking from my SurveyMonkey POC — and it is proving highly effective at validating deterministic core, agentic shell.

Here’s the broad strokes of what we’ve been working with:

+--------------------------------------------------+
|  Telnyx (telephony)                              |
|  caller <-> WebSocket <-> u-law 8kHz audio       |
+--------------------------------------------------+
|  Agentic Shell                                   |
|                                                  |
|  Mastra Agent + OpenAI Realtime (voice LLM)      |
|  - system prompt constrained by machine state    |
|  - available tools constrained by machine state  |
|                    |                             |
|                    | tool calls                  |
|                    v                             |
|  Mastra Tools (bridge)                           |
|  - translate agent intent -> machine event       |
|  - return machine state -> agent                 |
+--------------------------------------------------+
|  Deterministic Core                              |
|                    |                             |
|                    | events                      |
|                    v                             |
|  XState Machine                                  |
|  - states, transitions, guards, actions          |
|  - pure functions, trivially testable            |
+--------------------------------------------------+
|  Postgres (storage)                              |
+--------------------------------------------------+

If you have worked with Mastra before, there are some great framework primitives that will let you build a workflow, agents, and even individual tool calls; it’s very easy to put all your eggs in the Mastra basket, and it is good at what it does, but one of my hard-learned engineering principles is it’s better to minimize coupling to 3rd party systems and tools and put as much of your business logic into code that you own.

When the “system” is a sprawl of connections to 3rd party services or libraries, the surface area of your codebase becomes harder to test, change, and that much more of a liability.

Jason Collins, the CTO at Vendasta I mentioned earlier, used to say “show me the code!” whenever someone pitched an idea in a meeting.

I’ve always loved that — so in that spirit, here’s some code!

A Deterministic Core

Let’s use a simple identity verification flow as our example — the kind of thing you’d encounter at the start of any phone-based workflow.

Everything important lives in the machine definition:

// xstate 5 syntax
import { assign, setup } from "xstate";

export const sessionMachine = setup({
  types: {
    context: {} as {
      attempts: number;
      verifiedId: string | null;
    },
    events: {} as { type: "VERIFY"; id: string },
  },
  guards: {
    isValidId: ({ event }) => event.id === "6789",
  },
  actions: {
    incrementAttempts: assign({
      attempts: ({ context }) => context.attempts + 1,
    }),
    setVerifiedId: assign({
      verifiedId: ({ event }) => event.id,
    }),
  },
}).createMachine({
  id: "session",
  initial: "greeting",
  context: { attempts: 0, verifiedId: null },
  states: {
    greeting: {
      on: {
        VERIFY: [
          {
            guard: "isValidId",
            target: "verified",
            actions: ["incrementAttempts", "setVerifiedId"],
          },
          {
            target: "awaitingVerification",
            actions: ["incrementAttempts"],
          },
        ],
      },
    },
    awaitingVerification: {
      on: {
        VERIFY: [
          {
            guard: "isValidId",
            target: "verified",
            actions: ["incrementAttempts", "setVerifiedId"],
          },
          { actions: ["incrementAttempts"] },
        ],
      },
    },
    verified: { type: "final" },
  },
});

The guard (isValidId) is a pure function. The actions are deterministic assignments. The transitions are explicit. You can unit test this without any LLM, any network, any voice connection. Deterministic core.

Bridge: Tools That Connect Agent to Machine

The LLM doesn’t decide if the user is verified. The machine does. A Mastra tool is the thin bridge between them:

export const verifyIdentityTool = createTool({
  id: "verify-identity",
  description:
    "Verify the user's identity by checking their 4-digit verification ID.",
  inputSchema: z.object({
    id: z.string().describe("The 4-digit verification ID provided by the user"),
  }),
  execute: async ({ context, runtimeContext }) => {
    const sessionActor = runtimeContext.get("sessionActor") as SessionActor;

    // Agent -> event to the machine — machine decides what happens
    sessionActor.send({ type: "VERIFY", id: context.id });

    // Machine state is the source of truth
    const snapshot = sessionActor.getSnapshot();
    const verified = snapshot.value === "verified";

    return {
      verified,
      message: verified
        ? "Identity verified successfully."
        : `Incorrect verification ID. (Attempt ${snapshot.context.attempts})`,
      attempts: snapshot.context.attempts,
    };
  },
});

The tool translates what the agent heard into a machine event, and reports the machine’s verdict back. The agent can be “creative” with language (still working on this part, so YMMV here); the machine is authoritative on state.

Orchestration: Machine State Drives the Agent

This is where things get fun; dynamic tool swapping. The agent’s capabilities — both its instructions and its available tools — expand and contract based on the deterministic core’s state:

// Before verification: agent can ONLY verify
export const preVerificationTools = [
  mastraToolToOpenAI("verify-identity", verifyIdentityTool),
];

// After verification: agent gets financial tools
export const postVerificationTools = [
  mastraToolToOpenAI("verify-identity", verifyIdentityTool),
  mastraToolToOpenAI("getTransactions", getTransactionsTool),
  mastraToolToOpenAI("getAccounts", getAccountsTool),
];

When the machine transitions to “verified”, the agent’s entire configuration gets swapped in Mastra — different instructions, different tools:

// Create session state machine and inject into agent's runtime
sessionActor = createSessionActor();
const runtimeContext = new RuntimeContext();
runtimeContext.set("sessionActor", sessionActor);

await agent.voice.connect({ runtimeContext });

// Start with limited tools and verification-focused instructions
agent.voice.updateConfig({
  instructions: `You are a helpful financial assistant.
    The user has NOT been verified yet.
    Ask them to provide their 4-digit verification ID.
    Do NOT discuss any financial details until verified.`,
  tools: preVerificationTools,
});

// When the machine transitions to "verified", unlock everything
agent.voice.on("tool-call-result", (data) => {
  const { toolName, result } = data;
  if (toolName === "verify-identity" && result?.verified) {
    agent.voice.updateConfig({
      instructions: `You are a helpful financial assistant.
        The user has been successfully verified.
        Use get-transactions and get-accounts to help them
        with their finances. Be conversational and helpful.`,
      tools: postVerificationTools,
    });
  }
});

The machine transition (greeting → verified) is deterministic and testable. The consequence of that transition — swapping out the agent’s entire instruction set and available tools — is where the agentic shell gets reconfigured, and is also deterministic and testable.

Scaling: A Real Voice Workflow

The code above is simple on purpose, but early tests and spikes have me confident that the pattern works. Imagine a production voice agent where callers dial in and speak to an AI assistant to complete a multi-step workflow over the phone:

Telnyx handles telephony and streams audio over a WebSocket. Mastra connects to the OpenAI Realtime API for speech-to-speech — the caller speaks naturally and hears natural speech back. The round trip for a single turn looks like this:

1. Caller speaks
   → Telnyx streams audio (μ-law, 8kHz) via WebSocket

2. OpenAI Realtime processes speech
   → Mastra forwards audio to OpenAI
   → Speech-to-text + LLM reasoning

3. LLM calls a tool
   → take_action({ action: "SUBMIT_ID", params: { id: "123456" } })
   → Tool runs: machine.send({ type: "SUBMIT_ID", id: "123456" })
   → Machine transitions: greeting → verifying_identity
   → Tool returns: { success: true, nextStep: "ask_for_details" }

4. OpenAI generates response
   → Uses tool result to generate spoken response
   → "Thanks, I've got that. Now, could you tell me which
      account you'd like to work with?"

5. Audio plays to caller
   → Mastra → Hono API → WebSocket → Telnyx → phone

The agentic shell (Mastra + OpenAI) handles everything about the conversation — hearing the caller, interpreting speech, generating natural responses, handling “ums” and false starts and off-topic questions. Early signs are good, and one happy finding: the OpenAI realtime voice model supports interruptions and bi-directional voice communication out of the box.

The deterministic core (XState) handles everything about the workflow — what state we’re in, what’s valid, what happens next. The tools are the membrane between them.

Two tools do the heavy lifting: get_current_state lets the agent ask the machine “where are we and what can I do?”, and take_action lets the agent tell the machine “let’s move forward.” The machine enforces the rules — its guards reject anything that doesn’t satisfy the business logic. The agent is creative about how to have the conversation; the machine is authoritative about what happens next.

export const get_current_state = {
  name: "get_current_state",
  description: "Get current state of the workflow for this session",
  handler: async ({ sessionId }) => {
    const machine = getMachine(sessionId);
    return {
      state: machine.state.value,
      context: machine.state.context,
      availableActions: machine.state.nextEvents,
    };
  },
};

export const take_action = {
  name: "take_action",
  description: "Advance workflow by sending an event to machine",
  parameters: {
    action: {
      type: "string",
      enum: ["SUBMIT_ID", "SELECT_ACCOUNT", "CONFIRM",
             "ACCEPT_TERMS", "COMPLETE"],
    },
  },
  handler: async ({ sessionId, action, params }) => {
    const machine = getMachine(sessionId);
    machine.send({ type: action, ...params });
    return {
      success: true,
      newState: machine.state.value,
      message: getStateMessage(machine.state.value),
    };
  },
};

The system prompt constrains the agent’s conversational behavior, while the machine constrains the workflow:

export const SYSTEM_PROMPT = `
You are a professional assistant helping callers complete a workflow over the phone.

RULES:
1. Be clear and patient. Use simple language.
2. Always disclose important information upfront.
3. Never rush the caller. Frame every step as optional.
4. If you don't understand, ask for clarification.
5. If something goes wrong, apologize and offer a retry or human transfer.

CONVERSATION FLOW:
1. Verify the caller's identity
2. Present available options, caller selects
3. Run validation, explain if ineligible
4. Disclose any fees or terms, confirm acceptance
5. Execute the action, confirm completion

TOOLS:
- get_current_state: Check where we are in the workflow
- take_action: Advance the workflow
`;

The system prompt also tells the agent how to talk; the machine tells it when. Language lives in the shell, logic lives in the core.

Boundaries

Applying the core/shell lens has helped me think about where to draw the boundary between agent and state machine, and we are now in a world where it is trivial to produce working concepts that can validate where that line should be.

One way you might think about this is something like “let’s start with one Mastra tool, getMachine, and the only thing it was able to do was get the next state from the machine.”

This is a decent place to start, but kind of fell over in short order because there’s some amount of non-determinism you need in order for an agentic voice solution to work well — the agent needs to interpret messy human speech, handle interruptions, deal with off-topic questions, and generally be flexible about how it gets the information the machine needs.

What I’ve found works better is more guard-rails in the system prompt for the tools and workflow, but the principle of having the agent calling the machine to figure out what’s possible and constrain the states and tools that can be called next has proven extremely valuable.

The primary motivation should immediately be clear: get to determinism as fast as possible.

The voice agent interprets the caller’s input (non-deterministic), figures out what they’re trying to do (non-deterministic), and then immediately hands off to the machine (deterministic). The machine decides what’s valid, what the next step is, and what tools become available. The agent gets back a structured result and turns it into natural speech (non-deterministic again).

The non-deterministic parts are thin — the deterministic core is thick and does the actual work.

Lineage

I fell into an obsession with state machines as an architectural pattern by accident. I got to learn from Shawn, Jason, and Kevin at Vendasta, who were drawing on the pioneers that came before them. And I suspect there are more through-lines worth tracing, by revisiting old whitepapers and watching how ideas get refined as they move from theory to practice.

Every time I pick up a new codebase, I find the same thing hiding under the surface — a half-baked state machine that nobody drew on a whiteboard first. Events firing into the void, state scattered across a dozen files, transitions implicit in if/else chains or switch statements three levels deep.

If you believe the chatter online, then you might feel like we are dealing with a lot more non-determinism these days; the imperative I/O Gary uses to frame his version of the shell definitely feels a lot more deterministic to me than an LLM that can hallucinate, go off-script, or confidently tell a caller something that isn’t true.

That makes the case for a deterministic core a lot stronger.

I’ve been down a nostalgia rabbit hole lately, trying to track down a lot of my early internet artifacts—including my first website.

I had been hunting in the various internet archives for DM Designs™, the first website I published in 1998, but for the life of me I couldn’t remember the URL slug. I knew the GeoCities neighborhood was in Sunset Strip, but my searches in many of the remaining archives always came up empty. Those archives are themselves admittedly only partial snapshots of what once was.

As luck would have it, while digging around in my basement, I stumbled across a DVD-RW I had burned around 2003. It was a backup of many of my early web projects, and stuffed in a folder was a partial copy of the GeoCities site.

The code was a typical late ‘90s mess: invalid HTML, broken images, the works. Pretty common for the time, given that a lot of it was built with the GeoCities WYSIWYG editor. This was my very first foray into web development. Finding the site was great, but getting it up and running again turned into a fun experiment that became its own side-quest exploring what features existed in web browsers at the time.

Reflecting on this almost 30 years later, I’m struck by how robust HTML, CSS, and JavaScript are. It’s remarkable how well these languages and browsers have held up over time.

The hunt for missing pieces

I started exploring whether some of the assets missing from my local archive could be recovered from the Wayback Machine. There were a few hits for some of my other sites, but for this one a lot was missing. I still couldn’t find the original snapshot of my page either. It figures. A 17-year-old kid’s personal design page probably wasn’t notable enough for the Wayback Machine to have crawled, or for any of the GeoCities archives to have captured.

I’ve been using agentic coding tools like Claude and Codex regularly, and I often default to them for brainstorming sessions. Through the process of trying to resurrect the site, I discovered that they are surprisingly capable as partners in digital archival searches through the Wayback Machine. Early explorations helped me understand the Wayback Machine’s free JSON API. I could tweak URL parameters and use regex matching to quickly search for assets or even partial URL slugs. Using that, I was able to recover some of the things I was missing.

The stories GeoCities markup tell

The first thing I noticed when opening the site’s index page was actually the name of the file itself: index.htm. Opening it in Zed threw a UTF-8 error: stream did not contain valid UTF-8, which jogged my memory about early web browsers and encodings that may have predated UTF-8. The encoding markup in the HTML showed up as a Latin encoding, which surprised me, but the meta tags in the markup told more of the story:

<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="GENERATOR" CONTENT="Mozilla/4.04 [en] (Win95; I) [Netscape]">

The GENERATOR tag shows I was using Netscape Communicator 4.04, which at the time included Netscape Composer, a WYSIWYG HTML editor. [en] is for English.

Claude told me that in the late ‘90s, the Windows 95 English version I was using likely would have used Windows-1252 (Western European) as its default, and Netscape Composer defaulted to ISO-8859-1 for HTML files when set to the English locale. Interestingly, these two encodings are nearly identical and only differ in a few code points—but I knew none of that back in 1998.

As I explored the markup, I made a happy discovery: I was able to confirm the original GeoCities slug and neighborhood my page was part of, http://geocities.com/SunsetStrip/Palladium/4011. Alas, I still can’t seem to find an original copy in any of the online archives. GeoCities was huge though, and it seems like preservation efforts probably started too late for just how big that part of the internet was at the time.

My archive had a mix of 3-character and 4-character filename extensions for the markup itself—index.htm and contactpage.html. I was wondering why and did some digging. Claude told me this was another artifact of DOS v8.3 filename conventions, which had clearly influenced early web development. Part of me vaguely recalls Microsoft FrontPage 98 being fond of using 3 character extensions.

Anyway, I updated all the markup files to use a .html extension and updated all my internal links accordingly.

Claude as digital archivist

I ended up leveraging Claude as a kind of digital archivist to help me fix everything that was wrong with the site, but I instructed it to take extra care not to fundamentally change anything. We fixed the broken encoding, standardized filename extensions, dealt with missing images, cleaned up invalid characters that appeared when I converted to UTF-8, and added missing closing tags so my liberal use of <CENTER>, <FONT>, and <TABLE> would render properly again.

I had assumed this site predated the W3C validator, but a quick search showed it had just launched in late 1997. It clearly wasn’t on my radar at all, as I recall becoming much more concerned with validation only once XHTML and an official validator launched a few years later in 2000.

It was a similar story for CSS: the frequent use of stylistic attributes like border, bgcolor, height, and width at the time had me wondering if the site predated CSS, but a cursory glance at the W3C spec told me I just didn’t know about it yet and leaned heavily on the tools in my toolbox and what all the other codegen tools like GeoCities were doing at the time: inline styles, tables for layout, and gratuitous use of animated .gif files!

A whole bunch of JavaScript was still there trying to load and execute, and the popups still fired!

<SCRIPT LANGUAGE="javascript">
var cuid= "10199"; var keywords= "none";
var urlOfNewPop="http://www.geocities.com/ad_container/pop.html?cuid="+cuid+"&keywords="+keywords; oldPop=window.open(urlOfNewPop, '_popIt', 'width=515,height=125');

if (oldPop.location.href != urlOfNewPop) {
  if ((navigator.appName == "Netscape") && (parseInt(navigator.appVersion) == 3)) {
    setTimeout("oldPop.close()", 750);
    setTimeout("window.open(urlOfNewPop, '_popIt', 'width=515,height=125')", 1700);
  } else {
    oldPop.close();
    setTimeout("window.open(urlOfNewPop, '_popIt', 'width=515,height=125')", 1000);
  }
}
</SCRIPT>

The infamous GeoCities popup JavaScript, which still works if you allow popups.

As was the fashion at the time

Digging into some of the image files was illuminating. I discovered all kinds of format “carbon-dating” clues in the metadata: image dimensions and even which tools were used to create them! This was very helpful, because nearly all of the images in the markup had dimension attributes, and it was useful to correlate which ones matched the files on disk—many of which had different names for some reason. Perhaps an artifact of when I had uploaded them to GeoCities? I have very vague memories of using a web editor for pasting snippets of HTML and uploading images. God, I feel old.

Most of the JPEG files were .jpg and contained JFIF headers (JPEG File Interchange Format) at 72 DPI resolution—pretty much a standard for web graphics at the time. GIF files were version 89a, one of the first versions that supported transparency and animation. The animations were built with GIF Construction Set, and a lot of the artwork was done with Photoshop 4.0.1.

An exploration partner

This whole exploration was a lot of fun. I don’t know about you, but looking back that far into the past definitely brought me right back to where I was as a 17-year-old working on the core pieces of the web and not knowing a damn thing about it.

It was fun to have an LLM partner throughout my explorations too. I think there’s a lot of emphasis on using AI for generating new content, but the more I use these tools and default to them, the more I find them to be really powerful partners for exploration and, in this case, digital archaeology. They are able to recall a lot of information about obscure old tech, which isn’t surprising—they were trained on a lot of the public internet. The fact that they can quickly reference spec files, jump back in time, look at and understand deprecated APIs and historical file formats, read 25-year-old code, understand the intent with enough context, and help me fix it—that’s pretty cool.

This also got me thinking about other ways to use this capability. One of the other things I discovered on my backup disk was a lot of early internet art I had created—3D animations from a pirated copy of 3D Studio Max, an old hand-rotoscoped lightsaber test video, and even an old Half-Life 2 steam skin that crashed my web host in the early 2000’s. Those are stories for another time. But one of the things I’m interested in exploring next is maybe building a Claude Skill to time-warp back to the nineties—to impart enough knowledge in the skill definition that Claude could build a new site, but with all the constraints, techniques, and style of the time.

If nothing else, this convinced me that LLMs aren’t just for generating new stuff—they’re weirdly good at helping us dig up, understand, and preserve the old, broken, wonderful corners of the web too.

For now, I’m pretty happy that I found my backup archive and that I was able to restore my site. Here it is in all its glory: 1998.davemo.com

I spent a few hours yesterday experimenting with Sora 2, trying to figure out if it could help with asset pre-production for a game idea I’ve been noodling on for about a year. Turns out, it’s a pretty capable partner—with caveats. You can get solid consistency in format (camera angles, poses, framing) by being specific in your prompts, but aesthetic consistency—the actual look and feel across generations—is still a crapshoot.

The levers for locking down visual style just aren’t there yet, or at least not exposed. Still, for pre-production exploration and prototyping? It’s pretty damn useful if you set the right expectations.

My GameDev Origins

Before I became a full-time programmer, I was into 3D graphics and graphic design. We got our first family computer in 1996—an Acer Aspire with a knockoff Cyrix 6x86-P133 chip. I cut my teeth on pirated copies of Photoshop, Illustrator, and 3D Studio Max—downloaded over dialup—diving deep into the world of 3D modeling, texturing, and animation. I never quite mastered rigging (inverse kinematics confused the hell out of me), but I was obsessed.

Blender first got on my radar around this time, back when it was freeware, pre-OSS release (I think I started around version 1.6). I loved working in Blender because it was small, capable, and I didn’t have to sail the high seas. 🏴‍☠️

As I got deeper into 3D work, I naturally gravitated toward level design—first with Ken Silverman’s Build Engine (Duke Nukem 3D), then later with Valve’s Worldcraft (Half-Life). There was something satisfying about building playable spaces, not just rendering static scenes. That’s what eventually pulled me into the world of game modding.

Fast forward to 2003: S2 Games had just released Savage: The Battle for Newerth. I met a team of devs on Freenode who were building a mod on top of the Silverback engine, and I dipped my toes into the gamedev industry properly for the first time.

The mod didn’t really go anywhere—I can’t even remember the name of it—but I found myself far more interested in the web dev side and poured my energy into building our little modding team’s web presence and styling the phpBB installation we had going to manage development. Still, that experience gave me a taste of the gamedev pipeline: concepting, pre-production, asset production, prototyping, playtests. Some 25 years later, that foundational knowledge is proving useful as I re-engage with gamedev.

The Game Idea

The concept is simple: a dwarven-themed take on the excellent Moonlighter, with a twist. Instead of being a shopkeeper, you’re an innkeeper. You run an inn during the day—making food, breaking up bar fights between dwarven patrons, earning coin to invest in better furnishings. Instead of “moonlighting” in dungeons after hours, you go hunting, gathering, fishing, and foraging for higher-quality ingredients to feed your ever-hungry customers. Part sim, part survival-lite.

I have some lofty goals for multiple gameplay loops, but I’ve set those aside to just start prototyping and producing assets. That’s where Sora 2 comes in.

Lookdev and AI

Today, we have these incredibly powerful idea engines that let us produce assets in minutes—assets that would’ve taken days for a single artist just a few years ago. Sure, they’re low quality and better suited for prototyping right now, but the speed is transformative.

I started with a basic question: What type of prompt would I need to get Sora to produce lookdev assets for a dwarven innkeeper in a neutral pose?

Early Experiments: Finding the Aesthetic

My early prompts explored stylistic elements—texture, theme, stylized vs. photorealistic. I ping-ponged between Sora and GPT-5, using the latter to educate me on formal terminology I’d forgotten or never learned.

Here are some early attempts:

2.35:1 cinematic, 24fps. Opening wide shot (24mm) eye-level, slow push-in through bustling dwarven taproom with flagstones, oak beams, rune-carved pillars. Warm 3000K firelight + practical candles, soft rim lighting; gentle haze for light rays. Cut to medium (50mm) tracking left as innkeeper (braided beard, leather apron) slides tankard; shallow DOF. High-detail props: copper taps, carved mugs, stew steam. Final close-up (85mm) with rack focus at 8s from stew to ledger UI: minimalist diegetic overlay (gold coins + rep meter). Teal-orange grade, subtle 35mm film grain, Deakins-style naturalism. Natural motion blur, clean cut transitions.

One unbroken 10s take, 60fps, real-time gameplay capture. Low-poly dwarven tavern with toon outline shader and flat color ramps. Third-person shoulder cam follows the innkeeper from keg to bar; smooth, game-like acceleration/deceleration. Patrons in looping emotes (cheer, sit, wave). Lighting: high-key warm directional + baked bounce; crisp shadows. Foam as simple particle sprites; stew steam as billboard particles. HUD: thin outlined panels with bold pictograms; +10 coins tick at top-left, Order Served toast top-right. No cuts, no photo montage, no still frames. Loop seamlessly.

Real-time gameplay capture, continuous 10s, 60fps, no cuts. Third-person follow cam tracking a dwarven innkeeper jogging through misty conifer forest at blue hour; hand-painted terrain, stylized-PBR foliage; volumetric fog and light shafts. Subtle head-bob, camera eases around trees; IK foot planting. The character harvests glowing chanterelles (bioluminescent), +1 ingredient appears as tiny rune HUD near the mushrooms, then fades. Audio: soft wind, branch creaks, cloth rustle. Cool teal environment vs warm fungus glow; mild DOF; foamless motion blur. Seamless loop.

These experiments were fun, but not exactly what I was looking for. They helped me identify what I liked aesthetically and gave me a sense of Sora’s capabilities. As I refined my prompts, I realized there was likely a more structured approach that would help with look tests of characters, props, and animations.

The “What Ifs”

Like any good coding rabbit hole, my journey had me wondering:

• What if I could get consistent character outputs from Sora?
• What if I could use those to generate A-pose and other lookdev assets that a traditional 3D artist might use?
• What if I could take that asset and auto-generate a rough pre-production mesh?
• What if there existed a tool that could auto-edit and auto-rig the model so I could drop it into a test level?

I didn’t get to that last one (yet!), but I did accomplish the first three. And honestly? It’s really damn cool. We’re living in the most amazing version of “draw the rest of the owl” I ever imagined possible, and if you have any creativity whatsoever and a vision for what good looks like, it’s such an energizing time to be a builder of things.

Constraining Sora to Give Consistent LookDev Outputs

Once I realized you could constrain Sora with detailed prompts—including key lighting, temperature, intensity, materials, specific geometry, animation loops—I started experimenting more deliberately.

The thing is, I’d been away from game production for about twenty-five years, and the industry has evolved massively in that time. Back when I was doing this stuff, I was entirely self-taught, scraping together knowledge from forums and tutorials. There’s a whole professional vocabulary and standardized pipeline terminology now that didn’t really exist—or at least wasn’t codified—when I was working in 3D Studio Max and early Blender. I just didn’t know what I didn’t know.

So I leaned on GPT-5 as an educator. I found that shifting my role to that of a learner, and being specific about my background, was surprisingly effective context engineering. I told GPT-5 about my gamedev history, and it would respond with, “Ah yes, these are the pieces of the pipeline you’re describing, and here’s the modern terminology and keywords—here’s how to put it together in a prompt for Sora.”

The challenge was getting stylistic consistency. I went through many iterations, some decent but not quite the aesthetic I wanted. When I found something I liked, I doubled down on it, tweaking prompts to try and maintain a common theme.

The key insight here: specificity breeds consistency. The more detailed your prompt about lighting, materials, pose, and camera behavior, the more control you have over the output. When you don’t know what you don’t know, you can ask GPT-5 or another LLM to help bridge that gap, translating your intent into the precise terminology that makes prompts more effective.

Example Prompts That Worked

Real-time character lookdev test, one continuous take, 10 seconds, 60fps, no cuts, no still frames, no timelapse. Locked front orthographic camera, waist-up to full-body framing, centered. Painted-toon hybrid: low/medium-poly forms, hand-painted albedo with visible brush strokes, stylized-PBR lighting, thin 1–2px toon outlines, two-step shadow ramps (teal shadows, warm highlights). Dwarven innkeeper: braided beard, leather apron, oak-tone bracers, stout silhouette. Idle animation: slow breathing, slight weight shift, tiny head turn, natural blinks; beard/cloth secondary motion; subtle finger fidgets. Lighting: soft key 3000–3500K from screen-left, rim from screen-right 5000K, gentle bounce; neutral backdrop (oak-grain or mid-gray). No camera move, no DOF, natural motion blur minimal. Turn on shader cues: specular on brass buckles, matte leather roughness, painted wood grain. Loop seamlessly end-to-start.

Real-time character turnaround test, continuous single take, 10s, 60fps. Locked camera, character rotates 360° clockwise at constant speed on a turntable; no cuts. Painted-toon hybrid look: hand-painted textures, thin toon outlines, two-step shadow ramps, stylized-PBR highlights on brass fittings. Lighting: three-point—key 3200K, fill −1.5 EV, cool rim 5500K; floor card for bounce. Animation: subtle idle breathing while rotating; beard/cloth secondary motion; no exaggerated sway. Backdrop: neutral studio sweep; no DOF. Material callouts visible: leather roughness, wood grain on tankard at belt, brushed metal on belt hooks. Seamless loop end-to-start (rotation lands on front exactly at 10s).

Real-time lookdev test, one continuous take, 10 seconds, 60fps, no cuts, no still frames, no timelapse, seamless loop end-to-start. Painted-toon hybrid: low/medium-poly forms, hand-painted albedo with visible brush strokes, stylized-PBR lighting, thin 1–2px toon outlines, two-step shadow ramps (teal shadows, warm highlights). Locked front orthographic camera, full-body, centered trio. Neutral mid-gray studio backdrop, no DOF, natural motion blur minimal. Three dwarven innkeeper variants stand side-by-side at equal scale: Tankard-Bearer (broad shoulders), Smith-Chef (apron + skillet on belt), Scout (lighter kit). All perform subtle idle: slow breathing, slight weight shift, natural blinks; beards/cloth secondary motion. High-contrast rim so silhouettes read clearly; lighting constant; no camera move.

Getting Sora to Build Useful A-Pose Assets

Early on, I had pulled some Sora outputs into a demo version of Stable Fast 3D, then into Blender, but I ran into a few quality problems with the models. GPT-5 suggested a pre-production cleanup process I could do manually, but it felt tedious and I was really trying to optimize for speed over quality at this stage anyway.

I hadn’t considered this ahead of time, and my early assets didn’t have enough space between the arms, causing overlapping geometry that would make rigging difficult. So I went back to GPT-5 and learned about A-poses—neutral poses with arms at about 30-45 degrees from the body, rather than straight out (T-pose). This makes it easier for auto-rigging tools to identify limbs.

The frustrating part: as I refined the prompts for better poses, the aesthetic would sometimes shift. And despite my best attempts, Sora often wouldn’t complete a full 360° rotation within its 10-second limit.

A Happy Accident : Character Sheet Plates

I was trying to get a character turnaround and accidentally left in all of the parameters from a prompt that GPT had told me to use one at a time for separate Sora iterations; this ended up causing Sora to create a character sheet with multiple views. Instead of animating it, Sora generated a static image with front, right-profile, and back views—all on one plate. This turned out to be incredibly useful as a reference to snap screenshots from to use further on down the line for mesh generation.

Character sheet plate, single still. Dwarven innkeeper, painted-toon hybrid (hand-painted albedo, thin 1–2px outlines, two-step ramps, stylized-PBR highlights kept subtle). Stocky heroic proportions (broad shoulders, short legs), braided beard with clear air-gap from chest, bald crown with close-cropped sides, simple tunic, belt, pants, boots; no props/cape/apron/toolkit. A-pose: arms ~30° down from horizontal, hands open, fingers separated; legs shoulder-width, feet flat on ground plane, head level, neutral face. Camera: orthographic, full-body, perfectly [FRONT / RIGHT-PROFILE / BACK] facing, centered; framing consistent. Background: neutral mid-gray seamless; no DOF, no motion blur. Lighting: even studio three-point (key 5000K softbox, fill −1 EV, soft rim); shadows soft and minimal; materials unchanged. Exposure even, no vignetting, no color cast. Style anchors: warm, hand-painted base colors (muted earth palette: umber leather, slate cloth, iron accents), clean silhouettes, no grunge overlays, readable edge accents on folds and seams. Negative: no pose changes, no stepping, no crossed limbs, no props, no background objects, no text, no logos.

Other A-Pose Prompts That Worked

Character turnaround, 10s, 60fps, single continuous take, loopable. Painted-toon hybrid (hand-painted albedo, thin 1–2px outlines, two-step ramps, stylized-PBR). Locked orthographic camera, full-body, centered; neutral mid-gray cyc; no DOF, no motion blur. Dwarven innkeeper in A-pose (arms ~30° down from horizontal), legs shoulder-width, feet flat, hands open with fingers separated; head level; neutral face. No props, no cape, no apron, no toolkit. Beard volume clearly separated from chest. Elbows and hands fully visible (no self-occlusion). Character rotates 360° clockwise at constant speed on a turntable; framing constant. Lighting: neutral studio 3-point (key 5000K softbox, fill −1 EV, subtle rim), soft shadows; materials unchanged throughout.

I’ve since learned that the best angle for these lookdev assets is a 3/4 view—slightly offset (about 30 degrees) from the front—so you can see depth in the character model.

Generating Pre-Production Meshes with Stable Fast 3D

Once Sora was producing reasonably consistent assets, I wanted to convert them into 3D meshes I could use in Blender.

Enter Stable Fast 3D from Stability AI. You can use their demo app on Hugging Face if you don’t want to run anything locally, which is a pretty reasonable way to get pre-production meshes into your pipeline. But it runs pretty quick if you take the time to set it up—even on a Mac. I have an M3 Max with 36GB of RAM, and generating a mesh with 8,000–12,000 polys takes about 20 to 30 seconds.

~/code/innkeeper/stable-fast-3d git:(main)

PYTORCH_ENABLE_MPS_FALLBACK=1 uv run python run.py ./innkeeper_three_quarter.png --output-dir output/tris --remesh_option triangle --texture-resolution 2048
Device used:  mps
After Remesh 7309 14614
Peak Memory: 11525.65625 MB
100%|█████████████████████████████████████| 1/1 [00:29<00:00, 29.03s/it]

I followed the instructions on the Stable Fast 3D Repo to enable MPS support on my Macbook, and installed pre-requisites using uv. (You may need to install torch like this: uv pip install torch --no-build-isolation).

The Final Context

After many iterations, I had evolved my prompt to produce an asset with:

• Consistent character dimensions
• A proper A-pose
• Sequential camera cuts showing front, right-profile, back, and 3/4 views
• Proper framing (no clipping of hands, feet, or elbows)

Character sheet, 10s, 60fps, single take, sequential camera cuts; no motion blur, no DOF. Dwarven innkeeper, painted-toon hybrid (hand-painted albedo, thin 1–2px outlines, two-step ramps, subtle stylized-PBR). A-pose: arms ~30° from horizontal, hands open, fingers separated; legs shoulder-width, feet flat; head level, neutral face; beard with visible air-gap from chest. Camera & framing: orthographic, full-body, subject centered; lock zoom/orthographic size; character occupies ~70–80% of frame height; minimum 10% margin to all edges; no cropping of hands, feet, or elbows at any time; ground plane visible. Aspect 16:9. Background: neutral mid-gray seamless. Lighting constant (even studio three-point: key 5000K, fill −1 EV, soft rim). Timing: 0.0–2.5s = FRONT; 2.5–5.0s = RIGHT-PROFILE (90°); 5.0–7.5s = BACK (180°); 7.5–10.0s = 3⁄4 front-right (≈45°); loop resets to FRONT at 10s. Negative: no props/cape/apron/toolkit; no idle sway; no background objects; no text/watermarks;

The output format has been remarkably consistent, even if the aesthetics tend to drift between generations. For prototyping, that’s actually fine—I care more about format consistency than aesthetic consistency at this stage. What matters is being able to reliably screenshot these outputs, feed them into Stable Fast 3D, and get usable .glb meshes for rapid prototyping. The art style exploration is part of the process; having predictable poses and framing is what makes the workflow viable.

Automating the pipeline

I’ve always gravitated toward building internal tools in my software development career—there’s something deeply satisfying about being a toolmaker. So I asked myself: could I build a small web app that would:

1. Take an image via command line
2. Automatically generate a mesh
3. Preview multiple quality variants in a web browser

Turns out, you can. I mean, of course you can; we’re living in a world where tools you can imagine being useful are just a prompt away from existing, if you’re a sufficiently skilled context-smith.

I had GPT-5 and Codex whip up a python script that generates a webpage using Google Model Viewer to preview the .glb meshes that Stable Fast 3D generates (along with basic texture and lighting info). This script lets me batch-create models at varying quality levels and compare them instantly in a browser.

What I’ve learned

Here are some key takeaways from this experiment I wanted to leave you with:

1. Specificity is everything. The more detailed your prompt—lighting temperature, material roughness, camera behavior, pose angles—the more consistent your outputs.

2. Embrace happy accidents. The character sheet plate was a mistake, but it became one of the most useful outputs and guided me further down the rabbit hole.

3. Use AI as an educator. Shifting to a “learner” role with GPT-5 and providing context about my background made the guidance far more relevant and actionable.

4. Prototype fast, iterate faster. Even with limitations (10-second clips, inconsistent rotations), you can generate enough useful assets in a few hours to kickstart pre-production.

5. Build tools to reduce friction. Automating the image-to-mesh-to-preview pipeline makes experimentation far more enjoyable, and it took like 15 minutes of back and forth with GPT-5.

The barrier to entry keeps dropping, and I’m curious how far this type of workflow can scale.

What’s Next?

I still want to explore:

• Auto-rigging pipelines or Blender add-ons
• Using these meshes in Godot or Babylon.js for actual gameplay prototyping
• Experimenting with environment and prop generation

If you’re interested in trying this workflow yourself, everything lives in davemo/gamedev-preproduction-pipeline. Feel free to reach out if you want to chat about gamedev pipelines—I’m always happy to trade notes.

I’ve been having an absolute blast with the recently released Sora 2, pushing its limits, testing edge cases, and generally seeing what kind of creative chaos I can conjure.

Yesterday, I had a fun idea to create a spoof video of @searls and myself presenting at a fictitious conference. You know, the kind of thing that would get a chuckle at the next tech meetup.

The Future of “JS”

Original Prompt: dmosher and searls are panelists at a tech conference on the future of JS. Searls thinks the conference is about him because his initials are JS but it’s actually about the future of JavaScript.

It turned out better than I expected! I just wish they would have let the sentence continue so Justin could get to the punch line about getting to his autobiography, alas… perhaps in the next remix. 😂

A Quick Primer on Sora’s Onboarding

For those who haven’t been through it yet, when you onboard to Sora, it has you perform a rather sci-fi ritual. You read a sequence of numbers out loud while it captures an audio fingerprint—essentially training the model to understand the unique characteristics of your voice so it can synthesize it in generated videos. It does the same dance with your face, capturing a “likeness fingerprint.”

So, What Happened?

I was attempting to remix my own video and hit an odd error after a few attempts. The resulting draft played normally at first—same conference setting, same general vibe. But then Justin’s voice came through reading numbers. Those numbers. The exact sequence from what I think was likely his onboarding session.

Sora had somehow embedded what I think was Justin’s audio fingerprint into my remix. I immediately tried to export the video for documentation, but the publish button was grayed out, share options threw errors, and the download link was mysteriously absent. With the evidence literally vanishing before my eyes, I did the only thing I could—fired up PhotoBooth.app to capture this janky screen recording:

Shortly after I captured this, the draft mysteriously vanished from my dashboard entirely.

Remix Prompt: “Later in the panel discussion the two are talking about famed techno-prophet Gary Bernhardt’s prediction on the birth and death of JS (JavaScript) and @searls again has to be reminded the topic is not about him being born or dying”

Speculation

I’m only speculating here, but after spending some time thinking about the architecture, here’s my best guess at what went wrong:

Sora appears to be storing user ‘fingerprints’ (the audio and visual data from onboarding) in some kind of shared embedding space. When you remix videos multiple times in quick succession, especially when referencing other users, there seems to be a race condition or caching issue that can cause these embeddings to bleed into the generated content.

The fact that Sora immediately prevented any form of export suggests there’s some level of awareness built into the system about what constitutes protected data. But it seems like the safeguards kicked in after generation, not before—classic case of closing the barn door after the horse has bolted.

I’ll keep experimenting with Sora 2, because honestly, it’s incredible. Though next time I try to spoof a conference talk, I’ll make sure the only thing that gets leaked is Justin’s actual autobiography. Found any interesting edge cases yourself?

I recently finished reading The Courage to Be Disliked and thought it would be a good time to reflect on one of the more painful episodes in my career through the lens of resilience, trust, and the idea of separation of tasks that the book discusses so effectively.

The Website Redesign That Didn’t Ship

One of my last projects at Pulley was a full website redesign prototype. We had been struggling with the previous design and page sprawl for a while, and I was interested in taking a crack at a fresh vision in collaboration with our design team. Within about a week we had shipped a working prototype that included: a cleaner hero section, scroll-to-reveal animated content, an FAQ and comparison table, a clean pricing page, and a few templates and components intended to make shipping pages faster and more reliable.

We were experiencing both design and information architecture drift for a while, and it felt like a lot of that was being caused by our reliance on the CMS we were using at the time, Webflow. I could probably write a whole other post about all of the downstream impacts of choosing a WYSIWYG CMS for your marketing website, but this is not that blog post. Suffice it to say, we were dealing with a lack of cohesion in content building blocks, which resulted in many of the pages that were being created hitting that uncanny-valley effect that no founder wants to see on their web presence: off-by-one pixel counts on border radii, slightly varied colors that deviated from the design palette, or the dreaded font styling someone added to place emphasis at just the right place for their one-off marketing page.

It was a fun project—until it wasn’t. The closer we got to make it real, the fuzzier the ownership became. It was especially fun to work with my preferred publishing stack—Markdown and Astro—but a challenge emerged in my efforts to move the ball forward from prototype to production: it wasn’t clear who would decide or own the outcome.

The Dreaded DRI

A week after shipping the prototype we had a company offsite, during which there was a renewed charter and some tweaks to our company values. One of my favorite things about the culture at Pulley was the company values around “default public”, “ship it and iterate”, and “disagree and commit”. At the offsite, a new value landed for us: “stick your neck out.” Have a good idea? Move it forward. We needed people who would make bold bets and take risks with conviction. Heck yes, that was exactly what I needed to hear and I leaned into it pretty hard, hoping to be able to drive the website project forward.

I ended up making a pitch to leadership to be the DRI (directly responsible individual), and had received a yes, if somewhat reluctantly. This should have been my first clue that alignment on “stick your neck out” hadn’t translated into alignment on who owned the outcome.

Undeterred (or perhaps more accurately, painfully unaware) I kept moving things forward as best I could, looping in folks from the marketing department as well as our COO at the time to make my strongest pitch for moving off Webflow and into a Markdown-based publishing pipeline. This appeared to be well received, and it felt like I was being appropriately rewarded for having stuck my neck out.

My Misdiagnosis

A few weeks later, a contractor joined the project; someone our COO had worked with before, and was positioned as someone who would help me get the project over the finish line. At first I was glad to have the help, someone else to brainstorm with, and someone who was in favour of the approach I had selected, with a slight twist; they suggested we use a headless CMS. It started out fine, we collaborated to migrate my prototype over to use the headless CMS and Astro, and then meetings started to have more people involved. The COO began coordinating things and driving, deferring many of the decisions to the contractor. None of this happened with clear communication, no one said who was driving, and ownership started blurring right at the moment when I felt like we had forward momentum.

This was confusing to me, and I started to ask questions about things in our public channels, hoping to gain clarity about who owned the effort, leaning into our mostly healthy operating mode of “default public” for comms.

I ended up in a 1:1 with both the CEO and COO (or is that a 2:1?) where some honest feedback came for the first time: “Dave, you’re brilliant, but you’re not resilient, and what we really need are people who are resilient”. I tried to dig into that feedback to better understand it, but I was also struggling with having felt somewhat blindsided by the rapid shift in gears in a process and initiative I was helping drive forward to something being delegated to a contractor.

I ended up shutting down hard after that—pretty much fulfilling the prophecy from the 1:1 feedback. A month later, I ended up leaving the company.

In retrospect, the feedback wasn’t really the problem; it was that I couldn’t tell if we were in Disagree & Commit (a plan exists, follow it) or Ship & Iterate (we don’t have a plan, keep narrowing and we’ll decide later), and I didn’t actually ask—I just shut down. Leadership had quietly shifted us into Disagree & Commit on a contractor-led plan, and I was still operating as if we were in Ship & Iterate.

Not My Plan, Still My Job

One of the most useful lessons from Courage to Be Disliked is that of the separation of tasks. If you haven’t read it, the book is structured as a dialogue between a youth and a philosopher over five nights, exploring many of the themes in Adlerian Psychology—an excerpt on the topic of separating tasks from page 122:

PHILOSOPHER: One does not intrude on other people’s tasks.

The conversation in the chapter continues with much more specific examples, but this snippet is enough to illustrate what I think I was missing: a concrete understanding of what my task was in the website redesign project, and perhaps what it began as and how it changed as the situation evolved.

This has led me to a much more useful habit during introspection or when I sense myself getting frustrated: clearly separating my task from not-my-task. Reflecting on the website project:

• My task: propose, prototype, learn, align, and be useful within whatever route is chosen.
• Not my task: control final adoption; dictate the plan if a new DRI has been designated; personally resolve every stakeholder tension.

Had I been able to make that separation explicit, I think I would have done two things differently: (1) commit to the contractor-led plan and contribute high-leverage work inside of it, or (2) request a return to iteration mode with a timeboxed v2.0 and explicit decision checkpoints. Instead, I kind of just stayed stuck between modes and felt trapped by constraints.

Trust & Operating Modes

“Disagree & Commit” works when the goal is shared but the route is contested. It requires and rests on trust: I trust the decider’s taste and judgment enough to set aside my preferred route and still give my best efforts. “Ship & Iterate” is different: it assumes we’re still shaping the route, so the job is to show work, gather feedback, and narrow quickly.

In my case, I had trust in the person: I trusted my CEO’s judgment and sense of what good looked like. But I didn’t have trust in the system: the unclear DRI, the surprising plan change, and the hazy timebox made it challenging to know how to be useful.

This distinction matters. You can be resilient when the plan is clear but hard—when you know what operating mode you’re in and what’s expected. Unclear expectations make staying resilient far harder. Without clarity on operating mode and ownership, even well-intentioned people spin their wheels or shut down.

These are solvable alignment problems—if you name what mode you are operating in and ask for clarity:

Before we continue, can we confirm our operating mode? Are we in Commit to Contractor’s Plan or Iterate Toward a Decision? If it’s the former, I’d like to propose 2-3 contributions inside that plan by Friday. If it’s the latter, I’ll work on a few more iterations focusing on hero/information-architecture/templates—also by Friday.

What I like about this is it demonstrates curiosity, seeking to understand, and shows alignment and a bias for action regardless of what operating mode we’re in. Had I done this, I feel like I would have kept myself in the arena a lot longer and been much more resilient to changing dynamics; resisting the temptation to become frustrated over things that simply weren’t my task.

Maybe you find yourself in a similar position; your organization might be adopting another plan than the one you had in mind. If that’s true, then you have a choice to make about how you show up and engage, and if you want to be seen as someone who is resilient, then you need to have the right mindset going into that potential minefield of career-limiting moves.

Resilience is being useful within constraints and seeking the clarity you need to do so.

My Resilience Playbook

In the time between the website project and now I’ve worked with six other clients and made a return to working at Test Double as a full time consultant, and I’ve picked up a few tricks along the way that have helped me reframe my typical defensiveness when receiving feedback like I got from my CEO into a positive set of steps that help to keep me in the arena:

1. Pause Feeling defensive? Name it, out loud or in your head, or to the other person if trust is high. Give it 30 minutes or so.
2. Separate Tasks. What is mine to own right now? What is not?
3. Identify the Operating Mode. Are we in “Disagree & Commit” or “Ship & Iterate”, or maybe even “Stick Your Neck Out”? Ask explicitly.
4. Contribute, don’t Control. “What is the most useful thing I can contribute within these constraints?”
5. Close the Loop. Propose a timeboxed next step; confirm the owner and timeline.

If it helps, here’s a phrase you might use in such a scenario:

I’m hearing X and Y aren’t landing. I’ll regroup and come back with options by Friday EOD. Can you confirm whether we’re committing to Plan A or still in iterate-toward-spec mode? That changes how I’ll structure my next pass.

Staying in the Room

Reflecting on this honestly has helped me realize that the feedback that led to me quitting has been a pretty effective catalyst for growth; it has helped me distinguish trust in the plan from iteration stamina, separate my tasks from others’, and trade defensiveness for a contribution-within-constraints mindset when I’m not the decision maker.

As I’ve been considering what is necessary to move further into senior levels of leadership in my career, it seems to me that being the smartest person in the room is much less important than simply just being in the room, especially when the plan isn’t yours.

At Staff+, brilliance might open doors, but it’s resilience—staying useful and engaged even when the plan isn’t yours—that keeps you in the room and leads to more opportunities to learn and grow. The next time ownership blurs or plans shift, my hope is that you (and I) will know what to ask and how to show up.

I’ve been reading The Hard Thing About Hard Things by Ben Horowitz; it’s excellent and you should totally add it to your reading list if you haven’t already. Somewhere about the halfway point, there’s a chapter on the topic of “Management Debt” with some ideas I hadn’t considered before:

Like technical debt, management debt is incurred when you make an expedient, short-term management decision with an expensive, long-term consequence. Like technical debt, the trade-off sometimes makes sense, but often does not. More important, if you incur the management debt without accounting for it, then you will eventually go management bankrupt.

As a software developer, I’ve become all too familiar with the concept of technical debt, but this idea of management debt was new to me. Ben is writing from the perspective of the startup founder or business executive, but as I was reading I was struck by the idea that it would be useful to reflect on this cost from the perspective of the employee.

How Expensive Are You?

Say you’re about to join a new company, you’ve gone through the grueling interviews and conversations and finally negotiated an offer that you feel is fair, and so you accept. Congrats! Now you can kick back, relax, and wait until your starting date comes around, you’ve made it!

Well, that’s something you could do, and definitely something I’ve done in the past, but an older and wiser version of myself now sees this as somewhat of a wasted opportunity to consider the cost. No, I’m not talking about the cost of relaxing (which, to be clear, I have nothing against), but more specifically how much you are going to cost your new employer on the day you start.

Oh, you haven’t thought about this before? I hadn’t either, but you can be damn sure any CEO or startup founder worth her salt has likely read Ben’s book and has already thought about the cost and potential management debt that will be incurred by taking you on as an employee.

Now, the fact that they hired you is a good indicator that they think this is a net positive investment over the long term, so you’ve got that going for you, but what you might not realize is that most employees begin their employment with a slight deficit in their trust account.

Trust Capital

Think of your working relationship like a bank account. There are deposits, withdrawals, and someone somewhere is keeping a ledger of all the transactions (often mentally, sometimes literally). In the early days of your employment you begin with a slight deficit in this bank account, even though the company who hired you obviously values you – they wouldn’t have hired you if they didn’t!

You need to be onboarded, trained, potentially even certified all of which front loads your management cost and requires a non-trivial investment on the part of the business. There exists some amount of uncertainty about whether this investment will pay off, and the founder hopes that her vetting process and reference checks will give a good indication of the quality of the investment. However, there is always some amount of risk that this won’t pay off.

Of the many things that keep founders up at night, I suspect this is up there in the top 3.

The currency of this bank account is trust, and the fact that you begin your employment with that slight deficit means you should be highly aware of the need to move your balance sheet out of the red and into the black.

Build Your Own Onboarding Plan

One of the best ways to reduce your initial deficit is to come up with an onboarding plan. Yes, typically this is undertaken by your manager, but by thinking proactively about your version of that plan, you will set yourself up for success.

Consider what was communicated to you during interviews, what things are most important to the company? Where is the company heading? What would be most valuable? Then, come up with a 90 day plan and partition it into 30 day segments, writing down what you hope to achieve in each of those segments and questions to clarify with your manager. Here’s a few examples of questions that can help shape your plan:

• What milestones do you want to hit?
• What milestones does your manager want you to hit?
• How soon do you want to be ramped up on the business domain?
• When should you be able to push code to production?
• When should you meet with the other people on your team?
• What gaps exist in your knowledge of the tech?

The more detail and specificity you can add the better, just remember you need to be flexible as your plan might be less realistic than what your manager has in mind.

Sharing Your Plan

Once you’ve created a plan, you should figure out how you’re going to share it with your manager. You could do it proactively, before your start date, or you might want to wait until your first 1:1 session. Context matters, so use your best judgment.

Be careful not to be too prescriptive with this plan; present it humbly and ask to see if it resonates with what your manager has outlined. There will be things that are great and things that might be lower in priority; be willing to adapt to let your manager lead as you explore your plan together (they likely have more context than you do about what’s best for the company at this point).

Lastly, be positive, this is a great opportunity to collaborate!

Onboarding Tactics

Once you’ve begun onboarding, you should start thinking about what to do next. I’ve done this many times over my consulting career, onboarding with clients, and have a few lessons to share. Here are some do’s and dont’s:

• You should ask lots of questions and be curious, doing so demonstrates willingness to learn. (Asking questions as someone new is a superpower; you can see things others might not, and the act of asking the question can uncover improvements).

• You should take detailed notes during your onboarding experience and reiterate your understanding to multiple people along the way. This shows that you are engaged and actively looking to make sure you are in alignment with what the rest of the organization understands.

• You should not prescribe sweeping changes right away (or at all), as doing this constitutes a large withdrawal from your trust account.

• You should look for opportunities to improve things and write them down, you can even share them with your manager. (Be cautious here and consider whether your suggestions are reasonable. You might want to capture the notes first and then wait until you have a better understanding of the business’ needs).

• You should start small and surgical and then, if appropriate, move on to large and technical. (You might have the expertise and skill to make sweeping changes right away, but it’s better to lead into this type of thing slowly with many smaller deposits of trust capital).

• You should not make too many withdrawals in a short period of time. There’s an ideal ebb and flow to deposit/withdrawal that is unique to your company and you have to discover what it is. (This will also change over time as the company moves through various stages of growth; be ready to adjust your mental model).

Thinking this way significantly reduces your management cost right out of the gate, and will help you accumulate some assets, which can give you a significant infusion of trust capital that can often bring you into the black very quickly.

Reflect, Re-evaluate, Reinvest, Rinse, Repeat

Moving forward from here establishes a solid foundation on which to build. At this point it’s a good idea to keep your own ledger (mentally, or literally) of transactions in your trust account. Having self awareness about when you make deposits and withdrawals is to your advantage, especially in an age where so many companies have potentially trust-draining benefits like unlimited vacation.

Next, reflect on how you are doing on a monthly or quarterly cadence. This will help you ensure you are making the best investments to earn the most trust and compound interest as you grow in your role at the new company. It will also help you be strategic about when the right time to make that big withdrawal is.

At a startup these conditions are going to change more frequently than if you are joining an established larger company – you should understand this going in so that you aren’t surprised when priorities change and you need to diversify your trust investment portfolio.

The bottom line is, think of yourself as a savvy investor, trying to forecast, plan, and react to the market conditions in your new company as you make strategic investments to build up your trust account. 🚀

There’s been a lot of focus on the topic of DX or developer experience recently, which is great because it’s something that I’m passionate about improving on the teams I work with. So passionate, in fact, that I recently transitioned out of a customer-focused engineering role into one focused on developers, tooling, and infrastructure.

Now you might be asking yourself, ‘What does DX actually mean?’, and you would be justified in asking – I’ve heard varied interpretations of the term.

Jean Yang defines it this way in The Case for Developer Experience:

What I mean by developer experience is the sum total of how developers interface with their tools, end-to-end, day-in and day-out.

I think that a big part of the “how” there is the concept of feedback loops. Good developer tools provide affordances that reduce cognitive load, surface key information at the right time, but most importantly, good tools prioritize keeping feedback loops fast – and nothing depletes my cognitive batteries faster than slow, clunky tools.

Building Tools with Node.js

I first started working with Node around version 0.4 in 2011 and at the time I had absolutely no idea how anything worked. I came from a designer-first frontend background and my knowledge of JavaScript was limited to the context of the browser.

Jumping into Node was extremely disorienting for me, up was down and left was right, window was global and there was no XHR or DOM. Writing modules, packages, and command-line tools was all new to me, yet I was intrigued by the potential to leverage my knowledge of JavaScript within the world of tool building and systems programming.

Around this time I was working with Searls, who created Lineman after I shared frustrations with most of the frontend tooling I was using.

This experience was transformative for me because up until that point the tools I was using were clunky, written in other languages I didn’t know as well, and incredibly slow. Justin’s focus on the DX of Lineman was immediately apparent and the fact that it was written in JavaScript (ok, and a lot of CoffeeScript) made it easy for me to contribute to. It was empowering to have someone listen to my frustrations and translate them into magic on the command-line!

Since then, I’ve been able to work on some of my own tools, libraries, and plugins while picking up a few tips, tricks, and many opinions along the way. My hope is that sharing my experience will give you the starting place I wished for when learning to build tools in Node and lead you to a more enjoyable developer experience.

Ok, but how do I start?

Great question! It’s the same one I had when I was interested in writing command-line tools with Node. The rest of this post will be a guided tutorial that I think will help you get started and answer the following questions:

• How can I iterate on a node module locally without publishing to npm?
• How can I test what I’m building in another project?
• How can I debug the code while developing?
• How should I package things?

Prerequisites

You’ll need the following in order to follow along:

• an installation of node >= v16 and npm >= v8
• a terminal app like terminal.app or powershell
• your favorite code editor
• optional a checkout of the full example repo

If you prefer to follow along step-by-step, let’s get started by running the following commands to create a workspace folder for the tutorial.

I like to use ~/code/node, but you can use whatever you are most comfortable with.

$ # create tutorial directory
$ mkdir -p ~/code/node
$ cd ~/code/node

$ # create the my_project directory and initialize it with npm
$ mkdir my_project
$ cd my_project
$ npm init -y
$ cd ..

$ # create the my_tool directory and clone the workflow template
$ mkdir my_tool
$ cd my_tool
$ npx degit davemo/nodejs-tool-dev-template
$ npm i

Using the tree command, we can visualize the folder structure.

$ tree -L 2 ~/code/node

~/code/node
├── my_project
│   └── package.json
└── my_tool
    ├── dist
    ├── index.js          # executable entrypoint file
    ├── lib
    │   └── tool.js       # module that logs to STDOUT
    ├── node_modules
    ├── package-lock.json
    ├── package.json
    └── test

6 directories, 4 files

The my_tool directory comes with an entrypoint file index.js which will be our CLI tool. This entrypoint requires and executes lib/tools.js which logs a simple message to STDOUT and completes after a delay of 1 second.

// my_tool/index.js
#!/usr/bin/env node

require('./lib/tool.js')();

// my_tool/lib/tool.js
module.exports = function tool() {
  console.log('⏳ CLI tool: working ...')
  setTimeout(function() {
    console.log('✅ CLI tool: done!')
  }, 1000);
}

Iterating locally using npm install <dir>

Often you will want to test a node module in another project locally without having to publish it. There are a few ways to do this but we want to optimize for the best developer experience in our workflow, so I’m going to focus on using npm install <dir>.

Install my_tool from within my_project

$ cd my_project
$ npm install ../my_tool

added 1 package, and audited 3 packages in 634ms

Installing a package this way causes three side-effects that we should know about.

$ # a symbolic link for my_tool is created in node_modules
$ ls -la my_project/node_modules
my_tool -> ../../my_tool

$ # a symbolic link for the executable is created in node_modules/.bin
$ ls -la my_project/node_modules/.bin
my_tool -> ../my_tool/dist/index.js

$ # my_tool is added to dependencies mapped to a relative file path
$ cat package.json | grep -A2 dependencies
"dependencies": {
  "my_tool": "file:../my_tool"
}

The addition of the relative file: path in dependencies is nice because it serves as a physical reminder that we are in development mode; something we don’t get using npm link.

Now that we’ve got things installed, we’re ready to start iterating on my_tool and testing how it works inside of my_project.

Testing our tool in another project

To work with my_tool inside of my_project we’re going to use npm scripts inside my_project/package.json. Let’s add a script log_a_message that invokes my_tool.

// my_project/package.json
{
  "name": "my_project",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "log_a_message": "my_tool" // add this script
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

We can now execute npm run log_a_message to invoke my_tool.

$ npm run log_a_message

> my_project@1.0.0 log_a_message
> my_tool

⏳ CLI tool: working ...
✅ CLI tool: done!

What happens when we call npm run log_a_message?

To better understand what’s going on, let’s take a look at the name and bin keys within the scaffolded package.json that came with the workflow template.

// my_tool/package.json
{
  "name": "my_tool",

  // explicit mapping syntax
  "bin": {
    "my_tool": "dist/index.js" // bin.my_tool invokes dist/index.js
  },

  // implicit mapping syntax
  "bin": "dist/index.js" // package.name invokes dist/index.js
}

In our case, because the name of the package and the command are the same, we could simplify using the implicit syntax, but I generally prefer to use the explicit syntax just to make it clear to future readers.

The bin key in my_tool/package.json is a mapping of the command name (my_tool) to the local file name (dist/index.js) that will be executed.

With that in mind, this is roughly what happens when we run npm run log_a_message:

• npm looks for scripts.log_a_message in my_project/package.json
• npm appends my_project/node_modules/.bin to the shell’s pre-existing PATH
• npm resolves my_tool to my_project/node_modules/.bin/my_tool which is a symlink to my_tool/dist/index.js because of the bin mapping.
• my_tool/dist/index.js contains a unix shebang #!/usr/bin/env node
• the script is effectively executed with node my_tool/dist/index.js

Development workflow: (watch, build) + (debug)

With my_toolinstalled within my_project, the next thing we might want to do is debug our tooling code as we iterate while developing. There are a few included npm scripts in the workflow template that we’ll find within my_tool/package.json.

"scripts": {
  // watch: runs the build, compiling on every change
  "watch": "npm run build -- -w",

  // build: a one time compile using @vercel/ncc
  "build": "ncc build index.js -o dist --source-map",

  // debug: open ndb to debug our compiled tool
  "debug": "ndb dist/index.js"
}

Watch + Build with @vercel/ncc

The watch target here invokes build and passes along the -w parameter which tells ncc that it should watch for file changes and recompile every time a change is detected.

The build target invokes ncc, which traverses the dependency graph starting at our index.js entrypoint, and compiles everything it finds (including dynamically imported things) into a single file with all dependencies inlined, kind of like gcc.

To kick things off, run npm run watch from within my_tool.

$ cd my_tool
$ npm run watch

> my_tool@1.0.0 watch
> npm run build -- -w

> my_tool@1.0.0 build
> ncc build index.js -o dist --source-map "-w"

ncc: Version 0.31.1
ncc: Compiling file index.js into CJS
File change, rebuilding...
 2kB  dist/index.js
 2kB  dist/index.js.map
40kB  dist/sourcemap-register.js
42kB  [113ms] - ncc 0.31.1
Watching for changes...

I like starting with this “watch-build” workflow when developing for a few reasons:

• ✅ It compiles on every file change and compilation is effectively the first unit test; this gains us confidence that we aren’t introducing simple bugs as we develop.

• 📦 It sets us up to be ready to publish our module to npm as soon as we are done developing; the dist folder is the only thing we need to upload to npm.

• ⏭ ncc generates sourcemaps, so we get the same DX benefits of debugging against my_tool/index.js in concert with the compilation benefits above.

Debugging with ndb

The debug target spins up ndb pointing at our single compiled file in dist/index.js. The feedback loop when developing and debugging is nearly instant, and ndb includes some really nice DX features which we’ll look at shortly.

First, let’s add a debugger statement within my_tool/lib/tool.js knowing that it will be compiled into dist/index.js automatically by our Watch + Build process.

// my_tool/lib/tool.js
module.exports = function tool() {
  console.log('⏳ CLI tool: working ...')
  debugger; // add a debugger statement here
  console.log('another one');
  setTimeout(function() {
    console.log('✅ CLI tool: done!')
  }, 1000);
}

Then, in another terminal window let’s spin up ndb using npm run debug.

$ npm run debug

> my_tool@1.0.0 debug
> ndb dist/index.js

Downloading Chromium r624492...
Chromium downloaded to /Users/davidmosher/code/node/my_tool/node_modules/carlo/lib/.local-data/mac-624492
⏳ CLI tool: working ...

The first time you run the debug target ndb will download Chromium.

With this process launched you should see a Chromium window pop up.

This is where we can see some of the ndb DX specifics I mentioned earlier that make this workflow so nice:

• 📃 Notice the NPM Scripts tab? This allows you to repeatedly invoke any npm script right from the GUI. This is really handy for making changes and then testing them immediately.

• 🔁 ndb stays launched and available for you to re-run any of those npm scripts; this is much nicer than node --inspect-brk which exits after the current debug stack is completed.

• ⏭ We can see those sourcemaps I mentioned before at work here, lib/tool.js shows up in the source pane even though we’re debugging dist/index.js.

• 🐛 Finally, all of your other Chrome DevTools muscle memory applies just the same here as it does when debugging client-side JavaScript! The step-debugger, sources panel, and snippets can all come in handy working here in a node-based JS context.

Packaging with @vercel/ncc and npm

When you’ve finished creating your command-line masterpiece the next thing you may want to do is publish it to the npm registry so that others can npm install it. Thankfully, this step is easy because we already configured ncc to build for production in the dist folder via our build npm script.

The last thing to consider is how to minimize the size of our package when a user installs it with npm install, and this has already been done for us in the files key within my_tool/package.json key.

{
  "name": "my_tool",
  "version": "1.0.0",
  "description": "a template for developing node.js command-line tools",
  "main": "index.js",
  "bin": {
    "my_tool": "dist/index.js"
  },
  "files": [ // manages which files are published to npm
    "dist"
  ],
  // ...
}

NPM has some documentation on files that you may find helpful.

Limiting files to just the dist directory ensures that we keep our package as small as possible, which is already taken care of thanks to ncc producing a single file. If you want to double check to see exactly which files npm would add to the package, you can run npx npm-packlist.

$ npx npm-packlist

index.js
package.json
dist/index.js
dist/index.js.map
dist/sourcemap-register.js

Our last step before publishing will be to remove the private: true flag from package.json, which I added so nobody accidentally published the workflow template. Once that’s done, then you can simply npm publish and your minimal package will be uploaded to the npm registry.

Wrapping up

This concludes my tutorial on an enjoyable workflow for Node.js tool development. I hope you found this useful and that you find the experience of developing tools in this way enjoyable and the feedback loop fast and efficient.

If you have questions or wish to provide feedback, please reach out to me on twitter. I would love to hear from you!

Happy tool-making. 💚

Music composition and production is a large part of my life outside of software development, so much so that I often find myself thinking of ways to draw parallels between the two. One such parallel that has stuck with me over the past 6 months or so is the concept of the missing fundamental.

A harmonic sound is said to have a missing fundamental, suppressed fundamental, or phantom fundamental when its overtones suggest a fundamental frequency but the sound lacks a component at the fundamental frequency itself. This very concept of ‘missing fundamental’ being reproduced based on the overtones in the tone has been used to create the illusion of bass in sound systems that are not capable of such bass.

When I first learned about this concept, I couldn’t help but think of how it applied to the work we do as software engineers. In the same way that skilled audio engineers can leverage the concept of the missing fundamental to improve the characteristics of sound, skilled software engineers can use a similar set of skills to improve the performance of applications.

The Fundamentals of Software Performance

Ali and I were recently tasked with improving the performance of a legacy codebase that was deployed on Heroku using Node.js, MongoDB, and Angular 1. One of our first steps in evaluating the performance of code is to do an audit of dependencies and configuration; this is a great starting point because it can often lead to simple fixes for performance issues, like updating a database ORM adapter that can query things more efficiently.

In this case an audit of dependencies didn’t yield much and we had to dive further into configuration on Heroku to really make sense of the performance challenge. Significant discovery number one was that the application was configured using Performance L dynos on Heroku (the most expensive and highest tier available). This seemed strange since it did not appear commensurate with the surface area of the application; its purpose was to sync data using Heroku scheduler to pull from SalesForce into MongoDB.

One of our first steps was to reduce the dyno size and monitor logs to see if the Performance L size was warranted.

Fundamental #1: Investigate

Heroku makes it pretty easy to peek at logs for your app, which is what we started with: heroku logs --tail -a td-client-slow-app. This yielded the following trace:

app[web.1]:
app[web.1]: <--- Last few GCs --->
app[web.1]:
app[web.1]: <--- JS stacktrace --->
app[web.1]:
app[web.1]: ==== JS stack trace ==========
app[web.1]:
app[web.1]: 0: ExitFrame [pc: 0x1374fd9]
app[web.1]: Security context: 0x01f9540008a1 <JSObject>
app[web.1]: 1: getOwnPropertyNames [0x1f954001251](truncated...)
app[web.1]: 2: getOwnPropertyDescriptors [0x3dffde9f7229]
[/app/node_modules/mongoose/lib/helpers/document/compile.js:159]
app[web.1]:
app[web.1]: FATAL ERROR: Ineffective mark-compacts
app[web.1]: near heap limit Allocation failed -
app[web.1]: JavaScript heap out of memory

The slug for this application was ~74mb, which didn’t seem overly large to warrant running out of memory on the lowest tier dyno Heroku provides. That dyno allocates up to 512mb of RAM, so we dug into the code path that led to the above stacktrace to gain some more information.

Fundamental #2: Profile

Node.js has some pretty decent profiling tools for engineers who want to dive into performance profiling. TD-resident DevOps pro Micah showed me that you can add some flags to the node process on startup to influence how V8 manages garbage collection. This is useful if you aren’t getting consistency in your crashes and want to place constraints on the application runtime in order to suss out the source of the memory leak.

node --optimize_for_size --max_old_space_size=460 app.js

Artifically lowering the max heap size below what Heroku provisions for Standard dynos yielded the source of the leak was a method called getUsers which was responsible for querying a list of users and their permissions from MongoDB. Here’s a sample of what that code looked like as we first found it:

getUsers: function(req, res) {
  User.find({}, function(err, users) {
    var allUsers = users;
    var adminUsers = [];
    var corpUsers = [];
    var techUsers = [];
    var formalUsers = [];
    var searchUsers = [];
    users.forEach(function(user) {
      if(user.permissions.admin &&
         (user.permissions.admin.corpUsers ||
          user.permissions.admin.techUsers ||
          user.permissions.admin.formalUsers ||
          user.permissions.admin.searchUsers ||
          user.permissions.admin.superAdmin)) {
        adminUsers.push(user);
      }
      if(user.permissions.general.corpUsers) {
        corpUsers.push(user);
      }
      if(user.permissions.general.formalUsers) {
        formalUsers.push(user);
      }
      if(user.permissions.general.search) {
        searchUsers.push(user);
      }
      // this continued on for another 40 lines or so...
    })
  })
}

Node.js has performance tooling built-in that allows you to gain insight around memory and CPU usage, which is what we used next:

const { PerformanceObserver, performance } = require('perf_hooks')
const o = new PerformanceObserver((items) => {
  console.log(items.getEntries()[0]);
  performance.clearMarks();
})
o.observe({ entryTypes: ['measure']})

// to get the approx mem usage you can add this log line:
console.log(`
  The script uses ~
  ${process.memoryUsage().heapUsed / 1024 / 1024}
  MB
`)

# Output
PerformanceEntry {
  name: 'getUsers to res.status(200)',
  entryType: 'measure',
  startTime: 5486.524808,
  duration: 6559.275542
}
The script uses ~ 456.143798828125 MB

Looking at this code I couldn’t help but wonder if there was a more efficient way to query and aggregate this information.

Fundamental #3: Identify the Missing Fundamental

Returning to the idea of the missing fundamental, as audio engineers must ask themselves “what can I change about the frequencies in this mix in order to bring things into harmony?” the relevant question for software engineers is very similar: “what does this system need in order to bring harmony to its operation?”. In our case it was also helpful to consider that question in a historical context as “what fundamental were the original developers missing when they built this?”

In both cases, the answer for this application was how to query things more efficiently!

The getUsers method above was doing two things wrong; querying inefficiently for all the users in the system and then allocating large arrays to partition the data based on permissions. Once we understood the missing fundamental we had a path forward to try and optimize this poorly performing code: we should see if we can query things more efficiently. This is what we came up with using async/await and the MongoDB aggregation pipeline:

getUsers: async function(req, res) {
  const adminUsers = await User.aggregate([
    {
      $match: {
        $or: [
          { "permissions.admin.corpUsers" : { $eq: true }},
          { "permissions.admin.techUsers" : { $eq: true }},
          { "permissions.admin.formalUsers" : { $eq: true }},
          { "permissions.admin.searchUser" : { $eq: true }},
          { "permissions.admin.superAdmin" : { $eq: true }},
        ]
      }
    }
  ])

  const corpUsers = await User.aggregate([
    {
      $match: {
        "permissions.general.corpUsers" : { $eq: true }
      }
    }
  ])

  // ... etc...

  res.status(200).json({
    data: {
      adminUsers,
      corpUsers,
      ...
    }
  })
}

Once we had re-written and tested the query to make sure the output was the same, we re-ran our performance profiling to see what the difference was.

PerformanceEntry {
  name: 'getUsers to res.status(200)',
  entryType: 'measure',
  startTime: 496079.094306,
  duration: 270.1256
}
The script uses ~ 44.550048828125 MB

Using the Aggregation pipeline had yielded an order of magnitude less memory and CPU usage! Here’s a couple of screenshots from the Heroku dashboard for this app that show the before/after comparisons as well.

Before - request timeouts and high memory usage.

After - no timeouts, memory usage reduced by a factor of 10.

Conclusions

If you find yourself in a similar situation evaluating the performance of some legacy code I would encourage you to think about asking questions around what the missing fundamental(s) are. Walking through investigate, profile, identify (rinse. repeat.) has been useful for me, and I hope it is for you!

Thinking of web applications in terms of state machines is not a new idea; in fact, it has become so popular in the past few years that teams are spending increasingly more time breaking down their application into states managed by front-end frameworks.

Whether you use Redux, MobX, or even perhaps something framework-agnostic like xState, it is clear that thinking about web applications in terms of state machines is occurring much more frequently. With all this focus on state, transitions, and the benefits that come with structuring our applications like this, I’ve found there is still an area that is often overlooked when it comes to managing state in web applications: the visual or presentation layer.

CSS is incredibly powerful yet frequently misunderstood by most developers, which often leads to derision of the language. I think this is mostly due to a fundamental error in the way web developers manage presentation, often focusing their efforts on conditional logic in templates instead of a more flexible application of state-specific CSS selectors to HTML elements.

A Simple Example

Let us examine a simple example of a multi-selectable list for a user interface (UI) that a designer may have provided in mockup form for us as web developers to decompose into working code.

We can see there are a number of interactions at play, and at first glance these might seem simple enough that we would be tempted to solve the problem without putting much upfront thought into it. However, I think despite the simplicity of the example, there are enough complex states to enumerate that we should spend some time thinking about them before we dive into creating this UI.

Putting aside interaction design (IxD) and accessibility (a11y) concerns for the time being, after enumerating the states that we see here there is a lot to consider when building this UI! How should we manage the states? Should the logic live in our template or in our stylesheets? Let’s take a brief look at the first approach, using an implementation in Svelte.

Svelte is a compiler that takes as input one or more .svelte files with regions of functionality based on JavaScript, HTML, and CSS; with that input it produces the minimal amount of DOM API output in JavaScript to achieve the desired result. It’s a different take than something like React, Angular, or Ember, which ship substantial runtimes to the browser that execute application code. If you are interested in learning more I highly recommend watching this excellent talk called Rethinking Reactivity from Rich Harris introducing some of the core ideas. The code in the following examples is intended to be simple enough that you should be able to port the ideas represented to any other framework with minimal effort.

Implementation: Templates

One of the first places a web developer might start is by crafting the template that represents the UI mockup we received from our designer friend above. This seems like a logical place to start, given we need some way to represent the data in a web browser. Let’s build a template using svelte-infused HTML and see how it looks.

<table>
  {#each users as user}
  <tr>
    <td class="icon">
      {#if user.selected && hasSelection}
        {checkedBox}
      {:else if !user.selected && hasSelection}
        {uncheckedBox}
      {:else}
        {snowman}
      {/if}
    </td>
    <td>
      {user.name}
    </td>
    <td>
      {user.email}
    </td>
  </tr>
  {/each}
</table>

Aside from the svelte-specific things like the {#each} and {#if} blocks, this is probably close to what you might implement in any front-end or server-side templating solution.

We’ve taken the list of potential states that we extracted from the mockup above and encoded them as conditional logic in our templates in order to achieve the desired result. The one special case we needed to account for was the non-interactive state “1 or more Selections Active”; to do this we defined a local variable in our JavaScript region called hasSelection which is defined using Sveltes reactive declarations as

$: hasSelection = users.some(u => u.selected)

Although the code above satisfies most of the user experience (UX) as detailed in the mockup, there are two problems that shake out of an implementation like this that focuses on conditional logic in templates:

1. We didn’t capture all of the states enumerated, as we cannot effectively translate a user’s hover action in templates alone unless we get really creative and complex
2. This paradigm scales very poorly as our templates grow, mixing concerns of presentation and data in a template, resulting in code that is much harder to read and maintain over the life of a project

The scalability concern is the more worrisome of the two, yet is a common byproduct of developers using conditional logic in templates. Increasingly thorny conditionals can lead to missed acceptance criteria, which in turn can lead to stress and tension on a team. Rather than throw blame around, it’s worth focusing on whether that approach is healthy for a long-term project.

I think we can do better if we shift our focus from conditional logic in templates to thinking more in terms of leveraging CSS as the language we use to define the states in our presentational state machine and using JavaScript to manage when to apply those states. Let’s see what that looks like as we refactor the above example.

Implementation: Stylesheets

One of the first considerations we’ll need to make is how to address both the concerns raised in the previous section. We need to handle the hover state properly, and we also should strive for a solution that encodes data in the template and presentation in the stylesheets. Let’s start by refactoring the template to eliminate the conditional logic:

<table class:hasSelection="{hasSelection}" class="selectable">
  {#each users as user}
  <tr class:selected="{user.selected}">
    <td class="icon"></td>
    <td>
      {user.name}
    </td>
    <td>
      {user.email}
    </td>
  </tr>
  {/each}
</table>

The first thing you might notice is that we removed the conditional blocks replaced them with svelte’s class element directive. This is an elegant way to control toggling of a CSS class on an element via a boolean value, which we previously defined as {user.selected} and {hasSelection}. We also added a class=selectable to the root table element in order to allow us to better manage the complexity of the conditional logic for states in CSS. Let’s defer looking at the JavaScript that defines those values and instead look at what the definition of each state in our presentational state machine looks like when we encode it with CSS:

/*
  CSS variables in conjunction with escaped unicode or html
  entities are a great way to represent things like icons
*/
:root {
  --unchecked-box: "\02610";
  --checked-box: "\02611";
  --snowman: "\02603";
}

/*
  Managing the hover states to show a yellow background
*/
tr:hover,
tr:hover td {
  cursor: pointer;
  background-color: yellow;
}

/*
  Our first state, every icon should default to the snowman
*/
.icon:after {
  content: var(--snowman);
}

/*
  A complex state, if the table has a selection,
  then every selected items icon should be the checked box
*/
.selectable.hasSelection .selected .icon:after {
  content: var(--checked-box);
}

/*
  A combined selector to handle the alternative complex states:
  - for a table without a selection, when the user hovers, show the unchecked box
  - for a table with selections, swap the icon from the snowman to the unchecked box
*/
.selectable:not(.hasSelection) tr:hover .icon:after,
.selectable.hasSelection .icon:after {
  content: var(--unchecked-box);
}

With the combination of CSS and svelte-infused HTML we’ve achieved the result our designer was hoping for when they handed us the initial mockup, with an appropriate separation between the definition of our states (CSS) and the application of those states (HTML, and JavaScript).

For completeness, here is the entirety of the example as included in Application.svelte from the code on github:

<script>
  let users = [
    {name: 'Danika Dywtgowm', email: 'danika.dywtgowm@email.com'},
    {name: 'Erica Bule', email: 'erica.bule@email.com'},
    {name: 'Jim Snales', email: 'jim.snales@email.com'},
    {name: 'Daria Thorobox', email: 'daria.thorobox@email.com'},
    {name: 'Mendikant Hargrove', email: 'mendikant.hargrove@email.com'},
    {name: 'Ephraim Lischok', email: 'ephraim.lischok@email.com'},
    {name: 'Lera Nedialkova', email: 'lera.nedialkova@email.com'},
  ]

  function selectUser(user) {
    users[users.findIndex(u => u.name === user.name)] = {
      ...user,
      selected: !user.selected
    }
    console.log(`${user.name} was ${user.selected ? 'de-selected' : 'selected'}`);
  }

  $: hasSelection = users.some(u => u.selected)
</script>

<style>
  :root {
    --unchecked-box: '\02610';
    --checked-box: '\02611';
    --snowman: '\02603';
  }

  td {
    padding: 5px;
  }

  tr:hover, tr:hover td {
    cursor: pointer;
    background-color: yellow;
  }

  .icon, .template-icon {
    display: flex;
    justify-content: center;
  }

  .icon:after {
    content: var(--snowman);
  }

  .selectable.hasSelection .selected .icon:after {
    content: var(--checked-box);
  }

  .selectable:not(.hasSelection) tr:hover .icon:after,
  .selectable.hasSelection .icon:after {
    content: var(--unchecked-box);
  }
</style>

<h1>Complex Multi-Select</h1>

<table cellspacing=0 class:hasSelection={hasSelection} class=selectable>
  {#each users as user}
  <tr class:selected={user.selected} on:click={() => selectUser(user)}>
    <td class=icon height=20 width=20></td>
    <td>
      {user.name}
    </td>
    <td>
      {user.email}
    </td>
  </tr>
  {/each}
</table>

Closing Thoughts

This is how I have tended to manage the working relationship between HTML and CSS for the last 20 years, and I think the power of thinking in this way leads to cleaner code and easier to refactor web interfaces.

If this looks completely foreign to you and you found yourself considering that the template-based conditional-logic approach made more sense, I’d recommend learning more about the capabilities of CSS features like pseudo-selectors :not, variables, and the generated content: property.

I’ve found that teams who up their level of knowledge in CSS and tend to try to split concerns like we’ve done here will have web applications that are easier to change over the long term.

If you are interested in learning more about this approach and seeing a live coded version of this blog post, please check out the screencast posted to my YouTube channel; it walks through all the examples and touches on a few more svelte-specific things to consider.

Learning Resources

• Svelte Tutorial
• Complex-Multi-Select Code on Github
• CSS Variables
• CSS :not pseudo-selector
• CSS Generated Content
• HTML Entity Symbols

Recently, Michael Schoonmaker, Joshua Starkey, and myself got together to brainstorm some improvements we wanted to make to an open source library called Dependable that we had used on a client project.

Dependable is billed as “A minimalist dependency injection framework for node.js”, but I feel like it only took on the “minimalist” moniker after we shipped version 2.0 just a few weeks ago. As we sat down to discuss what we wanted to do there were a number of questions that shook out that I feel need to be asked by any team working on an open source project:

• How can we make this smaller?
• What features are core and what can we prune?
• How can we write the test-suite in a way that demonstrates real-world examples?
• What should the README communicate?
• How are we going to maintain this going forward?

As we worked towards the backlog of features, we built up a list of nice-to-haves and must-haves, choosing to defer the former and focus our efforts on the latter. Github Projects actually worked pretty well for a lightweight project management tool.

Limiting API Surface Area

One of our stated goals was to reduce the surface area of dependable in order to eliminate complexity in the codebase. The 1.0 release of dependable included a public API with 6 methods on the dependency inversion container and our rewrite whittled this down to 4. Choosing to have this discussion early allowed us to focus on what the most valuable parts of the API were, using our experience of real-world use within consulting projects to help guide us.

With the API sufficiently whittled down, we also set ourselves to renaming the methods in the public API in order to better reveal the intent for each method and avoid violating SRP (which some of the 1.0 API methods had done via overloading). container.register(hash) became container.constant(name, object), and container.register was renamed simply to container.factory. We deliberated for a while over naming but felt that .factory and .constant were terms that were familiar enough in the context of dependency injection and more descriptive than their 1.0 counterparts.

Rewriting Source and Test

Dependable 1.0 was written in CoffeeScript which we felt would limit options for potential future contributors. We chose to rewrite the library in ES6 and use standard to manage formatting the code for us. It becomes much easier for future contributors to submit patches when the tooling in an open source project handles formatting of the code.

The test-suite took slightly longer to rewrite due to the removal of container.load(fileOrFolder). We felt that this method complected the 1.0 codebase and was syntactic sugar for what could be accomplished by a user via loading one or many files externally using require or import and then invoking container.factory(name, func) within the context of a call to .map. In addition to translating the .coffee test sources to .js, we also reorganized the test examples themselves to better communicate the intended use of dependable.

A consistent use of objects from a real world scenario involving a Logger, Router, and Formatter in the context of an App was used throughout the test-suite. I’ve found that I’m much more likely to contribute to open source projects that have a well architected test suite with meaningful examples.

it('should register a factory with a single dependency', function () {
  subject.factory('logger', function () {
    return 'message'
  })
  subject.factory('app', function (logger) {
    return logger
  })
  assert.equal(subject.get('app'), 'message')
})

In addition, we added first-class support for isolation testing via container.getSandboxed which should be used during testing to ensure that a module under test has been completely isolated.

Closing Thoughts & Recommended Reading

At Test Double we are proud of our commitment to open source and we take pride in trying to be thoughtful in the way we approach open source stewardship. If you share these values and are interested in joining us you should reach out, we’re hiring!

If you aren’t familiar with the concept or benefits of Dependency Injection these are some great followup resources to get you thinking:

• Inversion of Control Containers and the Dependency Injection pattern
• Inversion of Control, The UI Thread and Backbone.JS Views

I wish we could have built it right the first time but we were learning along the way.

Ugh, we got handed this mess and now we have to figure out what to do with it.

What do you think we should do?

Sound familiar? If any of these statements resonates with you then you’ve most likely been in the shoes of the person making the statement or as a consultant engaged with a client in a similar predicament. The phrase “Software Consulting” generally evokes feelings of praise or disdain, or perhaps both depending on the circumstances. As consultants, we are often positioned as experts and typically engaged during times of crisis or uncertainty; so it should come as no surprise that our ability to deliver in the midst of that environment is the primary measure of our effectiveness.

Expectations about what we are hired to deliver generally set the scale by which we are judged; frequently that involves analysis, reporting, recommendations but also training, mentoring, and code. The world is a big enough place that it’s plausible to think of a scenario where the opening statements above were made by someone dealing with the aftermath of a poor experience with a previous software consultancy, typically measured by the state or quality of “the code”.

While it would be interesting to attempt an objective look at what typifies “quality” as it relates to “code”, I think there is a more interesting vector to approach the quality valuation that also pertains to “code” of a different kind; call it the code of conduct, or the way we interact with our clients as consultants, or perhaps even our ability to relate and help them rise out of the midst of chaos and uncertainty—this type of consultant’s code is less frequently explored, which is a shame, because I firmly believe it has a more direct impact on the physical code we write than our technical expertise.

Hostile Takeover

Consultants in any industry are often vilified from the start of an engagement due to a perceived “us vs them” mentality. Depending on the how the engagement was initiated, trust was likely extended from key decision makers but generally does not extend from the rank and file employees of the company. This puts an immediate burden of responsibility on the consultant to become adept at earning that trust inclusively and at discerning barriers which would prevent that from happening.

The key to building this trust starts with establishing relationships with everyone on the team. It might be easy to think of ourselves as only accountable to the primary stakeholders, but unless we earn the trust of our peers and treat them as equals, it will be challenging to avoid the perception of a hostile takeover. In essence we should think of ourselves as partners, invested in the success of the company and each member of the team. Finding opportunities to mentor team members and attribute wins to others is a great way foster a positive consulting environment.

Cooperation & Commiseration

At some point any consulting engagement will involve hearing about the technical failures, company problems, personnel issues, and every other negative thing under the sun; people like to complain. Good consultants know how to view these conversations as opportunities to build trust without becoming embroiled in corporate politics. It is a balancing act to be able to participate in just enough commiseration to demonstrate understanding without fostering a negative cycle. I believe a critical component to developing this skill is the ability to hone one’s capacity for active listening.

There is a fine line between commiserating and ruminating, and good consultants know how to empathize and then steer the conversation towards solutions instead of ruminating on past failures.

Technical Overdrive

Depending on the nature of the team and the consulting engagement, it may prove more effective to lead by example and jump into what I like to call “technical overdrive”. I’ve seen many teams become derailed in endless technical round-table discussions and talk themselves into a corner before ever trying to code up a prototype or draft solution. In these scenarios I have found it best to move forward with quick prototypes of solutions proposed by the team, enabling them to have more concrete discussion points and base decisions on actual implementations. Working code and a good demo can unblock many teams.

Like every piece of advice, when to jump into technical overdrive is highly context-sensitive; sometimes it is more appropriate to employ the opposite strategy and slow things down. Some teams have a bad habit of shipping their prototypes to production, and it is in this scenario where good consultants will try and introduce better habits around technical design, architecture discussions and how to effectively prototype. In some ways the slow-things-down scenario also involves technical overdrive, as it might mean taking the lead on demonstrating what these good habits look like for the team.

Closing Thoughts & Recommended Reading

As a consultant, building trust should be our number one goal, but we have to remember that it takes time and is a skill that needs to be cultivated over the course of many engagements. If you are interested in spending time honing your skills and improving your “consultant’s code” I would recommend investing in the following resources:

• The Secrets of Consulting: A Guide to Giving and Getting Advice Successfully
• More Secrets of Consulting: The Consultant’s Tool Kit
• Crucial Conversations: Tools for Talking When Stakes Are High
• Influencer: The New Science of Leading Change
• How to Win Friends & Influence People
• Spin Selling

Also, if you’re looking for a great opportunity to develop your consulting skills, they are hiring at Test Double, you should apply. :)

React is frequently touted as being performant due to the optimizations of its Virtual DOM technique, yet all to often this is used by developers as a crutch to avoid thinking about the performance of their code at all. This generally leads to performance problems in React apps of any significant scale.

With the proliferation of React applications in the wild, I thought it would be a good idea to examine some techniques for evaluating the performance of React Components.

This screencast covers a number of techniques for constructing components, but also shows how to evaluate performance objectively and make informed refactoring decisions.

Screencast Outline

1. Introduction

• A Common UI Scenario for React Components -> Large Data Tables
• Developer Default: Google/Stack Overflow Driven Development
• The Quest for a pre-built Library

2. Performance Goals

• Important Metrics: TTI (time to interactive), TTFMP (time to first meaningful paint)
• Setting a Frame Budget: (60fps / 16.7ms) might not always be feasible
• The Feedback Loop: Experiment, Evaluate

3. React Development Patterns

• Start BIG: 1 component, 1 render method, then profile
• Decompose: limit the surface area of your component by thinking hard about props, then profile
• Optimize by:
  • Reducing JS Execution: work the browser doesn’t have to do doesn’t need to be optimized
  • Reducing Surface Area for Change: small components, limited surface area, small lists of props
  • Keep render methods simple and mostly static

Code

https://github.com/davemo/react-performance-analysis

It has been but a few short years since web developers chose a side and took up arms in the holy war of build automation tools; this is only one of many wars that have been fought countless times since the dawn of computing. In the grim darkness of the far future, there is only war.

Meh. War is tiresome and I have had enough of war. This post is about some small usability improvements you can add to your Makefiles if you are using Make. Let’s dig in!

The Makefile

Make has been around for a long time. It has some neat features but it’s not always the friendliest to neophytes. Imagine you have a new developer joining your team, and your project has a Makefile that looks something like this

VERSION ?= $(shell cat VERSION)

.PHONY: version clean bump release

build:
  @echo "building..."
  # build the app here

clean:
  # rm -rf build

release:
  # bump
  # make push -e VERSION=$(shell cat VERSION)

push:
  # push the build artifact at a given version somewhere

version:
  # cat VERSION

bump:
  # using semver, bump the version by a major, minor or patch increment

At first glance this Makefile isn’t all that complicated but chances are your build automation process is composed of many more lines of code or even split into multiple places. The wise aged veteran developer on your team tells the new member “to build this project just clone this repo and run make“ after which the new developer sees

neophyte@newbie:~/code/project
$ make
building...

Now, assuming new dev didn’t encounter any snags with project setup and installing dependencies (hah! unlikely) this still doesn’t present a great picture of how the project is assembled or the bits of the lifecycle that are involved at first glance. Let’s see if we can improve this initial Makefile developer experience.

Step 1: Add a DEFAULT_GOAL

In Make semantics, goals are targets that make should strive to update. The docs give us a nice explanation of goals as well as some hints about how we can manage which goal is run first

By default, the goal is the first target in the makefile (not counting targets that start with a period). Therefore, makefiles are usually written so that the first target is for compiling the entire program or programs they describe. If the first rule in the makefile has several targets, only the first target in the rule becomes the default goal, not the whole list. You can manage the selection of the default goal from within your makefile using the .DEFAULT_GOAL variable (see Other Special Variables).

Even though the docs give us some informal conventions about the first target in our Makefile I think it makes (heh) for a better experience if we add some sort of help target that spits out some information to the terminal. Make doesn’t have any facility to display help messages like some other build automation tools, but it won’t be too hard to add one. First, let’s add a default goal of help

.DEFAULT_GOAL := help

help:
  @echo "Welcome to the Project!"

With this in place our new developer sees the following

neophyte@newbie:~/code/project
$ make
Welcome to the Project!

Ok, this is a little more friendly but still not very useful. We can do better!

Step 2: Annotate Makefile Targets

Let’s update our Makefile to add helpful annotations to some of our targets using comment blocks prefixed with ##

build: ## builds the application
  @echo "building..."

clean: ## gets you back to a clean working state
  # rm -rf build

release: bump ## bump the VERSION file, git tags, and push to github
  # make push -e VERSION=$(shell cat VERSION)

This annotation scheme works pretty well but doesn’t buy us anything on its own, to make this truly useful we need to parse the Makefile, look for lines prefixed with ## and format them in a pretty way and write them to stdout

Step 3: Parse Annotations

Make includes a handy list of special variables that can be used for all sorts of handy things. In this case we can use the MAKEFILE_LIST variable along with grep, sort and awk to get a list of annotated targets and display them on stdout in a user-friendly way

help:
  @grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-30s\033[0m %s\n", $$1, $$2}'

Wondering what the help target is doing? See this explain shell.

With that in place our new developer would run make from the command-line and see

neophyte@newbie:~/code/project
$ make
build           builds the application
clean           gets you back to a clean working state
release         bump the VERSION file, git tags, and push to github

Yay! This is a much nicer developer-experience than what we started with. To take it even further I might suggest annotating only a subset of tasks that are most commonly used.

Acknowledgements / Links

There are a few sources that come up when you google for “self-documenting makefile” along with a few different ways of solving this problem. I drew inspiration for this post mostly from this marmelab entry, but felt like there were enough interesting points and general make tips added that it was worth another post.

Other links you may find useful:

• Full Makefile as Gist
• VERSION bump script as Gist
• GNU Make
• Self Documenting Makefiles
• Explain Shell

Two years in the making; just released and fresh off the presses it’s yet another screencast covering everybodys favorite enterprise JS framework: Angular JS!

In all seriousness, I had a hard time considering whether or not to publish this screencast because I found myself questioning whether the content would still be relevant almost two years later. However, in revisiting all the comments and questions about the alluded-to “part 2” from the first video I felt like there were still valuable things to talk about. At the heart of this screencast is discussion around what I consider one of the most valuable features of angular: the ability to use custom elements as a domain-specific language (DSL) to ease the wrapping and use of 3rd party libraries.

This screencast continues the examination of some of the advanced features in Angular from Advanced Directives with Angular JS and expands by tackling some of the issues raised in Part 1 including:

• bugfixes for the inline editor
• auto toggling of editing state using CSS content generation and the angular $scope
• leveraging the DSL from the first screencast as an interface to a 3rd party JavaScript data grid component: js-grid

If you’re interested in some more context prior to watching, check out my other angular screencasts and an earlier post on the power of web components as abstractions.

Hopefully the next screencast series won’t take 2 years to complete ;)

Code

• bugfix: don’t re-add and compile editors https://github.com/davemo/advanced-directives-with-angular-js/commit/4efc9edfacc3cee791f155d52bf517a7ab251586
• feature: swap arrow on editor state https://github.com/davemo/advanced-directives-with-angular-js/commit/2f046f51dda4b54891353b7ec047b3a6e381792d
• feature: leverage a 3rd party lib using the same DSL https://github.com/davemo/advanced-directives-with-angular-js/pull/2/files

Resources

This is part of a screencast series on Angular JS

1. Intro to Angular JS
2. End to End with Angular JS
3. Security with Angular JS
4. Frontend Workflows with Grunt and Angular JS
5. Testing Strategies for Angular JS
6. Advanced Directives with Angular JS (Part 1)
7. Advanced Directives with Angular JS (Part 2)

2016 was a year of ups and (if you agree with many of the social media channels in the last month) a significantly larger number of downs. I’m not sure exactly what it is about this time of year that causes us to reflect so much more than the rest of the year; I suspect it mostly has to do with a heightened sense of awareness as we approach boundaries.

As children, we know they are there because our parents tell us and we frequently come close to them often by virtue of desiring to push past to see what is beyond. As parents, we know they are there because we create them; often intending to protect our children from things that are dangerous or to set parameters for healthy age-appropriate interaction. As software developers we encounter boundaries in tests, the design of our code as it relates to components in a system, API’s, the list goes on and on. Being in constant consideration of boundaries, whether consciously or subconsciously, has the side-effect of switching our brain into analysis and reflection mode.

I recently had some friends reach out for advice as they were contemplating a transition at a critical boundary: quitting their job.

It’s time for a change

I’ve been restless but I’m trying to be cautious

These exchanges, coupled with the time of year and my brain switching into reflection mode, have had me considering reasons I left jobs in the past. The more I thought about it the more my brain kept telling me that those reasons were often not valid reasons to justify leaving. I wrestled with it and tried to move onto other thoughts but it just kept nagging at me; this is usually an indication I need to deal with those thoughts in a positive way. So, here I am attempting to impart some wisdom about reasons not to quit your job in 2017 (but mostly just typing as an exercise to free my brain up to move onto other things!) 😉

Technology

On April 3rd, 2014 I wrote some of the reasons I left the last job I was at. I’ve re-read that post a dozen times trying to mine some fragments of wisdom, mostly to attempt to justify the decisions at that time. The only conclusions I’ve come up with at this point are: there are a lot of emotions in my writing that stemmed mainly from too much focus on Technology as one of the reasons I quit.

Technology is seductive; it attempts to woo us through flashy demos from paid evangelists or conference talks from influential heroes. Technology is like the trench-coat wearing crook on the street corner who promises us the Rolex and ends up delivering a cheap imitation. When the tech-stack you work with (or want to work with) becomes the justification for a career change it has the potential to snowball into a full on addiction to newness that sets your brain up for a dependence on switching tech in the same way a drug addict depends on their narcotic of choice.

The truth is that most systems that we work on move slowly towards disorder and complexity, so it is no surprise that the promise of a fresh-start in a new technology stack is appealing. An entire generation of software developers have become so accustomed to quitting their job based on technology that I suspect the coming decades will yield a significant need for people who have experience dealing with systems over a long period of time. If we don’t deal with our addiction to newness, the recruiter pitch of “Junior Developer Wanted, 5 years experience Node.JS/Rails/…” that we so frequently deride will turn into “Senior Developer Wanted, 10 years experience dealing with software entropy” and there will be no one to answer the call.

Don’t get sucked into the new tech cycle; learn to embrace what you’re working on now and strive to work the best with the technology constraints you’ve been handed.

Scale

So, you’ve thought considerably about your current job and ruled out technology as the main reason for leaving. In fact, you’re sticking with the tried and true tech-stack you’ve always known, but there’s just been that nagging feeling that you’d be so much better off if you could experience working with your stack at a larger scale. Oh to be able to understand the constraints of systems at an order-of-magnitude or two beyond where you are working now; that’s the ticket!

Scale is subtle; on the surface it promises a similar thing to the tech-stack switch: an opportunity to learn new things that your current scale of operations can’t teach you. Chances are you will probably actually learn some new technical things by increasing scale; the reality is that we tend to focus more on the technical and less on the human side of software development which often affects increases in scale much more. In his book, Predictable Success, Les McKeown gives some wisdom about how scale affects success in an operation:

Predictable Success is defined as one of the seven stages of organizational development. It can be attained by any organization or group that acts as a human machine, well-oiled, working in concert, not without challenge, but focused and “in the zone” where growth is attainable and sustainable. It is not about size or the age of an organization. It has nothing to do with resources, culture or industry.

I particularly like his focus on the “human” factors and exclusion of the traditionally technical things we tend to focus on. While the book is targeted towards business growth and the reasons why companies tend to fail, it can be broadly applied to any group or organization (including software development orgs). Ask the following questions about the organization you’re contemplating joining:

• What stage of organizational development are they in?
• Can I really grow the way I want to there, not just technically but also in my capacity as a human?
• Will the scale inhibit or promote my growth?
• What potential problems will exist at the new operating scale?

In my experience working as a consultant the only constant I’ve observed is that the same problems (and the same trend towards entropy) exist regardless of the scale of the company and how they use their technology.

Think hard about using scale as justification for quitting your job; chances are you haven’t considered the human factors involved.

Adversity

Ruling out both scale and technology as reasons for quitting a job is great, but to be honest they aren’t the most common reasons I’ve quit in the past. Typically that honor is reserved for a more sinister and damaging character-trait: an inability to handle adversity.

Adversity is double-edged; it shows us the best and worst of ourselves, sometimes at the same time! If I take a step back and examine times when I left jobs because things just got too challenging I truly feel like there was always an opportunity to overcome in spite of the challenging situation and emerge a stronger human. How we respond to challenges in our workplace says a lot about our character:

Before you quit your job because it’s just “too damned hard” I would encourage you to take a moment and step outside yourself; reflect on the situation and the facts – take the emotion out of it and be objective.

If you can turn a challenging situation from a painful pity-party into an opportunity to conquer and emerge better you will be amazed at what you can come through.

Still Ready to Quit?

There are valid reasons to quit your job, but I think we tend to dwell on them far too often before considering the reasons not to. My hope for you at the end of 2016 is that as you approach the boundary and transition leading into 2017 that you would be prompted to introspect and reflect on what it is you truly want. Set some goals, list some objectives for your life and above all ask yourself some tough questions before throwing in the towel; as a wise friend of mine once said:

If the grass is greener on the other side, then you’d better water your own lawn.

This screencast shows how to get setup, development workflow, and building the first cut at a ListView to show some images and meta data for Hearthstone cards in the React Native application workflow.

Resources

https://facebook.github.io/react-native/

Web developers are integration specialists—tying plugins, scripts and frameworks together into a web application that works. Thinking in terms of abstractions—by condensing many low-level ideas into fewer high-level ideas—allows us to simplify our code and reason about it with less cognitive overhead.

In this screencast, recorded at Prairie Dev Con 2015, we examine a few techniques for building abstractions on top of popular JavaScript frameworks by learning about Domain Specific Languages and bringing some convention to our code.

Code

https://github.com/davemo/jsdsl

This screencast examines some of the more advanced features in Angular, specifically Directives. We’ll see how we can leverage the power of custom elements and attributes to map Domain Specific concepts through HTML and translate those into Value Objects in our Domain, resulting in cleaner HTML output. Also discussed: complexity, creating a DSL with directives, debugging techniques, and other tips and tricks.

If you’re interested in some more context prior to watching, check out my other angular screencasts and an earlier post on the power of web components as abstractions. This screencast covers:

• html as a dsl
• abstractions in html
• $compile
• $templateRequest
• $templateCache
• directive definition object
• requiring other directives
• directive communication ($scope.$broadcast, $scope.$on)

Code

• https://github.com/davemo/advanced-directives-with-angular-js

Extra Credit

Some things in the screencast aren’t complete and some things could definitely done better. This section is a challenge to you, the reader/watcher to improve the code and level up your knowledge in the process! Try and tackle some of these challenges if you want:

• Bugfix: the editor currently shows up multiple times, fix it so this doesn’t happen (hint: maybe an ‘edit’ state that’s tracked could help the directive know if it should execute .insertAfter)
• Feature: make the expandy arrow thing point down when expanded and to the right when collapsed.

Resources

This is part of a screencast series on Angular JS

1. Intro to Angular JS
2. End to End with Angular JS
3. Security with Angular JS
4. Frontend Workflows with Grunt and Angular JS
5. Testing Strategies for Angular JS
6. Advanced Directives with Angular JS (Part 1)
7. Advanced Directives with Angular JS (Part 2)

Do software companies succeed because of the technology they use or is it due to the discipline and talent of their engineering and design teams? Is it the right combination of vision and direction from leadership that leads to success or just simply being in the right place at the right time? All of these things are potential contributing factors to the success of a software company but none of them stand alone as reasons for success.

The companies that succeed are the ones who solve their customers business problems with excellence, regardless of technology, leadership, or team composition. To attribute success to any one dimension, such as technology choice, is at best naive and at worst renders that dimension untouchable.

Alignment Challenges

When technology is given all of the credit for success it becomes the gravitational center that all further technology decisions must orbit. Evaluating alternatives becomes fruitless; anything that has the potential to drift too far from the ideals of this magnetic core is viewed with skepticism. This makes it challenging for companies to adapt, or even recognize, that there are changes in technology choice that could benefit them greatly.

Worse still, this magnetic core philosophy creates an environment where all choices are evaluated through the reality-warping magnetic field surrounding the technology core. Process changes, HR policies, culture; all these things are influenced by the gravitational pull of the core and often by the community of other companies who have chosen to embrace its ideals.

Untouchable Icons

In a company where the magnetic core of technology is viewed as the sole basis for success, there become untouchable icons (the technology, and its supporters) who are unable to be challenged. This often creates a sense of fear (and paralyzes objective thinking) as any idea that deviates from core ideals is viewed as having the potential to veer the company out of orbit and cause a cascade of failure.

There becomes an identity mismatch between team members who have good ideas and those who espouse core ideals; the framework of trust used to identify an individuals success often selects the latter.

Technical Depth at the expense of Empathy

Companies that align their success with the magnetic core of technology often over-value technical depth and promote from within those individuals who have demonstrated alignment with core ideals. When technical depth is put in this position of importance it creates an environment where empathy is seen as a less valuable skillset. The challenges that these companies face, and the challenges our industry continues to face are often a direct result of a commonly repeated failure that too closely aligns success with technology.

Moving Forward

All of this paints a pretty bleak picture about the state of technology and the software industry and it is easy to list problems without solutions; where do we go from here? I firmly believe that there are simple solutions to the problems listed above.

First, we need to balance the success equation in our companies so that it places more emphasis on people being the reason for success instead of technology choice.

Second, We need to foster an open environment that allows people to feel like they can contribute ideas, any ideas, without feeling paralyzed by the fear of being viewed as counter-culture when ideas differ from core ideals.

Lastly, and most importantly, we need to balance the framework we use to promote leaders in our companies so that empathy will be considered more important than technical depth.

This screencast examines low and high fidelity testing strategies for Angular JS, and demonstrates examples of how to write tests with these strategies using Protractor, Testem, and Jasmine.

Code

• https://github.com/davemo/lineman-angular-template

Resources

This is part of a screencast series on Angular JS

1. Intro to Angular JS
2. End to End with Angular JS
3. Security with Angular JS
4. Frontend Workflows with Grunt and Angular JS
5. Testing Strategies for Angular JS
6. Advanced Directives with Angular JS (Part 1)
7. Advanced Directives with Angular JS (Part 2)