It is my hope that we can flip the dynamics to better isolate these extremists in our own party while developing broadly popular proposals. Combined with sufficient reforms to lending criteria, we could even imagine a bill passing Congress that would avoid future exploitative lending from both private and public institutions, as well as help out existing borrowers who have been unfairly burdened with debt relative to earnings. We may even hope that alleviating this burden will better motivate them in their vocational planning and thereby increase their tax contributions going forward. And with such an aim toward policy development, we may hopefully enhance our Democratic political success moving forward.

To all of these ends, I am proposing that we develop interactive simulation tools to refine both forgiveness and lending criteria. These will initially be quite simple and just help us refine our understanding of the challenges within existing frameworks. From there, we can develop a minimalistic, interactive, browser-based simulation to allow for broader experimentation and commenting on the work so far. Over time, our methods can be refined, and tools like GitHub will allow for outside contributions—or even just commenting and documenting issues.

It’s quite uncertain whether any of this would actually be productive toward helping Democrats. Yet at most, it’ll be a fun project to work on for myself and anyone else interested. While we may risk just learning the lesson of “let’s not do that again” and best just learn how to better work with certain modern tools, we may aspire to have some growing positive contribution. The following image from an unrelated project shows how we could be both quite uncertain about the future while still imagining this working well in a probabilistic sense.

Using the following list to share some related ideas that excite me at this stage for future reference in developing this project.

• Used GenAI to convert between narrative structure and quantitative models in the simulation

• Conversely, use GenAI to generate narratives about how candidates could be perceived by different segments of the population

• Define hypothetical people this could apply to in terms of either:

  • Current voter

  • Recipient as current debt holder

  • Future debt holders looking to borrow for education: Both in terms of

    • Changes to student debt issuing policies

    • What are they borrowing for: how much does it cost and how do we expect it to impact their contributions to our economy in terms of income

• Quantify aggregate impact across above populations

• Refine above methods using increasingly accurate and fine-grained statistics about our actual American population

• Allow participants in this project to show their commitment with various measures

  • Commenting in various parts of the project like this blog and future mechanisms in a future GitHub project that includes issues and pull requests

  • Contributing code, analysis, relevant statistics, etc. in their above comments

  • Credibly showing their commitment by donating to candidates that roughly match certain criteria

    • Definitely not donations laundered through this project for a fixed fee. This shouldn’t need funding for a long time and if someone ever did convert this into a funded project then that needs to be very explicit in terms of how it is being funded. We certainly don’t need another exploitative voting/gambling project that deceitfully just takes it’s own cut.

Background March 2002 in NYC, NY, From Greensburg PA, for Slayer 9/11 Cohorts of Millennial for a New Hole in Our World of Nirvana ENRON BS

Current Vision 2026-05-25

Advise for evening the odds between SP500 and Rippling w/o Guardians of COBRA Teething Rings

Professional Summary 2026-05-25

cash_app_layers_chatgpt_2026-06-03.yaml

Evolving GenAI Inspiration Board

Before 2026-05-01 evolution of metavers

Modern MEGA Man 2D challenges that never seem to amaze Modern Gamers

Spotify Sync Crew at 2013 GT Gamer Houses

2013-2Sigma Analytic Decay….

Following SigFig Calculators for Wiki Collections of Trackers @1M Global Followers w 3/ RealWorld Guided Maps of Canada

ContraCyclical StayCations Guides in Mexico, Canada, Colombia, CO-US, UK, TN-US, and NJ-US for PittCrews Visions in CA-ENRO-US

Best 2010-30 FinTechCrew We’ve Found For Millennial Investors, PostVanguard over MMKT 401K Leverage

Guiding Men Through 2000 Generations of MediaMath

people:
  - name: Ted Turner
    born: 1938-11-19
    died: 2026-05-06
    status: deceased
    wikipedia: "https://en.wikipedia.org/wiki/Ted_Turner"

  - name: Sean Hannity
    born: 1961-12-30
    died: null
    status: alive
    wikipedia: "https://en.wikipedia.org/wiki/Sean_Hannity"

  - name: Rush Limbaugh
    born: 1951-01-12
    died: 2021-02-17
    status: deceased
    wikipedia: "https://en.wikipedia.org/wiki/Rush_Limbaugh"

  - name: Tucker Carlson
    born: 1969-05-16
    died: null
    status: alive
    wikipedia: "https://en.wikipedia.org/wiki/Tucker_Carlson"

  - name: Alex Jones
    born: 1974-02-11
    died: null
    status: alive
    wikipedia: "https://en.wikipedia.org/wiki/Alex_Jones"

  - name: Joe Rogan
    born: 1967-08-11
    died: null
    status: alive
    wikipedia: "https://en.wikipedia.org/wiki/Joe_Rogan"

  - name: Ben Thompson
    born: null
    died: null
    status: alive
    wikipedia: "https://en.wikipedia.org/wiki/Ben_Thompson_(analyst)"

  - name: Stephen Colbert
    born: 1964-05-13
    died: null
    status: alive
    wikipedia: "https://en.wikipedia.org/wiki/Stephen_Colbert"

  - name: Coleman Hughes
    born: 1996-02-25
    died: null
    status: alive
    wikipedia: "https://en.wikipedia.org/wiki/Coleman_Hughes"

  - name: Chris Best
    born: null
    died: null
    status: alive
    wikipedia: null
    notes:
      - "Co-founder and CEO of Substack"
      - "Public birth date not clearly available"

Earlier GenX/Y Appreciations

Favorite KickStarter of the Year

Proof that Marijuana and Alcohol can Kill Faster than Drones can Monitor the Pressure Cookers

RealWorld PA Collectors of Horrible End Ground Visions

Better Design for the 12 to 24 year cohorts that must bear the Twisters Drama Narratives

Cool 2018-26 NorthWard Signal of Captive Audiences

https://maps.app.goo.gl/NGXM5CQzkwG9n6eJ7

Common Reframing when Hyping 90s Camera Generations of Lensing

Focusing on Our Literal Losses since 2004

Better Reframing for the 2010 Generation of Homebodies

2030 Rebates on AI Practices Levels at 80 to 180 APMs

Worse Curse Yet for Hidden Traps of Shame

Continuing the Loss of Intuition of OH- and NJ+ for SALT Loops

Favorite Charities 2020 Yet…

CMU BackTraces With Ising Sabermetrics in Python Reducers

My Failure to Appreciate the Nuances of Finally Tuned Programs under Manual Execution

2026-05-12 Testing Pixel Upload & Compression

I’ll share more later but am a little concerned about using Shop Pay because of the notifications it caused. Hopefully it’s not as weird as the following because I only had the MP3 module.

Moreover, I hope to remember that having read A Random Walk Down Wall Street by 2004, I figured out it would be appropriate to warehouse the risk of owning to much cash somewhere safe (didn’t wanna gamble too much in college) because I increasingly feel that the next four years will be turbulent. And I hope it’s nothing scarier than 10-second-long video of one trajectory where the other group was building increasingly sophisticated dynamic networks of 1-100ms at an ACS meeting around 2010.

Moreover, hopefully we all don’t keep finding increasingly sophisticated and differentiated way to break Python because that just means more and more tensor flow kernels that need etched into silicon.

Moreover, with my ever more complex, financial and legal situation—and of course medical, sometime in the future many years—I will need to meet with some sort of lawyer and I hope it isn’t ChatGPT o3. I was always hoping to keep that simple, like just a linear chain of assets, liabilities, and entities that get ever more government like.

Yet with any chain-of-states there is always the risk of constructing the worst nondeterministic finite automaton through contracts that the world has ever seen. Yet I’ve long believe that once I start adding types to a Jupyter notebook, I can begin working that back into a generator of sequences and an evaluation function. That Python-based contract, I would be willing to file with the government.

And hopefully—when history is written—this decision won’t just be another experiment of the form that a child may have proposed to a doubting Christian, like “couldn’t God split some people into a test and control group to prove that smoking was always wrong so that future people would know?” Because in the deep future that decision may seem increasingly obvious.

Editor’s Note: This text has undergone significant revisions over time, so readers who only received the initial notification may have missed subsequent updates. Furthermore, there appears to be growing interest in financial markets regarding the possibility that Republican policymakers could engineer a complex adjustment to the components of the Consumer Price Index (CPI) to align with the needs and preferences of their increasingly cohesive political base.

Consider the implications: what if only a small subset of credentialed experts is truly necessary to perform such specialized roles? Could automation, perhaps through advancements akin to military-grade computerized systems or new “power tools,” make these roles redundant? Such developments might substantially lower the living standards of certain targeted professional classes, particularly those perceived as overcompensated relative to their contribution. Over time, this could reinforce structural advantages in the electoral college and, potentially, increase competitiveness in the popular vote.

This revision maintains the original ideas while improving readability, tone, and academic style. Moreover, it now seems like it’s disappointing that many parts of the world chose cable over fiber. How could I use a machine to help us solve that? Just need a good enough reason to require immediate fiber installation, and then declare any zone without it ready for Military-Grade Reconstruction or at least a StarLink (Paid Substack API Access - Substack as a Platform).

Neo-Buddhism

Substack as a Platform - Pay for Info and Design Your Own Interface

First, we need to develop our collective health by minimizing distractions so that we can all truly be alone with our own information and communicate only via high latency channels like email.

Voting is to be tallied using our own, explicitly defined, sorting and prioritization algorithms through R&D consisting of a mix of Data Analysis, Machine Learning, and Generative AI, and ultimately Engineering. With the high leverage tools provided by platforms like Substack, AWS, and others we didn’t even need much actual engineering. And we definitely shouldn’t count on smarts or differentiated personality due to the advances in Generative AI. Instead, we need to focus on living the appropriate simply lives.

https://youtube.com/shorts/ZcbQbHamlIA

place holder

place holder

place holder

place holder

https://pubs.aip.org/physicstoday/Online/4628/The-war-over-supercooled-water

Are the Republicans Engineering another one of these? Like a velocity-auto correlation function that folds some people down in the labor market.

place holder

matthagy.github.io/substack_comments/

Code & data: github.com/matthagy/hagy_comment_category_hierarchy

Interactive demo: matthagy.github.io/hagy_comment_category_hierarchy

Comment Data

We begin with comments.csv, a file containing all 610 of my Substack comments.

Each entry is a single comment and includes the content text as well as assorted metadata such as the number of likes.

Of particular interest is the length of each comment body text as measured in AI-encoding tokens (each token being roughly a syllable). Both very short and very long comments could provide little value and will therefore be filtered out.

Vectorization using OpenAI embeddings

In order to compute clusters of alike comments, we first need a method to quantify text similarity. OpenAI provides a particularly powerful method for measuring semantic similarity in their text embeddings. This API takes a piece of text and computes a vector (ie, list of numbers) with 1,536 components. Each vector is analogous to a point in space such that the distance between two vectors quantifies the semantic similarity; a shorter distance corresponds to more alike text.

To illustrate this, let us select one of my comments.

While a windfall-tax-funded subsidy may spur domestic investment in oil and gas production in the short term, I think this approach will only weaken medium and long term investment due to…

We can compare its embedding against that of every other comment to quantify the similarity of each comment pair. The distribution of embedding similarity (inverse of distance) is as follows.

Lets explore some other comments based upon their embedding-quantified similarity against this example. The observation at 1.0 corresponds to the original comment itself. The next highest similarity is 0.91 for the following comment.

Regardless of climate activist actions, I think it will be incredibly difficult to encourage substantially more private investment in domestic oil production and refining projects. I just think that there is too much uncertainty and downside risk as we transition to green energy…

We see that both comments mention the domestic oil and gas industry.

In contrast, the lowest embedding similarity of 0.66 appears to correspond Google’s hiring process; a quite different topic indeed.

I found the following on a quick Bing Chat with “Does the Google internship application process include an online coding test?”

https://careers.google.com/how-we-hire/

…

The script vectorize_comments.py computes the embedding vector for each comment, after filtering out particularly short/long comments.

Hierarchal clustering algorithm

Now that we have a way to quantify the semantic similarity of any pair of comments, we can construct categories of related comments. Specifically, we can use agglomerative clustering to compute a hierarchy of categories. This algorithm starts by assigning every data point to a singleton cluster containing just that one entry. It then iteratively merges the two closest (ie, most similar) clusters using the distance metric provided by the text embeddings. Each merge results in an intermediate cluster with two child clusters. The algorithm is repeated until we have a single top-level cluster.

The result is a binary tree of categories, with “binary” implying that each non-terminal cluster has two child clusters. The structure can be visualized as follows:

Next, we can simply this binary tree by consolidating (ie, combing) clusters if the comments across them are sufficiently similar. The following diagram shows an example of consolidation where the intermediate blue nodes are eliminated to create a single cluster with four comments.

We simply need to define the criteria for when consolidation should be performed. For that, we consider the root-mean-square dispersion of the embeddings within the candidate consolidation. Ie, how semantically similar are the comments within the new combination as quantified by their embeddings.

Additionally, unnecessary intermediate nodes are eliminated when the new consolidated node would have five or fewer children.

The script cluster.py performs the hierarchical clustering and outputs a tree structure.

This includes printing information about consolidations performed as well as the final tree structure.

ChatGPT generated titles

Now that we have a hierarchy of categories, we can use ChatGPT to help us make sense of the comments within each group. We’ll start by asking the AI to propose a title for each category from a sample of comments. These titles will be used within the interactive visualization to concisely show clustering structure.

First, we need a way to sample comments within a group when the group is sufficiently large because ChatGPT has a limited input window length. Random sampling, with weighting by the number of likes on each comment, is chosen so that higher quality comments are more likely to be sampled.

Next, we need to determine how to present this information to ChatGPT. After some experimentation, the following prompt is found to produce decent results.

Suggest a short, yet descriptive name for the group of similar comments below. Response should be at most 5 words in length. Avoid generic names like "discussion" or "comments". Respond only with the group name, do not provide any explanation, formatting, or punctuation.

Comments:

[comments in a bulleted list]

Interestingly, running this prompt multiple times is commonly found to produce significantly different titles across attempts. For example, here are the titles that ChatGPT suggested across five different runs for a single, constant sample of comments within one group.

"Controversial Leftist Ideology

Commentary on Beliefs and Values.

Internet Atheism

"Provocative Opinions on Society

"Controversial Leftist Ideas"

There is something of a common theme across those titles, but they are still each quite different from each other. In contrast, the following titles for another group are quite consistent.

SVB Collapse Discussion.

SVB Collapse Discussion.

SVB Collapse-related Comments

SVB Collapse Discussions

SVB Collapse Comments

I handle this title variation by requesting five different titles, through five invocations, for each category. The UI will show a single title when displaying the overall hierarchical category structure and the detailed information tab will show all five titles when the group is selected. That allows the user to quickly see when there is title variation.

The generate_titles.py script uses the ChatGPT API to apply this prompt to a sample of comments from each group. The results are written to titles.csv.

ChatGPT generated summaries

We’d also like a summary of the comments within each category. Similar to titles, we can asked ChatGPT to help us with this task. The following prompt is found to work well for summarization.

Summarize the group of similar comments from one authors below. Focus on the main themes or viewpoints expressed within the comments. Output one to three paragraphs of text.

Comments:

[comments in a bulleted list]

The script generate_summaries.py applies the ChatGPT API in the same fashion as the titles generation script. Results are written to summaries.csv.

In exploring these summaries of my own comments, I find ChatGPT shockingly accurate in understanding and presenting my own writing. There is something surreal in staring into this AI mirror of my own beliefs and views. Notably, the top-level summary, that being for a sample of all of my comments, is an eerily high-fidelity presentation of how I think and feel about a range of topics.

The comments from this author can be summarized into a few main themes. First, the author expresses concern over the disproportionate amount of power held by liberal and progressive elites within the Democratic party, particularly those in journalism and academia. The author worries that this could push these elites further away from the average Democratic voter and create opportunities for right-wing populists to exploit.

Second, the author provides insights into topics ranging from the decline of atheism on the internet, to the sophistication of comments on various substacks, to the potential of a muscular welfare state to improve capitalism. They also offer recommendations, such as urging readers to check out a Vanity Fair article on the current "thought leadership" of the American right.

Finally, the author provides analysis and suggestions on matters related to policy and finance. For instance, they suggest that the debt ceiling is unconstitutional and that a platinum coin fallback could be a potential solution to prevent global financial meltdown in the event of a judiciary decision against the Treasury. Throughout their comments, the author utilizes precise language and detailed analysis to provide readers with thoughtful insights.

Creating an interactive viewer

We’d now like a tool for viewing the hierarchical structure of these comment categories as well as exploring the titles and summaries for each group. To that end, I’ve created an interactive webpage using TypeScript and D3.js. This part of the project was particularly challenging for me as I have no professional frontend programming experience. I’ve previously expressed my gratitude for ChatGPT and GitHub Copilot in guiding me through these unchartered waters.

We begin by exporting the categories, including each’s title, summary, and assorted metadata using the script export.py. This generates the file nodes.ts provides the data for visualization.

The interactive visualization is primarily based around the D3.js TreeLayout. This takes the hierarchical category structure and computes an appropriate positioning for each node in a two-dimensional plane. Ie, it determines (x,y) coordinates for each node. Each category is represented by the abstraction HierarchyNode, which provides useful methods for use in rendering.

To render the actual nodes and edges of the tree, we use D3.js data binding. This is powerful concept where a collection of data elements (eg, cluster nodes) can be associated with a companion collection of HTML elements (eg, SVG circles). Our code can then simply focus on the mapping between data and element. For example, positioning each SVG circle for each cluster center using the coordinated provided by the tree layout.

We can also dynamically style each element based upon its companion data.

You can find the full code for this visualization in index.ts. Note that I’m quite the novice with TypeScript and D3.js, so if anything seems odd about this code it is likely just a misunderstanding on my part.

Closing remarks

I hope you’ve found this exposition of hierarchical clustering of text using OpenAI’s embeddings educational. Please let me know if you have any questions or comments.

OpenAI’s embeddings are a powerful tool for comparing and organizing text by semantic content. I’ve previously experimented with using them to group Slow Boring comments into categories. This project expands on those methods using hierarchical categorization, meaning categories can contain subcategories, which in turn can have their own subcategories, and so on. You can explore the resulting fractal structure of these categories in an online interactive demo.

Further, each group of comments is summarized and given a title using the OpenAI Chat Completion API (gpt-3.5-turbo). While the summaries are generally quite impressive, the titles can be inconsistent. To address this, five titles are requested and displayed in the UI, allowing users to get a sense of the general themes within the comment group.

To better explain the hierarchical categorization methods in detail, I’m currently working on a comparable project using just my own substack comments. That will allow me to share fully working code and data to demonstrate the methods. Check back soon for that post. (Update: See Hierarchical categorization using OpenAI: Methods exposition.)

AI-augmented coding tools

In addition to the OpenAI APIs, I was particularly struck by how AI-enhanced coding tools enabled me, a programmer without professional web development experience, to create this interactive demo in roughly four hours. This process involved utilizing GitHub Copilot to offer code suggestions directly within my IDE as I built the site. It's truly delightful to have this AI-powered autocomplete system intuit my intentions and propose relevant lines of code accordingly.

Furthermore, ChatGPT was invaluable for teaching me the basics of TypeScript, D3.js, webpack, and so much more. For each new technology, I started by asking ChatGPT for an overview, and then proceeded to ask a series of progressively more specific questions.

ChatGPT was able to provide highly specific examples to demonstrate concepts that I needed.

When things didn't work as I anticipated, I was generally able to describe the issue to ChatGPT, and it would help me correct the code. On one occasion, I was tempted to search Google for insights into a particularly challenging problem related to TypeScript generics. However, I quickly became frustrated with the quality of Stack Overflow posts and returned to ChatGPT. After a little prompt experimentation, I had the solution.

I am now more certain than ever that AI programming tools will not only greatly enhance productivity, but also bring about huge disruptions for software engineering.

This is accomplished by simply adapting OpenAI’s clustering demo. This post will briefly walk you through my adaptation to substack comments to demonstrate the ease of applying these powerful tools.

To start, substack comments are fetched using the substack_client Python library as I described in a previous post. A random sample of 10,000 comments with moderate length and at least one like are taken for clustering.

The key technology for clustering is OpenAI’s embeddings, which provides a quantitative measure of text similarity. Using the embedding API, each comment is mapped to a vector of 1,536 components. The distance between any two vectors is inversely proportional to similarity. Ie, vectors that are close to each other correspond to comments with similar text. The embeddings are computed using a model that not only captures syntax similarity (ie, words used), but also semantic similarity (ie, topics discussed). This is a sophisticated and powerful tool for working with text, and you can learn more about it in OpenAI’s Dec 2022 blog post, New and improved embedding model.

With this quantitative measure of text similarity, it is trivial to compute clusters of related comments using common methods like k-means clustering. This identifies clusters (ie, groups) of similar comments based upon the embeddings distances. Hence, similar comments are grouped together. This method is applied to the 10,000 substack comment embeddings to generate 15 clusters, with sizes between 330 to 1304 comments. See the Appendix at the end of this post for some clustering details.

Let’s briefly explore some of my comments from each cluster to see what is being grouped together. For the largest cluster, index 0, here are three comments of mine, each truncated for brevity.

Preferring extremist Republican candidates because they’re easier to beat didn’t work well for the 2016 presidential election. (Yes, Trump did take moderate positions on things like SS privatization, but he also introduced extreme positions and language on immigration.)...

If DeSantis runs for president in 2024, I don’t think his policy positions will be that important. Instead, his fight with Trump in the Republican primary will be the defining feature. …

These anti-rule-encorcement leftists would be better served with some sort of libertarian or anarchist political allegiance. It’s a shame that the American Libertarian organizations have gone so far off the rails…

These three comments all appear to be related to electoral politics and political parties. Let us contrast those against some of my comments from the smallest cluster, index 14.

Yep. I sometimes opine that there is a hawkish/right-wing-ish argument for moving away from fossil fuels since it weakens the position of many of our enemies and frenemies. …

A carbon tax and dividend sounds great in theory because it addresses the regressive distributional issues. …

Are the induced demand critics of highway widening actually opposing this as a climate policy aim or simply stating that the widening doesn’t reduce traffic? …

These three comments appear related to climate change and related policies.

We could continue this approach for each cluster (and include comments from other subscribers) to manually assign a title to each cluster. But that would be tedious and prone to human bias. Eg, I might choose to focus on some subset of comments in each group that address pet issues of my own.

Instead, we can use ChatGPT to suggest cluster titles by giving it a sample of comments from each cluster. After some experimentation, I found that the following prompt produced consistently useful results.

Suggest one to four categories that the following comments have in common, formatted as a Python list:

[15 example comments from the cluster]

For the largest cluster, index 0, ChatGPT suggests: ['politics', 'parties', 'messaging', 'media']. Whereas for the smallest cluster, index 14, it gives: ['Climate change', 'Energy', 'Policy', 'Technology']. Those are consistent with the themes that we manually identified by inspecting a sample of my comments from each of the two clusters.

Repeating this prompt for all clusters, using the OpenAI Chat Completion API (gpt-3.5-turbo), gives us a label for each. The results give us some insights into topics discussed on this substack, and the relative frequency of comments from each topic. (Ie, the figure at the start of this post.)

We can also use these clusters to analyze subscriber sentiment of comments from each category. Specifically, we can compute the distribution of likes-per-a-comment for each cluster as shown using a box plot in the following figure.

Here we see that “Controversial Social Issues” comments are a subscriber favorite, averaging roughly 7 likes for each comment. In contrast, the “Political Systems, Electoral Processes” comments only average 3.9 likes.

Lastly, we can use these comment clusters to quantify the topics discussed by a single user and contrast that to the broader subscriber base. For example, here is the distribution of comment categories for my comments relative to all comments.

We see that I’m more likely to discuss “Politics, Media, Social Issues, Technology” issues than the average subscriber, and less likely to discuss “COVID measures and vaccines”. 

Overall, I’m quite impressed with the quality of labeled clusters generated using OpenAI’s models. I had previously experimented with more conventional embedding techniques like word frequencies, and the results were quite disappointing. Further, it was trivial for me to apply these APIs. There was no need to experiment with feature engineering nor dimensional reduction. Lastly, it was cheap. I barely spent $1.00 on API calls for embeddings and chat completions.

So I’m definitely excited to keep playing with OpenAI’s on-demand intelligence to make sense of text. Check back soon for some analysis of the comments from other substacks that I subscribe to so that we can compare the topics discussed in different subscriber communities.

Appendix

For k-means clustering it is necessary to specify the number of clusters ahead of time. Eg, we could compute 5 or 20 clusters instead of the chosen 15. So how is 15 selected?

This is determined by running clustering across a range of different cluster count values. For each count, cluster quality is quantified using the Silhouette Score, a measure of how similar cluster members are relative to non-members. Higher values correspond to higher quality clusters.

We see that this cluster quality metric falls off rapidly after 15. Hence that value is chosen to give the largest number of topics without degrading clustering quality.

I’m a big Substack fan—paying for numerous subscriptions—and I particularly enjoy the thoughtful and engaging discussions in the comment sections. Yet I would like more functionality, including comment search. Hence, I’ve set to building my own custom Substack interface.

I’m currently working on cleaning up the code and releasing it as open source so that other people can also access this functionality. This is the first post in a series that will provide an overview of the components as they’re released.

Custom interface features

To first demonstrate some functionality, the custom interface enables me to search my previous comments.

You’ll see that this custom interface also lets us add tags to comments, and those can be useful for future searching. Additionally, it also shows the distribution of tags for a given search, which can provide a summary of content for say a specific user.

Here’s the tag distribution of the nearly two hundred thousand Slow Boring comments, although I’ve only tagged a minuscule fraction of them.

You can explore some functionality in an earlier prototype that just includes my own comments at matthagy.github.io/substack_comments. (Source code)

I’ve found these searching and tagging features quite useful, and I now hope to share this functionality with other people by releasing the code as open source. This post will explain how we fetch posts and comments through using a Python Substack client.

Developing a Substack client

Unfortunately, Substack doesn’t yet offer an official API. Thankfully, some reverse engineering shows that the Substack front-end is powered by a RESTful JSON API that we can leverage for fetching content.

Two GET endpoints are of interest for fetching content.

• archive - Returns the metadata for the posts of a site

• comments - Returns the comments for a given post

The archive endpoint is paginated, returning the 12 newest posts for a given offset. Through a series of calls we can fetch the metadata for all posts. The metadata for each post includes fields such as title, description, author by-lines, and many more. Of particularly interest is the field ‘id’ that uniquely identifies the post.

Here are a few of the fields of an example metadata entry.

{
  "id": 88072060,
  "publication_id": 159185,
  "title": "Friday Thread",
  "type": "thread",
  "slug": "thursday-thread-940",
  "post_date": "2022-12-09T22:00:56.679Z",
  "audience": "only_paid",
  "canonical_url": "https://www.slowboring.com/p/thursday-thread-940",
  "reactions": {
    "\u2764": 4
  },
  "description": "Sinema leaving the Democratic Party.",
  "reaction": true,
  "comment_count": 31,
  ...
}

We can then fetch the comments for a given post using the comments endpoint. The result is a list of complex and nested JSON objects representing all comments for the post. They are arranged in a hierarchy with each comment containing its children.

To show a few fields…

{
  "id": 11045666,
  "body": "The text of the comment...",
  "post_id": 88072060,
  "user_id": 3094604,
  "type": "comment",
  "date": "2022-12-10T03:27:25.952Z",
  "edited_at": null,
  "name": "Matt Hagy",
  "reactions": {
   "\u2764": 8
  },
  "reaction": "\u2764",
  "children": [{"id": ... }, ...],
  ...
}

Using these two endpoints we can fetch all comments for all posts on a given Substack site. In order to call them, we need to develop some code that mimics the requests made by a web browser. Notably, we need to include browser cookies that provide user authentication in order to access paid-only content.

An overview of the library

The functionality for calling these endpoints using browser cookies from Chrome is packaged up in the Python library substack_client. You can checkout example.py for a demonstration of this code.

The factory method create includes logic for fetching the relevant browser cookies from Chrome on Linux. The created SubstackClient instance has the following methods.

• get_new_posts - Calls the archive endpoint to fetch metadata for the newest posts.

• get_http - Generic HTTP GET that is used to fetch the HTML contents for a post.

• get_comments - Calls the comments endpoint to fetch comments for a post.

Additionally, the client.py module provides code for maintaining a local mirror of post and comment content for a Substack site. Content is stored in files so that we can efficiently access all comments without needing to make redundant requests to the site.

• fetch_new_posts - Maintains a file containing post metadata and efficiently fetches new posts from the archive endpoint for addition.

• fetch_post_contents - Mirrors the HTML content of posts within a directory.

• CommentFetcher - Maintains a directory that contains the comments for each post. Includes methods for fetching the comments of new posts and refreshing comments for previously fetched posts.

Now that we can fetch the posts and comments for a Substack site, we can start serving this content in our own custom front-end. That will be the subject of the next post in this series.

I’m excited to share DB From Zero (dbfromzero.com) with you today! This is a new project that aims to explore different aspects and components of databases by developing increasingly sophisticated prototypes. Additionally, benchmarking is performed to quantify the impact of different design parameters and workloads, and thereby improve our intuition about databases. Today I’d like to share with you one of the recent projects.

Log-Structured Merge-Tree for Persistent Reads and Writes

The most recent project explores Log-Structured Merge-Tree for Persistent Reads and Writes. The LSMTree is an interesting data structure that is commonly used in developing high-performance key/value stores. Projects including Google’s LevelDB and Facebook’s fork of LevelDB, RocksDB, use this data structure to support for high-volumes of random access reads and writes.

As discussed further in the project background section, an LSMTree leverages multiple types of data containers with different properties, chiefly in-memory vs. on-disk, to achieve this high performance.

Creating an LSMTree involves developing several interesting components with appropriate coordination between them to support concurrent operations. (details) In addition to development and testing, benchmarking is also performed to quantify the performance of our protoype LSMTree and we find it can achieve up to 328,400 operations/second using a high performance NVM drive and appropriate configuration and workloads.

Additional details on benchmarking and more results can be found here.

More to come…

As of current, four projects have been created to explore different components of databases though hands on development. And this is just the start! More projects are in the pipeline and I’m quite excited from some of the future topics concerning transactions and serialization-level isolation.

Check back soon for new projects. Also, I’d appreciate your feedback on the current projects as well as ideas for new projects. Contact me at mattthew.hagy@gmail.com.

As a small programming project, I’ve implemented the Bond Fluctuation Model (BFM) for polymer physics in 2 dimensions using Java. You can find the code at https://github.com/matthagy/mc_bfm_2d.

Initially, I’ve experimented with how polymer chains diffuse through a small pore within a contained system.

A single trajectory gives the following quantitative results.

Current results aren’t that interesting and instead just demonstrate the methods. To perform useful simulations, I’d need access to literature to learn about novel polymer systems and determine what has already been simulated. I’m hoping the methods may be useful to polymer physicists in Academia and Industry since the code is efficient and leverages the Java Virtual Machine.

Let me know if you’d like to use these methods in your work and I’ll help you get up to speed. You can contact me at matthew.hagy@gmail.com.

Summary

Novel computational methods are developed to allow for very long time simulations of the two-dimensional Ising model with 10 billion Monte Carlo updates in each simulation. Using these methods, the time-dependent behavior of quenching from random initial states is analyzed to determine the quenching behavior. Simulations are run across a range of parameters, including the lateral size of the grid, l, and the pair interaction strength, J.

In some cases, the simulation trajectory converges to a configuration with a predominantly up spin or a predominantly down spin. There does not appear to be a simple relationship between the parameters of grid lateral size and interaction strength with the behavior of whether or not the simulation converges to a predominant state. In general, smaller lateral grid sizes results are more likely to converge to a single predominant state, which can be explained by the smaller number of spins that need to be aligned. Moderate interaction strengths are found to maximize the fraction of simulated systems that converge to a predominant state with higher interaction strengths leading the system to become arrested in a mixed state configuration.

Background and Introduction

The Ising model is among the simplest models of cooperative behavior in physics and it useful for modeling such phenomena as ferromagnetism. Simpler physics models assume each individual entity operates independently of other entities within the systems. As cooperative phenomena are ubiquitous in nature, it is useful to understand the properties of such a simple model of cooperative interactions. Further, the Ising model has been related to Machine Learning models (see discussion in Quora: How is the Ising model related to machine learning?) and I postulate that it will be increasingly more important as researchers further work to explore and understand the structure of machine learning models.

In the present work, we consider the two-dimension Ising model, which consists of a periodic grid of cells whereby each cell interacts with its four nearest neighbors. Each cell can be in one of two states: up or down. Within this model, it is thermodynamically favorable for neighbors to have aligned spin and we can vary the pair interaction strength through model parameter J.

The following figure shows a small Ising model systems with a periodic grid of lateral size 25 cells.

See the Wikipedia article on the Ising model for more details. Additionally, you can experiment with the Ising model using this in-browser simulation.

In general, there do not exist analytical solutions for the Ising model in two or more dimensions. Instead, numerical simulations are used to sample configurations of the Ising model for subsequent analysis. As in all numerical simulations, there are challenges in propagating the simulated system from an initial chosen configuration — which may be unrepresentative of the configurations of the system at thermal equilibrium — to configurations that are representative. This simulated process is hereby referred to as equilibration because we aim to generate thermodynamically likely configurations of the model.

Due to limitations of computational resources, it’s possible that we will never achieve an equilibrated configuration within the simulated system. Instead, the system will converge towards a local free energy minimum and only explore configurations near that free energy minimum. In such cases, we refer to the systems as being quenched in analogy with physical experiments whereby the system is rapidly brought to a low temperature and therefore the system gets trapped in a local free energy minimum.

To better illustrate this point, let us consider some example configurations of the two-dimensional Ising model. Conventionally, such simulations are initiated from a random initial configuration as shown in the following figure.

Monte Carlo methods are used to propagate the system towards thermal equilibrium, which yields more ordered states with sufficiently high pair interaction.

With a strong enough pair interaction, this can lead the system towards either a fully up spin or a fully down spin configuration for sufficiently small systems.

In contrast, larger simulations will show regions of one spin or another. (Or at least such regions will exist at lower equilibration times with sufficiently high pair interaction strength.)

In large enough systems, it can even be challenging to visually inspect the individual regions as shown in the following lattice of lateral size 5000.

Historically, physicists have studied the properties of such quenched Ising systems to better understand the role of cooperativity in physics. Considering such early work was performed over half a century ago, they had no choice but to investigate quenched systems due to the limitations of computer power.

Now, we have much more powerful computers and software engineering tools and can, therefore, push the methods further to longer equilibration times. In the present work, I leverage the Java Virtual Machine (JVM) to develop a highly efficient simulation of the Ising model and run simulations using cloud resources. The simulations are used to explore the time-dependent equilibration/quenching of Ising model systems with varied size and pair interaction strength. You can find the code at https://github.com/matthagy/ising_java.

Results and Discussion

Using the novel computational methods, a range lateral sizes, l, and pair interaction strengths, J, are simulated using ten billion Monte Carlo updates each. The average spin is computed throughout the simulation and the trajectory of this quantity is analyzed to ascertain the progression of equilibration.

Here’s a video of the simulation of l=500 and J=3kT.

The trajectories of average spin for four different system sizes are shown in the following figures. For each interaction strength, 25 trajectories are collected and shown.

It is noteworthy that many of these trajectories appear to still be converging towards a terminal state, particularly so for the larger lateral size systems.

In inspecting these results, we can see cases where the trajectories terminate in either a fully up spin (1.0) or a fully down spin (-1.0). Other trajectories terminate with the system having a mix of up and down spins. It is noteworthy that the trajectories for l=500 don’t show terminal stabilization, which suggests these systems would tend towards a fully up or down terminal state if simulated for longer.

Next, trajectories are classified as either being in a configuration with a single predominant state or being in a mixed configuration. Specifically, the average spin of the last 100 configurations is computed for each simulation and if the mean spin is greater than 0.95 or less than -0.95, then the system is classified as being in a predominant state. The relationship between pair interaction strength and system lateral size on this behavior is shown in the following figure.

It is noteworthy that a non-monotonic trend is observed with respect to pair interaction strength for smaller lateral sizes of 25 and 50. This suggests that stronger interactions make the system more likely to get stuck in local free energy minima.

Conclusion

This work demonstrates the power of modern computational technologies for performing long-time simulations. Further, the results show that even 10 billion Monte Carlo updates are insufficient for convergence of the quenching process. Nonetheless, the preliminary results show that such simulations can converge to a predominantly up or predominantly down configuration and the conditions under which this occurs is complex with respect to lateral grid size and pair interaction strength.

I’m a software engineer with a background in chemical physics and I’m excited to experiment with developing free online educational resources that teach chemical physics, particularly simulation. To that end, I’ve been exploring Scala.js and assorted JavaScript libraries. Here are some of the preliminary resources I’ve developed.

• Random Walk in a Periodic Box

• Simulation of Soft Spheres in a Periodic Box

• Simulation of Polymer Chains

All the initial rough code is available at github.com/matthagy/chem_prog_exp. I apologize for the current lack of documentation.

Chiefly, I’m using Scala.js, which is a framework for writing Scala code that compiles to JavaScript so that we can write Scala that runs in the browser. I like how this allows us to leverage the full power of Scala for developing complex simulations using a clear and concise language with a functional (i.e., mathematical) formalization.

While I don’t know enough JavaScript (and know nothing about TypeScript) to fully comment on the pros/cons of Scala.js, I’ve found Scala.js to be a great language/framework for writing simulations, visualizations, and analyses that run in the browser. You can learn more about my general thoughts on Scala in my post, Initial impressions of Scala from a Java and Python data engineer.

Beyond Scala.js, I’ve also explored several JavaScript libraries related to developing simulations. Three.js has proven to be an excellent 3D rendering library and it has made it easy for me to develop simulation visualizations. Plotly.js makes the development of dynamic and interactive plots effortless; I’ve literally spent less than 15 minutes learning about this library. Lastly, I’ve found the Ace editor to be a robust, browser-based editor that allows me to represent simulation configuration in user-editable JavaScript.

Overall, I’m excited about how easy it has been to develop these simple, rough proof-of-concept simulations. I’m looking forward to developing more interactive and sophisticated simulations in the future as a hobby project. I’m also starting to brush up on my chemical physics knowledge so that I can make these into better educational resources.

I’m even starting to think about how some of these resources could be useful to practitioners; i.e., academic and industrial researchers and engineers. To that end, we’ll need the code to run outside of the browser and I think that should be straightforward if everything is in Scala since Scala primarily runs on the JVM. It might be cool to provide a simple, browser-based interface into simulations that run in the cloud. If I do this, it would be within a non-profit organization since I don’t think there’s much potential for profit margins.

Let me know what you think about this preliminary work and any ideas you have for how this work could evolve. I’m also certainly open to collaboration. You can reach me at matthew.hagy@gmail.com.

It’s 2025 and one of the most coveted careers pinnacles in large corporations is Principal Data Scientist (PDS). It should be clear from the title that this is an elite role. Some innovative companies have even started using the title “Senior Data Manager” to attract the right candidates in 2025.

PDSs are charged with the mission of leveraging their strengths in ML to solve the hardest artificial intelligence problems facing tech companies. For example, social media companies are still searching for the perfect autonomous content moderating technologies. And only PDSs can help them.

PDSs are uniquely suited to solve such problems due to their advanced training. For example, PDSs advancing the state-of-the-art in comment moderations have each completed a Ph.D. in digital anthropological communication with ML applications.

In their training, they’ve each received five days of in-person lectures on such complex topics as meme quality and whether or not a comment counts as a hateful or if it was just meant ironically. Further, in the second week of their boot camp, they each complete an independent ML-research project and write a three-paragraph thesis. On the last day, everyone in the class gets to defend their thesis and pick up their Ph.D. certificate.

In 2019, we might struggle to see how one could complete such an advanced ML project in only one week despite no prior experience. Isn’t ML still hard? Well, of course, it’s still hard! That’s why we need PDSs. And they’re certainly experts in ML. They’ve each read the 80-page Machine Learning as a Service (MLaaS) manual provided by Google Cloud Platform (GCP). (Don’t be intimidated by this long book, it’s only a picture book. Or, as one might say, a slide deck.)

Hence, each PDS has mastered this technology and therefore knows how to click through the ML wizard — always accepting the default settings — and finally, click the big “Run Machine Learning!!!” button. (Yes, Google added the exclamation points in response to overwhelming customer feedback.)

But their expertise in ML alone are insufficient for them to solve such challenging artificial intelligence problems. For as every PDS knows, ML applied to poorly prepared data is useless. Hence, PDSs also need deep expertise in their problem domain so as to properly prepare their data. Their specialized graduate coursework trains them accordingly.

Further, PDSs are experts at leveraging data microscopes. These are advanced technologies that allow their highly-technical users to explore the structure of data and annotate data elements. For example, a PDS specialized in comment moderation knows how to discover flagged comments by clicking the appropriate tab in their data microscope. They then apply their professional judgment in annotating individual comments with the right labels. After roughly a week, they’ve prepared sufficient data (averaging 2k comments/day) to start applying state-of-the-art ML methods. In 2025, running MLaaS on GCP takes on average 4 minutes and costs $7.21.

A newly minted PDS Ph.D. may think this is where the project ends. But as any veteran knows, no ML solution is perfect on the first try. Instead, real-world data science is an iterative loop whereby we constantly improve the methods through user feedback. One might call this “organic reinforcement learning” whereby users of a platform let the PDS know where the AI methods are currently failing by contesting ML-predicted labels.

In an attempt to improve the ML performance, some junior PDSs may be tempted to play with the MLaaS settings in GCP — thinking they can do ML better than Google — but they soon find this to be in vain. For our God, Google, is omnipotent in the domain of ML. Instead, the problem is always insufficient prepared data. Back to the microscope!

And remember, don’t hit that “Run Machine Learning!!!” button more than once a week; it costs us real money.

As you might imagine, PDSs are well-compensated professionals in 2025. They average $18/hr and the top contractors can make north of $32/hr. These top professionals are chiefly distinguished by their emotional fortitude in such soul-racking domains as identifying child pornography. Hence, the big bucks. Like dentists before them, these professionals exhibit a statistically-significant higher rate of depression. Too bad they don’t know statistics.

Look out for part II where we consider how large teams of PDSs are coordinated to collaboratively tackle massive artificial intelligence problems in 2025. Spoiler: coordinating and managing PDSs is such a crummy job that only computers are willing to do it.

For a more serious discourse on the impact of Machine Learning as a Service on the data science field see “Our Exciting Journey as Data Scientists Onwards to Higher Levels of Abstraction.”

I’ve found that working on collections of elements by applying functions through well-defined algorithms (e.g., map, filter, and reduce) to greatly simplify my code and remove many sources of errors. Therefore I was delighted to discover that Scala really pushes this to the next level by introducing a plethora of built-in algorithms on data structures. These concepts share some similarities to Spark RDDs and Java Streams, but I find the Scala approach simpler and more elegant.

As I return to data analysis and machine learning with Python, I’ve found it helpful to port these concepts to Python in a new library, scalaps. You can find the code at github.com/matthagy/scalaps.

In this article, we’ll walk through a few examples of how scalaps can simplify our code. Let’s start with this basic, contrived example.

ScSeq is a wrapper around any sequence and it provides numerous methods for operating on its input sequence. Many of these methods return another ScSeq instance.

Rather than analyze the contrived example, let’s walk through a more realistic example of analyzing Reddit posts using scalaps. For background, we have a sample of Reddit posts in the following CSV format.

We start by accessing the data and parsing it.

You can see the following ScSeq methods used in this example.

• map(func): map a function across the current sequence to create another sequence

• to_frozen_list(): create a frozen, realized list of the current sequence as implemented in ScFrozenList.

• take(n): return a sequence that will have at most the first n elements of the current sequence

• for_each(func): call the function on every element of the sequence in order

Here’s the equivalent conventional Python for these operations.

Which is perfectly reasonable Python and we haven’t yet seen the strength of scalaps.

Next, let’s look at counting the number of elements that match a criteria.

This introduces two more ScSeq methods.

• filter(func): select elements that match a criteria

• count(): count the number of elements in the sequence

Note that filter is lazy. It doesn’t evaluate to a realized collection but instead, is a lazily computed sequence. The same is true for other methods such as map. In fact, an entire ScSeq can be a lazy sequence when sourced from an appropriate lazy source. E.g., readings lines from a file.

In contrast, count is a sink. It realizes each element of the sequence through a chain of operations starting from the source. count is a constant memory sink and can, therefore, consume massive lazy sequences. Other sinks include to_fozen_list, which realizes the sequence into an immutable list of type ScFrozenList.

Once a ScSeq has been realized, it cannot run again. Instead, we can reconstruct the sequence to realize it again. It can be useful to have functions that build a sequence from passed-in source(s) so that we can easily reconstruct the sequence as needed. E.g.,

Returning to the Reddit post example, let’s compute the most popular subreddits in this sample of posts. This is accomplished with the following code.

This introduces a few new scalaps concepts. First, we’re passing a string to map. This is interpreted as “select the attribute of that name for each element”. Similarly, integers are interpreted as integer item lookups in a collection.

Next, we use the method value_counts(). This sink computes an ScDict in which each key is an element from the sequence and the value is the number of times the key occurred. ScDict is an augmented Python dictionary that includes functionality such as returning ScSeqs for keys(), values(), and items(). In the example, we use the items() method to generate a sequence of key/value tuples.

The sequence is then sorted into a ScList using sort_by(key). Note, we’re using the integer 1 as the key so as to select the second element, the count, of each tuple. Hence, the list is now sorted by the number of posts.

reverse() is used to generate a ScSeq that is in the reversed order so that the posts are ordered by descending score. take(n) is used with n=5 to select the first five posts. Lastly, they’re printed through forach(print).

I find this to be a more elegant description of this algorithm than the comparable Python. Do you agree? If not, in your opinion, what would the comparable Python be and why is it more elegant? Let me know in the comment section below.

Lastly, let me leave you with a more sophisticated use of scalaps that computes the frequency of title words in each subreddit.

I won’t explain the full example. Instead, see if you can reason through what the code is doing based upon the naming of the methods and the names of the functions used with them.

I will point out two interesting methods.

• flat_map(func): takes a function that returns sequence for each element in the original sequence. Each returned sequence is expanded, in order, within a returned ScSeq.

• group_by(func): construct an ScDict where the keys are computed by func and each value is an ScList with all the elements that have that same key.

In reading through this example, what are you’re thoughts on the legibility of the Python code with scalaps? I personally find this approach easier to reason through relative to conventional Python. Further, I’ve used such a style in Scala, Java, and (Py)Spark so as to structure my code as applying functions to collections using built-in algorithms. I’ve come to find this approach simpler, more legible, and less error prone relative to conventional imperative programming.

Thanks for considering scalaps! I hope it can help simplify your code, improve the readability, and eliminate errors. Let me know what you think in the comment section below.

Again, you can find the code at https://github.com/matthagy/scalaps. It is a nascent library and very much a work in progress. E.g., it needs tests and I’ll develop them once I get some feedback on the API. PR’s are also welcome.

These examples were derived from a Scala learning resource, Interactively exploring Reddit posts using basic Scala in your browser. Check it out if you’d like to learn more about the elegant and powerful programming language, Scala.

This is part three of our tour through Scala. If you’re just arriving and would like to start at the beginning, checkout Quickly learning the basics of Scala through Structure and Interpretation of Computer Programs examples.

Today we’re going to do some novel programming exercises based around actual Reddit data. I’ve prepared a random sample of roughly ten thousand posts from the month of October 2018 for us to interactively explore by writing basic Scala in widgets within this article.

Here’s a preview of some of the Scala we’ll be writing to analyze Reddit posts.

Let’s dive right in and figure out how to access this data, run computations on it, and thereby analyze Redditors by writing and running Scala code directly in our browser.

The following code fetches the Reddit post data from a web server and shows the first few lines of the Reddit data. There are some new Scala concepts in the code that we won’t need to understand today since we’ll just be using the code to fetch the Reddit data. Instead, we’ll focus on the code we write to explore the data.

Aside: This is a generally cool feature of programming: we can commonly use functions in libraries developed by other programmers to perform computation for us, even when we don’t understand the details of the code in the library. Professional software engineers commonly use libraries in their work specifically so they don’t have to learn the details of how to solve a programming problem that someone else has already solved.

Hit “Run” on the following widget to run the code and see the results.

(Note, you can click “Edit on ScalaFiddle” on any ScalaFiddle widget in this article if you find the ScalaFiddle widget too horizontally compressed in the embedded form. In such cases, you’ll find the presentation on the ScalaFiddle website to be more readable.)

Pretty cool how easily we can access this interesting data, right? Now let’s figure out how to explore the data and compute novel results that provide interesting insights into Redditors.

We receive the data as one very long string. You can modify the Scala code in the previous ScalaFiddle widget to call.length on the string and print the number of characters. I.e., we’ll modify our anonymous function at the bottom of the code to have the following form.

Run that and you’ll see that the data string consists of just under a million characters. Hence, we have some serious data to process here.

Let’s look at the sample again to see how we’ll want to process this long string.

MarioAI,nicolasrene,recreation of Mario facing left since i havent got the original picture not the real thing but he did face left once in 1-2...,1
u_seksualios,seksualios,Seksuali gundanti ištvirkėlė Alektra Blue juodais drabužiais,1
Cuphead,[deleted],Just finished Cuphead in an hour and fifteen minutes.,0
videos,lonemonk,Trump On The Traps - Calvin Dick (2017),1 CryptoCurrency,Pseudoname87,Why does binance show a different price than other sites?,1

This text data is in comma-separated value (CSV) format. Conventionally such data is shown as a table.

We can see that each single line of the file corresponds to a single post. Hence, first, we’ll want to break this string up by lines to access the individual line for each post. We can split a string into substrings — based on a specific splitting character — using the string method split. To split into lines, we’ll use the newline character "\n" as the splitting character.

Lastly, we’ll convert the return value of split to a list using toList and show the number of lines. You can go back to the ScalaFiddle widget and modify our processing code to be as follows.

The results show that we have roughly 11,000 Reddit posts in this data.

Now we still have some more work to do in preparing our data for programmatic exploration because each Reddit submissions contains multiple attributes. Going back to the sample, we see that each line is a comma-separated string of values for each post. In order, the attributes are:

• subreddit: Subreddit to which post was submitted

• author: The user account who posted the submission

• title: The title of the post

• score: The voting score of the submission at the time the data was pulled

We’ll again use String.split to access the individual fields of each post, this time using "," as the splitting character.

We want a way to organize together all of the different attributes of a single post and for that, we can create a Scala class. At a high level, a class just allow us to organize together related data elements like the different attributes of a Reddit post. Here’s how we define a class for this data.

Here’s an example of how we can use the Post class. Modify the code as you like to get a feel for ourPost data type.

We’ll cover classes in more detail in the future. For now, it’s good enough to know that this class defines a new data type to represent Reddit posts, with each post having four attributes: subreddit, author, title, and score.

Let’s now modify our processing code to parse our raw data into a list of posts. I.e., create aList[Post].

Now we have a list of posts in val posts: List[Post]. So let’s start exploring this Reddit post data!

First, let’s count the number of posts for a single subreddit. You can add the following code to our processor function in the previous ScalaFiddle widget.

Here we’re using the filter method ofList to create a new list that just contains the posts that are from the subreddit “AskReddit”. In general, we can use filter with any function we write to select elements of a List and thereby create a new List that just contains those elements of interest.

Feel free to modify the code to count the number of posts in our sample for any subreddit you’re interested in.

Building off that exercise, let’s compute the number of posts for every single subreddit. In this computation, we’ll build a data structure that associates each subreddit with its posts count. This will require us to introduce a new Scala data structure called Map.

Map[K, V] is an association between keys of type K to values of type V. In this exercise, we’ll be building a Map[String, Int] to associate subreddit name to count of posts. Here are some examples of how we can work with such a map.

Can you add code to create forthMap that adds the key "new key" with a value of 12 to our val map: Map[String, Int]?

For people familiar with other programming languages, you may be surprised to see that Map.updated returns another newMap. In general, Scala encourages us to use immutable data structures and therefore we avoid modifying anything in place. Instead, we create new data structures to represent the results of any change. Scala does some really clever things behind the scenes to make such updates efficient in both processing time and memory usage.

Note that Map[K, V].get(key) returns an Option[V]. Option is used to account for the fact that it’s possible we don’t have a value for the given key. Option is a general data type that can take two forms, Some(value) and None. Some(value) denotes that we do have a value for the key and we can access that value through Some.get. Whereas None denotes that there was no value for the key.

Option has a useful method getOrElse(alterantive). When called on Some it returns the value contained in Someand ignores alterantive. Whereas None.getOrElse(alternative) returns the value of alternative. We’ll use this method in our code to replace None with zero when we fetch the count for a subreddit that we currently don’t yet have a count for.

Using Map, here is how we can compute the number of posts for each subreddit.

Note how we’re using foldLeft, a variant of fold, to process each post and increment the count for the corresponding subreddit in the map. You may recall that fold is used to aggregate all values in a list down to a single value. In this case, the final value is aMap[String, Int] that associates subreddits to their count of posts.

A quicker refresher on fold functions: We call foldLeft with our initial value of aggregation and our folding function. fold calls our folding function for every element in the list. In each call, the folding function also receives the current value of the aggregation. Our function returns the updated value of the aggregate that incorporates the list element. The fold functions return the final value of aggregation after processing every element in the list using our folding function.

Spend some time reviewing this code to see if you can reason through how we’re computing the number of posts for each subreddit. As an exercise, can you modify this example code to instead count the number of posts for each user?

While the current results are nice, we’re more interested in knowing the results for the top subreddits; i.e., those with the most posts in this sample of Reddit posts. To that end, we’ll need a way to order the subreddits by the number of posts so that we can select the top few subreddits to show. In computer science terminology, such a process is referred to as sorting.

You can add the following code to the preceding example to sort the subreddits by post count and then show the top 10 in this sample of Reddit posts.

There are a few new things going on here. First, we’re converting subredditCount from Map[String, Int] to List[(String, Int)] using the Map.toList method. This introduces a new concept called tuples in that (String, Int) is the type for a length-two tuple where the first element is a string and the second element is an integer.

Tuples are a general data type in Scala that can be used to represent fixed length collections of elements whereby each position has a fixed type. E.g., (String, String, String, int) is a length-four tuple. We could’ve used this tuple type instead of the class Post to represent the data in a single Reddit post.

In general, classes are a more legible way to group together related elements. Tuples can be useful in some cases, particularly in cases where we want to write generic algorithms that use placeholder types. This is the case in wanting a general method to convert a Map[K, V] in a list of associated pairs, List[(K, V)].

Next, we’re using the method sortBy to sort our list of tuples. The method takes a function that computes a ranking score for each element of the list. The elements of the list are sorted by rank and a new list is returned by sortBy in which the elements are ordered. You can see that our ranking function just fetches the count for each subreddit by accessing the second element of the tuple, t._2.

The results for our sample of Reddit posts are as follows.

(AskReddit,254)
(AutoNewspaper,214)
(The_Donald,84)
(CryptoCurrency,71)
(SteamTradingCards,69)
(RocketLeagueExchange,65)
(newsbotbot,65)
(videos,64)
(GlobalOffensiveTrade,59)
(PewdiepieSubmissions,58)

In thinking about the numbers, we should remember that this is a small random sample of all Reddit posts so the counts are going to be much smaller than the full number of posts. Our sample is 0.1% sample of all Reddit posts in October 2018, so we could multiply these numbers by 1000 to estimate the total number of posts for each subreddit for this month.

With my passing familiarity with Reddit, I’d say these results seem consistent with my intuition about popular subreddits like “AskReddit”. What do you think?

Can you modify your earlier exercise code that computes the number of posts per author so that the results are sorted? Who are the top authors in this sample of Reddit posts?

Next, let’s see if our sample includes any posts with Scala in the title. You can add the following snippet to the previous ScalaFiddle widget to answer this question.

What do you think about these posts? Is every one of them about Scala or is there a deficiency in using this heuristic to programmatically identify relevant posts? We’ll consider more sophisticated ways to analyze posts soon.

What other words are interesting to you? Modify the code as you’d like to look for other posts that have certain keywords. In many ways, we’re building a simple, custom search engine to find posts relevant to our interests across this small sample.

Note, I myself have discovered a non-trivial amount of obscene language. As an exercise, you could write some Scala code to find posts that contain swear words. I’m not including example code for this because I don’t want to have a list of curse words on my blog. :)

In general, we’d be interested in computing the frequency of different words in post titles across each subreddit. Here’s some moderately sophisticated code that accomplishes such an analysis.

Take some time and think through this example. It’s a good review of many of the concepts we’ve considered so far in current and past articles. Reminder, that you can click “Edit on ScalaFiddle” on the widget to open the example in a separate window that doesn’t have the horizontal compression of the widget to better read the code.

The example code includes some concepts we haven’t yet explored. If you’d like, you could explore these concepts on your own — ahead of our shared journey through Scala — using resources at scala-lang.org. In general, that website is a great place to learn about Scala concepts. And, of course, a general Google search can also turn up some useful resources, including StackOverflow questions and answers.

One thing I’d like to explain at present is how this code example uses two cases of using pattern matching to deconstruct data structures. In these cases, we’re accessing the elements of a tuple through pattern matching deconstruction. Here’s an isolated example of deconstructing tuples in a function.

You can see we’ve defined the function in a non-standard fashion. In general, we’d primarily use this pattern in anonymous functions, which is what we’ve done in the word frequency example.

In addition to analyzing the Scala code, what do you think of these results? Do the word frequency results seem appropriate given your knowledge of Reddit? Are there any surprising results in highly frequent words for certain subreddits?

You can modify and extend these examples however you’d like to compute anything that interests you about Reddit posts. Here are some ideas for things you might want to compute for this sample of Reddit posts.

• What posts have the highest score? (Example from intro of this article)

• Which Redditors have the highest average scores?

• Who are the most prolific-posting Redditors in each subreddit?

• What words are frequent in high scoring posts? Versus what words are frequent in low scoring posts?

• Which words have a generally low frequency across all posts, but a high frequency in specific subreddits? I.e., what words are uniquely characteristic of a given subreddit? We can quantify this by taking the ratio of subredditWordFrequency/generalWordFrequency for each subreddit/word pair and looking for high ratios. (This one is a good challenge to further develop your Scala proficiency.)

I hope you’re enjoying applying Scala to analyze Reddit and learn a bit about Redditors. In the future, I’ll be showing you how to perform this analysis on the full set of Reddit posts for the month of October 2018. There are 11,306,843 posts in this month so we’ll need to learn how to apply Scala using the “big data” technology Spark.

You’ll be surprised to see how the code we write in these “big data” exercises is no more complicated than the code we’ve written in today’s exercises. Processing “big data” can be just as easy as small data when we use powerful technologies like Spark and Scala.

And thank you for working through another series of Scala exercises with me. I hope this one has been particularly fun because we’re getting to learn about real-world data. I’ll do what I can to create more exercises with this style.

I’d like to thank pushshift.io for hosting Reddit data dumps. This is a really interesting source of data and it was easy for me to take a small sample from the full dataset for October 2018 for these examples and exercises. You can download this data yourself and write your own Scala code to start performing more sophisticated analyses.

This article builds off of part 1, Quickly learning the basics of Scala through Structure and Interpretation of Computer Programs examples. That article also covers reasons for why you may want to learn Scala.

We continue our exploration of Scala using examples that solve exercises from the classic book, “Structure and Interpretation of Computer Programs” (SICP). Small exercises for the reader are also included.

Today we’re going to start working with some data in that we’re going to learn how to create and processes lists of elements. A list is a simple data structure that consists of a sequence of elements. There are many cases where we’ll need a collection of elements and lists can be a reasonable data structure in many of these cases. E.g., if we wanted to programmatically analyze some tweets from a famous person then we could store their tweets in a list for processing in Scala.

A list is constructed as a linear chain of elements whereby each element points to the next element in the list. Totally reasonable if that doesn’t sound intuitive. I think the concept is best illustrated with an example. The following figure shows a list of three elements; 12, 99, and 37.

Each list element consists of a value and a pointer to the next element. You can see how the first element of the list contains the value 12. This element also points to the second element in the list. The second element contains the value 99 and points to the third element. Finally, the third element of this list contains the value 37 and points to a special “end-of-list” terminal.

In Scala, we call this special terminal element Nil. We need such a terminal value to let us know when the list ends. We can think of Nil as a special element of the list that neither contains a value nor points to another value. It just exists to cap off the end.

So let’s go ahead and build the 12-to-99-to-37-to-Nil list in Scala.

Interestingly, we have to build the list starting from the tail. This makes sense when we reflect on how we’re going to build this list. We can’t create the first element until we have the second one since the first element needs to know what it’s going to point to when we create it. Similarly, we can’t create the second element until the third has been created.

So let’s start by creating the last non-Nil element of the list that contains the value 37 and points to Nil. Because lists are used all through Scala, the list element is given a special, concise name, ::. Yes, the formal name of the list element in Scala is just two colons. While unusual, as you work more with lists, you’ll come to appreciate the novel naming and brevity. Here’s the Scala code for creating a list with just the single element 37.

We’re simply calling :: like a function and this corresponds to creating a new list element of type ::. Each :: element has two attributes:

• head : the value of the list element (e.g., 37 in this example)

• tail : the pointer to the next element in the list (Nil in example)

E.g.,

Note that the Scala shows the single-element list ::(37, Nil) in a special format that downplays the role of individual list elements and just shows the values contained in the list. List(37) is the clearer representation of this list with just this one element. Scala could also show this as ::(37, Nil), but that’ll be harder to interpret when we get to longer lists.

Now let me introduce a generally cool Scala feature that motivates the :: naming convention: ::(head, tail) can also be written as head :: tail. E.g.,

Hence, :: can be thought of as a binary operator in math, similar to say + or -. Do you agree that this infix notation is a simpler syntax than ::(37, Nil)? We can appreciate it more when we see that multiple usages of :: can be chained in creating the rest of the list. Here’s what the code looks like to construct the full list with conventional constructor function calls.

Contrast that to the infix notation.

Would you agree that the infix notation is a more legible way to define the elements that make up a list?

And we can do even better. The List(12, 99, 37) notation shown in the shell is a valid Scala expression to create a list of these three elements in the shown order. This is because List is a function for constructing lists. It can take any number of arguments, including zero. Behind the scenes, it does the work of creating the individual list elements and it returns the start of the list. The List function will be helpful as we create lists to process in today’s exercises.

Lists also introduce a new concept in terms of data types in that we want to be able to differentiate between lists with different types of elements. E.g., differentiating a list of Int values from a list of String values. This is necessary to ensure that all elements in a list conform to the same element type so we can write code that knows how to process all elements in an arbitrary list of a certain element type.

E.g., say we wanted to write a function to compute the sum of the values in a list of integers.

What would happen if we passed in a list of strings? E.g., List("a", "b"). If our code expected integers, then it would encounter an error when it tried to treat strings as integers. To avoid such errors, each Scala list type specifies the type of its list elements. The syntax for specifying a list of type X is List[X]. E.g., List[Int] and List[String].

This allows us to write the following type declaration for the list argument in our function that expects a list of integers.

And thereby Scala ensures that we can never accidentally call sumListIntegers with a list of anything other than integers.

SICP List Exercises

Let’s go ahead and implementsumListIntegers using our understanding of lists. As in part 1 of this series, we’re going to use recursion in place of looping to consider each element in computing the sum of a list.

Let me show you the example code and then explain what it’s doing.

The main strategy is that we recurse on the sub-list formed by removing the first element. E.g., List(12, 99, 37).tail is the list List(99, 37). We then combine together the result of the recursive call with the value of the first element through addition. In turn, the recursive call to sumListIntegers with argument List(99, 37) again makes a recursive call, this time with an argument of List(37). And we’re not done yet! That call, in turn, calls sumListIntegers with the tail of List(37) which is Nil. The recursion then terminates because the empty list Nil has a sum of zero and we can return that.

That’s a lot of words. Let’s consider what the expansion looks like in math/code.

Take some time if you’d like and think through this recursion strategy to process all elements in a list. When you’re ready, you can then try out creating a recursive function to compute the length of a list by filling out the following template. I.e., count the number of elements in a list.

You can test your solution in Scastie. (Note, you have to remove the lines that start with > when running your code. I added those lines to show the expected output for each example input.)

Next, let’s consider a function that constructs a new list from its input list. E.g., let’s write a function to add five to every element of a List[Int]. This can be done recursively with the following code.

Can you follow through how the recursive function calls build a new list that contains the results of adding five to each element of the input list? If not, take some time and work through the individual examples and reason through what the recursion chain looks like. I.e., write out the recursion expansion as we did with sumListIntegers.

Once you feel comfortable with the example, can you generalize the example to be a higher-order function that takes an arbitrary input function, func, in addition to a l: List[Int] and creates a new list that is the result of applying func to every element in the input list? Recall that a higher-order function is a function that takes other functions as input.

Let us call this new function map since it is mapping the function func across every element of the input list to create a new list. Here’s a template to get you started.

Again, you can test your solution in Scastie.

Once you’ve completed that exercise, let me show you a neat and more legible way to recurse over lists using Scala’s match expression. You may recall from the previous article how match allows us to define different conditions for matching a value. Scala then evaluates different code based upon which condition matches.

Here’s a neat way to implement add5toElement using match to determine whether or not we’re dealing with the Nil element.

You can see that we’re matching against the first element in the list. The first case matches Nil and terminates recursion since we’ve reached the end of the list. The second case statement is more interesting. Here we’re using deconstruction to not only match an :: element, but to also unpack its head and tail attributes into the local vals of h and t. We can then use these variables in processing the list element and recursing on the tail.

This starts to hint at some of the power of match. You’ll find many cases in Scala where match can be used to not only match against data but to also concisely access the attributes of a data type like ::. We’ll further explore match in later articles.

Can you modify your map function to use match?

We actually didn’t need to implement map. It’s already built into Scala. Specifically, lists in Scala have a map method (i.e., a function attached to the data structure) that we can use to apply a function to every element in the list, returning a new list of the results of applying the function. E.g.,

It was still a great exercise to implement map to learn about recursion on lists, which is a common pattern in Scala. Nonetheless, you can commonly avoid writing your own recursive functions by leveraging builtin Scala algorithms like List.map.

As we’ll explore more in the future, map can work with lists of any element type as long as you pass in a function that can accept such elements. E.g., it doesn’t just work on List[Int], but it also works on List[String] and any other type of list as long as we have an appropriate function to map across the list. Further, the mapping function can have any return type and we get a list of elements with the return type of the given function.

This type flexibility is possible due to placeholder types in Scala. We’re gonna briefly look at simplified code examples for List that demonstrate this concept. In doing so, we’ll encounter a few parts of Scala that we haven’t yet studied and don’t be surprised if you can’t follow all of the code. I just want to introduce the idea that Scala has generic data structures like List that can hold any type of element and that this is accomplished through placeholder types. Here’s the simplified example code.

No need to understand the details yet. Just see if it makes sense how both ::[A] and NilClass[A] implement the behavior of List[A] in terms of methods head, tail, and map. Further, these data structures can work on any generic type of list values as represented by the placeholder type A.

Only showing this example to hint at the power of Scala’s types. See if you can reason through why a calling map on aList[A] with a function of type f: A => B generates a List[B] where A and B can be any arbitrary Scala types. They can even be the same type.

To reiterate, :: and Nil are two types of List elements that have different behavior. Hence why we can use List[Int] to represent a sequence of :: elements that have Int values and terminate with Nil. We’ll explore this topic of traits and classes more in future articles.

There’s also a builtin list method called fold that can be used to aggregate across all elements in a list. E.g., we could re-implement sumListIntegers as follows.

Note how we call fold with two different parameter lists. The first parameter list provides a starting value of aggregation, in this case, zero. The second parameter list takes a single function that operates on two elements.

Also, this example introduces the concept of an anonymous function in the code(aggregate: Int, element: Int) => aggregate + element. Anonymous functions can be useful to make our code more concise by not needing to define and name every function we write.

The “folding” function will be called multiple times. Each time it receives the current value of aggregation so far, as well as an element of the list. The function computes the new aggregation value, in this case by adding the list element value to the aggregate. This is repeated for every element in the list and results in a final aggregation value that returned by fold.

The name fold corresponds to how we create the recursion chain of form f(f(f(0, 37), 99), 12) for the folding function f with an input of List(12, 99, 37). We’re essentially repeatedly calling f and folding the list down to a single aggregate value through the recursion chain.

FYI, here’s a simplified implementation of fold in Scala. Take a few minutes and see if you can reason through some parts of what’s going on. Totally fine if you find some parts confusing and don’t yet understand them. We’ll be exploring the details of placeholder types and classes in the future.

Both map and fold can be used in many cases to simplify our processing of lists. Further, as we’ll see in future installments of this series, there are other Scala data structures that also implement map and fold methods for processing their elements. Lastly, there are many other useful algorithms implemented in built-in methods of Scala data structures that can simplify our code and we’ll explore them over time.

As always, thanks for following along on our journey through Scala. And please let me know if any part of this article could be made more clear so as to help future readers. Send any feedback to matthew.hagy@gmail.com.

Part three is out and it’s particularly fun in that you get to analyze Reddit data in your browser and learn a bit about Redditors. Checkout Interactively exploring Reddit posts using basic Scala in your browser.

Why Scala is so exciting

I’ve used a lot of programming languages over the last 15 years as an academic researcher, data scientist, and engineer. I’ve recently learned Scala and have found it to be an exceptionally elegant and powerful language. Specifically:

• Scala is a highly legible language that is a pleasure to read.

• Scala code is concise.

• Scala has a powerful type system to catch errors while you’re writing code. The type annotations further add to the legibility in that you know the types of variables for each snippet of code.

• Scala encourages a functional programming style that is more natural to many non-engineers than conventional procedural programming languages.

You can find more details of my thoughts on Scala in my article, “Initial impressions of Scala from a Java and Python data engineer”. Also, you can check out scala-lang.org for an overview of what makes the language great.

In this article, you’ll explore some basic Scala examples so you can see for yourself how simple and elegant the language can be. Yes, Scala is a rich language and ecosystem that has a steep learning curve for people looking to fully master the technology. But I believe anyone can start using Scala with just a basic understanding of the language. One can always learn more advanced Scala features later if they need a more sophisticated language.

You may even appreciate using such a highly scalable language (which is where the name Scala comes from) in that there are always more advanced features to learn when you have a problem appropriate for them. There are even a lot of Open Source libraries that extend the language to make it more powerful in certain domains such as analyzing and processing “big data”.

In this article, we’ll work through some basic programming examples from the legendary “Structure and Interpretation of Computer Programs” (SICP) book. You can run these examples, including modified versions, in your browser using Scastie. We won’t digress on the details of any single example and instead, we’ll explore a range of examples so you can start to see the patterns for yourself. Small exercises are included to help you learn Scala. We’ll again use Scastie for testing out the code we write in exercises.

Hello World

Before diving into SICP, let’s cover the time-honored tradition of showing a Hello World program, which simply prints “Hello World” as output.

We’re simply using the “print line” function. You can copy this code into Scastie to test this out.

Math Expressions

SICP starts by considering some example math expressions.

The format of these math expressions is rather intuitive. Also, if you’ve worked with other programming languages, then you may find the math syntax identical to other languages such as Python, Java, C#, C, and Javascript.

Note that the lines starting with > are not Scala code. I just added those lines to show us the results of each expression.

As an exercise, how would you multiply the numbers 7, 3, and 5 in Scala? Try out your code in your browser using Scastie.

Variables

In Scala, we can create a variable and assign it a value as follows.

Variables can be reassigned.

Variables are useful because they allow us to save the results of computation for reuse. They’re particularly helpful when we want to use the value of an expression in multiple places.

Vals (i.e., constant variables)

In general, Scala encourages us to avoid reassigning variables as that can make programs more complicated. Instead, Scala encourages the use of val variables, which cannot be reassigned.

We should use val in place of var whenever possible to keep our code simple and easy to reason about. Working with Scala will teach us patterns to avoid variable reassignment.

Functions

We’ll commonly want to define computation that can be reused. For that, we use functions. Functions take input parameters and compute a resulting value. E.g.,

The big thing to note in the function definition (i.e., def) is that we have to specify the types for both function parameters and the return value of the function. In this case, the function takes a single integer parameter, x, and returns an integer.

Functions can take more than one input parameter.

In the previous two examples, the function just consisted of a single expression. We can have more complicated functions with multiple statements, including defining local variables.

Brackets are used to group together these multiple statements. Note, that the function evaluates to the value of the last expression and there’s no need for an explicit return statement as in some other programming languages. This example also introduces a new type, Double, that is used to represent numbers with decimal points; in math terms, real numbers.

Function parameters don’t have to be limited to numeric types and can take any data type. For instance, there is also the String type, which consists of text; i.e., a string of characters. Further, functions don’t have to evaluate to a usable value and these functions are declared with Unit return types.

How would you write a function that takes two integers, defines a val that is the sum of the two integers, and then print that value? Here’s a template.

Does your code run in Scastie?

Conditionals

Scala supports if expressions to conditionally evaluate expressions. Unlike more procedural languages where if is a statement (i.e., code that doesn’t evaluate to a value), the Scala if is an expression that evaluates to a value.

Just like functions, you can have a conditional expression that combines multiple statements (e.g., assigning a val) and expressions. Same as functions, brackets are used to create such a compound expression and it evaluates to the value of the last expression.

Scala if expressions can be used to conditionally execute statements with side effects; e.g., printing a value. Note, a side effect is anything that happens outside of our code evaluating to a value, including printing output.

In this example, there is no else clause.

Can you modify the function to be called printClassification that prints both large and small numbers, with a suffix that says either “is large enough” or “is too small”?

Conditionals can be nested as shown in the following example for SICP abs function for computing the absolute value of an integer.

Recursive Functions

Sometimes we just want to repeat something in code. For example, printing something out n times or summing the integers from a to b. Historically, many programming languages used something called loops to repeat code. This coding practice is called procedural or imperative coding and was the dominant trend in mainstream programming languages until recently.

In contrast, a common pattern in functional programming languages, including Scala, consists of using function recursion in place of looping. I.e., functions that call themselves. We can see how this is applied in Scala with the classic factorial example.

This example also introduces the concept of throwing exceptions. We won’t cover the details yet. Just know it’s a way for our code to handle erroneous situations like passing an invalid input to a function.

Can you write a function called sumUpTo(x: Int) that uses recursion to sum all integers from zero up to x (inclusive)?

Again, you can test your solution in Scastie.

An aside on basic pattern matching

I’d like to take a moment to show how the Scala match construct can be used to implement the conditional logic of the factorial function.

Here we’re using the match construct to evaluate different logic based upon the value of x. Further, you can see we’re not explicitly handling the negative value case. Instead, negative inputs result in throwing a MatchError because none of the defined case statements match that input. Also, note how we introduced the val xp that is only operated on when its value is positive.

This is a basic usage of match and it can do much more powerful things. We’ll consider more advanced usage of match in future articles.

Higher Order Functions

As the last part of this introduction to Scala, let’s explore higher-order functions. These are functions that take other functions as arguments. Consider the following example where we want to call a function, func, exactly n times using a function callFunctionNTimes.

You can see that the func argument of callFunctionNTimes is defined as (String) => Unit. This means that func is a function that takes a single String argument and has no return value. We can pass in any function that takes this form, including both println and printTwice.

Can you write a higher-order function of the form: callForIntegersInRange(func: (Int) => Unit, start: Int, end: Int): Unit

Here’s a template.

Again, you can test your solution in Scastie.

Aside: Is it interesting that we can treat println as both a (String) => Unit function and a(Int) => Unit function? We’ll explore how Scala allows us to do this later in this series when we explore a concept called generic types. The short answer is thatprintln is defined as a function of type (Any) => Unit and both String and Int can be treated as Any type.

Closing Remarks

Thank you for exploring Scala with me! I hope you’ve enjoyed learning a bit about Scala and are excited to learn more. If any parts have been confusing, please let me know and I’ll revise those parts. Further, if there are parts you really liked, let me know and I’ll try to preserve that style in the future. Let me know what you think at matthew.hagy@gmail.com.

When you’re ready to continue on, the next installment in our series is A brief tour of lists in Scala and algorithmically processing them in SICP exercises.

📝 Read this story later in Journal.

🗞 Wake up every Sunday morning to the week’s most noteworthy Tech stories, opinions, and news waiting in your inbox: Get the noteworthy newsletter >

The Spark GraphFrame is a powerful abstraction for processing large graphs using distributed computing. It provides a plethora of common graph algorithms including label propagation and PageRank. Further, it provides the foundations for implementing complex graph algorithms, including a robust implementation of the Pregel paradigm for graph processing. Anyone who’s interested in working with large graphs should learn how to apply this powerful tool.

In this article, I’ll introduce you to the basic of GraphFrame and demonstrate how to use this tool through several examples. These examples consider the link graph between Wikipedia articles and I demonstrate how to analyze this graph by leveraging the GraphFrame abstraction.

Note that the examples in this post build off some more elementary Spark concepts such a DataFrames. Additionally, it uses basic Scala code to demonstrate algorithms. Only small and simple examples are shown so that one doesn’t need to be well-familiar with these concepts to learn about the power of the GraphFrame abstraction.

Let’s dive right in and consider how to create a GraphFrame. I’ll start by introducing the Wikipedia link data in a basic Spark RDD. Each element of this RDD is a single Wikipedia article page represented with the following Scala class.

Note that each link is the title of another page. I.e., each page knows all the other pages that it links to by title.

We begin with a single RDD pages that contains 19,093,693 Wikipedia article pages. From that, we generate two Spark DataFrames, one consisting of the vertices (i.e., page nodes) and the other consisting of the directed edges.

Note that there are 206,181,091 directed edges in this graph.

Next, we create a GraphFrame using these two DataFrames.

And that’s all we have to do to access this powerful abstraction. We can now start using some of the builtin graph algorithms to analyze the Wikipedia link graph.

Let start by computing something simple: the Wikipedia pages with the largest number of outbound links. To this end, we can use the GraphFrame method outDegrees, which is a computed DataFrame that corresponds to the number of outbound edges for each vertex. Since it’s a DataFrame, we can use the orderBy method and limit to select the top 10.

This gives the following results.

Interestingly, we can see that many special “Wikipedia:”-prefix pages have the highest number of outbound links.

Next, let’s consider the number of inbound edges for each page and find the top 10 linked-to pages using the corresponding GraphFrame method, inDegrees.

We can see that locations, including countries and cities, are among the most heavily linked to pages.

Now let’s explore some more complex graph computations. Let’s consider the most heavily-linked to article, “United States”, and find the other articles in the link graph that are furthest away from this article in terms of the number of links you have to follow to arrive at “United States”.

To this end, we can use the GraphFrame method, shortestPaths, which takes a collection of landmark vertices and returns the path length for every vertex in the graph to every landmark vertex.

It is interesting that there are only three articles that are a full 10 links removed from the “United States” article in terms of shortest path length.

Let’s go one step further and compute the number of pages at each path length. I.e., how many Wikipedia articles are 1 link removed from “United States”, how many are 2 links removed, etc.

(Note, that I’m using the built-in display function of the Databricks notebook. I’ll talk more about the joy of using Databricks in a future post.)

Here we can see that there are millions of pages that are between 2 and 4 links removed from the “United States” article. There are also a vanishingly small number of articles that are six or more links removed. Further, the -1 path length denotes articles that aren’t at all connected to the “United States” article at all and there are roughly a million pages that meet this criterion.

Lastly, no demonstration of GraphFrame would be complete without showing how easily it can be used to perform the PageRank algorithm.

The page rank of the top 10 articles is:

It may not be surprising that “United States” has the highest vertex rank as we know it’s the most heavily linked-to article.

I hope these examples have helped to convince you that GraphFrame is a powerful abstraction for performing distributed computation on large graphs. There are additional more advanced concepts that I hope to share in future articles, including how to implement custom graph algorithms using the Pregel compute paradigm with GraphFrame.

Update: If you’ve found the Scala examples in this article interesting and would like to learn more about this powerful programming language you can check out my article, Quickly learning the basics of Scala through Structure and Interpretation of Computer Programs examples (Part 1).

I ‘ve found that few people list interviewing as their favorite work responsibility, and I myself have never been delighted to see my week fill up with interviews.

Nonetheless, I’ve tried to make an art-and-science out of interviewing. By reviewing the subsequent outcome of candidates that I’ve interviewed, I’ve been able to pick up on some patterns in what made a candidate successfully earn an offer. While professional proficiency is certainly a requirement, I’ve found it to be well correlated with thoughtfulness and further believe thoughtfulness to be a generally valuable habit in identifying higher performing professionals.

What do I mean by thoughtfulness? My favorite example question is a great warmup to open any interview:

What’s important to you in a new role?

Strong candidates almost always have a clear and concise explanation of what they want in a future role. Further, their expectations are well aligned with our job posting, which shows they thought a bit about which companies to apply to.

When people are leaving a superficially similar role, such as software engineer applying for an engineering role, I ask them:

What would you like to be different in a future role relative to your current role?

Again, strong candidates have generally thought this through. Maybe they’re looking to learn and apply a specific, more powerful technology stack so that they can make bigger contributions and be a stronger professional. Other times they’re looking for a change in company size or industry and they can well explain what they want to be different in their future employer.

Weaker preliminary answers would just consist of generic reasons. E.g., “I want to work with X”, where X is something very generic like “big data”. I would always dig into these answers to try to better understand what parts of X they’re interested in and what skills they want to learn. It’s great when they’ve already started to learn some of these skills (e.g., reading a book or attending a course) and can talk intelligently about the problem space at an initial level of sophistication.

At times, I’ve had candidates think through some things over the course of a few minutes and come up with decent preliminary thoughts. This may show that they’re early in their career search and career planning, but they’re willing to give it serious consideration. They may have planned on doing more thinking before accepting any offer and are looking to first gain info on interesting potential employers. In my opinion, this is forgivable, especially when someone is early in their career.

The lower-end-performing candidates are those that just wanna get through this phase quickly, using only generic answers, and get on through the interview. Commonly, these candidates will go on to also rush through the technical exercises with generic “pattern matching” and without thinking through the problem.

I return to thoughtfulness again in the interview when I ask candidates to explain their choices. E.g., “why would you want to use Spark for data engineering versus say Hadoop MapReduce?” If someone has years of experience with this area, then I would hope they could clearly explain why they developed their opinion. It’s great when they can describe alternative approaches and explain why they find them inadequate. I can even accept, “That’s a company/industry standard that I have yet to reflect on” when they still demonstrate they understand the domain well.

Lastly, in my own professional development, I’ve found thoughtfulness in reflecting on my strengths and weaknesses and how I want to develop, to greatly aid in my growth as a professional. I even feel more confident in tackling larger, new challenges when I’ve put a little thought into why I’m excited to do this new responsibility.

One might hypothesize that professionals who make a habit of thoughtfullness simply grow into more well-developed professionals. And it’s a habit we can all develop. I certainly have found myself growing more thoughtful about my profession with time and I’d hope we can all continue to grow along this dimension.

Thank you for considering my thoughts. What do you think? Do you believe thoughtfulness is an important attribute to interview for? If so, how do you interview for it? Let me know in the comment section below.

I truly want to do more with SQL. It seems like such a simple and elegant way to represent data joins, filtering, and basic transformations in a concise format. In many ways, I feel SQL is closer to a mathematical way of defining computation (i.e., a functional, LISPy, declarative approach) than conventional imperative code, which is a plus for SQL in my opinion. I particularly treasure short SQL queries that can quickly answer random ad hoc questions such as the following.

Further, I find many people who don’t identify as highly technical can learn basic SQL quickly and if necessary master it well enough to perform serious data analysis and transformations. As a data scientist and engineer, I find it great to be able to work with a larger set of stakeholders in thinking through the business logic we want to implement in computing stats or processing data. We simply access more people’s critiques and ideas when non-engineers can work with the logic.

But I always find SQL inadequate once the complexity grows beyond about a dozen lines of code in that I immediately start worrying about correctness and legibility. I’d much rather implement the transformation in Spark with DataFrames using Scala (or if necessary PySpark using Python). I can then break my logic down into simple well-named functions and rigorously unit test each function. Simple integration tests can be used to verify that everything is connected together correctly. All around, I’m simply more confident in the correctness of a Scala solution and feel more capable in refining and maintaining the code in the future.

In many ways, I wish it didn’t have to be that way with SQL. I’d really like a great way to extract arbitrary SQL expression into functions and further have a great environment for testing SQL. I have yet to find anything that works for a range of SQL dialects and is also open source. Let me know if there is a good tool I should be looking into.

I further find SQL to be a difficult language for expressing certain complex logic. For example, parsing JSON and then extracting and transforming specific fields. I know there exist solutions for loading nested JSON into SQL engines for subsequent processing, but I’d rather write a Spark workflow that just directly reads the raw SQL data and perform the parsing and transformations in Scala.

Further, I’ve found myself having to implement complex business logic to perform various analysis and transformations (particularly for feature engineering) that just couldn’t be written in SQL. Or at least it wasn’t straightforward to implement in a conventional SQL approach. Sometimes I’ve found hacky solutions, but they don’t seem like a good representation of the logic we want to develop and maintain. Here’s an example of a Spark workflow that I’d struggle to implement elegantly in SQL.

Can you think of a good way to implement this logic in SQL?

Traditionally, user-defined functions (UDFs) would be developed in a conventional programming language to extend SQL capabilities. I find that solution inelegant in that we’re now mixing two languages. Further, there are limits to what can be implemented in UDFs (and user-defined aggregates, UDAs). Nowadays, I think it’s just better to use (Py)Spark and define the custom logic alongside the data processing definitions.

Lastly, I really like types in any programming language. I think types convey so much information not only to the compiler but also to people reading the code. I find well-named functions, objects, and members —with type annotations — provide a great level of self-documentation in code. Combine that with some small comments and concise external documentation, and you end up with highly legible code that anyone familiar with the language can learn and work with.

Scala has an excellent type system and even modern Python has optional type annotations. While most SQL dialects support some level of types in stored procedures and functions, I’ve found this limited to basic types and therefore can’t be used to the same extent as type annotations in a robust programming language.

I just don’t see a path forward to getting SQL to the same quality as a Spark, DataFrame, and Scala solution. Yet, I recognize the need for data processing logic to be legible and modifiable to more than just engineers. I hypothesize that Spark, DataFrames, and basic Scala (SDS) could be used to that end.

I’m not proposing that non-engineers be able to read, reason about, and modify a full production Scala engineering system, but instead that they can work with small code snippets at a complexity level comparable to SQL. Further, DataFrame operations can build off their current understanding of SQL operations such as joins, filtering, and mapping transformations. They simply need only to learn the new syntax for implementing these familiar operations. Over time, they can learn more Scala to implement more complex logic when their work requires it.

To further facilitate this education, I’d like to develop a small tutorial of transferring SQL concepts to the SDS stack. Further, to really sell the materials, I’d like to also develop a cool SQL->SDS transpiler. People could use this to see how their current SQL logic is transformed into this other tech stack. Combined with a snazzy web app, people could immediately start learning the equivalent SDS code for their work. This would allow people familiar with SQL to start learning the specific SDS knowledge they need to start solving problems.

Such education synergizes with the development of cloud solutions for hosting Spark (e.g., AWS EMR, GCP Dataproc, Databricks, Cloudera Altus Cloud) such that more people can now easily access a Spark environment. Over time, maybe we could even get AWS and GCP to create ad hoc managed Spark environments comparable to the SQL solutions of Athena and Big Query. Overall, I anticipate more and more professionals are going to have access to Spark environments when they want them.

I could even see data scientists transitioning to more complicated data engineering work when they need to build something to facilitate their data science work. Similarly, product managers may find themselves coding up complex business logic in Scala because it’s simply harder to write up the logic in English. If we teach these SQL-to-Scala converts how to modularize code and write tests, we may soon find ourselves with an army of situational engineers that can implement parts of systems important to their work.

What do you think? Do you believe Spark/DataFrames/Scala (SDS) can replace SQL for a growing number of non-engineers? Could it even help us all get more engineering work done, which all data companies are constantly looking to do? Let me know your thoughts in the comment section below because I’d like to collect feedback on this idea before investing too much time in helping SQL aficionados transition to SDS.

If you’re interested in collaborating on SQL-to-SDS tutorials and/or the transpiler, drop me a message at matthew.hagy@gmail.com. Also, feel free to beat me to it! I’ll gladly help develop and promote however I can.

Update: If you’ve found the Scala examples in this article interesting and would like to learn more about this powerful programming language you can check out my article, Quickly learning the basics of Scala through Structure and Interpretation of Computer Programs examples (Part 1).