Codestin Search App

Introduction

Scala 3 is pretty great already. Even so, one of the vastly underrated features of Scala is the ability to do type-safe, compile-time metaprogramming, which allows us to generate, expand, synthesize, or otherwise manipulate code at compile time, while preserving the full expressiveness of the language and the strictness of the type system.

Metaprogramming certainly isn’t new: Lisp derivatives allow the manipulation of code while it’s being interpreted due to the structure of the code — which is the same as that for data, so it makes things much easier — while other languages offer various forms of text expanders, text processors, compiler pluginsm, quasi-quotes or other hacks to inject code into already running (or compiling) code.

However, Scala is unique in that we can use the same language, the same data structures, the same standard library, to turn correct code into other correct code. If you want the best of both worlds (safety of the type system with the homoiconic representation of code as data), then Scala is the closest to perfection.

This article is about Scala 3 macros, the most general form of metaprogramming in Scala. It follows the previous one on inlines, so it would be great if you check that out first. It’s optional, though: all you need to know about inlines for the purpose of this article is that

inline methods replace all their invocations with their body at call site
inline arguments replace all their uses in the method body with the expression passed at call site

Setup

If you want to follow the code (highly recommended!), all you need is a plain Scala project with the following compiler flags added in build.sbt:

1
ThisBuild / scalacOptions ++= Seq(
2
  "-Xprint:postInlining",
3
  "-Xmax-inlines:100000"
4
)

Because we’re going to deal with inlines a lot, we’ll need to keep track of what final code the compiler arrives at. I’ve also added an irresponsibly large max-inlines limit, just in case we get into recursive expansions and other undesired things.

Abstract Syntax Trees (ASTs)

Let’s forget Scala for a moment, and let’s imagine a much simpler “programming language”: a scientific calculator.

In a scientific calculator, you can type arbitrarily complex (but well formed) mathematical expressions, you hit ”=” (or “run”) and the calculator would produce a result. In order for the calculator to return a final value, though, it needs to turn a mathematical expression into some data structure that can be predictably evaluated according to the rules of maths. Let’s imagine the sort of data structures that we can use to describe the possible operations the calculator can perform:

1
trait Expr
2
case class Num(value: Double) extends Expr
3
case class Sum(lhs: Expr, rhs: Expr) extends Expr
4
case class Sub(lhs: Expr, rhs: Expr) extends Expr
5
case class Mul(lhs: Expr, rhs: Expr) extends Expr
6
case class Div(lhs: Expr, rhs: Expr) extends Expr
7
case class Sin(expr: Expr) extends Expr
8
case class Cos(expr: Expr) extends Expr
9
// ... and everything else

If we define something like this, then an expression of the sort 2 + 3 / 4 + 2 * 8 * sin(30) can be turned into an instance of Expr, as follows;

1
val computation =
2
    Sum(
3
        Num(2),
4
        Sum(
5
            Div(Num(3), Num(4)),
6
            Mul(
7
                Num(2),
8
                Mul(
9
                    Num(8),
10
                    Sin(Num(30))
11
                )
12
            )
13
        )
14
    )

This data structure makes things predictable. We can even write an algorithm to evaluate this data structure:

1
def evaluate(expr: Expr): Double = expr match {
2
    case Num(v) => v
3
    case Sum(lhs, rhs) => evaluate(lhs) + evaluate(rhs)
4
    case Sub(lhs, rhs) => evaluate(lhs) - evaluate(rhs)
5
    case Mul(lhs, rhs) => evaluate(lhs) * evaluate(rhs)
6
    case Div(lhs, rhs) => evaluate(lhs) / evaluate(rhs)
7
    case Sin(expr) => Math.sin(evaluate(expr)) // or whatever the low-level implementation is
8
    // ... and everything else
9
}

Of course, the act of parsing the “natural” mathematical expression into this Expr thing is the most important point. I would even go as far as to say that the Expr data structure is the “more natural” form of the mathematical expression than the stuff we write on paper, but I digress.

The Expr data type is called an abstract syntax tree, or AST for short. Evaluating a math expression is therefore a two-part process:

turn the math expression (syntactically correct) into an AST
evaluate the AST

The compilation process of a real programming language is not too dissimilar:

take the piece of text we write (and by “we” I also include the LLM assistants)
turn that text into an AST
turn that AST into a binary

This high-level compilation process has been essentially unchanged since the first compiler. The details make all the difference, obviously. The Scala compiler has several dozen phases for specific parts of this process, but the principle stays the same. Of course, there is a much higher variety of AST nodes that we have available, because a programming language is that much more complex and we need to account for everything:

“import” statements
class/interface/enum/object definitions
fields, methods, abstract types, type restrictions
generics
arguments
value definitions
method calls
the “regular” math-style expressions
and everything else

What is metaprogramming, then?

For Scala specifically, metaprogramming means the manipulation of ASTs and reinjecting ASTs back into well-typed code, so you can use their results. This sounds straightforward, but it’s all but impossible. The reason is that once you get your AST, you aren’t in “code land” amymore and you lose access to the host language. How can you access an AST from source code, when the source code is the one that produces the AST?

In Scala, this is possible. We can do two things:

turn an expression into an AST, which is called quoting
turn an AST back into an expression, which is called splicing

Let’s see how that looks like.

The First Scala 3 Macros

A macro always has both parts:

build an AST, usually by quoting expressions in the code
splicing that AST

We cannot quote expressions or otherwise use ASTs as we would use other data structures, in our regular code. Even though we can use the Scala language to its fullest and create classes, methods, values, and everything else, the objective is always to splice a final AST and go back to regular types.

Let’s explore a “hello world” of macros:

1
import scala.quoted.*
2

3
inline def firstMacro(number: Int, string: String): String =
4
    ${ firstMacroImpl('number, 'string) }
5

6
def firstMacroImpl(numExpr: Expr[Int], strExpr: Expr[String])(using Quotes): Expr[String] =
7
    Expr("This is my first macro")

There’s already a lot to unpack. Let’s take everything in turn.

Firstly, all macros are necessarily inline methods. Our objective is to manipulate ASTs at compile time and inject them (or rather, what they represent) back into the code, at compile time. Therefore, the inline keyword is compulsory to allow the compiler to run the macro implementation.

Secondly, the implementation of the macro is always the combination of both steps described above. We need to build the AST, then splice it. So if the act of building the AST was buildAST and the act of splicing was a fictitious method splice, then the value returned by the method would be something like

1
splice(buildAST(myParam1, myParam2, ...))

In our case, the fictitious splice method is actually separate syntax: we write ${ myAST } instead of splice(myAST), which means that our macro implementation is written as

1
${ buildAST(myParam1, myParam2, ...) }

Usually, if we write a macro called myMacro, the function that builds the AST is called myMacroImpl. This is a naming convention that you’ll see everywhere in macro-based libraries in Scala. So our function is now

1
${ firstMacroImpl(myParam1, myParam2, ...) }

What about the arguments? We said that the whole point of a macro implementation is to manipulate ASTs. We can build ASTs out of thin air, but we often need to transform ASTs into other ASTs, which is why the macro implementation function usually takes ASTs as arguments. As mentioned earlier, we can build ASTs out of regular code by quoting. To quote an expression, we put a single quote before the expression. So if we want to quote our number parameter of the firstMacro function, we turn it into an AST by writing 'number. So if we want to take both our arguments as ASTs and pass them on to the macro implementation function, we’ll arrive at our final expression

1
${ firstMacroImpl('number, 'string) }

That’s the first part. Let’s go to the next part, and specifically to its signature. By quoting an expression of type A, we obtain a typed AST described by the type Expr[A], in a similar style to the toy programming language of the scientific calculator, only that we keep the types (this is Scala, after all!). The goal of the method is to obtain another AST which the main macro will need in order to splice, so we’ll need to return an Expr[SomethingElse]. Because we said that the main macro returns a String, we need the macro impl to return and Expr[String]. Therefore, our macro impl has the signature

1
def firstMacroImpl(numExpr: Expr[Int], strExpr: Expr[String]): Expr[String]

In order to use the Expr type and the macros API, we need the import scala.quoted.*. It’s the only import you need to have (or remember).

Okay, what about that Quotes thing? The Quotes is the one giving us the API to build ASTs, transform them, synthesize types, values, or everything else. Besides that, the Quotes instance has a special package of “reflection” which allows us to inspect the properties of the expressions or types in question, as well as create arbitrarily complex ASTs manually, as we will see a bit later. Therefore, the complete signature will be

1
def firstMacroImpl(numExpr: Expr[Int], strExpr: Expr[String])(using Quotes): Expr[String]

and the implementation is a simple expression: Expr("This is my first macro"), which means that when spliced, the compiler will produce the value “This is my first macro”.

Okay, how do we use the macro?

Due to how macros are compiled, we can’t use the macros in the same file where they were defined. That’s never a big deal, since we usually use macros written in libraries, so we naturally separate their use from their definition.

So if the macro was written in a file

1
package com.rockthejvm.macros
2

3
import scala.quoted.*
4

5
class MacrosDemo {
6
    inline def firstMacro(number: Int, string: String): String =
7
        ${ firstMacroImpl('number, 'string) }
8

9
    def firstMacroImpl(numExpr: Expr[Int], strExpr: Expr[String])(using Quotes): Expr[String] =
10
        Expr("This is my first macro")
11
}

Then in another file, we can use the macro

1
package com.rockthejvm.macros
2

3
import MacrosDemo.*
4

5
object MacrosDemoUsage {
6
    val firstMacroCall = firstMacro(42, "Scala")
7
}

Running an SBT console and hitting ~compile, we can keep track of what the compiler expanded, especially in the usage file. In our case, we see something like this:

1
[info] package com.rockthejvm.macros {
2
[info]   import com.rockthejvm.macros.MacrosDemo.*
3
[info]   final lazy module val MacrosDemoUsage: com.rockthejvm.macros.MacrosDemoUsage
4
[info]      = new com.rockthejvm.macros.MacrosDemoUsage()
5
[info]   @SourceFile("src/main/scala/com/rockthejvm/macros/MacrosDemoUsage.scala")
6
[info]     final module class MacrosDemoUsage() extends Object() {
7
[info]     this: com.rockthejvm.macros.MacrosDemoUsage.type =>
8
[info]     private def writeReplace(): AnyRef =
9
[info]       new scala.runtime.ModuleSerializationProxy(
10
[info]         classOf[com.rockthejvm.macros.MacrosDemoUsage.type])
11
[info]     val firstMacroUsage: String = "This is my first macro":String
12
[info]   }

Look at that last line. The compiler computed that

the value of firstMacroUsage is “This is my first macro”, as returned by the macro function
the type of that value is String

In other words, the macro is computed at compile time. It must be so, because that’s the whole point: to build ASTs and inject them into our regular code.

Congratulations: you’ve written your first macro, and used it for the first time!

Inside a macro implementation, we can run arbitrary computations. Let’s consider one: if the number is bigger than 3, then repeat the string n times, otherwise take n / 2 characters from that string. A toy example, but it makes a point and raises a question: how can we get the value of an Expr?

ASTs of the type Expr[A] may or may not have a value, in the sense that the compiler may or may not know (at compile time, of course) what value of the expression was originally. For example, literal strings, numbers or simple math operations like 2 + 3 are knowable by the compiler. An expressions like getMeaningOfLife() is not, even though we know it’s 42 and the method returns it as such.

The macro API allows us to try to get the value of an Expr, or trigger a compiler error if that value is not known. The code with our toy logic therefore becomes

1
def firstMacroImpl(numAST: Expr[Int], stringAST: Expr[String])(using Quotes): Expr[String] = {
2
  val numValue    = numAST.valueOrAbort // will trigger a compile error if this is not computable at compile time
3
  val stringValue = stringAST.valueOrAbort
4
  val newString =
5
    if (numValue > 3) stringValue.repeat(numValue)
6
    else stringValue.take(numValue / 2)
7

8
  Expr("The macro impl is: " + newString)
9
}

Compiling this again gives us the following output in SBT:

1
[info] [[syntax trees at end of              postInlining]] // /Users/daniel/dev/rockthejvm/blog-projects/scala-macros-demo/src/main/scala/com/rockthejvm/macros/MacrosDemoUsage.scala
2
[info] package com.rockthejvm.macros {
3
[info]   import com.rockthejvm.macros.MacrosDemo.*
4
[info]   final lazy module val MacrosDemoUsage: com.rockthejvm.macros.MacrosDemoUsage
5
[info]      = new com.rockthejvm.macros.MacrosDemoUsage()
6
[info]   @SourceFile("src/main/scala/com/rockthejvm/macros/MacrosDemoUsage.scala")
7
[info]     final module class MacrosDemoUsage() extends Object() {
8
[info]     this: com.rockthejvm.macros.MacrosDemoUsage.type =>
9
[info]     private def writeReplace(): AnyRef =
10
[info]       new scala.runtime.ModuleSerializationProxy(
11
[info]         classOf[com.rockthejvm.macros.MacrosDemoUsage.type])
12
[info]     val firstMacroUsage: String =
13
[info]       "The macro impl is: ScalaScalaScalaScalaScala":String
14
[info]   }
15
[info] }

The last line is what we want to see: the compiler ran the logic of the macro at compile time and we get what we expect.

Worth repeating: we can run arbitrary code in a macro. This is insanely powerful. An obvious implication: the more complex the computations in your macros, the longer your compile time.

Pattern Matching Expressions

In real life, the expressions we want to process and the ASTs we want to manipulate are often quite big. The Scala 3 macros API allows us, for example, to pattern match expressions and extract pieces that we can manipulate.

Let’s consider a simple example. Let’s describe an Option at compile time, depending on how that Option was built:

1
inline def pmOptions(inline opt: Option[Int]) =
2
    ${ pmOptionsImpl('opt) }
3

4
def pmOptionsImpl(opt: Expr[Option[Int]])(using Quotes): Expr[String] = ???

The calculator example was pretty easy because we can pattern match the case classes of the possible values of the expression being evaluated. Scala being Scala, we can pattern match much more complex things and extract sub-expressions with quote patterns.

Before we look at the implementation, please note that the opt parameter is marked inline. The reason is that we want access to the full expression in the macro implementation. With that out of the way, let’s look at the impl.

Here’s an example for our pmOptionsImpl:

1
def pmOptionsImpl(opt: Expr[Option[Int]])(using Quotes): Expr[String] = {
2
  val result = opt match {
3
    case '{ Some(42) }                   => "got the meaning of life"
4
    case '{ Some($x) }                   => s"got a variable: ${x.show}"
5
    case '{ ($o: Option[a]).map[b]($f) } => "mapping an option"
6
    case _                               => "got something else"
7
  }
8
  Expr(result)
9
}

Whoa.

Let’s look at the syntax first. We can pattern match an Expr[A] with a case that is of the structure '{ }, and inside the {} you can write (almost) the original code that made that expression. Our first case is '{ Some(42) }, so if we use pmOptions(Some(42)), that case will match.

The second case contains an expression variable. The rules for pattern matching here are almost identical to the rules for pattern matching regular values, except we use $variable to capture the expression. We can also call expression.show to display the code this expression corresponds to. This is useful for our SBT-based verbose session.

The third case is pretty wild. We match expressions of the kind Option(x).map(f) if they occurred exactly like that in the code. We isolate the first expression Option[a], where a is a type variable that the compiler can figure out. Then we can “call” .map on it, which is not a method call per se, but rather a pattern. We add another type variable b and extract the function expression $f, if we need them. There are many things that we can do inside quoted cases: add type variables and restrictions, extract sub-expressions, match generic types, enforce the equality of types in complex chains (for example, match a chained .map only if the resulting type is the same every time). We have a few dedicated lessons on what you can do with quoted matches in the Scala macros course.

Let’s use it now - in the usage file, add the following

1
val optionDescription   = pmOptions(Some(2))
2
val optionDescription_2 = pmOptions(Option(2))
3
val optionDescription_3 = pmOptions(Option(10).map(_ + 1))

We get the following output (as before, the last lines are the important ones):

1
[info] package com.rockthejvm.macros {
2
[info]   import com.rockthejvm.macros.MacrosDemo.*
3
[info]   final lazy module val MacrosDemoUsage: com.rockthejvm.macros.MacrosDemoUsage
4
[info]      = new com.rockthejvm.macros.MacrosDemoUsage()
5
[info]   @SourceFile("src/main/scala/com/rockthejvm/macros/MacrosDemoUsage.scala")
6
[info]     final module class MacrosDemoUsage() extends Object() {
7
[info]     this: com.rockthejvm.macros.MacrosDemoUsage.type =>
8
[info]     private def writeReplace(): AnyRef =
9
[info]       new scala.runtime.ModuleSerializationProxy(
10
[info]         classOf[com.rockthejvm.macros.MacrosDemoUsage.type])
11
[info]     val firstMacroUsage: String =
12
[info]       "The macro impl is: ScalaScalaScalaScalaScala":String
13
[info]     val optionDescription: String = "got a variable: 2":String
14
[info]     val optionDescription_2: String = "got something else":String
15
[info]     val optionDescription_3: String = "mapping an option":String
16
[info]   }
17
[info] }

So we see how the macro was evaluated at compile time in all cases, including the map structure. Quoted matches are incredibly powerful.

But notice something: the Option(2) was not matched by any case! This is important: even though Some(2) and Option(2) produce equal objects at runtime, they are not the same expression! The code is what we care about, these bits of code are different, and the compiler (and the pattern match) knows it.

Compile-Time, Type-Safe Reflection

There is a lot to talk about macros, but one other example worth mentioning is the ability to synthesize arbitrary ASTs. This is the subject of “reflection”.

When we say “reflection”, we generally think of the Java API that allows us to inspect fields and methods at runtime, crash the program if they don’t exist, add annotations that create all sorts of magic dependency injection for us in the style of Spring, and so on. In our case, “reflection” means that we can inspect and manipulate ASTs dynamically but at compile time and with all type safety intact.

Let’s consider a classical use-case of reflection: calling a method by a string name.

We’re now familiar with the macro pattern, so we can start the macro entry point:

1
inline def callMethodDynamically[A](instance: A, methodName: String, arg: Int): String =
2
    ${ callMethodDynamicallyImpl('instance, 'methodName, 'arg) }

By calling this method, we would like the compiler to find the method given by methodName and invoke it on the argument arg. Of course, at compile time. So if we create some class (let’s add it in the usage file)

1
case class SimpleWrapper(x: Int) {
2
    def magicMethod(y: Int) =
3
        s"This simple wrapper called a magic returning ${x + y}"
4
}

Then by creating a value

1
val magicCall = callMethodDynamically(SimpleWrapper(2), "magicMethod", 10)

we would like the compiler to turn that into

1
val magicCall = SimpleWrapper(2).magicMethod(10)

and of course, we want the compiler not to compile the code if the method in question does not exist on that class.

In order to build arbitrary ASTs, we will look at something new. If Expr[A] is a typed expression in the code that returns a value of type A when injected (and run) back in the code, we cannot build arbitrary Expr[A] manually, but only by quoting them from code. An arbitrary AST may be an expression returning a value, but not necessarily, because expressions producing values are not the whole language. Type definitions, fields, arguments, references to variables, these are examples of ASTs that must be represented, but that may not produce values per se. The general type of an AST is called Term.

Producing arbitrary ASTs means producing Terms and manipulating Terms. The final result will be an Expr[Something], which we produce after the careful handling of Terms.

For our “dynamic” method call, we need to produce a Term signifying a method invocation, and once the Term is fully built, we need to convert it back to an Expr[String] in our case.

Let’s build the impl step by step:

1
def callMethodDynamicallyImpl[A](instance: Expr[A], methodName: Expr[String], arg: Expr[Int])(using
2
    q: Quotes
3
): Expr[String] = {
4
    import q.reflect.*
5

6
    // TODO
7
}

We have some changes already. We have our given Quotes named now, because we need to import q.reflect.*, that API which allows us to build and inspect arbitrary ASTs. The reflect package gives us access to extension methods that help convert between Expr and Term.

We need to do the following:

obtain a Term out of the instance which is currently an Expr
create a Term that signifies a method reference
create a Term that invokes the method on that instance
turn that Term back into an Expr

Step by step - turn the instance into a Term:

1
val term = instance.asTerm // turn the Expr into a Term

then, find the method on that term:

1
val method = Select.unique(term, methodName.valueOrAbort)

then call the method:

1
val invocation = Apply(method, List(arg.asTerm))

and finally, return the term as a desired Expr type:

1
invocation.asExprOf[String]

The full code looks like this:

1
inline def callMethodDynamically[A](instance: A, methodName: String, arg: Int): String =
2
  ${ callMethodDynamicallyImpl('instance, 'methodName, 'arg) }
3
def callMethodDynamicallyImpl[A](instance: Expr[A], methodName: Expr[String], arg: Expr[Int])(using
4
    q: Quotes
5
): Expr[String] = {
6
  import q.reflect.*
7
  // turn the Expr into a Term
8
  val term       = instance.asTerm
9
  // find the method
10
  val method     = Select.unique(term, methodName.valueOrAbort)
11
  // create a method invocation
12
  val invocation = Apply(method, List(arg.asTerm)) // == instance.method(arg)
13
  // return the final Expr
14
  invocation.asExprOf[String]
15
}

And sure enough, in the usage file if we write

1
val result = callMethodDynamically(SimpleWrapper(10), "magicMethod", 42)

we see the following compiler output:

1
[info]     val result: String =
2
[info]       {
3
[info]         val instance$proxy1: com.rockthejvm.macros.MacrosDemoUsage.SimpleWrapper
4
[info]            = com.rockthejvm.macros.MacrosDemoUsage.SimpleWrapper.apply(10)
5
[info]         instance$proxy1.magicMethod(42):String
6
[info]       }

which is equivalent to SimpleWrapper(10).magicMethod(42).

Obviously, if we pass a method name that does not exist on the class, the compiler shows an error:

1
[error] -- [E008] Not Found Error: /Users/daniel/dev/rockthejvm/blog-projects/scala-macros-demo/src/main/scala/com/rockthejvm/macros/MacrosDemoUsage.scala:18:44
2
[error] 18 |  val result        = callMethodDynamically(SimpleWrapper(10), "stupidMethod", 42)
3
[error]    |                                            ^^^^^^^^^^^^^^^^^
4
[error]    |value stupidMethod is not a member of com.rockthejvm.macros.MacrosDemoUsage.SimpleWrapper
5
[error] one error found
6
[error] (Compile / compileIncremental) Compilation failed

All happening at compile time!

This process of finding a method appropriate for a method invocation is not that different from what we normally do in the code: an invocation of the sort SimpleWrapper(10).stupidMethod(42) triggers the exact same error, because the compiler does the exact same search.

The Power of Macros

As you can see, macros are incredibly powerful. Here are some examples of things that you can do with macros:

synthesize types
pattern match on types
derive type class instances
summon or create givens, either one at a time or in bulk
create new values, methods, or arbitrary code
report compile errors in specific places
optimize code
make otherwise normal code intentionallly fail to compile, for example if you want to enforce best practices
run arbitrary code that helps you compile better, e.g. read files, parse schemas or even connect to databases

and much, much more.

Conclusion

In this article we explored the basic structure of macros and hopefully outlined some reasons why they’re useful. We talked about ASTs, the macro pattern, new syntax, building expressions, obtaining values, pattern matching on quotes and various code structures, and even built our own AST with the help of the compile-time “reflection” package.

I hope this article got you curious to learn more about Scala macros, because aside from the Scala compiler team and a few other library authors who learned on their own, skills with macros are very rare, and it would be great to have more people in the Scala community learn about them and create amazing tools and libraries with them.