Dynamic Applications From the Ground Up ∗

Don Stewart Manuel M. T. Chakravarty

Programming Languages and Systems
School of Computer Science and Engineering
University of New South Wales
{dons,chak}@cse.unsw.edu.au

Abstract thus, some functionality cannot change either. In the case of Emacs,
Some Lisp programs such as Emacs, but also the Linux kernel the editor primitives and any other primitives provided by the Lisp
(when fully modularised) are mostly dynamic; i.e., apart from engine are fixed. This includes the choice of user interfaces, which
a small static core, the significant functionality is dynamically can be text-based or depend on one or more GUI toolkits (such as
loaded. In this paper, we explore fully dynamic applications in Xwindows).
Haskell where the static core is minimal and code is hot swap- An entirely different example of a mostly dynamic application
pable. We demonstrate the feasibility of this architecture by two is the Linux kernel. It can load and unload so-called kernel mod-
applications: Yi, an extensible editor, and Lambdabot, a plugin- ules at runtime using an in-kernel module loader. Just like Emacs,
based IRC robot. Benefits of the approach include hot swappable Linux contains fixed functionality in its static core; some of this
code and sophisticated application configuration and extension via functionality (such as the process scheduler, the I/O scheduler, and
embedded DSLs. We illustrate both benefits in detail at the example the virtual memory subsystem) would benefit from being dynam-
of a novel embedded DSL for editor interfaces. ically configurable. In contrast to Emacs, both the core as well as
the modules of Linux are implemented in C. As a consequence,
Categories and Subject Descriptors D.3.3 [Programming Lan- kernel configuration is achieved via mechanisms independent of
guages]: Language Constructs and Features dynamically loaded code—which is in contrast to Emacs, where
configuration files are just stylised Lisp files.
General Terms Design, Languages
In this paper we move beyond applications, such as Emacs and
Keywords Dynamic applications, Hot swapping, Dynamic up- Linux, by exploring a fully dynamic software architecture whose
date, Extension languages, Functional programming static core is minimal; i.e., the core contains only what is needed
to load the rest of the application at runtime. This was not a fea-
1. Introduction sible design for Emacs, due to hardware and operating systems
constraints, which required an interpreted Lisp implementation
Dynamic applications are able to add, remove, and exchange code when Emacs was designed in the late 1970s. Interpreted byte code,
at runtime. This increases application flexibility by facilitating run- in turn, was not sufficiently efficient for the implementation of
time configuration, user extension, hot code swapping, runtime performance-sensitive editor primitives. In contrast, we demon-
meta programming, and application configuration by EDSLs (em- strate by benchmarks of the extensible editor Yi that an editor in
bedded domain specific languages). a modern, compiled functional language structured as a fully dy-
Stallman’s Emacs editor [27] is a widely known example of a namic application can have excellent performance.
dynamic application from the Lisp world—Stallman calls it an on- We experimented with two fully dynamic applications: the ex-
line extensible system. Emacs consists of a small core, implemented tensible editor Yi and the IRC1 robot Lambdabot. We found two
as a conventional C program, which is extended to a fully-fledged main advantages over applications that are only mostly dynamic.
editor by a large body of Lisp code, which is byte-compiled and Firstly, with a smaller static core, we gain more flexibility and ex-
dynamically loaded. The application core contains the Lisp engine tensibility from dynamic code loading and code swapping, as fewer
as well as those editor primitives that would not be sufficiently features depend on the static core. A case in point are user inter-
efficient in interpreted byte code. Such an application is mostly faces in Emacs and Yi. As already mentioned, the user interfaces
dynamic, but the code of the static core cannot change dynamically; supported by Emacs depend on the primitive operations realised in
∗ This
Emacs’ static core. In contrast the minimal static core of Yi is inde-
work was funded by the Australian Research Council under grant
pendent of the choice of user interface, which enables completely
number DP0211793.
dynamic interface configuration.
Secondly, fully dynamic applications enable hot code swapping.
We demonstrate this by presenting a method for replacing the cur-
rently executing application code by a variant without losing the ap-
plication state. Dynamic code replacement is always a tricky busi-
Permission to make digital or hard copies of all or part of this work for personal or ness; in a purely functional language, the semantic consequences
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation are even more subtle. We achieve it by redoing the dynamic boot
on the first page. To copy otherwise, to republish, to post on servers or to redistribute process while preserving a handle to the application state. This pro-
to lists, requires prior specific permission and/or a fee. cess is fast in practice and avoids a large range of semantic issues
Haskell’05 September 30, 2005, Tallinn, Estonia.
Copyright c 2005 ACM 1-59593-071-X/05/0009. . . $5.00.
1 Internet Relay Chat (IRC) is a standardised online chat system.
 (3) configuration plugins the binary itself consists solely of a static core of around 100 lines
C D of Haskell. The purpose of the static core is to provide an interface
A to the dynamic linker and to assist in hot swapping. The static core
Config keymap extensions ee has no application-specific functionality. Lambdabot is structured
B
similarly, with the difference that the dynamic application is a tree
joe emacs
of dependent modules, rather than a single archive of objects like
uis
curses wx nano vi vim the Yi main library.

(2) dynamic editor library 2.1 Overview

entry point The static core, with the help of the dynamic linker, loads any con-
figuration data (as plugins) followed by the main application code.
Control is then transferred to the main application’s entry point—
(1) static core dynamic linker its main function—and the program proper starts up. In Yi files are
opened, buffers allocated and the user may begin editing. Lambd-
abot connects to an IRC server node and begins serving user re-
Figure 1. Structure of yi quest. Note that the Yi configuration plugins and the main applica-
tion are dynamically linked against each other so that plugins may
call application code (and vice versa). For clarity we omit these
that other approaches to dynamic software updates incur. Dynamic details from Figure 1.
software updates, and also dynamic extension and configuration, During execution the user may invoke the Haskell dynamic
are generally convenient, but they are of special value in 24/7 ap- linker library [24] to recompile and reload any configuration files,
plications like servers that are in continual use. or to reload the application itself. The state of the application is
Lisp applications, such as Emacs, pioneered the idea of con- preserved during reloading (hot swapping), with the result that new
figuration files based on the extension language itself—in this case, code can be integrated or existing code be replaced, without having
Lisp. This approach is lightweight as it avoids a parser and analyser to quit the application. In the rest of this section, we discuss the
for yet another language. Moreover, configuration languages have process of booting in more detail; in the following section, we
the tendency to slowly grow to include ad hoc abstraction facilities discuss hot swapping.
and control structures. These features are available in a system-
atic way right from the beginning if the extension language is used
for configuration. This idea is even more powerful when combined 2.2 Dynamic Bootstrapping
with the concept of embedded domain specific languages (ED- The single purpose of the minimal static core is to initiate the dy-
SLs) [13, 14]. Then, configuration files can be tailored to the task namic bootstrapping process, by arranging the dynamic linking of
and be used for complex configurations. For example, the editor Yi the application proper. Hence, we seek a structure that it is generic
uses self-optimising lexer combinators (i.e., an embedded language in its functionality (so that the structure is reusable), efficient and
of regular expressions) to define editor command sequences such type safe. The following simplified code illustrates such a structure:
that different configuration files emulate the interface of various
existing editors (e.g., Nano, Vim, Emacs). This approach is more import System.Plugins
lightweight than, e.g., Emacs’ keymaps. It has the added benefit
that configuration files are statically type checked, and it illustrates main = do
how configurable components can be structured to achieve flexible, status <- load "Yi.o" [] [] "main"
modular extensions. case status of
In summary, our contributions are the following: LoadFailure e -> return ()
LoadSuccess m main’ -> main’
• a fully dynamic software architecture (Section 2);
• hot code swapping that preserves application state (Section 3); Here Yi.o is the main module of the editor library, which we load
• practical experience with the new software architecture in two using the plugin infrastructure described in our previous work [24].
applications, namely Yi and Lambdabot (Sections 2 and 4); Yi.o is the root of a tree of modules determined by module depen-
• application configuration and extension via plugins coded in dencies. The plugin infrastructure takes care of recursively loading
embedded DSLs (Section 5) including a novel EDSL for editor all required modules and packages, such that the entire program is
interfaces; and loaded and linked. The final argument to load—i.e., "main"—is
• performance evaluation of a fully dynamic editor (Section 6).
the symbol that represents the application’s entry point. If loading
is successful, the plugin infrastructure presents us with the Haskell
We discuss related work in Section 7. The source code for Yi2 and value represented by the given symbol. Hence, we transfer control
Lambdabot3 is publicly available. to the dynamic portion of the application by evaluating that value.
On the dynamic side, the entry point is a conventional main
2. Dynamic Architecture function:
The foundation for extensibility in Yi and Lambdabot is a dynamic main = do
architecture based on runtime code loading. Figure 1 illustrates setupLocale
this architecture, as implemented in the Yi editor. It is partitioned args <- getArgs
into three main components: (1) a static bootstrap core; (2) an mfiles <- do_args args
editor library; and (3) a suite of plugins. Under this framework, ...
2 http://www.cse.unsw.edu.au/ ∼ dons/yi.html
In this case, the Yi editor entry point is called, and execution
3 http://www.cse.unsw.edu.au/ ∼ dons/lambdabot.html continues as if the dynamic main had been invoked directly at
 all dynamic linker operations, including plugin loading, in our
architecture. All invocations of the dynamic linker by dynamic code
go via the static core; i.e., the dynamic code calls into the static
core, which in turn forwards the request to the dynamic linker. As
we discuss soon, this structure is crucial for the reloading process.
To be able to pass configuration values through to the dynamic
code, we need extend the simple dynamic main function illustrated
in Section 2.2 slightly. Rather than jumping to a constant main
function, we instead pass the configuration values as arguments to
a modified entry point:
main’ :: Config -> IO ()
To initiate reloading of configurations—or any other plugins—
from the dynamic application, we need to interact with the dynamic
linker. As already explained, the linker is not visible from the
dynamic code: it is linked to the static core. This structure is
crucial to enable the hot swapping of the entire dynamic application
code, which we will discuss shortly. Hence, the static core needs
to provide the dynamic application the required functions of the
Figure 2. Screenshot of Yi’s ncurses interface dynamic linker as function arguments to the dynamic entry point.
Overall, the dynamic entry point has the following type:
type DynamicT =
startup. A screenshot of Yi running under its ncurses interface4 is (Maybe State, -- editor state
in Figure 2. Maybe Config, -- configuration values
Maybe State -> IO (), -- ‘reboot’ function
3. Hot Code Injection IO (Maybe Config)) -- ‘reconf’ function
Once the application is up and running there are two operations that
we would like to be able to perform: dynmain :: DynamicT -> IO ()

1. reloading of plugins and The first component is an optional editor state value for when we
wish to preserve state over hot code swapping (Section 3.2). The
2. hot swapping of application code. second field is a configuration record retrieved from the configura-
The first operation means that changes to configuration files (imple- tion plugins. The final two components constitute dynamic linker
mented as plugins) can affect the running application without hav- functions, which we call reboot and reconf, respectively.
ing to restart the program. The second is more profound: we want The function reconf is a simple wrapper around the dynamic
to replace application code while the editor is running, without los- linker’s reloading primitive, reload, which checks for changes
ing state—allowing us to “fix-and-continue” or to incorporate new to the configuration files. If there are changes, reload triggers
functionality on the fly. We will first consider how to reload config- recompilation and reloading of the configuration modules. The
uration files in a dynamic architecture, which then leads us to the reconfiguration function has the following type:
second, more difficult problem.
reconf :: IO (Maybe a)
3.1 Dynamic Reconfiguration The type is polymorphic so that we can enforce statically that
We follow the Lisp tradition of expressing configuration files as reconf does not depend on the representation of the values ex-
source code in the extension language, in our case Haskell. The tracted from configuration files—that is the concern of the con-
reasons for using the extension language rather than special pur- sumers of configuration values.
pose configuration languages are very much the same as those for When reconf is called from the dynamic code a new config
the use of embedded domain specific languages (EDSLs) over stan- value is retrieved by the static core, which passes this value back
dalone DSLs [13, 14]. However, we also inherit the drawbacks of to the caller in the dynamic application. The dynamic code then
the EDSL approach, such as error messages that are harder to com- uses the new value to update the application’s state. For example,
prehend for end users. On the positive side, we shall see in Sec- Yi calls reconf by defining an editor action in the dynamic code:
tion 5 how configurations can naturally and conveniently grow into reloadE :: Action
EDSLs in our approach. reloadE = do
In any case, users specify their own configuration settings by modifyEditor_ $ \e -> do
writing Haskell code in configuration files. These files are compiled conf <- reconf
and dynamically loaded. The values they provide are used to update return $ case conf of
a default configuration record type; in other words, we follow the Nothing -> e
scheme for application configuration via plugins described in our Just (Config km sty) ->
earlier work [24]. e { curkeymap = km, uistyle = sty }
Configuration files are dynamically loaded when the static core UI.initcolours
of Yi begins executing. The user-defined configuration values are
retrieved and passed to the dynamic editor core, which uses these The function reloadE atomically (with respect to the dynamic ap-
values to set initial states for the various components of the editor. plication’s global state) invokes the static linker’s reconf function.
It is important to remember that it is the static core that performs Then, control is passed back to the static core where the actual re-
compilation checks and dynamic reloading takes place. Once the
4A prototype interface based on wxHaskell [22] is also being developed new configuration data is returned reloadE uses it to set the cur-
 besides the desire to modify all functionality of the application at
runtime. We use the static core to keep all global state while the
reloadE() dynamic code dynamic linker reloads plugins. For this to work it is crucial that all
requests to load or re-load plugins go via the static core, as men-
tioned previously. Ultimately, the static core is the only safe place to
new keep the global state during reloading, as the whole dynamic appli-
config
cation code may be replaced. In other words, we structure the entire
reconf()
dynamic application so that it takes its state as an argument, mak-
dyn linker
static core ing it purely functional again, and hence, enabling state-preserving
hot swapping.
Figure 3. Reloading configuration files from dynamic code At this point, Haskell offers significant advantage over lan-
guages which encourage the use of global state. Unrestricted use
of global state leads to a multitude of values scattered throughout
the program; hence, bookkeeping of such a dispersed state becomes
rent user interface style and key mappings. In this way we can in- difficult and potentially infeasible. Hot swapping is only practical if
ject new (stateless) code dynamically, enabling changes in config- the application already has disciplined use of global state—as is the
uration files to be immediately reflected in the running application. case by default in Haskell programs. Without restricted state, run-
This process is illustrated in Figure 3. time system or operating system support seems necessary to deal
There is a question of type safety when importing code dynam- with the state problem.
ically. We are able to check configuration files using techniques
developed in our earlier work on type safe plugins [24]. Configu- 3.2.2 Redoing main
ration files may be checked prior to loading either through the use So far, we established that hot swapping needs to be via the static
of dynamic typing, or by employing the type checker at runtime to core, while passing any global state from the old to the new instance
check the interface between static and dynamic code. through the static core. We realise this by the reboot component
We use dynamic reconfiguration for similar purposes in Lambd- of the configuration argument of type DynamicT passed to the
abot. Lambdabot was originally developed as a static binary of dynamic entry point, as discussed in Section 3.1. The static core
some 1500 lines of Haskell; in addition, it used dynamically loaded will pass the following function as a concrete value for reboot:
plugins as an extension mechanism. We refactored Lambdabot to
use a minimal static core which loads the actual Lambdabot ap- remain :: a -> IO ()
plication dynamically. In doing so, we needed to make all further It takes a state value and ‘returns’ a nominal () value—in fact it
loading of plugins go via the static core for the reasons just dis- never returns to the caller, it instead reloads all the application code
cussed. The new static core of Lambdabot is less than a 100 lines. (which is fast, c.f Section 6). The input state is then passed back
to the new main function we just loaded. The new dynamic entry
3.2 Pulling the Rug Out from Underneath point then restarts the editor and uses the state parameter to restore
In the discussion so far, we neglected an important problem: during the previous environment. The state value lets us keep any files and
reloading a plugin, be it the main body of the dynamic application buffers open in Yi, for example.
or an extension, we lose any state maintained in that plugin. In our We utilise polymorphism in remain, as with other interfaces to
experience, it is not unusual for plugins to require private state (for the static core. This simplifies state transfer, as we can be sure that
example, in the form of IORefs), which is reinitialised on reload- the static core does not depend on the representation of the state
ing. The loss of this state may be acceptable for small plugins or component.
configuration files, but is unacceptable if we attempt to dynami- The complete sequence of operations performed by remain is:
cally reload large amounts of code, or indeed the application itself. 1. the dynamic code calls remain with the current state as an
Hence, we will now turn to discussing an approach to maintain plu- argument,
gin state across reloads by state injection from the static core.
2. static core unloads the dynamic application,
3.2.1 Keeping state 3. the (new) dynamic application is loaded, and
When updating code dynamically, we wish to continue execution 4. the core calls the application’s entry point, passing the old state
from exactly the point at we reached prior to the dynamic update. as an argument.
To do this, we must reconstruct any relevant application state built In the static core, we implement this as follows:
by the code before the update. Reconstructing an application’s state
remain :: a -> IO ()
in Haskell is usually simple. By default entire Haskell programs
remain st = do
are purely functional, so simply re-entering the application via its
unloadPackage "yi"
normal entry point is enough to recreate the application’s state from
status <- load "Yi.o" [] [] "main"
a previous run. There is no need to actually store state between runs,
case status of
as we are always able to reconstruct it.
LoadFailure e -> return ()
However, some applications require large amounts of mutable
LoadSuccess m main’ -> main’ st
state, and reproducing the mutations to that state may be infeasible.
Yi is such an application. Editor buffers are potentially very large The function unloadPackage unloads the entire dynamic applica-
flat byte arrays, constructed over the entire running time of the tion (packaged as an archive). We then reload the code from object
application. Any copying is prohibitively expensive. Buffers in Yi files, and reenter the code via its normal entry point, supplying the
are stored in a single global state component of the editor. What we old state value as an argument. The dynamic code then inspects
need to do is capture this state during the update, and then inject it this value, and does a quick restart—avoiding the initial applica-
into the new updated code, continuing exactly where we left off. tion start up—and instead directly entering the editor main loop.
This brings us to a central reason for using a dynamic applica- In Yi we invoke the hot swap event by getting a handle to the
tion structure centred around a minimal static core; i.e., a reason current state, shutting down the editor (which involves terminating
some threads, and exiting the user interface). We then jump into the all the same features and is usable in less supported environments.
static code by calling reboot (i.e. calling into the static core), and Such an application will lack the ability to dynamically load con-
passing down the state. figuration files or new code.
Producing a static version of the dynamically loaded appli-
rebootE :: Action
cation can be achieved quite simply by adding a conventional
rebootE = do
main :: IO () to the main module of the dynamic application,
st <- getEditorState
and arranging for the compiler to treat this as the normal entry
Editor.shutdown
point of the program, leaving the boot stub out of the compilation
reboot (Just st)
process. Both Yi and Lambdabot can be compiled as static-only
When the new application code is entered after the reload, the old applications in this way.
state is injected back into the application, and we short circuit the This is possible as the static core is minimal—it contains no
usual initialisation steps. application code—so even when it is left out, we still arrive at a
We can increase the performance of dynamic reloading by struc- working application. This clean separation of all dynamic linking-
turing the dynamic code as a single library, rather than a tree of related code into a single static core is another reason for forcing
separate modules. Doing so means that the dynamic linker doesn’t the dynamic application to delegate all invocations of the dynamic
have to read interface files for each module it needs to load—to linker to the static core.
chase dependencies—and instead loads the entire dynamic linker
in one load. In Section 6 we discuss the performance of this opera- 4. Adapting Lambdabot
tion in detail.
Lambdabot is an Internet Relay Chat (IRC) [1] robot, or bot for
3.3 Immortal Values short. IRC features messaging between individual users as well as
channels where multiple users can meet to engage in conversation.
Replacing code dynamically while still preserving state works well In the chat network, bots appear as normal user clients that provide
when extending functions. New functionality, bug fixes and exten- a wide range of services by reacting to commands sent to them di-
sions can all be integrated without having to quit the application. rectly or issued on a channel monitored by the bot. Some bots with
However, reinjection of global state is only safe if the definition of special privileges authenticate users, award operator privileges to
the state data type is unchanged in the new code. If, for example, selected users, and stop network abuses. Other bots simply provide
the buffer type changes to include an extra field and we restore the services to the users of a particular channel. Lambdabot belongs to
old state containing an old buffer value lacking this field, the new the later category and is, for example, used on the #haskell chan-
code will crash—it simply is not compiled to handle a value of the nel of the irc.freenode.net network. Lambdabot services are
old type. What we need is a safe way of injecting an old state value implemented by a suite of dynamically loaded service plugins that
into a new, different state type. This problem, state transfer, has include dictionary queries, weather forecasts, notification services,
been considered in work relating to operating systems supporting but also an online Haskell type checker and interpreter.
hot swapping [26, 5] as well as in the work of Hicks [12]. The first version of Lambdabot had a fairly large static core
We tackle this problem by reducing it to a form of parsing: we (about 1500 lines) implementing IRC protocol support. The ser-
define an error-handling parsing function to inject values of the old vice plugins amounted to another 5000 lines of Haskell. The main
type into the new type. To transfer values between types we convert incentive for the use of plugins was ease of configuration and the
the state value into a binary representation, via serialisation. Prior ability to upgrade plugins without needing to shut down the bot.
to rebooting the application, we encode the old state value in this Channels, such as #haskell, almost always have active users who
binary format. After injecting this binary state value into the new might want to use the bot’s services. Hence, hot code updates are
code, the value is parsed to construct a state value of the new type, an attractive option. Unfortunately, the original architecture with
where missing, or extra, fields are either ignored or replaced with the fairly large static core suffered from two drawbacks:
default values. In this way global state values can be preserved over
data type changes, and the state value is effectively immortal. 1. Patches to the core of Lambdabot required a complete restart—
A more complex solution could use a version tag in the state and the core is quite large (nearly 25% of the application’s
data type and a class of injection functions to achieve binary com- code).
patibility between arbitrary versions of the state type. 2. GHC’s runtime system requires all libraries that are both used in
statically and dynamically linked code to be in memory twice.
3.4 Persistent State This is clearly a waste of resources.
The problem of preserving state during dynamic code reloading is We refactored Lambdabot to use the dynamic architecture de-
tightly connected to the general problem of persistence. Applica- scribed in this paper. Now, the static core has less than 100 lines of
tions that are able to extract and inject their entire state can be ex- Haskell, which are linked statically against the hs-plugins linker.
tended to support persistence via the usual mechanisms—e.g. se- This static core when invoked dynamically loads the Lambdabot
rialisation of values to binary representations [30]. When passing main module, triggering cascading dynamic loading of the rest of
the state value from dynamic code to the static core, the core can what was formerly the static binary and is now the dynamic applica-
arrange to write the state to disk, and on rebooting, reread this state, tion code. After loading, control is transferred to the dynamic entry
passing it back to the dynamic main. point of Lambdabot, passing a set of linker functions as an argu-
The problem of preserving state across code loading effectively ment. These functions are then used to dynamically load Lambd-
forces us to separate and sanitise use of state in the application. abot’s suite of service plugins. At this point the application con-
Extending the state preservation to full persistence is relatively nects to the server. Under this new architecture, we avoid both of
simple once we reach that point. the above problems.
Refactoring Lambdabot proved to be quite painless, despite
3.5 Static Applications Lambdabot having been developed without any thought of struc-
Dynamic linking is not supported in all environments that the com- turing it as a dynamic application. The responsibility of loading
piler, GHC, runs in. For portability, it is convenient to be able to plugins was transferred from Lambdabot’s simple object loader, to
build a statically linked version of the application that provides the static core linked against the hs-plugins library. This was a
matter of exporting linker functions in a manner similar to Yi, and The use of a domain-specific language for key bindings has
in fact reduced and simplified the main application code, as all dy- enabled us to create specifications for the often obscure grammars
namic loading glue was factored into the static core. of existing editor interfaces with little effort. We suspect that key
Our experience with Lambdabot suggests that adapting other binding construction via DSLs will allow cleaner and more intuitive
Haskell applications to the dynamic architecture we propose is fea- interfaces to be constructed, as the gap from formal specification to
sible. Dynamically loading any Haskell application can be achieved implementation is less.
with only small modifications to the application. In particular, the
static entry point is simply hoisted to become the dynamic entry 5.2 Overview of Key Bindings
point. Most editors have a fixed mapping of character sequences to edi-
tor actions. Users interact with the editor by typing character se-
5. Configuration and Extension quences which trigger specific functions in the editor. This method
As already mentioned in Section 3.1, the Lisp experience shows of interaction is analogous to an interpreter—keystrokes are lexed
that it is attractive to realise configuration files in a dynamic envi- and parsed producing phrases in the editor language which specify
ronment as dynamically loaded code. After that, it is a small step editor state changes. These phrases are then executed, causing the
from configuration files to extension modules and the use of em- corresponding changes to occur.
bedded domain specific languages (EDSLs) [13, 14] for both ap- It is important to note that the input to the lexer is possibly
plication configuration as well as extension. Languages that are infinite, making it essential to use a lexer that consumes input lazily
used to extend applications dynamically are called extension lan- (lexer generators that consume input strictly are not suitable). We
guages. Many programs written in conventional languages opt to can model a lazy keymap lexer as a function:
implement extension languages by embedding interpreters for new keymap :: [Char] -> [Action]
application-specific languages—Vim can be configured and ex-
where Action is the type for editor actions. The return type of
tended in Vim script, for example. An alternative is to embed an
keymap is a list of such actions, as each key sequence generates
existing scripting or extension language in the application, for ex-
a discrete action event, and there are an infinite number of such
ample Perl [29], Lua [16, 17], or Scheme.
events.
Lisp programs, such as Emacs, allow configuration and exten-
In Yi the keybinding lexers have precisely this type. Multiple
sion of the application in the application language itself. Yi and
functions of this type can be loaded as a plugins to the editor, and
Lambdabot are built following this model, as dynamically reload-
lexer definitions can be implemented as separate combinator frag-
able plugins directly provide extension and configuration via com-
ments spread across multiple modules. Different editor interfaces
piled code. Moreover, we use EDSLs for more complex configura-
are emulated by writing different lexers. The result is an elegant
tions and extensions.
combinator language for writing [Char] -> [Action] editor in-
To illustrate our approach to creating modular, flexible exten-
sions, we use as a running example the self-optimising lexer com- terfaces.5
binators used by Yi. These are an embedded regular expression lan- We briefly summarise the lexer combinator language. It con-
guage for implementing editor keystroke interfaces (or keymaps) sists of a set of regular expression combinators, and a number of
based upon a lexer combinator library. When combined with dy- combinators for binding regular expressions to actions, to produce
namically reloadable configuration files and hot swapping, the use lexer fragments (action, meta), joining regular expressions (>|<),
of a configuration and extension EDSL allows us to rapidly extend combining lexer fragments (>||<), and running lexers over in-
keystroke interfaces for Yi. Indeed, with very little programming put (execLexer). The semantics of these operations are described
effort, Yi is able to emulate, to differing degrees of completeness, in [9].
several existing editors, including Vi [28] and Vim [8] (different Regular expressions Construction and control
editors of the Vi family) as well as Emacs [27], Nano [23] and epsilon action
Ee [15]. char meta
Configuration files in Yi are conventional Haskell modules string >|<
which Yi arranges to compile and dynamically load, bringing user star >||<
configuration data and code into the editor. The user may thus im- plus execLexer
plement small interface extensions, or indeed completely new in- quest
terfaces, by extending their configuration files. We use this EDSL alt
to construct new lexers emulating existing editor interfaces, and
make use of advanced lexer features, such as threaded lexer state 5.3 A Simple Example
storing nested lexer histories, and lexer table elements that trigger Here is a simple example of how Yi users would dynamically
monadic lexer switching, to develop sophisticated interfaces. extend the editor interface via code fragments in configuration files.
The user writes a configuration file, Config.hs. This file will
5.1 The Lexer Language
be loaded by the static core when the application is invoked, and
The interface extension language of Yi is an embedded lexer com- reloaded on demand as the user extends or modifies the code:
binator language based on the self-optimising combinators de-
scribed in our earlier work [9]. We use these as the foundation module Config where
of a novel EDSL for the construction of key bindings. User-written
lexer fragments are written in configuration files which are then import Yi.Yi
appended to, or replace, default key bindings at runtime. The use import Yi.Keymap.Vim
of a DSL for specifying key bindings allows full interfaces to be 5 The original lazy lexer combinators described in Chakravarty [9] have
constructed by those unfamiliar with the implementation of the ap- been modified in two ways to support key binding programming: firstly,
plication. Additionally, interfaces written in a combinator style can lexers immediately return tokens once they are uniquely determined (not
be constructed from fragments spread over multiple files, allowing waiting till the next input character). Secondly, lexer composition with over-
code reuse. This enables users to conveniently specify their own lapping bindings is permitted, with the new bindings overriding previous
custom mappings. bindings.
 default interface, and may be extended while the editor is running.
yi = settings { keymap = keymapPlus bind } The brevity of the code provides an indication of the power of
domain specific extension languages for this task, and we believe
bind = char ’n’ ‘action‘ \_ -> the lexer combinator EDSL to be of general utility for programming
Just $ mapM_ insertE "--" application keystroke interfaces:
In the above example, settings is a set of default configuration keymap cs = fst3 $ execLexer lexer (cs, ())
values, and yi is the distinguished value the dynamic loader ex-
pects to find in any config file. The keymap field specifies which lexer = insert >||< command
lexer the editor is to use as its keystroke handler—in this case the
Vim interface augmented with a binding for the character ‘n’, that insert = any ‘action‘ \[c] -> Just (insertE c)
when triggered inserts the Haskell comment token “--” into the
buffer at the current point. The function keymapPlus augments the command = cmd ‘action‘ \[c] -> Just $ case c of
default binding with new lexer fragments, allowing us to compose ’\^L’ -> leftE
the Vim lexer with new user-supplied code. ’\^R’ -> rightE
User’s may thus write their own lexer bindings in configuration ’\^U’ -> upE
files using the EDSL, gaining the safety and expressiveness of ’\^D’ -> downE
Haskell in the process. Configuration files that fail type checking ’\^B’ -> botE
are rejected, and default values are substituted in their place. ’\^T’ -> topE
’\^K’ -> deleteE
5.4 Lexer Table Elements ’\^Y’ -> killE
Key bindings, under our scheme, are lexers from strings to action ’\^H’ -> deleteE >> leftE
tokens. Keystroke interfaces use regular expressions to construct ’\^G’ -> solE
self-optimising lexer tables specifying what editor actions to invoke ’\^O’ -> eolE
given a particular series of keystrokes. The elements of the lexer ’\^X’ -> quitE
tables are functions combining primitive editor operations. These _ -> undefined
are built from a set of around 80 primitives providing the following Where any and cmd are patterns that match any character, and the
functionality: set of command characters, respectively. At the top level, a key
• Movement commands: left, right, up, down, goto, ... binding takes a lazy list of input characters, and runs the lexer on
• Buffer editing actions: insert, delete, replace
this list. This causes the lexer table to be dynamically constructed.
The lexer is built from two lexer fragments: one for self-inserting
• Reading portions of the buffer: read, read line characters, and another for a small set of control characters.
• Multiple-buffer actions: next, prev, focus, unfocus, close, split Partially defined extensions, such as those including undefined,
• Undo/redo, yank/paste, search/replace or code that throws other exceptions [20], may be caught by the
• File actions: new, write application, and dealt with in the usual manner. For example, Yi
• Meta actions: reboot, reload, quit, suspend
catches and prints exceptions via the message buffer, before resum-
ing execution in the main loop. Furthermore, the effect of malicious
• Higher-level actions: map, fold extensions can be mitigated somewhat using techniques described
These functions return Action type, a synonym for IO (). Our in our previous work [24].
extension language thus allows us to construct bindings from char- We now consider more sophisticated interfaces utilising thread-
acter input to monadic expressions, specifying state changes to the ed recursive state, lexer switching and finally monadic lexer switch-
editor. These actions values can be composed with >>= (or other ing.
monadic combinators), in the usual way.
5.6 Threaded State
When passed input, keymaps return a series of (unevaluated)
editor actions. For example, in Vi or Vim emulation mode, user A powerful feature of the lexer combinator language upon which
input of “jl2x” generates the following list of actions: we base our extension language is the ability to thread state through
the lexer as it evaluates input. This has proved to be invaluable,
[downE, leftE, replicateM_ 2 deleteE] and we use the state, for example, to communicate results between
These actions are self-explanatory. As the user types characters a different regular expressions, to implement command history and
lazy list of these editor actions is produced. These actions in turn line editing, as well as stackable dynamic key mappings.
need to be forced, to generate their effects, so the main editor loop We take as a simple example the task of emulating the prefix
body is as follows: repetition arguments to commands used in the Vi family of ed-
itors, including Vim and Vi. Many commands can be optionally
sequence_ . keymap =<< getChanContents ch prefixed with numerical repetition arguments. For example 3x is
The character input reader runs in a separate thread, returning a the sequence to delete 3 characters. We need a way to parse any
lazy list of keystrokes via a channel to the main thread. The list numerical prefix n in a digit lexing fragment, but make that value
of keystrokes is passed as an argument to the current keymap, our available later on, once we’ve decided which action to perform.
(pure) key binding lexer. The resulting list of actions are evaluated We implement this by threading an accumulator as a state com-
as they are produced, causing immediate state changes in the appli- ponent through the lexer. Digit key sequences can be appended to
cation. this lexer state by the digit lexer fragment (as well as, perhaps,
echoed to a message buffer), until a non-digit key is pressed. Con-
5.5 A Complete Interface trol is then transferred to a command character lexer. Once the full
command has been identified, the digit state value is retrieved and
We now present a basic keybinding definition for the ee [15]
the command replicated by that value. The digit lexing code is
editor, implemented in our combinator lexer extension language.
This interface can be written to a configuration file, replacing the nums :: Lexer String Action
 nums = digit ‘meta‘ \[c] s -> ’a’ -> doI $ rightOrEolE 1
(msgE (s++[c]), s++[c], Just lexer) ’o’ -> doI $ eolE >> insertE ’\n’
’O’ -> doI $ solE >> insertE ’\n’ >> upE
The lexer state now consists of a String value. Rather than us- ’C’ -> doI $ readRestOfLnE >>= setRegE >> killE
ing the ‘action‘ combinator for binding regular expressions to }
actions, we instead access a lexer state component when the lexer
table element is retrieved. This is achieved via the ‘meta‘ combi- These bindings all cause a mode switch to the insert mode lexer
nator, which allows us access to the lexer state component, as well (insert) where keys are inserted into the buffer by default. The
as specifying (1) any token to return, (2) a new state value, and (3) first binding (’i’ -> doI nopE) causes a direct switch, whilst the
a lexer to continue execution with. other bindings all perform actions of various complexity prior to the
In the above code, s is the state component of the lexer. When switch. For example, ’A’ moves the cursor to the end of the line,
a digit is matched in the input stream, we extract the existing and then enters insert mode.
lexer state. We immediately echo the input value, and any previous We may also pass state from one mode to another via the
input digits to the message buffer (via msgE), append the current state component of the lexer. State is sometimes useful over mode
character to the state, and continue execution with the default lexer. switches. For example, when exiting from a line editing mode,
In this way digits will be accumulated, as well as being echoed to hitting Ctrl-M causes the final edited input to be passed from the
the screen each time they are pressed. line editor, to a sub-mode that can then interpret the edited string.
The state can be used later by the command lexer fragment: We now consider how to add new bindings that dynamically
extend editor key bindings.
command = cmd ‘meta‘ \[c] s ->
(msgClrE >> fn c (read s), [], Just lexer) 5.8 Dynamic Mappings
where fn c i = case c of
’\^L’ -> replicateM_ i leftE Many applications allow users to dynamically extend the table of
’\^R’ -> replicateM_ i rightE key mappings by binding keys to a new sequence of keystrokes.
... In Vi or Vim this is achieved by :map and :unmap commands.
We may also remap previously defined mappings, shadowing the
Here we first clear the message buffer, then construct a new editor former definition with a new definition.
action using the digits stored in the state to specify the repetition. In order to implement dynamically extensible mappings cleanly
Finally, we return this action, along with a new empty state, and we would like to be able to update the application’s lexer table at
continue with the default lexer. runtime. This is not possible if our lexer is either a hand-written
The state is convenient for other values we extract from the parse function (i.e. the lexer table is compiled code) or if our lexer
key input stream. For example, suppose we wish to maintain a is an statically generated lexer table. One solution is to use dynamic
command history. We can keep track of all editor input in a list reloading to recompile the lexer on the fly, however this is a rather
value inside the lexer state, and later retrieve or index this history, sledgehammer approach when we simply want to update a lexer
when we receive user input to do so. table.
These simple examples illustrate the close fit between our lexer An alternative is to construct new lexer combinators at runtime.
combinators, and the domain of interface keybindings. We continue This approach—implementing dynamically extensible key map-
by describing how to model more difficult features of key binding pings via runtime constructed combinators—cleanly and elegantly
syntax. allows us to implement dynamic mappings.
Let us consider user input in a vi-like editor of :map zz d4j.
5.7 Modes This binds the key sequence zz to the actions produced by typing
Most editors consist of a set of modes, or distinct sets of key d4j. This sequence deletes the next four lines down from the
bindings. Usually only one such set is in operation at any point. current line. How do we implement this using the lexer combinator
Even in modeless editors there are submodes introduced when extension language? What we need to be able to do is to inspect the
certain keys are pressed. current lexer table to find what actions the input d4j are bound to.
There is a direct encoding of modes as independent lexer frag- We can then use the resulting actions to build a new lexer fragment
ments in our extension language, with the result that the distinction dynamically.
between moded and modeless editors evaporates—modeless edi- To inspect the lexer table we need to run the current lexer, on the
tors just consist of one large default lexer, with smaller lexer frag- side, with d4j as input. This will produce a list of Actions. We can
ments for submodes. The main problem is actually how to achieve then use a fold to join the list of actions into a single action. This
mode switching. That is, how to bind a keystroke to an action that action we then use on the right hand side of a newly constructed
causes control to pass to a new lexer. lexer fragment (i.e. a new lexer table entry), bound to zz, which we
The usual way to switch modes is via a distinguished mode use to augment the current lexer.
switching sequence. There are often multiple ways to enter a new Assuming we wish to bind our zz to the command mode lexer,
mode, the difference being that particular actions are performed we can inspect the lexer table elements bound to an arbitrary input
prior to the mode switch. These rules are directly implementable in string, inp, like so:
the lexer framework, via the meta action. lookupC inp = fst3 $ execLexer command (inp, [])
For example, we may specify a lexer fragment for the Vi
and Vim editors that specifies mode switching into ‘insert’ mode This builds a new lexer on the fly, with inp as input, returning a list
(equivalent to the default self-insertion mode in Emacs): of actions that are associated to the given input. We can construct
a single action from the list of actions as. This can be achieved by
switchChar ‘meta‘ \[c] st -> folding the monadic >> over this list, with a no-op to start the fold
let doI a = (with a, st, Just insert) off. We then construct a new lexer fragment binding “zz” to the
in case c of { resulting Action:
’i’ -> doI nopE
’I’ -> doI solE bind zzs = string zzs ‘action‘ \_ ->
’A’ -> doI eolE Just (foldl (>>) nopE as)
The last step is to transfer control from the currently executing lexer history of key bindings or to write the current set of bindings to a
to a new lexer augmented with the bind we have just constructed. file, so that dynamic mappings can persist beyond the current editor
The right hand side of the meta binding for the :map command is session.
thus:
5.10 Higher Order Keystrokes
(msgClrE, [], Just $ command >||< bind)
A number of existing editor interfaces have commands that take
where new bindings in the right argument of >||< replaces any other commands as arguments. For example, it is common for
existing binds in command. simple editing commands to be parameterised by movement com-
One issue remains, however. When we combine two lexers we mands that when executed define a region of the buffer for the edit-
create a new lexer value to which we pass control. If we ever call ing command to operate over. We can implement such semantics in
:map again, we will add any new binding to the original lexer, a similar way to dynamic mappings: we run a new lexer with the in-
losing any dynamic bindings. The solution to this is to store the put key sequence to find the sequence of actions an input sequence
most recent lexer, with any dynamic mappings it has, in the lexer maps to, and then use that to construct a composite action.
state, so that bindings will not be lost on later mappings. We can Extracting actions from the table is as before. For example, if
extend the lexer state to store the current lexer, along with the we take the Vi or Vim key sequence d4j (delete four lines down),
accumulator, like so: we then run a new lexer with d4j as input, returning a list of actions
data State = State { bound to d4j.
acc :: String, We can now take these actions and use them to construct a
cmd :: Lexer State Action new expression of type Action to find the range of the buffer a
} command is to operate on. The following expression uses as, the
actions bound to d4j, to calculate the start and end position in the
Now new key mappings will be bound to a meta action as before, buffer of a movement command:
and we also record this augmented lexer in the lexer state. Rather
than hard coding which lexer control passes to a meta action, we do p <- getPointE
instead transfer control to the lexer found in the threaded state. foldr (>>) nopE as
Now, we are in a position to consider how to implement command q <- getPointE
unmapping. when (p < q) $ gotoPointE p
return (p,q)
5.9 Nested Mappings
We save the current point, and force the actions retrieved from the
It is often desirable to allow users to unmap commands, and most lexer. This yields a new point in the buffer. Then, we move back to
extensible editors support this. Two main approaches may be con- where we started, and return a pair defining a buffer range. Finally,
sidered. The first is to only allow one level of mapping to exist at we can call the editor’s deletion primitive with the range as an
any point. That is, if the sequence zz is mapped twice, the second argument:
mapping will overwrite the first mapping, and it will be forgotten –
this is Vi’s behaviour. Unmapping zz will then revert to the map- deleteNE (max 0 (abs (q-p) + 1))
ping for zz found in the default lexer, i.e. no mapping at all.
We are thus able to implement parameterised bindings via lexer
We can implement this simple unmap similar to map. We run the
table elements that are defined via the results of running other
default lexer with the sequence we wish to unmap as input. Three
elements of the lexer table.
possible values will result, either:
• No actions are bound to the sequence 5.11 Monadic Lexer Switching
• Exactly one action One problem that we have not approached yet is how to implement
• Multiple actions are bound to the sequence lexer switching where the switch is based on the result of evaluating
an element of the lexer table—that is, an Action.
If no actions were bound to some sequence we are unmapping, Switching lexers based on the lexer input is easy—it’s a pure
then we can unmap this sequence from the current mapping by operation, and we can use lexer meta actions to transfer control
binding it to the empty action: nopE. If exactly one action is bound (and even pass control to lexers threaded through the lexer state).
to the sequence, then that input did have a previous binding, and However, there are some features that require a mode switch based
we restore this by binding the input to the action we have just on the outcome of an editor action. That is, we need to pass control
retrieved. The difficult case is if some input string s results in to a specific lexer based on the value returned when evaluating an
multiple actions—this implies that components of the string are element found in the current lexer table.
individually bound. In this case we really need to delete the entry An example of such an action is text searching. Applications
for s from the table entirely (mapping s to nopE won’t do, as it adds often provide a search function. If a user types ‘/’ in many editors
a binding). Direct deletion of table elements is not yet possible, they enter a line editor submode for regular expression searching.
however we can emulate deletion by rebinding dynamically added On entering a complete search phrase, a buffer search takes place.
bindings to the original lexer, filtering out the entry we wish to If a search successfully finds a match, the editor displays a prompt
delete. and switches to a sub lexer where the user can answer yes or no to
With this model it is possible to implement stackable dynamic trigger individual text replacements. If there is no match, an error
mappings. Rather than new bindings overwriting previous dynamic message is displayed, and control continues as normal.
bindings, we can instead revert to the last bind (which may have The problem, then, is how to specify that if we find a match
been the result of a :map) for a particular sequence. To do this (the result of an IO event), control should be transferred to the
we extend the lexer state to carry a table whose keys are input search key binding lexer. We solve this by introducing the metaM
character sequences, and whose elements are lists of bindings. We combinator, an editor action for lexer continuations. It has the type
can then implement an :unmap that reverts to the innermost binding
for a particular sequence, by popping the top element of the list of metaM :: ([Char] -> [Action]) -> IO ()
bindings for that sequence. It would also be possible to browse the metaM km = throwDyn (MetaActionException km)
where the dynamic exception is defined as Yi startup as buffer size varies
16
Dynamic Yi
newtype MetaActionException = Static Yi

MetaActionException ([Char] -> [Action]) 14

deriving Typeable
12

The metaM function connects the elements of the lexer table to the

Startup time (seconds)

10
currently running lexer. Such an action allows a table element to
cause control to be passed to a new lexer. When a metaM action 8
is evaluated, it throws a dynamic exception which wraps the new
lexer to which control is transferred. 6
To actually use this new lexer, we need to catch this exception
in the main loop of the application, and replace the current keymap 4

with the thrown keymap. The top-level key input code for Yi is
2
thus:
0
do let run km = catchDyn 0 20 40 60 80 100 120 140 160 180 200
(sequence_ . km =<< getChanContents ch) Buffer size (MB)

(\(MetaActionException km’) -> run km’) Figure 4. Dynamic architecture startup cost
run dflt
The simple main loop code is now wrapped in a handler that passes
control to a thrown keymap. We begin evaluation of user input with 6. Performance
the default keybinding, dflt.
The metaM action is a powerful function—it allows elements of We discuss the performance of our dynamic architecture by way of
the lexer table to cause control transfers to lexers specified by the results obtained with the Yi editor. All measurements were made
table element. We justify its inclusion by outlining some common on a Pentium-M 1.6Ghz laptop running OpenBSD 3.7, with 256M
editor behaviours only implementable in our framework via metaM. RAM.
Startup time. We first evaluate the startup cost of the dynamic
5.12 Prompts and User Interaction architecture by comparing the startup time for the Yi editor with
A common editor action requiring monadic lexer switch are com- and without the dynamic architecture in Figure 4. Caches were
mand prompts. In the Nano editor [23], a user can begin a search primed with a dummy run before each run.
by typing Ctrl-W. This causes a mode switch to a submode for We instrumented Yi to measure the time from entering the static
searching, providing a line editor and some related commands (for main until the editor main loop, located in the dynamic application,
continuing, or cancelling the search, for example). On hitting re- commences. The startup time increases linearly with the buffer size
turn, the search is performed. (from the cost of loading the buffer into memory). The results
After completing a search, any further searches behave as be- demonstrate that the dynamic architecture has a startup cost of
fore, except that the previous search string is used to set a prompt, around 30 milliseconds on the benchmark platform for typical
and a default search value. The default search value is a compiled editing jobs.6 In other words, the cost of loading buffers by far
regular expression produced as an intermediate result of a previous exceeds the startup costs of the dynamic architecture. As the buffer
search. The problem is how to script this behaviour. size increases, factors unrelated to dynamic linking come into play,
When switching to search mode we need to check if a previous in particular, after around 100M the amount of real memory has an
expression has been searched for. In Yi, the global editor state impact as the machine begins to swap.
stores the most recent compiled regular expression, and the string
that it was generated from. When we switch to search mode the Hot code swapping. We display the costs of state-preserving hot
switch action extracts any previous regular expression and uses it code swapping in Figure 5. The displayed time includes hot swap
to generate a prompt, which then dynamically creates a new lexer the entire application code, reinjecting the preserved state, and
with the prompt as its default state. When this action is evaluated, returning to the dynamic main loop.
we use metaM to switch control to the custom lexer generated by The results indicate a constant cost of 10 milliseconds for hot
the action. swapping, independent of the Yi state size (the state increases as the
buffer size increases), up to buffer sizes of around 120M, at which
char ’\^W’ ‘action‘ \_ -> Just (a >>= metaM) point the test machine starts to run out of physical memory, and
where performance rapidly deteriorates as hot swapped code competes
a = do for space with the large buffer state. The cost of hot swapping
mre <- getRegexE is independent of the size of Yi buffers as we pass references to
let prompt = case mre of buffers to the static core, not buffers themselves.
Nothing -> "Search: "
Just (pat,_) -> "Search ["++pat++"]: " Comparison with other editors. To evaluate Yi’s overall perfor-
msgE prompt mance, we compare it to a range of common editors in Figure 6.
return (mkKM prompt) The benchmark involves the execution of a specific editing se-
quence submitted to the keystroke interfaces of each editor. We
Here, mkKM constructs a new search keymap via execLexer, with then test performance as buffer size increases. The measured time
the prompt as its default state. In this way we are able to program includes editor start, GUI initialisation, execution of the edit se-
lexer mode switches based on the result of monadic actions, en- quence, and editor shutdown. It might have been preferable to mea-
abling us to implement many common interface features. In effect
we have lexers that build new lexers on the fly, whose elements are 6 Wedefine typical as file under 20M in size. The average source file in the
actions that trigger switches between lexers. Haskell hierarchical libraries is 10k; so, 20M is a generous limit.
 Time to hot swap Yi versus buffer size due to lack of operating system support. Moreover, Emacs pre-
4
loads the most essential Lisp byte-code at build time and dumps
the resulting runtime image into a binary file for faster startup.
3.5

Erlang. Hot code swapping (aka dynamic software updating) is

3
an important feature of Erlang [2, 3, 11, 18], which has been
2.5
part of the language for several years. Hot swapping in Erlang
Time (seconds)

is used, for example, in telephony applications that must have

2 very high availability. Recent work has looked at formalising the
hot code swapping capabilities of Erlang [6]. Hicks [12] cites
1.5 Erlang’s functional nature as contributing to the simplicity of its hot
swap implementation, making reasoning about global state much
1
simpler.
0.5
Java and Eclipse. The architecture of the Eclipse workbench
0
revolves around a hierarchy of dynamically loaded plugins [7]. At
0 20 40 60 80 100 120 140 160 180 200 the core is the Eclipse platform together with a number of core
Buffer size (MB)
plugins, which are always available. In recent versions of Eclipse,
Figure 5. Performance of hot swapping the core is close to minimal, permitting the use of the Eclipse plugin
architecture for non-IDE applications. However, in contrast to Lisp
applications and our proposal, Eclipse configuration is via XML
meta data. Moreover, there seems to be no simple facility for hot
100
Comparitive editor performance
swapping code. Hot swapping has also been used in the context of
Yi
XEmacs
Java to implement dynamic profiling [10].
Vim
mg
Linux and other operating systems. The Linux kernel can be
80
compiled such that most of its functionality is in kernel modules
than can be loaded and unloaded at runtime. When fully modu-
larised, only functionality that is crucial during booting the kernel
Edit time (seconds)

60
itself, such as memory management and process scheduling, are in
the static core. Everything else, including support for the file sys-
40 tem used for the root partition, is loaded at runtime; the root file
system module comes from an initial RAM disk. Linux’ level of
modularisation is limited as it began as a monolithic operating sys-
20 tem with support for kernel modules added much later. In contrast,
micro kernel operating systems provide a higher degree of modu-
larisation and, in some cases, hot swapping capabilities [4, 5].
0
0 20 40 60 80 100 120 140 160 180 200
Buffer size (MB)
8. Further work
Figure 6. Comparative editor performance There still remains much work to be done in the area of exten-
sible, statically-typed applications. Extensions to Yi and Lambd-
abot, for example, must currently be loaded as compiled plugins.
sure each operation in isolation, but the interface of some editors In the case of Yi, the time taken to compile quick scripting jobs—
does not support such isolated measurements. such as evaluating a simple Haskell expression—is tedious when
It is interesting to observe that Yi (both dynamic and static) out- the code is immediately discarded once a result is produced. An
performs the mature extensible editors Vim and Emacs. It seems alternative would be to embed a Haskell interpreter in the applica-
as if dynamic applications consisting of compiled functional code tion and use that for small scripting jobs, whilst retaining compiled
have a significant performance advantage over interpreted extensi- code for larger, more permanent application extensions. Haskell
ble systems. The small non-extensible editor mg also had good per- users already follow this pattern, writing throw-away code in an
formance. We also tested several other small static editors (nano, interpreter such as GHCi, but resorting to compilation for more
ee and vi). They have good performance on small data sizes, but long-lived code. Ideally we would like to be able to mix compiled
this advantage deteriorates rapidly as file sizes increase. We have code with interpreted code. To this end, we plan to integrate the
not included them in the figure to improve readability. GHCi interpreter, with its ability to mix compiled and interpreted
code [25], into Yi, effectively adding a form of interactive runtime
meta-programming.
7. Related Work When extending the state of a dynamic application, consumers
Emacs. Emacs [27] was the first editor whose design revolved of the state, in particular the state transfer functions (which must
around facilitating extensibility. This implied an extension lan- preserve state over changes in the state type) need to be modified
guage beyond simple editing macros and the ability to dynami- to handle the new state type. We speculate that a state design
cally load extension code that can add new or override existing based upon extensible generic programming may reduce the cost of
functionality. Limitations of the hardware and operating systems of state extension and allow extensible state transfer functions. Recent
the time, encouraged interpretation of extension code, and thus, re- work on open, extensible, functions [21] seems promising.
quired the most performance-sensitive functionality to be included When extending the state type whilst hot swapping, we are cur-
in the static core (which is implemented in C in current flavours of rently forced to step outside of the type system—by translating the
Emacs). Stallman [27, Section 4], in fact, explicitly lists dynamic state value into a binary representation—and then reconstructing
linking as an alternative approach, but dismisses it as impractical this value after the swap. Ideally, we would like to be able to auto-
matically extend the state type, without requiring the binary encod- [10] M. Dmitriev. Profiling Java applications using code hotswapping and
ing or requiring a new state type. Extensible records seem suitable dynamic call graph revelation. In WOSP ’04: Proceedings of the
in this regard [19]. The transfer of values as the state type is ex- Fourth International Workshop on Software and Performance, pages
tended is achieved via a transform from a type τ to the same type 139–150, New York, NY, USA, 2004. ACM Press.
augmented with new record fields. The result would be that pre- [11] B. Hausman. Turbo Erlang: Approaching the speed of C. In
serving state over extensions to the state type can be typed inside E. Tick and G. Succi, editors, Implementations of Logic Programming
Systems, pages 119–135. Kluwer, Dordrecht, 1994.
the language without resorting to untyped binary intermediaries.
We avoid a large number of semantic concerns by hot swapping [12] M. Hicks. Dynamic Software Updating. PhD thesis, Department
of Computer and Information Science, University of Pennsylvania,
all dynamic code, which obviates the need to worry about refer-
August 2001.
ences into old code persisting in hot swapped code. However, more
[13] P. Hudak. Building domain-specific embedded languages. ACM
fine grained hot swapping may be desirable in some applications.
Computing Surveys (CSUR), 28(4es):196, 1996.
We intend to investigate further issues relating to fine grained hot
[14] P. Hudak. Modular domain specific languages and tools. In
swapping in lazy languages.
P. Devanbu and J. Poulin, editors, Proceedings: Fifth International
Beyond the general research issues raised by this paper, there Conference on Software Reuse, pages 134–142. IEEE Computer
are a number of features we would like to see developed in Yi, Society Press, 1998.
including [15] Hugh Mahon. The Editor ’ee’. http://mahon.cwx.net/.
• parser-based syntax highlighting and language-aware editing, [16] R. Ierusalimschy, L. H. de Figueiredo, and W. Celes. The evolution
• support for embedding other Haskell applications in the editor, of an extension language: A history of Lua. In M. A. Musicante
such as refactoring support or class derivation tools, and and E. H. Hausler, editors, V Simpósio Brasileiro de Linguagens de
Programação, pages B–14–B–28, Curitiba, May 2001.
• compiler support whilst editing; e.g., for incremental typing.
[17] R. Ierusalimschy, L. H. de Figueiredo, and W. C. Filho. Lua-an
extensible extension language. Softw., Pract. Exper., 26(6):635–652,
9. Conclusion 1996.
We presented an architecture for fully dynamic applications in [18] E. Johansson, M. Pettersson, and K. Sagonas. A high performance
Haskell, based upon a minimal static core. We discussed a prac- Erlang system. In PPDP ’00: Proceedings of the 2nd ACM SIGPLAN
International Conference on Principles and Practice of Declarative
tical design and implementation of hot swapping for such dynamic
Programming, pages 32–43, New York, NY, USA, 2000. ACM Press.
applications, and demonstrated the feasibility of this architecture
[19] M. P. Jones and S. Peyton Jones. Lightweight extensible records for
with two applications, Yi and Lambdabot. We then explored so-
Haskell. In Proceedings of the 1999 Haskell Workshop. Published in
phisticated dynamic application configuration and extension via Technical Report UU-CS-1999-28, Department of Computer Science,
an EDSL, as implemented in Yi for dynamic interface extensions. University of Utrecht, Sept. 1999.
Finally, we discussed some performance benchmarks that suggest [20] S. P. Jones, A. Reid, F. Henderson, T. Hoare, and S. Marlow. A
that the costs of the dynamic architecture and hot swapping are semantics for imprecise exceptions. In PLDI ’99: Proceedings of the
negligible, and also that dynamic applications written in modern ACM SIGPLAN 1999 Conference on Programming Language Design
compiled functional languages can have excellent performance. and Implementation, pages 25–36, New York, NY, USA, 1999. ACM
Press.
Acknowledgements. We are grateful to Gabriele Keller, Stefan
[21] R. Lämmel and S. Peyton Jones. Scrap your boilerplate with
Wehr and Simon Winwood for feedback on drafts. The develop-
class: extensible generic functions. Draft, submitted to ICFP 2005.
ment of Yi benefited from several discussions with Tuomo Valko- http://www.cwi.nl/ralf/syb3/, Apr. 2005.
nen, in particular the monadic lexer switching model.
[22] D. Leijen. wxHaskell – a portable and concise GUI library for
Haskell. In ACM SIGPLAN Haskell Workshop (HW’04). ACM Press,
References Sept. 2004.
[1] Internet Relay Chat (IRC) help archive. http://www.irchelp. [23] Nano Core Development Team. GNU Nano Text Editor. http:
org/, 2005. //www.nano-editor.org/.
[2] J. Armstrong. Erlang — a Survey of the Language and its Industrial [24] A. Pang, D. Stewart, S. Seefried, and M. M. T. Chakravarty. Plugging
Applications. In INAP’96 — The 9th Exhibitions and Symposium on Haskell In. In Proceedings of the ACM SIGPLAN Workshop on
Industrial Applications of Prolog, pages 16–18, 1996. Haskell, pages 10–21. ACM Press, 2004.
[3] J. L. Armstrong. The development of Erlang. In International [25] J. Seward, S. Marlow, A. Gill, S. Finne, and S. P. Jones. Architecture
Conference on Functional Programming, pages 196–203, 1997. of the Haskell Execution Platform (HEP). http://www.haskell.
[4] A. Baumann, G. Heiser, J. Appavoo, D. D. Silva, O. Krieger, R. W. org/ghc/docs/papers/, 1999.
Wisniewski, and J. Kerr. Providing dynamic update in an operating [26] C. A. N. Soules, J. Appavoo, K. Hui, R. W. Wisniewski, D. D. Silva,
system. In Proceedings of the 2005 USENIX Technical Conference, G. R. Ganger, O. Krieger, M. Stumm, M. Auslander, M. Ostrowski,
pages 279–291. USENIX Association, 2005. B. Rosenburg, and J. Xenidis. System support for online reconfigura-
[5] A. Baumann, J. Kerr, J. Appavoo, D. Da Silva, O. Krieger, and tion. In Proc. of the Usenix Technical Conference, 2003.
R. W. Wisniewski. Module hot-swapping for dynamic update and [27] R. M. Stallman. EMACS: the extensible, customizable self-
reconfiguration in K42. In Proceedings of the 6th Linux.Conf.Au, documenting display editor. In Proceedings of the ACM SIGPLAN
Canberra, Australia, Apr. 2005. To appear. SIGOA Symposium on Text Manipulation, pages 147–156, 1981.
[6] G. Bierman, M. Hicks, P. Sewell, and G. Stoyle. Formalizing dynamic [28] Sven Verdoolaege and Keith Bostic. The Berkeley Vi Editor.
software updating. In Proceedings of the Second International http://www.bostic.com/vi/.
Workshop on Unanticipated Software Evolution (USE), April 2003. [29] L. Wall. Programming Perl. O’Reilly & Associates, Inc., Sebastopol,
[7] A. Bolour. Notes on the Eclipse plug-in architecture. http://www. CA, USA, 2000.
eclipse.org/articles/Article-Plug-in-architecture/ [30] M. Wallace and C. Runciman. The bits between the lambdas: binary
plugin architecture.html, 2003. data in a lazy functional language. In ISMM ’98: Proceedings of
[8] Bram Moolenaar. The Vim Editor. http://www.vim.org/. the 1st International Symposium on Memory Management, pages
[9] M. M. T. Chakravarty. Lazy lexing is fast. Fourth Fuji International 107–117, New York, NY, USA, 1998. ACM Press.
Symposium on Functional and Logic Programming, LNCS 1722:68–
84, 1999.