Nektar++: An open-source spectral/hp element framework

C. D. Cantwella,∗, D. Moxeya , A. Comerforda , A. Bolisa , G. Roccoa , G. Mengaldoa , D. de Graziaa , S. Yakovlevb , J.-E. Lombarda ,
D. Ekelschota , B. Jordia , H. Xua , Y. Mohamieda , C. Eskilssonc , B. Nelsonb , P. Vosa , C. Biottoa , R. M. Kirbyb , S. J. Sherwina
a Department of Aeronautics, Imperial College London, London, UK
b Schoolof Computing and Scientific Computing and Imaging (SCI) Institute, Univ. of Utah, Salt Lake City, UT, USA
c Department of Shipping and Marine Technology, Chalmers University of Technology, Gothenburg, Sweden

Abstract
Nektar++ is an open-source software framework designed to support the development of high-performance scalable solvers for
partial differential equations using the spectral/hp element method. High-order methods are gaining prominence in several engi-
neering and biomedical applications due to their improved accuracy over low-order techniques at reduced computational cost for
a given number of degrees of freedom. However, their proliferation is often limited by their complexity, which makes these meth-
ods challenging to implement and use. Nektar++ is an initiative to overcome this limitation by encapsulating the mathematical
complexities of the underlying method within an efficient C++ framework, making the techniques more accessible to the broader
scientific and industrial communities. The software supports a variety of discretisation techniques and implementation strategies,
supporting methods research as well as application-focused computation, and the multi-layered structure of the framework allows
the user to embrace as much or as little of the complexity as they need. The libraries capture the mathematical constructs of
spectral/hp element methods, while the associated collection of pre-written PDE solvers provides out-of-the-box application-level
functionality and a template for users who wish to develop solutions for addressing questions in their own scientific domains.
PROGRAM SUMMARY
Manuscript Title: Nektar++: An open-source spectral/hp element framework
Authors: C. D. Cantwell, D. Moxey, A. Comerford, A. Bolis, G. Rocco, G. Mengaldo, D. de Grazia, S. Yakovlev, J.-E. Lombard, D. Ekelschot,
B. Jordi, H. Xu, Y. Mohamied, C. Eskilsson, B. Nelson, P. Vos, C. Biotto, R. M. Kirby, S. J. Sherwin
Program Title: Nektar++
Journal Reference:
Catalogue identifier:
Licensing provisions: MIT
Programming language: C++
Computer: Any PC workstation or cluster
Operating system: Linux/UNIX, OS X, Microsoft Windows
RAM: 512 MB
Number of processors used: 1-1024
Supplementary material:
Keywords: spectral/hp element method, computational fluid dynamics
Classification: 12 Gases and Fluids
External routines/libraries: Boost, METIS, FFTW, MPI, Scotch, PETSc, TinyXML, Loki
Nature of problem: The Nektar++ framework is designed to enable the discretisation and solution of time-independent or time-dependent partial
differential equations.

Solution method: spectral/hp element method

Restrictions:

Unusual features:

Additional comments:

Running time: The tests provided take a few minutes to run. Runtime in general depends on mesh size and total integration time.

Keywords: High-Order Finite Elements, Spectral/hp Elements, Continuous Galerkin Method, Discontinuous Galerkin Method,
FEM.

Preprint submitted to Computer Physics Communications February 4, 2015

1. Introduction scaling on large HPC clusters is challenging. Semtex [4] imple-
ments the 2D spectral element method coupled with a Fourier
Finite element methods (FEM) are commonplace among a expansion in the third direction. The implementation is highly
wide range of engineering and biomedical disciplines for the efficient, but can only be parallelised through Fourier-mode de-
solution of partial differential equations (PDEs) on complex ge- composition. Nek5000 [5] is a 3D spectral element code, based
ometries. However, low-order formulations often struggle to on hexahedral elements, which has been used for massively par-
capture certain complex solution characteristics without the use allel simulations up to 300,000 cores. Hermes [6] implements
of excessive mesh refinement due to numerical dissipation. In hp-FEM for two-dimensional problems and has been used in a
contrast, spectral techniques offer improved numerical charac- number of application areas. Limited high-order finite element
teristics, but are typically restricted to relatively simple regular capabilities are also included in a number of general purpose
domains. PDE frameworks including the DUNE project [7] and deal.II
High-order finite element methods, such as the traditional [8]. A number of codes also implement high-order finite el-
spectral element method [1], the p-type method [2] and the ement methods on GPGPUs including nudg++, which impl-
more recent spectral/hp element method [3], exhibit the con- ments a nodal discontinuous Galerkin scheme [9], and PyFR
vergence properties of spectral methods while retaining the ge- [10], which supports a range of flux reconstruction techniques.
ometric flexibility of traditional linear FEM. They potentially Nektar++ provides a single codebase with the following key
offer greater efficiency on modern CPU architectures as well features:
as more exotic platforms such as many-core general-purpose
graphics processing units (GPGPUs). The data structures which • Arbitrary-order spectral/hp element discretisations in one,
arise from using high-order methods are more compact and lo- two and three dimensions;
calised than their linear finite element counterparts, for a fixed
number of degrees of freedom, providing increased cache co- • Support for variable polynomial order in space and het-
herency and reduced memory accesses, which is increasingly erogeneous polynomial order within two- and three-
the primary bottleneck of modern computer systems. dimensional elements;
The methods have had greatest prominence in the structural
• High-order representation of the geometry;
mechanics community and subsequently the academic fluid dy-
namics community. They are also showing promise in other • Continuous Galerkin, discontinuous Galerkin and hy-
areas of engineering, biomedical and environmental research. bridised discontinuous Galerkin projections;
The most common concern cited with respect to using high-
order finite element techniques outside of academia is the im- • Support for a Fourier extension of the spectral element
plementational complexity, stemming from the complex data mesh;
structures, necessary to produce a computationally efficient im-
plementation. This is a considerable hurdle which has limited • Support for a range of linear solvers and preconditioners;
their widespread uptake in many application domains and in- • Multiple implementation strategies for achieving linear al-
dustries. gebra performance on a range of platforms;
Nektar++ is a cross-platform spectral/hp element framework
which aims to make high-order finite element methods acces- • Efficient parallel communication using MPI showing
sible to the broader community. This is achieved by provid- strong scaling up to 2048-cores on Archer, the UK national
ing a structured hierarchy of C++ components, encapsulating HPC system;
the complexities of these methods, which can be readily ap-
plied to a range of application areas. These components are • A range of time integration schemes implemented using
distributed in the form of cross-platform software libraries, en- generalised linear methods; and
abling rapid development of solvers for use in a wide variety of
• Cross-platform support for Linux, OS X and Windows op-
computing environments. The code accommodates both small
erating systems.
research problems, suitable for desktop computers, and large-
scale industrial simulations, requiring modern HPC infrastruc- In addition to the core functionality listed above, Nektar++
ture, where there is a need to maintain efficiency and scalability includes a number of solvers covering a range of application ar-
up to many thousands of processor cores. eas. A range of pre-processing and post-processing utilities are
A number of software packages already exist for fluid dy- also included with support for popular mesh and visualisation
namics which implement high-order finite element methods, formats, and an extensive test suite ensures the robustness of
although these packages are typically targeted at a specific do- the core functionality.
main or provide limited high-order capabilities as an extension. The purpose of this paper is to expose the novel aspects of the
The Nektar flow solver is the predecessor to Nektar++ and im- code and document the structure of the library. We illustrate
plements the spectral/hp element method for solving the in- its use through a broad range of example applications which
compressible and compressible Navier-Stokes equations in both should enable other scientists to build on and extend Nektar++
2D and 3D. While it is widely used and the implementation is for use in their own applications. We begin by outlining the
computationally efficient on small parallel problems, achieving mathematical formulation of the spectral/hp element method
and discuss the implementation of the framework. We then
∗ Correspondingauthor present a number of example applications and conclude with
Email address: [email protected] (C. D. Cantwell) a discussion of our development strategy and future direction.
2
2. Methods or more of the coordinate directions are collapsed creating sin-
gular vertices within these regions [11, 12]. Operations, such
In this section we introduce the mathematical foundations as calculating derivatives, map the coordinate system to a non-
of the spectral/hp element methods implemented in Nektar++. collapsed coordinate system through a Duffy transformation
A more detailed overview of the mathematical theory can be [13] — for example, ωT : T → Q maps the triangular region
found in [3] and is beyond the scope of this paper. Nektar++ T to the quadrilateral region Q — to allow these methods to be
supports both continuous and discontinous discretisations in well-defined.
one, two and three dimensions, but the majority of the formula- A local polynomial basis is constructed on each reference el-
tion which follows is generic to all cases, except where stated. ement with which to represent solutions. A one-dimensional
We consider the numerical solution of partial differential order-P basis is a set of polynomials φ p (ξ), 0 ≤ p ≤ P, de-
equations (PDEs) of the form Lu = 0 on a domain Ω, which fined on the reference segment, S. The choice of basis is usu-
may be geometrically complex, for some solution u. Practi- ally made based on its mathematical or numerical properties
cally, Ω takes the form of a d-dimensional finite element mesh and may be modal or nodal in nature. For two- and three-
(d ≤ 3), consisting of elements Ωe such that Ω = Ωe and
S
dimensional regions, a tensorial basis may be used, where the
Ωe1 ∩ Ωe2 = ∂Ωe1 e2 is the empty set or an interface of dimension polynomial space is constructed as the tensor-product of one-
d̄ < d. The domain may be embedded in a space of equal or dimensional bases on segments, quadrilaterals or hexahedral
higher dimension, d̂ ≥ d. We will solve the PDE problem in regions. In particular, a common choice is to use a modified
the weak sense and, in general, u|Ωe must be smooth and have hierarchical Legendre basis, given as a function of one variable
at least a first-order derivative; we therefore require that u|Ωe is by
in the Sobolov space W 1,2 (Ωe ). For a continuous discretisation,  1−ξ
we additionally impose continuity along element interfaces. 

 2  p = 0,
  1−ξ 1+ξ
 1,1
φ p (ξ) =  < p < P,

Our problem is cast in the weak form and, for illustrative  2 2 P p−1 (ξ) 0
purposes, we assume that it can be expressed as follows: find  1+ξ


p = P.
2
u ∈ H 1 (Ω) such that
which supports boundary-interior decomposition and therefore
a(u, v) = l(v) ∀v ∈ H 1 (Ω), improves numerical efficiency when solving the globally as-
sembled system. Equivalently, φ p could be defined by the
where a(·, ·) is a bilinear form, l(·) is a linear form, and H 1 (Ω) Lagrange polynomials through the Gauss-Lobatto-Legendre
is formally defined as quadrature points which would lead to a traditional spectral el-
ement method.
H 1 (Ω) := W 1,2 (Ω) = {v ∈ L2 (Ω) | Dα u ∈ L2 (Ω) ∀ |α| ≤ 1}.
On a physical element Ωe the discrete approximation uδ to
To solve this problem numerically, we consider solutions in a the solution u may be expressed as
finite-dimensional subspace VN ⊂ H 1 (Ω) and cast our problem X
uδ (x) =
  
ûn φn χe −1 (x)
as: find uδ ∈ VN such that
n
δ δ δ
a(u , v ) = l(v ) (1) where ûn are the coefficients from Equation 1, obtained through
projecting u onto the discrete space. Therefore, we restrict our
∀vδ ∈ VN , augmented with appropriate boundary conditions. solution space to
For a projection which enforces continuity across elements, we n o
impose the additional constraint that VN ⊂ C 0 . We assume the V := u ∈ H 1 (Ω) | u|Ωe ∈ PP (Ωe ) ,
solution can be represented as uδ (x) = n ûn Φn (x), a weighted
P
sum of N trial functions Φn (x) defined on Ω and our problem where PP (Ωe ) is the space of order-P polynomials on Ωe .
becomes that of finding the coefficients ûn . The approximation Elemental contributions to the solution may be assembled to
uδ does not directly give rise to unique choices for the coeffi- form a global solution through an assembly operator. In a con-
cients ûn . To achieve this we place a restriction on R = Luδ that tinuous Galerkin setting, the assembly operator sums contribu-
its L2 inner product, with respect to the test functions Ψn (x), is tions from neighbouring elements to enforce the C 0 -continuity
zero. For a Galerkin projection we choose the test functions to requirement. In a discontinuous Galerkin formulation, such
be the same as the trial functions, that is Ψn = Φn . mappings transfer flux values from the element interfaces into
To construct the global basis Φn we first consider the contri- the global solution vector.
butions from each element in the domain. Each Ωe is mapped
from a standard reference space E ⊂ [−1, 1]d by a paramet- 3. Implementation
ric mapping χe : Ωe → E given by x = χe (ξ), where E is
one of the supported region shapes, and ξ are d-dimensional In this section, we provide an architectural overview of Nek-
coordinates representing positions in a reference element, dis- tar++, sufficient to enable other scientists to leverage the frame-
tinguishing them from x which are d̂-dimensional coordinates work to develop application-specific PDE solvers using high-
in the Cartesian coordinate space. The mapping need not nec- order methods. In doing so we summarize the salient features
essarily exhibit a constant Jacobian, supporting deformed and of the code and how the mathematical constructions from Sec-
curved elements through an isoparametric mapping. The ref- tion 2 are represented in the library.
erence spaces implemented in Nektar++ are listed in Table 1. In designing Nektar++, strong emphasis has been placed on
On triangular, tetrahedral, prismatic and pyramid elements, one ensuring the code structure strongly mirrors the mathematical
3
 Name Class Domain definition
Segment StdSeg S = {ξ1 ∈ [−1, 1]}
Quadrilateral StdQuad Q = {ξ ∈ [−1, 1]2 }
Triangle StdTri T = {ξ ∈ [−1, 1]2 | ξ1 + ξ2 ≤ 0}
Hexahedron StdHex H = {ξ ∈ [−1, 1]3 }
Prism StdPrism R = {ξ ∈ [−1, 1]3 | ξ1 ≤ 1, ξ2 + ξ3 ≤ 0}
Pyramid StdPyr P = {ξ ∈ [−1, 1]3 | ξ1 + ξ3 ≤ 0, ξ2 + ξ3 ≤ 0}
Tetrahedron StdTet A = {ξ ∈ [−1, 1]3 | ξ1 + ξ2 + ξ3 ≤ −1}

Table 1: List of supported elemental reference regions.

APPLICATION DOMAIN operations onto these regions,

SolverUtils • MultiRegions: list of physical regions comprising Ω,
global assembly maps which may optionally enforce con-
tinuity, construction and solution of global linear systems,
SPECTRAL ELEMENT METHOD extension of local core operations to domain-level opera-
MultiRegions tions,
• SolverUtils: building blocks for developing complete
LocalRegions solvers.
We now outline in detail the different aspects of the implemen-
tation, the division of functionality across the libraries, the re-
SpatialDomains lationships between them and how together they construct the
Helmholtz operator H which forms an essential component in
1 solving many elliptic PDE problems.
StdRegions
-1
-1 1 3.1. Input format
AUXILIARY Nektar++ uses one or more XML-structured text files as in-
put for simulations. These describe both the discretisation of
LibUtilities
the domain (the finite element mesh) and the specification of the
PDE problem in terms of the necessary boundary conditions,
variables and parameters required to solve a specific problem.
Figure 1: Nektar++ architecture diagram. Shows the relationship between the A large number of example XML input files are provided with
individual libraries in the framework and the mathematical constructs to which the source code in the Examples and Tests subdirectories of
they relate.
the library demonstration programs and solvers. The input for-
mat and comprehensive list of the available options are docu-
formulation. The implementation is partitioned into a set of six mented in full in the user guide (see Appendix A.14).
libraries, each of which encapsulates one aspect of the above The Nektar++ mesh specification format is of a hierarchical
construction. This division maintains separation between math- type in which one-dimensional edges are defined in terms of
ematically distinct parts of the formulation, provides an intu- the vertices they connect, two-dimensional faces are defined in
itive means to organise the code and enables developers to eas- terms of the bounding edges and three-dimensional elements
ily interject at the level most appropriate for their needs. The in terms of the bounding faces. A composite is defined as a
overall architecture of Nektar++ is illustrated in Fig. 1. In sum- collection of mesh entities which have a common shape, but
mary, the six libraries cover the following aspects of the math- need not necessarily be connected. Composites are used for
ematical formulation given in Section 2: specifying the extent of the domain and for defining boundary
regions on which constraints can be imposed (see Section 3.7).
• LibUtilites: elemental basis functions ψ p , point distri- Meshes are typically generated by third-party mesh generation
butions ξi and basic building blocks such as I/O handling packages and the necessary XML specification is generated by
and mesh partitioning, the Nektar++ MeshConvert utility.
• StdRegions: reference regions E along with integration,
differentiation and other core operations on these regions, 3.2. LibUtilities library
• SpatialDomains: the mappings χe , the geometric factors The primary function of the LibUtilities library is to
∂χ provide the fundamental mathematical and software constructs
∂ξ , and Jacobians of the mappings,
necessary to implement the spectral/hp element method. In par-
• LocalRegions: physical regions in the domain, compos- ticular, this includes the description of Q-point coordinate dis-
ing a reference region E with a map χe , extensions of core tributions, ξ j , on the standard segment S and the construction of
4
suitable polynomial bases, ψi , to span PP (S). Each type of basis and is equivalent to solving for û the linear system
is encapsulated in a class which, when augmented with a point
distribution, provides the P × Q basis matrix BS [i][ j] = ψi (ξ j ), Mû = B> Wf.
the P × Q basis derivative matrix DBS [i][ j] = ∂ψ i
∂ξ |ξ j , the Q × Q The StdRegions classes also describe the mapping of basis
diagonal quadrature weight matrix WS [i][i] = wi and the Q co-
functions to the vertices, edges and faces of the element, which
ordinates ξi of the points on S. As well as providing these ba-
are necessary to assemble elemental contributions and construct
sic mathematical objects, the LibUtilities library also pro-
a global system for the domain.
vides a range of other generic functionality. In particular, paral-
lel communication routines, Fourier transforms, linear algebra
containers and design pattern implementations are all incorpo-
rated into this part of the framework. 3.4. SpatialDomains library
The classes defined in SpatialDomains fall into three main
3.3. StdRegions library hierarchies which together describe the geometric information
Reference regions E ⊂ [−1, 1]d provide core element-level needed to represent the problem domain and the elemental en-
operations and are implemented in the StdRegions library. tites which comprise it. Geometry classes capture the physi-
Classes are defined for each of the reference regions, as given cal geometry of an individual element Ωe . There are separate
in Table 1, and class inheritance is used to share common classes for each elemental region and the class hierarchy fol-
functionality. An example of the class hierarchy for two- lows a parallel structure to the StdRegions classes, shown in
dimensional elements is shown in Fig. 2(a). We assume each re- Fig. 2(c). The GeomFactors class, instantiated by each geome-
gion is equipped with a basis φn extended from one of the basis try object, defines the parametric mapping χe between the phys-
functions ψ p . In two- and three-dimensional elements, the basis ical geometry of the element Ωe and the corresponding standard
is often constructed as a tensor product of one-dimensional hier- region. This mapping is implemented in a generic manner and
archical bases; for example, φn(p1 ,p2 ) (ξ1 , ξ2 ) = ψ p1 (ξ1 ) ⊗ ψ p2 (ξ2 ) does not require dimension-dependent subclasses.
for a quadrilateral region or φn(p1 ,p2 ) (ξ1 , ξ2 ) = ψ p1 ⊗ ψ p1 ,p2 for a MeshGraph classes read the mesh definition from the input
triangular region. However, nodal and non-tensorial bases are file and construct the domain Ω, instantiating a corresponding
also supported. In a similar way, coordinates in two- and three- Geometry object for each Ωe . The derived classes are again
dimensional elements are given by ξq=q(i) . Expansion orders dimension-dependent. Finally, the BoundaryConditions
may be different for each of the ψ pi bases, although constraints class manages the association of specific mathematical con-
are imposed on simplex regions to ensure a complete polyno- straints to each boundary of the domain for a given variable.
mial space. The core operators defined on the reference element
are then:
• BwdTrans: û 7→ u(ξ) = n ûn φn (ξ), 3.5. LocalRegions library
P

which evaluates the solution represented by û, on the basis Physical regions, shown in Fig. 2(b), are an extension of a
φn , at the quadrature points ξ. This operation requires the reference element augmented with geometric information and
basis matrix B and therefore evaluates the result as u = a mapping between the two regions. As such, the physical el-
Bû. ement types implemented in the LocalRegions library inherit
their StdRegions counterparts and override core operations,
• IProductWRTBase: f 7→ fˆn = E f (ξ)φn (ξ) dξ,
R
such as integration and differentiation, to incorporate the geo-
which computes the inner product of a function with re- metric information. For example, the inner product operation
spect to the basis. The discrete approximation of integra- becomes
P
tion, Gaussian quadrature, leads to f̂[i] ≈ q wq f (ξq )φi (ξq ) Z
which can be expressed in matrix form as f̂ = B> Wf, f (x) 7→ fi =
ˆ f (x)φi (x) dx
Ωe
where W is a diagonal matrix containing the integration
weights wq .
which, in discrete form is evaluated as f̂[i] ≈
∂u
P
• PhysDeriv: u 7→ ∂ξi , q Jw q f (ξ )φ (ξ
q i q ). Here we have incorporated J, the de-
which computes the derivative of u with respect to the terminant of the Jacobian of χe . Similarly, for differentiation,
∂u P ∂ξ ∂u
d coordinates of the element, with matrix representation the chain rule gives rise to u 7→ ∂xi
= nj=1 ∂xij ∂ξ j
. Both J and
∂ξ
u0 = Du. the terms ∂xij are provided by the GeomFactors classes.
These operators can be combined to produce more complex op- To identify the relationship between the different libraries
erators such as the mass matrix, and their respective contributions to the above core operations,
Fig. 2(d) illustrates how the different libraries examined so far
M = B> WB. contribute to the implementation of the backward transform on
a given element Ωe .
With this we can project a solution onto the discrete space using
the FwdTrans operation. This requires solving the projection
3.6. MultiRegions library
problem uδ (ξ) = f (ξ). In the weak sense, this has the form
Z Z So far, operations have only been defined on the physical el-
vδ (ξ)uδ (ξ) dξ = vδ (ξ) f (ξ) dξ emental regions; to define operators on the entire domain the
E E elemental regions are assembled. The MultiRegions library
5
 (a) StdRegions (b) LocalRegions (c) SpatialDomains

StdExpansion Expansion Geometry

StdExpansion2D Expansion2D Geometry2D

StdQuadExp QuadExp QuadGeom

StdTriExp TriExp TriGeom

(d) Backward Transformation Inheritance

Cross-library
inheritence GeomFactors
Member of

Figure 2: Example of class inheritance structure for two-dimensional regions. Each LocalRegions class (b) inherits base functionality from the corresponding
StdRegions class (a). Inheritance is used to minimise code duplication. The Expansion class contains member pointers to Geometry and GeomFactors objects
(c). Operations, such as a transform from coefficient to physical space (d), on a physical element are constructed as a composition of functionality from the different
libraries.

encapsulates the global representation of a solution across a ditions are frequently Dirichlet, Neumann or Robin constraints,
domain comprising of one or more elemental regions. While depending upon the application area, other more complex con-
the same type of basis functions must be used throughout the ditions can be implemented by the user if needed. The spec-
domain, the order of the polynomials for each expansion may ification of boundary conditions in the input file is generic to
vary between elements. Assembly is the process of summing support this.
local elemental contributions to construct a global representa- • Boundary regions are defined using one or more mesh
tion of the solution on the domain. The information to con- composites. For example, the following XML describes
struct this mapping is derived from the elemental mappings of two regions constructed from three composites on which
modes to vertices, edges and faces of the element. Mathemat- different constraints are to be imposed:
ically, this operation can be represented as a highly sparse ma-
trix, but it is practically implemented as an injective map in <B ID = " 0 " > C [1 ,3] </B >
<B ID = " 1 " > C [2] </B >
the AssemblyMap classes. Different maps are used for differ-
ent projections; in particular, the AssemblyMapCG class sup-
• The conditions to be imposed on each boundary region for
ports the exchange of neighbouring contributions in continuous
each variable are described using XML element tags to
Galerkin projections, while the AssemblyMapDG supports the
indicate the underlying type of condition:
mapping of elemental data onto the trace space in the discon-
tinuous Galerkin method. – D: Dirichlet;
The resulting assembly of the elemental matrix contributions – N: Neumann;
leads to a global linear system in the GlobalLinSys classes. – R: Robin.
This may be solved using a variety of linear algebra techniques
including direct Cholesky factorisation and iterative precondi- For example, the following excerpt defines an in-flow
tioned conjugate gradient. Substructuring (multi-level static boundary using a high-order Neumann boundary condi-
condensation) allows for a more efficient solution of the ma- tion on the pressure:
trix system. As well as a traditional Jacobi preconditioner, spe- < REGION REF = " 0 " >
cialist Preconditioner classes tailored to high-order methods <D VAR = " u " VALUE = " 0 " / >
are also available[14]. These include a coarse-space precon- <D VAR = " v " VALUE = " 0 " / >
<D VAR = " w " VALUE = " y *(1 - y ) " / >
ditioner, block preconditioner and low-energy preconditioner
<N VAR = " p " US E RD EF IN E DT YP E = " H " VALUE = " 0 " / >
[15]. Performance of the conjugate gradient solver is depen- </ REGION >
dent on both the efficiency of the matrix-vector operation and
inter-process communication. The rich parameter space of a The USERDEFINEDTYPE attribute specifies the user-
high-order elemental discretisation may be leveraged by pro- implemented condition to be used. The REF attribute cor-
viding multiple implementations of the core operators, each of responds to the ID of the boundary region.
which perform efficiency across a subset of the parameter space The list of boundary regions and their constraints is man-
on different hardware architectures. This has been extensively aged by the BoundaryConditions data structure in the
explored in the literature [16, 17, 18]. The gather-scatter op- SpatialDomains library. The enforcement of the bound-
eration necessary for evaluating operations in parallel is imple- ary conditions on the solution is implemented at the
mented in the gslib library [19], developed within Nek5000. MultiRegions level of the code (and above) during the con-
Finally, a PETSc interface is available which provides access to struction and use of domain-wide operators. User-defined
a range of additional solvers. boundary conditions are implemented in specific solvers. For
example both high-order Neumann boundary conditions and a
3.7. Boundary Conditions radiation boundary condition are supported by the incompress-
Boundary-value problems require the imposition of con- ible Navier-Stokes solver.
straints at the boundaries of the domain. Although these con-
6
3.8. SolverUtils library 3.10. Implementing solvers using Nektar++
The SolverUtils library provides the top-level building To conclude this section, we outline how one can construct a
blocks for constructing application-specific solvers. This in- time-dependent solver for the unsteady diffusion problem using
cludes core functionality, such as IO, time-stepping [20] and the Nektar++ framework, solving
common initialisation routines, useful in quickly constructing
∂u
a solver using the Nektar++ framework. It contains a library = ∇2 u on Ω
of application-independent modules for implementing diffusion ∂t
and advection terms as well as a number of Driver mod- u = gD on ∂Ω.
ules which implement general high-level algorithms, such as an
This can be implemented with the following key steps as out-
Arnoldi method for performing various stability analyses [21].
lined below. Only the key statements are shown to illustrate the
use of the library. Full source code for this example is included
3.9. Solvers
in Appendix A.12.
Nektar++ includes a number of pre-written solvers for some
• Create a SessionReader object to load and parse the in-
common PDEs, developed for our own research. Some exam-
put files. This provides access for the rest of the library to
ples, outlined in the next section, include incompressible and
the XML input files supplied by the user. It also initiates
compressible Navier-Stokes equations, the cardiac electrophys-
MPI communication if needed.
iology monodomain equation, shallow water equations and a
solver for advection-diffusion-reaction problems. The modu- session = LibUtilities::SessionReader
::CreateInstance(argc, argv);
lar nature of the code, combined with the mathematically mo-
tivated class hierarchy allows the code to be adapted and ex- • Create MeshGraph and field objects to generate the hierar-
tended to rapidly address a range of application and numerical chical mesh entity data structures and allocate storage for
questions. the global solution. The mesh provides access to the ge-
To illustrate the use of the framework, we first consider the ometric information about each element and the connec-
solution of the Helmholtz equation, since this is a fundamen- tivity of those elements to form the domain. In contrast,
tal operation in the solution of many elliptic partial differential the field object represents a solution on the domain and
equations. The problem is described by the following PDE and provides the finite element operators.
associated boundary conditions:
var = session->GetVariable(0);
∇2 u − λu + f = 0 on Ω, mesh = SpatialDomains::MeshGraph
::Read(session);
u = gd on ∂Ω, field = MemoryManager<MultiRegions::ContField2D>
∂u ::AllocateSharedPtr(session, mesh, var);
= gn on ∂Ω.
∂x • Get the coordinates of the quadrature points on all ele-
We put this into the weak form and after integration by parts, ments and evaluate the initial condition
this gives,
Z Z Z Z field->GetCoords(x0,x1,x2);
icond->Evaluate(x0, x1, x2, 0.0, field->UpdatePhys());
∇u · ∇v dx + λ uv dx = f v dx + v · ∇u dx. (2)
Ω Ω Ω ∂Ω
• Perform backward Euler time integration of the initial con-
Approximating u and v with their finite-dimensional counter- dition for the number of steps specified in the session file,
parts and substituting into Equation 2 we obtain where epsilon is the coefficient of diffusion.
XX Z XX Z
ûn v̂m ∇Φm · ∇Φn + λ ûn v̂m Φn · Φn for (int n = 0; n < nSteps; ++n)
Ω Ω {
m n m n
Z Z Vmath::Smul(nq, -1.0/delta_t/epsilon,
X XX
field->GetPhys(), 1,
= v̂n f · Φn + ûn v̂m Φn · ∇Φn , field->UpdatePhys(), 1);
n Ω n m ∂Ω

which can be expressed in matrix form as field->HelmSolve(field->GetPhys(),

field->UpdateCoeffs(),
v̂> (DB)> W(DB)û + λv̂> Mû = v̂> B> Wf. NullFlagList,
factors);
The matrix H = (DB)> W(DB) + λM is the Helmholtz matrix,
and the system Hû = f̂, where f̂ = B> Wf is the projection of f field->BwdTrans (field->GetCoeffs(),
field->UpdatePhys());
onto VN , is then solved for û.
}
The above is implemented in Nektar++ through the
HelmSolve function, which takes physical values of the forcing • Write out the solution to a file which can be post-processed
function f and produces the solution coefficients u, as and visualised.
field->HelmSolve(forcing->GetPhys(),
field->UpdateCoeffs(), fld->Write(outFile, FieldDef, FieldData);
NullFlagList,
factors); Here, fld is a Nektar++ field format I/O object.
where factors is a data-structure which allows us to prescribe A second example is provided in the supplementary material
the value of λ. Appendix A.13 which elicits the use of the time-integration
framework to support more general (implicit) methods.
7
 plied to the front wing, body and rotating wheels (Ωw ), a high-
order outflow boundary condition [22] is imposed on the out-
let (Ωo ) and a constant free-stream velocity f is applied on the
inlet and far-field boundaries of the domain (Ωi ). Due to the
high Reynolds number of Re = 2.2 × 105 , based on the chord
of the front-wing main plane, the solution is particularly sen-
sitive to the spatial resolution, requiring the use of techniques
such as spectral vanishing viscosity (SVV) [23] and dealiasing
in order to efficiently maintain the stability of the solution. To
improve convergence of the conjugate gradient solver, we ap-
ply a low-energy block preconditioner [15] to the linear systems
for velocity and pressure. We apply, in addition, a coarse-space
preconditioner, implemented as a Cholesky factorisation [14],
Figure 3: Flow past a front section of a Formula 1 racing car at a Reynolds
number of 2.2 × 105 . Streamlines show the flow trajectory and are coloured
on the pressure field using an additive Schwarz approach. A
by pressure. The simulation has 13 million degrees of freedom at polynomial high-order operator-splitting scheme [24] is used to decouple
order P = 3. the system into four linear equations together with consistent
pressure boundary conditions [25], although care must be taken
to avoid instabilities arising from this formulation [26]. Fur-
4. Applications
ther examples of the capabilities of the incompressible Navier-
In this section we illustrate, through the use of the pre-written Stokes solver are given in Section 4.7.
solvers, key aspects of the Nektar++ framework through a num- In contrast, the CompressibleFlowSolver encapsulates
ber of example scientific problems spanning a broad range of two different sets of equations forming the cornerstone of aero-
application areas. The source files used to generate the figures dynamics problems: the compressible Euler and compressible
in this section are available in the supplementary material (apart Navier-Stokes equations. In these equations, the compressibil-
from Figure 3, due to commercial considerations). Although ity of the flow is taken into account, leading to a conservative
these problems primarily relate to the modelling of external hyperbolic formulation,
and internal flow phenomena, the framework is not limited to ∂U
this domain and some examples extend into broader areas of + ∇ · F(U) = ∇ · Fv (U),
∂t
biomedical engineering.
where U = [ρ, ρu, ρv, ρw, E]> is the vector of conserved vari-
4.1. External aerodynamics ables in terms of density ρ, velocity (u1 , u2 , u3 ) = (u, v, w), E is
One of the most challenging problems in next-generation the specific total energy, and
aerodynamics is capturing highly resolved transient flow past
ρu ρv ρw
 
bluff bodies using Direct Numerical Simulation (DNS). For ex- 
 p + ρu2

ρuv ρuw 
ample, understanding and controlling the behaviour of vortices
F(U) =  ρuv ρv2 + p ρvw  ,
 
generated by the front-wing section of a racing car (see Fig. 3)  ρuw ρvw ρw2 + p

is critical in ensuring the stability and traction of the vehicle.  
u(E + p) u(E + p) v(E + p)
Apart from the computational complexity of the simulation, a
number of mesh generation challenges need to be overcome to where p is the pressure. To close the system we need to specify
accurately capture the flow dynamics. These include high-order an equation of state, in this case the ideal gas law p = ρRT
curvilinear meshing of the CAD geometry and the generation of where T is the temperature and R is the gas constant. For the
sufficiently fine elements adjacent to the vehicle surfaces to re- Euler equations, the tensor of viscous forces Fv (U) = 0, while
solve the thin boundary layers and accurately predict separation for Navier-Stokes
of the flow.
Using the IncNavierStokesSolver, flow is modelled un-  0 0 0 
 
 τ
der the incompressible Navier Stokes equations,  xx τyx τzx 

Fv (U) =  τ xy τyy τzy  ,
∂u  τ
+ (u · ∇)u = −∇p + ν∇2 u,  xz τyz τzz 

(3a)
∂t A B C
∇ · u = 0, (3b)
∂p 
! with
u, = 0, (3c)
∂n ∂Ωw

A = uτ xx + vτ xy + wτ xz + k∂ x T,
∂u 
! 
,p  = 0, (3d) B = uτyx + vτyy + wτyz + k∂y T,
∂n ∂Ω
o C = uτzx + vτzy + wτzz + k∂z T,
(u, p) |∂Ωi = (f, 0), (3e)
where in tensor notation the stress tensor τ xi x j = 2µ(∂ xi ui +
where u is the velocity of the flow, p is the pressure and ν is ∂ xi u j − 13 ∂ xk uk δi j ), µ is the dynamic viscosity calculated using
the kinematic viscosity. No-slip boundary conditions are ap- Sutherland’s law and k is the thermal conductivity.
8
 (a) (b)

(c)

Figure 4: Examples of external aerodynamics problems solved using the CompressibleFlowSolver. (a) Flow over a cylinder at Re = 3, 900. (b) Euler simulation
of flow over a NACA0012 aerofoil at Ma∞ = 0.8. (c) Temperature of flow passing over a T106C low-pressure turbine blade at Re = 80, 000. Simulation input files
are provided in Appendices A.15, A.16 and A.17, respectively.

To discretise these equations in space, we adopt an approach technique which makes use of artificial viscosity to damp os-
which allows for the resolution of discontinuities and shocks cillations in the solution, in conjunction with a discontinuity
that may appear in the solution at transonic and supersonic flow sensor adapted from the approach taken in [30] to decide where
speeds. We therefore use approximations to our solution com- the addition of artificial viscosity is needed.
prised of functions which are not continuous across element Figure 4 shows representative results from compressible flow
boundaries. Traditionally, we follow a Galerkin approach by simulations of a number of industrially relevant test cases. We
utilising the variational form of the equations in order to ob- first highlight two simulations which utilise the Navier-Stokes
tain the discontinuous Galerkin method. One of the key fea- equations. Fig. 4(a) demonstrates the three-dimensional ver-
tures of Nektar++ is the ability to select a wide range of nu- sion of the compressible solver showing flow over a cylinder
merical options, and to this end we support both discontinuous at Re = 3,900. In this figure we visualise isocontours of the
Galerkin and flux reconstruction spatial discretisations, which pressure field and colour the field according to the density ρ.
have various numerical equivalences [27] but may possess dif- To demonstrate the shock capturing techniques available in the
ferent performance characteristics. In the flux reconstruction code, Fig. 4(b) shows the results of an Euler simulation for
formulation, we instead use the equation in differential form in flow over a NACA0012 aerofoil at a farfield Mach number
combination with correction functions which impose continuity Ma∞ = 0.8 and a 1.5◦ angle of attack. The transonic Mach
of fluxes between elements. number of this flow leads to the development of a strong and
In either case, information is transferred between elements weak shock along the upper and lower surfaces of the wing re-
by solving a one-dimensional Riemann problem at the interface spectively. This figure shows isocontours of the Mach number
between two elements. For the non-viscous terms there is sup- where the presence of the shocks are clearly identified. Finally,
port for a wide variety of Riemann solvers, including an exact in Fig. 4(c), we visualise the temperature field from flow pass-
solution or a number of approximate solvers such as HLLC, ing over a T106C low-pressure turbine blade at Re = 80,000
Roe and Lax-Friedrichs solvers [28]. For the viscous terms, we to highlight applications to high Reynolds number flow simula-
utilise a local discontinuous Galerkin method (or the equivalent tions.
flux reconstruction version). Boundary conditions are imple-
mented in a weak form by modifying the fluxes for both the 4.2. Transitional turbulent flow dynamics
non-viscous and viscous terms [29]. Various versions of the Transient problems in which turbulence dominates the flow
discontinuous Galerkin method which are available throughout domain, or in which the transition to turbulence dominates the
the literature, mostly relating to the choices of modal functions simulation, remain some of the most challenging problems to
and quadrature points, can also be readily selected by setting resolve in computational fluid simulations. Here, accurate nu-
appropriate options in the input file. merical schemes and high resolution of the domain is critical.
Given the complexity and highly nonlinear form of these Moreover, any choice of scheme must be efficient in order to ob-
equations, we adopt a fully explicit formulation to discre- tain results in computationally feasible time-scales. Tradition-
tise the equations in time, allowing us to use any of the ex- ally, highly resolved turbulence simulations, such as the Taylor-
plicit timestepping algorithms implemented through the gen- Green vortex problem, lie firmly in the class of spectral meth-
eral linear methods framework [20], including 2nd and 4th order ods. However, spectral methods typically lead to strong geom-
Runge-Kutta methods. Finally, in order to stabilise the flow etry restrictions which limits the domain of interest to simple
in the presence of discontinuities we utilise a shock capturing shapes such as cuboids or cylinders.
9
 (a)

(b)

(c)

Figure 5: Contours of velocity magnitude in a periodic hill simulation at Re =

2800. Simulation input files available in Appendix A.18

Figure 6: Linear stability analyses of two-dimensional flow past a circular

Whilst spectral element methods may seem the ideal choice cylinder at Re = 42. Illustrative plots of (a) streamwise (left) and transverse
for such simulations, particularly when the domain of interest (right) components of velocity for the dominant direct mode, (b) streamwise
(left) and transverse (right) velocity for the dominant adjoint mode and (c)
is geometrically complex, they can be more computationally structural sensitivity to base flow modification (left) and local feedback (right).
expensive in comparison to spectral methods. However, when Simulation input files are provided in Appendix A.19.
the domain of interest has a geometrically homogeneous com-
ponent – that is, the domain can be seen to be the tensor prod-
uct of a two-dimensional ‘complex’ part and a one-dimensional Time-periodic base flows are sampled at regular intervals and
segment – we can combine both the spectral element and tradi- interpolated.
tional spectral methods to create a highly efficient and spectrally The linear evolution of a perturbation under Equs. (4) can be
accurate technique [4]. expressed as
We consider the application of this methodology to the prob-
lem of flow over a periodic hill, depicted in Fig. 5, where the u0 (t) = A(t)u0 (0),
flow is periodic in both streamwise and spanwise directions.
This case is a well-established benchmark problem in the DNS for some initial state u0 (0), and we seek, for some arbitrary time
and LES communities [31], and is challenging to resolve due to T, the dominant eigenvalues and eigenmodes of the operator
the smooth detachment of the fluid from the surface and recir- A(T ), which are solutions to the equation
culation region. Here we consider a Reynolds number of 2800,
A(T )ũ j = λ j ũ j .
normalised by the bulk velocity at the hill crest and the height
of the hill, with an appropriate body forcing term to drive the The sign of the leading eigenvalues λ j are used to establish the
flow over the periodic hill configuration. global stability of the flow. An iterative Arnoldi method [33] is
The periodicity of this problem makes it an ideal candidate applied to a discretisation M of A(T ). Repeated actions of M
for the hybrid technique described above. We therefore con- are applied to the discrete initial state u0 using the same time-
struct a two-dimensional mesh of 3626 quadrilateral elements integration code as for the non-linear equations [21]. The re-
at polynomial order P = 6, and exploit the domain symmetry sulting sequence of vectors spans a Krylov subspace of M and,
with a Fourier pseudospectral method consisting of 160 collo- through a partial Hessenberg reduction, the leading eigenval-
cation points in the spanwise direction to perform the simula- ues and eigenvectors can be efficiently determined. The same
tion. This yields a resolution of 20.9M degrees of freedom per approach can be applied to the adjoint form of the linearised
field variable and allows us to obtain excellent agreement with Navier-Stokes evolution operator A∗ (T ) to examine the recep-
the benchmark statistics, which are available in reference [32]. tivity of the flow and, in combination with the direct mode,
identify the sensitivity to base flow modification and local feed-
4.3. Flow stability
back. The direct and adjoint methods can also be combined to
In addition to direct numerical simulation of the full identify convective instabilities over different time horizons τ
non-linear incompressible Navier-Stokes equations, the in a flow by computing the leading eigenmodes of (A∗ A)(τ).
IncNavierStokesSolver supports global flow stability This is referred to as transient growth analysis.
analysis through the linearised Navier-Stokes equations with To illustrate the linear analysis capabilities of Nektar++, we
respect to a steady or periodic base flow. This process identifies use the example of two-dimensional flow past a circular cylin-
whether a steady flow is susceptible to a fundamental change der at Re = 42, just below the critical Reynolds number for
of state when perturbed by an infinitesimal disturbance. The the onset of the Bénard-von Kármán vortex street. This is a
linearisation takes the form well-established test case for which significant analysis is avail-
∂u0 able in the literature. We show in Fig. 6 the leading eigen-
+ (u0 · ∇)U + (U · ∇)u0 = −∇p0 + ν∇2 u0 (4a) modes for the direct (A) and adjoint (A∗ ) linear operators for
∂t
∇ · u0 = 0, (4b) both the streamwise and cross-stream components of velocity.
The modes are characterised by the asymmetry in the stream-
where U is the base flow and u0 is now the perturbation. wise component and symmetry in the cross-stream component.
The time-independent base flow is computed through evolving We also note the spatial distribution of the modes with the
Equs. (3) to steady-state with appropriate boundary conditions. leading direct modes extending far downstream of the cylin-
10
der, while the adjoint modes are predominantly localised up- into a scalar Helmholtz type equation and solved for the aux-
stream but close to the cylinder. This separation is a result of iliary variable z = ∇ · ∂t (Hu). The conservative variables are
the non-normality of the A operator. We also show the struc- recovered in a subsequent step.
tural sensitivity of the flow to base flow modification and local A frequently used test-case for Boussinesq models is the
feedback. The latter highlights regions where localised forcing scattering of a solitary wave impinging a vertical cylinder. Here
would have greatest impact on the flow. a solitary wave with nonlinearity  = 0.1 is propagating over a
still water depth of 1m ( = A/d, where A is the wave ampli-
4.4. Shallow water modelling tude). The initial solitary wave condition is given by Laitone’s
first order solution. The cylinder has a diameter of 4m, giving
The ShallowWaterSolver simulates depth-averaged wave a Keulegan-Carpenter number well below unity and diffraction
equations, often referred to as “long-wave” approximations. number on the order of 2. Hence, the viscous effects are small
These equations are often used for engineering applications while the diffraction and scattering are significant.
where the vertical dimension of the flow is small compared We compute the solution in the domain x ∈ [−25 , 50] me-
to the horizontal. Examples of applications include tidal flow, ters and y ∈ [−19.2 , 19.2] meters, discretized into 552 triangles
river flooding and nearshore phenomena such as wave-induced using P = 5. Snapshots of the free surface elevation at four dif-
circulation and wave disturbances in ports. ferent times are shown in Fig. 7. In Fig. 7(a) the solitary wave
The governing equations are derived from potential flow: reaches its maximum run-up on the cylinder, while in Fig. 7(b)
the Laplace equation inside the flow domain and appropriate the peak of solitary wave has reached the center of the cylin-
boundary conditions at the free surface and bottom. The two der and a depression in the free surface around the cylinder is
key steps are (i) the expansion of the velocity potential with re- clearly visible. The propagation of the scattered waves, and
spect to the vertical coordinate and (ii) the integration of the those later reflected from the side walls, are seen in Figs. 7(c)
Laplace equation over the fluid depth. This results in sets of and 7(d).
equations expressed in horizontal dimensions only. Depending
on the order of truncation in nonlinearity and dispersion, nu- 4.5. Cardiac electrophysiology
merous long-wave equations with different kinematic behaviour
The cardiac electrical system in the heart is the signalling
have been derived over the years [34, 35, 36].
mechanism used to ensure coordinated contraction and efficient
Many depth-averaged equations can be written in a generic
pumping of blood. Conduction occurs due to a complex se-
form as
quence of active ion exchanges between intracellular and extra-
∂U cellular spaces, initiated due to a potential difference between
+ ∇ · F(U) + D(U) = S(U) , (5) the inside and outside of the cell exceeding a threshold, pro-
∂t
ducing an action potential. This causes a potential difference
where U = [H , Hu , Hv]T is the vector of conserved variables. across boundaries with adjacent cells, resulting in a flow of ions
The horizontal velocity is denoted by u = [u(x, t) , v(x, t)]T , between cells and triggering an action potential in the adjacent
H(x, t) = η(x, t) + d(x) is the total water depth, η is the free cell. Disease, age and infarction lead to interruption of this sig-
surface elevation and d the still water depth. The flux vector nalling process and may produce abnormal conduction patterns
F(U) is given as known as arrhythmias. Clinically, these can be treated using
  catheter ablation, however accurately selecting the most effec-
 Hu Hv  tive substrate modification to restore normal rhythm is often
F(U) =  Hu2 + gH 2 /2 Huv  , (6)
 
particularly challenging and may benefit from insight derived
Huv Hv2 + gH 2 /2

from computer modelling.
The CardiacEPSolver models the conduction process us-
in which g is the acceleration due to gravity. The source term ing the monodomain equations
S(U) contains forcing due to, for example, Coriolis effects, bed-
∂u
!
slopes and bottom friction. Importantly, D(U) contains all the
β Cm + Iion = ∇ · σ∇u,
dispersive terms. The actual form of the dispersive terms differs ∂t
between different wave equations and the term can be highly
complex with many high-order mixed and spatial derivatives. where the Iion term captures the complex movement of ions
At present, the ShallowWaterSolver supports the non- in and out of cells and is itself modelled as a set of ordinary
dispersive shallow-water equations (SWE) and the weakly dis- differential equations. Additionally, σ captures the potentially
persive Boussinesq equations of Peregrine [34]. The SWE are heterogeneous and anisotropic nature of the tissue which gov-
recovered if D(U) ≡ 0 while for the Peregrine equation the ex- erns the speed of electrical propagation. While full 3D simula-
pression is: tions of myocardium are traditionally performed, the left atrium
is sufficiently thin that it can be reasonably represented as a
 
 0  two-dimensional manifold embedded in three dimensions and
D(U) = ∂t  (d3 /6)∂ x (∇ · (Hu/d)) − (d2 /2)∂ x (∇ · (Hu))  . solved at significantly reduced computational cost [38].
 
(d /6)∂y (∇ · (Hu/d)) − (d2 /2)∂y (∇ · (Hu))
 3
An example of conduction propagation over the left atrium
(7) using the monodomain equations is illustrated in Figure 8.
Electrophysiological characteristics vary spatially, with regions
The Boussinesq equations are solved using the wave conti- of scar and fibrosis more resistive to activation, resulting in ac-
nuity approach [37]. The momentum equations are first recast tivation patterns of greater complexity. The geometry is derived
11
 (a)

(b)

Figure 8: Illustrative simulation of a depolarising electrochemical wavefront on

a two-dimensional manifold representation of a human left atrium. Blue areas
denote regions of unexcited (polarised) tissue, while green denotes areas of
(c) excited (depolarised) cells. The red areas highlight the wavefront. Simulation
input files are provided in Appendix A.21.

form is given by
∂U ∂U
+H = S, (8)
∂t ∂x
" # " # " #
U U A 0 
U= , H = ∂p , S= 1f ,
A ρ ∂A U ρ A −s
(d)
in which A is the Area (related to pressure), x is the axial coor-
dinate along the vessel, U(x, t) the axial velocity, P(x, t) is the
pressure in the tube, ρ is the density and finally f the frictional
force per unit length. The unknowns in Equ. (8) are u, A and
p; hence, we must provide an explicit algebraic relationship to
close this system. Typically, closure is provided by an algebraic
relationship between A and p.
Figure 7: Solitary wave impinging a stationary cylinder. The colours and sur- For a thin elastic tube this is given by
face deformation illustrate the height of the surface at times (a) t = 4.5s; (b)
t = 5.5s; (c) t = 8.5s; and (d) t = 12.5s. Simulation input files are provided in
√
√ p  πhE
Appendix A.20. p = p0 + β A − A0 , β = , (9)
(1 − ν2 )A0

where p0 is the external pressure, A0 is the initial cross-

from segmented magnetic resonance imaging (MRI) data, while
sectional area, E is the Young’s modulus, h is the vessel wall
tissue heterogeneity is prescribed based on late gadolinium-
thickness and ν is the Poisson’s ratio. Other more elaborate
enhanced MRI. A human atrial ionic model [39] is used to com-
pressure-area relationships are currently being implemented
pute the Iion term and represents the exchanges of ions between
into the framework. Application of Riemann’s method of char-
the interior and exterior of the cell, along with other cellular
acteristics to Equs. (8) and (9) indicates that velocity and area
biophysics.
are propagated through the system by forward and backward
travelling waves. These waves are reflected within the network
4.6. Arterial pulse-wave propagation by appropriate treatment of interfaces and boundaries (see for
example [40, 41]). The final system of equations are discretised
1D modelling of the vasculature (arterial network) represents in the Nektar++ framework using a discontinuous Galerkin pro-
an insightful and efficient tool for tackling problems encoun- jection.
tered in arterial biomechanics as well as other engineering prob- To illustrate the capabilities of the PulseWaveSolver, a 1D
lems. In particular, 3D modelling of the vasculature is relatively geometry is created by extracting the centreline directly from a
expensive. 1D modelling provides an alternative in which the 3D segmentation of a carotid bifurcation. The extracted centre-
modelling assumptions provide a good balance between physi- line with the segmented geometry overlaid is shown in Figure 9.
ological accuracy and computational efficiency. To describe the At the inlet a half-sinusoidal flow profile is applied during the
flow and pressure in this network we consider the conservation systolic phase, whist during the diastolic phase a no-flow con-
of mass and momentum applied to an impermeable, deformable dition is applied. Although this profile is not representative of
tube filled with an incompressible fluid, the nonlinear system the carotid wave, it is useful for demonstrating essential dynam-
of partial differential equations presented in non-conservative ics of the system e.g. reflection of backward travelling waves
12
 We next solve the advection-diffusion equation,
∇ · (−D∇c + cu) = 0,
to model transport of oxygen along the arterial wall, again using
the ADRSolver. Here, c is the concentration and u is the steady-
state flow solution obtained previously. D = 1/Pe is the diffu-
sivity of the species considered where we use Pe = 7.5 × 105
for oxygen. This value corresponds to a Schmidt number (rel-
ative size of mass transfer and momentum boundary layers)
of 3000, which is typical of species such as free oxygen or
adenosine triphosphate (ATP). For most of the domain, the non-
dimensional concentration remains constant at c = 1. However,
a particularly high gradient in concentration forms at the wall.
Biologically, it is this non-dimensional concentration gradient,
the Sherwood number (Sh), in the vicinity of the cells that line
the arterial wall which is of particular interest. This is given by
Figure 9: Geometry used for the simulation. Insets show how flow and pressure
vary with time and different locations in the geometry. Simulation input files Sh = 2∇c · n,
are provided in Appendix A.22.
where c is the non-dimensional concentration and n is the local
wall normal.
only during the diastolic phase. At the outflow RCR boundary The existing mesh used for the flow simulation is unable to
conditions are utilised [41]. The RCR model is an electrical resolve the concentration gradients close to the wall. We there-
analogy consisting of two resistors (total peripheral resistance) fore refine the boundary layer in the wall-normal direction, with
and a capacitor (peripheral compliance). This boundary con- element heights following a geometric progression, using an
dition takes into account the effects of the peripheral vessels isoparametric refinement technique [42, 43] to naturally curve
(e.g. small arteries, arterioles and capillaries) on pulse wave the resulting subelements in such a way as to guarantee their
propagation. The pressure and flow results are shown in Fig. 9. validity. This technique is implemented in the MeshConvert
The insets demonstrate that the pressure needs about 4 cycles utility, which acts as both a way to convert meshes from other
to reach a periodic state. The boundary condition is responsi- formats such as Gmsh [44] and Star-CCM+, but also to apply
ble for establishing the correct pressure-flow relationship on the a variety of processing steps to the mesh in order to make it
outflow and throughout the domain. suitable for high-order computation.
To minimise computational cost, we reduce the polynomial
order of the prisms in the directions parallel to the wall, since
4.7. Vascular mass transport the concentration shows negligible variation in these directions.
This exploits the rich nature of the spectral/hp discretisation
To conclude this section, we illustrate the flexibility of the and removes the need for a potentially expensive remeshing and
software in combining two solvers to understand mass trans- interpolation step. Furthermore, the domain-interior tetrahedra
port in the aorta. We consider the simulation of blood-flow may also be discarded and a Dirichlet c = 1 condition imposed
through a pair of intercostal branches in the decending aorta. on the resulting interior prism boundary.
The three-dimensional geometry is derived from CT scans and Fig. 10(c) shows the resulting Sherwood number distribution
is meshed using a combination of tetrahedral and prismatic el- on the surface of the arterial wall. Regions of reduced mass
ements. Prisms are used to better capture the boundary layer flux are observed upstream of the intercostal branches, while
close to the wall, while tetrahedra fill the remaining interior of elevated mass flux are observed downstream. Fig.10(d) shows
the domain. The embedded manifold discretisation code can close-ups of the branches to illustrate this. These patterns are
be used to compute boundary conditions for three-dimensional driven directly by the blood flow mechanics in these regions. In
simulations where the complexity of the geometry precludes particular, upstream of the intercostal branch the mass transfer
the use of analytic conditions. The resulting inflow condition is boundary layer grows due to a growth of the momentum bound-
shown in Fig. 10(a). ary layer as it negotiates the sharp bend into the branch, forcing
The flow is modelled using the incompressible Navier-Stokes flow away from the apex of the bend; the bulk of the flow con-
equations (see Equs (3) in Section 4.1). In this case, the inlet tinues down the aorta with only a small proportion entering the
flow-profile f in Equ. 3(e) is the solution of the Poisson prob- branch. Progressing into the branch, as shown in the top inset
lem, computed using the ADRSolver, on the two-dimensional of Fig.10(d), the mass flux is slightly elevated as the boundary
inlet boundary surface for a prescribed body force. The result- layer shrinks and the flow is directed towards the wall of the
ing profile is imposed on the three-dimensional flow problem as branch, causing the mass transfer boundary layer to shrink. In
illustrated and the steady-state velocity field from the flow sim- the main aorta, distal to the intercostal branch, mass flux to the
ulation at Reynolds number Re = 300 is shown in Fig. 10(b). arterial wall is elevated. This is associated with a flow stagna-
A single boundary layer of prismatic elements is used for this tion region that forms due to the impingement of flow crossing
simulation and both the prismatic and tetrahedral elements use the branch mouth onto the wall, as illustrated in Fig.10(b). Be-
a polynomial expansion order of P = 4. This is sufficient to low the impingement zone the boundary layer grows, leading to
capture the boundary layer at the walls. a reduction in the mass flux.
13
 (a)

(b) (c) (d)

Figure 10: Calculation of mass transport in an intercostal pair. (a) Inflow boundary condition for fluid simulation computed by solving a Poisson problem on
the inflow surface. (b) Flow computed using the incompressible Navier-Stokes equations. (c) Mass transport of low diffusion coefficient species simulated using
the advection-diffusion solver with Pe = 7.5 × 105 . Non-dimensional gradient of concentration (Sherwood number) at the wall is shown. (d) Detailed view of
non-dimensional concentration gradient at intercostal branches. Simulation input files are provided in Appendix A.23.

5. Discussion & Future Directions code merged into the main codebase, thereby always maintain-
ing a stable distribution. New bugs and feature requests are
The Nektar++ framework provides a feature-rich platform recorded using the Trac [46] issue-management system. To en-
with which to develop numerical methods and solvers in the able cross-platform compatibility Nektar++ uses CMake [47]
context of spectral/hp element methods. It has been designed to manage the creation of build scripts, which also allows the
in such a way that the libraries reflect the mathematical abstrac- automatic download and compilation of additional third-party
tions of the method, to simplify uptake for new users, as well libraries and simplifies the configuration and installation for the
as being written in a modular manner to improve the robustness end-user. Boost [48] data structures and algorithms are used
of the code, minimise duplication of functionality and promote throughout the code to simplify complex data management, im-
sustainability. prove code modularity and avoid the introduction of memory
leaks. While the templated nature of many of the Boost libraries
significantly adds to compilation times, we consider the bene-
Development & Tools fits to code robustness justify its use.
The development of a complex and extensive software Testing is a critical part of the development cycle for any
project such as Nektar++ necessitates the adoption of certain software project and regression tests ensure new features do not
development practices to enable developers to easily write new break existing functionality and ensures the code base remains
code without breaking the existing code for other users of the stable when new features are implemented. Continuous integra-
framework. The code is managed using the git distributed ver- tion using a publically accessible buildbot service [49], builds
sion control system [45] due to its performance, enhanced sup- and executes these tests after each update to the master branch
port for branching, as well as allowing off-line development. of the code, across a range of operating systems, architectures
All development is performed in branches and only after rig- and configuration options. The system may also be used by de-
orous multi-architecture testing and internal peer-review is new velopers to test other branches prior to inclusion in the main
14
codebase. (a) 140
Nektar++ makes extensive use of C++ programming pat-
120
terns to decouple and manage components of the code and
the creation of objects at runtime. As well as limiting inter- 100
dependencies within the source code, it improves compila-

Speedup (T32/T)
tion times, enforces modularity and simplifies compile-time 80
selection of features and functionality. Design patterns for-
60
malise many aspects of writing high-quality, robust code and
we briefly outline some of the key patterns used within Nek- 40
tar++.
The Template method pattern provides separation between 20
Ideal
algorithms and specific implementation. A general algorithm is HECTOR
0
implemented in a C++ base class, while particular aspects of 0 500 1000 1500 2000 2500 3000 3500 4000 4500
the algorithm implementation are overridden in derived classes Nproc
through the use of protected virtual functions. These derived
classes could correspond to specific element shapes or Galerkin (b) 140

projections, for example. The Factory method pattern allows

120
dynamic key-based object creation at runtime, without prescrib-
ing the particular implementation choice a priori within the 100

Speedup (T24/T)
code at compilation time. The technique is used extensively
within the libraries as a means to decouple components of the 80

code and manage multiple implementations of an algorithm. 60

Additional implementation modules can be added to the code
at a later date without needing to modify those routines which 40
instantiate the objects. Finally, Managers provide a templated
20
mechanism to keep track of large numbers of similarly-typed Ideal
objects during program execution and avoid duplication where ARCHER
0
possible and so minimise memory usage. Their underlying data 0 500 1000 1500 2000 2500 3000 3500
structure is a static map. For each manager, a functor is held Nproc

which can be used to instantiate objects which have not previ-

ously been allocated. Figure 11: Strong scaling of the Nektar++ diffusion solver on an intercostal pair
simulation similar to the one presented in section 4.7. (a) Performance scaling
Performance on HECToR where each node has 32 cores and (b) performance scaling on
ARCHER where each node has 24 cores. In both cases, results are normalised
Modern high-performance computing has transformed in re- by the performance on a single node.
cent years as processor clock speeds have reached practical lim-
itations and vendors have been forced to increase parallelism in
order to support greater computation. This has resulted in an ters. The code has been tested on large clusters such as HEC-
increase in the number of cores per processor die and subse- ToR and ARCHER, the UK National High Performance Com-
quently an effective reduction in available cache per core. It is puting Facilities, and shows excellent scaling for both two-
becoming ever more expensive to move data within a computer dimensional and three-dimensional problems. Figure 11 shows
and modern software must therefore be engineered to optimise strong scaling for the implicit diffusion solve for an intercostal
algorithms to make the most of data while it is on the CPU. flow simulation similar to that presented in section 4.7. This
High-order methods naturally increase data locality by produc- mesh contained approximately 8,000 tetrahedral elements, re-
ing tightly coupled blocks of data. This enables greater cache sulting in only 2 or 3 elements per core in the most parallel
coherency and therefore supports a larger number of floating case, which accounts for the reduced efficiency at higher core
point operations to be achieved per cache line than conventional counts.
linear finite element methods, when using high polynomial or-
ders.
Future work
For each of the key finite element operators, the Nektar++
architecture supports multiple implementations. This allows Although the current version of Nektar++ supports variable
the code to be targeted at a variety of architectures in an effi- and heterogeneous choices of polynomial order, it does not
cient manner based on the choice of input parameters, such as yet support adaptive polynomial order during time advance-
polynomial orders and mesh configuration. Although mathe- ment (p-adaptivity). This is one of the next features to be
matically equivalent, these implementations may lead to slight implemented in the code. Mesh refinement (h-adaptivity) is
differences in the numerical result due to the order of floating- a well-established technique in many other finite element re-
point operation evaluation. However, for most applications, search codes, and we believe hp-adaptivity will provide sub-
double-precision arithmetic provides more than sufficient pre- stantial performance benefits in a wide range of application ar-
cision to ensure this is not a practical problem. eas.
Nektar++ is designed to work on a wide range of computer Finally, it is becoming increasingly costly for individual in-
systems from laptops to large high-performance compute clus- stitutions to purchase and maintain the necessary large-scale
15
HPC infrastructures to support cutting-edge research. In re- File: user-guide.pdf
cent years cloud computing has become increasingly prevailent Figure A.14: User guide for Nektar++ detailing compilation, installation, input
and is potentially the approach by which extensive computa- format and usage, including examples.
tional resources may be obtained for simulation in the future.
Nektar++ is embracing this infrastructure shift through the de- File: cylinder.zip
velopment of Nekkloud [50] which removes the complexities
Figure A.15: Nektar++ input files for simulating compressible flow over a
of maintaining and deploying numerical code onto cloud plat- cylinder at Re=3, 900 using the CompressibleFlowSolver.
forms.
File: naca0012.zip
Figure A.16: Nektar++ input files for simulating Euler flow over a NACA0012
6. Availability
aerofoil at Ma∞ = 0.8 using the CompressibleFlowSolver.

Nektar++ is open-source software, released under the MIT File: t106c.zip

license, and is freely available from the project website
Figure A.17: Nektar++ input files for simulating compressible flow
(http://www.nektar.info). While the git repository is freely ac- over a T106C low-pressure turbine blade at Re=80, 000 using the
cessible, discrete releases are made at milestones in the project CompressibleFlowSolver.
and are available to download as compressed tar archives, or
as binary packages for a range of operating systems. These File: 2dhill.zip
releases are considered to contain relatively complete function-
Figure A.18: Nektar++ input files for simulating incompressible flow over a
ality compared to the repository master branch. periodic hill at Re=2, 800.

File: stability.zip
Acknowledgments
Figure A.19: Nektar++ input files for computing the leading direct and ad-
joint eigenmodes of incompressible flow past a cylinder at Re=42 using the
Nektar++ has been developed over a number of years and we IncNavierStokesSolver.
would like to thank the many people who have made contri-
butions to the specific application codes distributed with the File: solitary.zip
libraries. In particular, we would like to acknowledge the Figure A.20: Nektar++ input files for simulating a solitary wave impinging on
contribution of Christian Roth for initial developments on the a stationary circular cylinder using the ShallowWaterSolver.
pulse-wave solver, Kilian Lackhove for work on extending the
acoustic perturbation equations solver and Rheeda Ali, Eugene File: atrium.zip
Chang and Caroline Roney for contributing to the cardiac elec- Figure A.21: Nektar++ input files for simulating the propagation of an acti-
trophysiology solver. vation wavefront across a heterogeneous anatomically realistic model of the
The development of Nektar++ has been supported by human left atrium using the CardiacEPSolver.
a number of funding agencies including Engineering and
Physical Sciences Research Council (grants EP/L000407/1, File: pulsewave.zip
EP/K037536/1, EP/K038788/1, EP/L000261/1, EP/I037946/1, Figure A.22: Nektar++ input files for modelling pulse wave propagation along
EP/H000208/1, EP/I030239/1, EP/H050507/1, EP/D044073/1, the centreline of a carotid bifurcation using the PulseWaveSolver.
EP/C539834/1), the British Heart Foundation (grants
FS/11/22/28745 and RG/10/11/28457), the Royal Society File: mass-transport.zip
of Engineering, McLaren Racing, the National Science Foun- Figure A.23: Nektar++ input files for calculating mass transport in
dation, the Army Research Office, the Air Force Office of an intercostal pair using the IncNavierStokesSolver, ADRSolver and
Scientific Research and the Department of Energy. FieldConvert utility.

References
Appendix A. Supplementary material [1] A. T. Patera, A spectral element method for fluid dynamics: laminar flow
in a channel expansion, Journal of computational Physics 54 (3) (1984)
468–488.
[2] I. Babuska, B. A. Szabo, I. N. Katz, The p-version of the finite element
File: Diffusion.cpp method, SIAM journal on numerical analysis 18 (3) (1981) 515–545.
[3] G. E. Karniadakis, S. J. Sherwin, Spectral/hp element methods for CFD,
Figure A.12: Implementation, using the Nektar++ framework, of a solver for Oxford University Press, 2005.
the unsteady diffusion problem advanced with backward-Euler time integration. [4] H. M. Blackburn, S. Sherwin, Formulation of a galerkin spectral element–
fourier method for three-dimensional incompressible flows in cylindrical
geometries, Journal of Computational Physics 197 (2) (2004) 759–778.
[5] P. Fischer, J. Kruse, J. Mullen, H. Tufo, J. Lottes, S. Kerkemeier,
File: DiffusionTimeInt.cpp
Nek5000–open source spectral element cfd solver, Argonne National
Laboratory, Mathematics and Computer Science Division, Argonne, IL,
Figure A.13: Implementation of a solver, using the Nektar++ framework,
see https://nek5000. mcs. anl. gov/index. php/MainPage.
for the unsteady diffusion problem using the general linear methods time-
[6] T. Vejchodskỳ, P. Šolı́n, M. Zı́tka, Modular hp-FEM system HERMES
integration approach.
and its application to Maxwells equations, Mathematics and Computers
in Simulation 76 (1) (2007) 223–228.

16
 [7] A. Dedner, R. Klöfkorn, M. Nolte, M. Ohlberger, A generic interface for in: 7th AIAA Theoretical Fluid Mechanics Conference, AIAA Aviation,
parallel and adaptive discretization schemes: abstraction principles and American Institute of Aeronautics and Astronautics, 2014.
the DUNE-FEM module, Computing 90 (3-4) (2010) 165–196. [30] P.-O. Persson, J. Peraire, Sub-cell shock capturing for discontinuous
[8] W. Bangerth, R. Hartmann, G. Kanschat, deal.II–a general-purpose Galerkin methods, AIAA 112.
object-oriented finite element library, ACM Transactions on Mathemat- [31] M. Breuer, N. Peller, C. Rapp, M. Manhart, Flow over periodic hills–
ical Software (TOMS) 33 (4) (2007) 24. numerical and experimental study in a wide range of Reynolds numbers,
[9] J. S. Hesthaven, T. Warburton, Nodal discontinuous Galerkin methods: Computers & Fluids 38 (2) (2009) 433–457.
algorithms, analysis, and applications, Vol. 54, Springer, 2007. [32] ERCOFTAC QNET-CFD Database for test case UFR 3-30, 2D Periodic
[10] F. Witherden, A. Farrington, P. Vincent, PyFR: An open source frame- Hill Flow: database of numerical and experimental results (2014).
work for solving advectiondiffusion type problems on streaming archi- URL http://qnet-ercoftac.cfms.org.uk/w/index.php/UFR 3-30 Referen
tectures using the flux reconstruction approach, Computer Physics Com- [33] W. E. Arnoldi, The principle of minimized iterations in the solution of
munications 185 (2014) 30283040. doi:10.1016/j.cpc.2014.07.011. the matrix eigenvalue problem, Quarterly of Applied Mathematics 9 (1)
[11] M. Dubiner, Spectral methods on triangles and other domains, Journal of (1951) 17–29.
Scientific Computing 6 (4) (1991) 345–390. [34] D. H. Peregrine, Long waves on a beach, Journal of Fluid Mechanics 27
[12] S. J. Sherwin, G. E. Karniadakis, A triangular spectral element method; (1967) 815–827.
applications to the incompressible Navier-Stokes equations, Computer [35] P. Madsen, H. Schäffer, Higher-order Boussinesq-type equations for sur-
Methods in Applied Mechanics and Engineering 123 (1-4) (1995) 189– face gravity waves: derivation and analysis, Philosophical Transactions
229. of the Royal Society London A 356 (1998) 3123–3184.
[13] M. G. Duffy, Quadrature over a pyramid or cube of integrands with a [36] M. Brocchini, A reasoned overview on Boussinesq-type models: the in-
singularity at a vertex, SIAM journal on Numerical Analysis 19 (6) (1982) terplay between physics, mathematics and numerics, Philosophical Trans-
1260–1262. actions of the Royal Society London A 469.
[14] H. M. Tufo, P. F. Fischer, Fast parallel direct solvers for coarse grid prob- [37] C. Eskilsson, S. Sherwin, Spectral/hp discontinuous Galerkin methods for
lems, Journal of Parallel and Distributed Computing 61 (2) (2001) 151– modelling 2D Boussinesq equations, Journal of Computational Physics
177. 212 (2006) 566–589.
[15] S. J. Sherwin, M. Casarin, Low-energy basis preconditioning for ellip- [38] C. D. Cantwell, S. Yakovlev, R. M. Kirby, N. S. Peters, S. J. Sherwin,
tic substructured solvers based on unstructured spectral/hp element dis- High-order spectral/hp element discretisation for reaction-diffusion prob-
cretization, Journal of Computational Physics 171 (1) (2001) 394–417. lems on surfaces: Application to cardiac electrophysiology, Journal of
[16] P. E. Vos, S. J. Sherwin, R. M. Kirby, From h to p efficiently: Imple- Computational Physics 257 (2014) 813–829.
menting finite and spectral/hp element methods to achieve optimal perfor- [39] M. Courtemanche, R. J. Ramirez, S. Nattel, Ionic mechanisms underly-
mance for low- and high-order discretisations, Journal of Computational ing human atrial action potential properties: insights from a mathematical
Physics 229 (13) (2010) 5161–5181. model, American Journal of Physiology-Heart and Circulatory Physiol-
[17] C. D. Cantwell, S. J. Sherwin, R. M. Kirby, P. H. J. Kelly, From h to p ogy 44 (1) (1998) H301.
efficiently: strategy selection for operator evaluation on hexahedral and [40] S. Sherwin, L. Formaggia, J. Peiro, V. Franke, Computational modelling
tetrahedral elements., Computers & Fluids 43 (2011) 23–28. of 1D blood flow with variable mechanical properties and its applica-
[18] C. D. Cantwell, S. J. Sherwin, R. M. Kirby, P. H. J. Kelly, From h to p tion to the simulation of wave propagation in the human arterial system,
efficiently: selecting the optimal spectral/hp discretisation in three dimen- International Journal for Numerical Methods in Fluids 43 (6-7) (2003)
sions, Math. Mod. Nat. Phenom. 6 (2011) 84–96. 673–700.
[19] P. Fischer, J. Lottes, D. Pointer, A. Siegel, Petascale algorithms for reactor [41] J. Alastruey, K. Parker, J. Peiró, S. Sherwin, Lumped parameter outflow
hydrodynamics, in: Journal of Physics: Conference Series, Vol. 125, IOP models for 1-D blood flow simulations: effect on pulse waves and param-
Publishing, 2008, p. 012076. eter estimation, Communications in Computational Physics 4 (2) (2008)
[20] P. E. Vos, C. Eskilsson, A. Bolis, S. Chun, R. M. Kirby, S. J. Sher- 317–336.
win, A generic framework for time-stepping partial differential equations [42] D. Moxey, M. D. Green, S. J. Sherwin, J. Peiró, An isoparametric
(PDEs): general linear methods, object-oriented implementation and ap- approach to high-order curvilinear boundary-layer meshing, Computer
plication to fluid problems, International Journal of Computational Fluid Methods in Applied Mechanics and Engineering 283 (2015) 636–650.
Dynamics 25 (3) (2011) 107–125. doi:10.1016/j.cma.2014.09.019.
[21] D. Barkley, H. Blackburn, S. J. Sherwin, Direct optimal growth analysis [43] D. Moxey, M. Hazan, J. Peiró, S. J. Sherwin, On the generation of curvi-
for timesteppers, International journal for numerical methods in fluids linear meshes through subdivision of isoparametric elements, to appear in
57 (9) (2008) 1435–1458. proceedings of Tetrahedron IV (Jan. 2014).
[22] S. Dong, G. E. Karniadakis, C. Chryssostomidis, A robust and accu- [44] C. Geuzaine, J.-F. Remacle, Gmsh: A 3-d finite element mesh gener-
rate outflow boundary condition for incompressible flow simulations ator with built-in pre- and post-processing facilities, International Jour-
on severely-truncated unbounded domains, Journal of Computational nal for Numerical Methods in Engineering 79 (11) (2009) 1309–1331.
Physics 261 (2014) 83–105. doi:10.1002/nme.2579.
[23] R. M. Kirby, S. J. Sherwin, Stabilisation of spectral/hp element methods [45] L. Torvalds, J. Hamano, GIT: Fast version control system (2014).
through spectral vanishing viscosity: Application to fluid mechanics mod- URL http://git-scm.com
elling, Computer methods in applied mechanics and engineering 195 (23) [46] Trac integrated SCM & project management (2014).
(2006) 3128–3144. URL http://trac.edgewall.org
[24] G. E. Karniadakis, M. Israeli, S. A. Orszag, High-order splitting meth- [47] CMake (2014).
ods for the incompressible Navier-Stokes equations, Journal of Computa- URL http://cmake.org
tional Physics 97 (2) (1991) 414–443. [48] Boost C++ libraries (2014).
[25] S. A. Orszag, M. Israeli, M. O. Deville, Boundary conditions for incom- URL http://www.boost.org
pressible flows, Journal of Scientific Computing 1 (1) (1986) 75–111. [49] Buildbot (2014).
[26] E. Ferrer, D. Moxey, S. J. Sherwin, R. H. J. Willden, Stability of projec- URL http://www.buildbot.net
tion methods for incompressible flows using high order pressure-velocity [50] J. Cohen, D. Moxey, C. Cantwell, P. Burovskiy, J. Darlington, S. J. Sher-
pairs of same degree: Continuous and Discontinuous Galerkin formula- win, Nekkloud: A software environment for high-order finite element
tions, Communications in Computational Physics 16 (3) (2014) 817–840. analysis on clusters and clouds, in: Cluster Computing (CLUSTER), 2013
doi:10.4208/cicp.290114.170414a. IEEE International Conference on, IEEE, 2013, pp. 1–5.
[27] D. de Grazia, G. Mengaldo, D. Moxey, P. E. Vincent, S. J. Sherwin, Con-
nections between the discontinuous Galerkin method and high-order flux
reconstruction schemes, International journal for numerical methods in
fluids 75 (12) (2014) 860–877. doi:10.1002/fld.3915.
[28] E. F. Toro, Riemann solvers and numerical methods for fluid dynamics: a
practical introduction, 3rd Edition, Springer, Berlin, New York, 2009.
[29] G. Mengaldo, D. De Grazia, J. Peiro, A. Farrington, F. Witherden, P. E.
Vincent, S. J. Sherwin, A guide to the implementation of boundary con-
ditions in compact high-order methods for compressible aerodynamics,

17