Thanks to visit codestin.com
Credit goes to www.scribd.com

0% found this document useful (0 votes)
18 views21 pages

Neural Network Verification With Pyrat: Augustin Lemesle, Julien Lehmann, Tristan Le Gall November 1, 2024

Uploaded by

Patrick Winnett
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
18 views21 pages

Neural Network Verification With Pyrat: Augustin Lemesle, Julien Lehmann, Tristan Le Gall November 1, 2024

Uploaded by

Patrick Winnett
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 21

Neural Network Verification with PyRAT

Augustin Lemesle, Julien Lehmann, Tristan Le Gall


November 1, 2024

Abstract
arXiv:2410.23903v1 [cs.AI] 31 Oct 2024

As AI systems are becoming more and more popular and used in various critical domains (health,
transport, energy, ...), the need to provide guarantees and trust of their safety is undeniable. To this
end, we present PyRAT, a tool based on abstract interpretation to verify the safety and the robustness
of neural networks. In this paper, we describe the different abstractions used by PyRAT to find the
reachable states of a neural network starting from its input as well as the main features of the tool
to provide fast and accurate analysis of neural networks. PyRAT has already been used in several
collaborations to ensure safety guarantees, with its second place at the VNN-Comp 2024 showcasing its
performance.

1 Introduction
There is no doubt that Artificial Intelligence (AI) has taken over an important part of our lives and its recent
popularisation with large language models has anchored this change even more in our current landscape.
The use of AI is becoming more and more widespread reaching new sectors such as health, aeronautics,
energy, etc., where it can bring tremendous benefits but could also cause environmental, economic, or human
damage, in critical or high-risk systems. In fact, numerous issues are still being uncovered around the use
of AI, ranging from its lack of robustness in the face of adversarial attacks [1, 2], to the confidentiality and
privacy of the data used, the fairness of the decisions, etc.
Faced with these threats and the exponential growth of its use, regulations have started to emerge with
the European AI Act. Not waiting for regulations, tools have already been developed to respond to and
mitigate these threats by providing various guarantees from the data collection phase to AI training and
validation of the AI. Our interest here lies in this last phase, the formal validation of an AI system, and
more specifically a neural network, to allow its use in a high-risk system.
Neural networks are a subset of AI models and are also one of the most popular and widely used ar-
chitectures. They include a large variety of architectures (DNN, CNN, GNN, Residual, Transformers, etc.)
applicable to various use cases such as image classification, regression tasks, detection, control & command,
etc. Similarly, to a classical software, these neural networks, when implemented in critical systems, should
be tested and validated in order to provide strong guarantees regarding their safety. While there are many
approaches, such as empirically testing the robustness to small input variations, adversarial attacks or even
metamorphic transformations [3], we focus here on using formal methods to provide strong mathematical
guarantees on the safety of a neural network.
To this end, PyRAT, which stands for Python Reachability Assessement Tool, has been under develop-
ment at CEA since 2019, in the hope of simplifying the validation process of neural network. It provides an
simple interface and an easy step to formally prove the safety of neural network, which can often be seen
as an arduous task. To easily interface with most of the neural network training frameworks (Tensorflow,
PyTorch, etc.) PyRAT is written in Python and handles several standard AI formats (ONNX, NNet, Keras,
etc.). PyRAT also provides several possibilities and interfaces to evaluate the safety of a network through
standard safety property files (VNN-LIB or similar format) or through several Python APIs.
PyRAT has been time-tested in several national and international projects as well as in multiple col-
laboration with industrial partners. It has also participated in the international neural network verification
competition, the VNN-Comp, in 2023 and 2024 [4] achieving third and second place respectively.

1
2 Related tools
Even before the widespread use of machine learning, classical programs had to be formally verified and/or
certified to avoid any erroneous behavior when used in critical systems. Thus, many verification techniques
used for AI were initially developed to increase trust in critical systems, including model-checking [5, 6],
which represents the program as a state transition system and checks whether the model built satisfies a
formal property, often using Satisfiability Modulo Theory (SMT), or abstract interpretation [7, 8], which
abstracts the semantics of a program to compute an approximation of what the program does (e.g. an
over-approximation of the reachability set).
As the need arose, these methods were later adapted to the verification of neural networks with specific
modifications, e.g. no loops, no memory to allocate, but instead linear and non-linear operation on huge
matrices or tensors. The first tools developed for this purpose were SMT-based methods [9, 10, 11], which
rely on SMT formulations and solvers to compute the feasibility of an input-output property on small neural
networks. Instead of using SMT, some tools transformed the verification task into a constrained optimisation
problem in the Mixed Integer Programming (MIP) setting [12, 13]. Both of these techniques are known to
be computationally expensive.
To avoid these costly computations, methods based on abstract interpretation have been developed [14,
15]. In general, tools using these methods represent the inputs they want to verify as a set which is then
passed along the neural network and it yields a set that over-approximates the output which is used to
evaluate the property. The formalism used to represent the set along the neural network is called an abstract
domain. The accuracy of an analysis by abstract interpretation depends heavily on the abstract domain
chosen. If the result of the analysis is too imprecise, one way to improve the precision is to combine several
abstract domains [8, 16].
We briefly present a non-exhaustive list of tools aimed at the formal verification of neural networks with
abstract interpretation and we refer to Urban and Miné [17] for a more comprehensive survey. Many of these
tools, including PyRAT, accept the Open Neural Network Exchange (ONNX) format to describe the neural
network and the VNN-LIB format (based on SMT-LIB) to describe the properties to be verified and have
participated to at least one instance of the annual VNN-Comp. They are marked with (*).

• (*)alpha-beta-CROWN [18, 19, 20] uses the CROWN-domain which is mathematically equivalent
to the polytope domain with a tailored abstractions for ReLU and other non-linear functions, and
branch and bound verification strategy. This tool won the yearly VNN-Comp from 2021 to 2024.
• (*)MN-BaB [21] is a dual solver built on the DeepPoly domain [22] leveraging multi-neural relaxation
of the ReLU along with branch and bound verification strategy.
• (*)NNV [23, 24] and (*)nnenum [25, 26] use the star set domain that uses linear programming to
optimize ReLU abstractions for the Zonotopes domain
• Libra [16] also uses abstract interpretation, although for checking fairness properties (unlike PyRAT
and other tools that analyse reachability and safety properties). Like PyRAT, Libra can combine
several abstract domains to increase the precision of its analysis. Its domains include intervals, as well as
domains from the Apron [27] library or domains reimplemented from Symbolic [28], Deeppoly [22],
and Neurify [29].
• Saver [30, 31] verifies properties on support vector machines (SVMs), focusing on robustness or vul-
nerability properties of classifiers. Its domains include Intervals and Zonotopes.

PyRAT also relies on abstract interpretation techniques to verify the safety of neural networks. It im-
plements several abstract domains such as intervals, zonotopes, zonotopes paired with inequality constraints
or a DeepPoly reimplementation. These domains can be combined and optimised by PyRAT with multiple
strategies such as branch and bound.

2
3 Key Notions
3.1 Neural Networks
Neural networks are functions N : Rn → Rm which are generally defined in terms of layered architectures
alternating between linear and non-linear operations.
We represent a neural network as a directed graph of layers resulting in a composition of the operations
corresponding to each layer. For example, a simple feedforward neural network with an activation function
between each layers without any loop or branching in the network as shown in Figure 1.

Figure 1: Feedforward neural network.

It can also be defined as the composition of each layer of the network with x ∈ Rn the input and y ∈ Rm
the output of the network:
y = N (x) = σ2 (l2 (σ1 (l1 (x)))).
Typically, the activation functions σ1 , σ2 and σ3 can be the Rectified Linear Unit (ReLU), the Sigmoid
or the Tanh functions which are defined below. While the layers l1 , l2 and l3 can be matrix multiplication
or convolution layers.
ReLU(x) = max(x, 0)
1
Sigmoid(x) =
1 + e−x
ex − e−x
Tanh(x) = = 2 × Sigmoid(2x) − 1
ex + e−x
More complex architectures can include some residual connections between layers to model more complex
behaviors, as represented in Figure 2.

Figure 2: Residual neural network.

The representation in Figure 2 is equal to the following formula:

y = N (x) = σ2 (l2 (σ1 (l1 (x)))) + σ1 (l1 (x)).

Overall, we classify the layers of a neural network into three categories which will lead to different
treatment in PyRAT:
• Linear layers such as affine or convolutional layers.

• Non-linear layers such as the various activation functions but also layers implementing multiplication
or division between different layers (Mul, Div, Pow, ...).
• The layers that do not directly influence the values of the data but may be only modify their shape
such as concatenation, flattening or reshaping layers.

3
3.2 Reachability Analysis
Let N : Rn → Rm be a neural network. We define the notion of safety property by a set of constraints on
the inputs and outputs of N . Through set inclusion, we express the satisfiability of this safety property:
given the input set X ⊆ Rn and the output set Y ⊆ Rm defined by the safety property, the safety property
is satisfied if and only if N (X) ⊆ Y .
Ideally, we would like to define a safety property such that for all inputs x ∈ Rn of a given neural network
N , the output y is correctly classified by N . However, in practice, we cannot define the correct classification
for all inputs since this is precisely the task we are trying to solve using a neural network. Therefore, we can
only reduce the scope of what we aim to verify by defining more general properties.
The first type of safety property here consists of properties on the behavior of the models. For example,
given a certain input space X, we know that the output will always be y. These properties are well suited
for control-command neural networks such as ACAS-Xu [32] where the data are structured around some
physical values and their semantic is clear. For ACAS-Xu, Katz et al. [33] defined 10 such safety properties
on the neural networks used. For example, property 1 is expressed by:
• Description: If the intruder is distant and is significantly slower than the ownship, the score of a
Clear-of-Conflict advisory will always be below a certain fixed threshold.
• Input constraints: ρ ≥ 55947.691, vown ≥ 1145, vint ≤ 60

• Desired output property: the score for Clear-of-Conflict is at most 1500.


More generally for the ACAS-Xu, as we possess a fully defined specifications through a look up table
(LUT), the safety property can be formalised like in Damour et al. [34] by saying that the decision of the
neural network should be contained in the decision of the LUT:

∀x ⊆ R5 , N (x) ⊆ LU T (x)

On the other hand, with unstructured data such as images or sound, it is harder to define a semantically
meaningful set as the input space that we want to verify, e.g. "there is a pedestrian on the image". Defining
what a pedestrian is on an image is an open question and approaches such as using simulators or image
generation may answer this in the future [35]. For now, we focus on a more local approach to the safety
on such data. For example, considering a classification task, we can define a local robustness property as
follows:
Local robustness: Given an input x0 ∈ Rn and a modification size ϵ > 0, we want to prove or invalidate
the following property
∀x′ ∈ B∥.∥∞ (x0 , ϵ), arg max N (x0 )i = arg max N (x′ )i
i i

where B∥.∥∞ (x0 , ϵ) := {z | ∥x0 − z∥∞ ≤ ϵ}.


In this example, we have a set X := B∥.∥∞ (x0 , ϵ) and a set Y := {y ′ | arg maxi N (x0 )i = arg maxi yi′ },
and the local robustness property holds if N (X) ⊆ Y . The local robustness property can be extended to a
regression task, i.e. without the argmax by adding a threshold on the distance between the two outputs.
Computing N (X) for any given input set X ⊆ Rn is called a reachability analysis, and its exact compu-
tation is generally hard. For example, when the activation function of N are only ReLU, then computing
exactly N (X) is NP-complete [11]. Therefore, PyRAT uses abstract interpretation to compute an over-
approximation of the reachability set N (X).

3.3 Abstract Interpretation


Abstract Interpretation [7, 8] is a theoretical framework that can be applied to the verification of computer
programs and neural networks. Its principle is to abstract the semantics of a program and thus to compute
an over-approximation of its reachability set by the following method:
• Choose an abstract domain Λ♯ , and two operators α : P(Rd ) → Λ♯ and γ : Λ♯ → P(Rd ) forming a
Galois connection [8]

4
• Determine, for each layer Nj , a safe abstraction Nj♯ : Λ♯ → Λ♯ such that α ◦ N j ◦ γ ⊑ Nj♯ :

• Compute N ♯ (α(X)) by applying successively the abstraction of each layer to either α(X) (if it is the
first layer) or to the outputs of (the abstraction of) the previous layer
• The Galois connection ensures that N (X) ⊆ γ(N ♯ (α(X)))
The classical Abstract Interpretation framework assumes that Λ♯ is a complete lattice [8] (e.g. the Interval
lattice); however, the approach works even when the lattice Λ♯ is not complete (e.g. convex polyhedra) and
even when Λ♯ is not a lattice (e.g. zonotopes), as long as we can define safe abstraction of each operator.
Therefore, this approach is parameterised by the abstract lattice Λ♯ and how we abstract each layer,
especially when the layer implements a non-linear function. We will explain these choices in PyRAT in the
next sections.
Through the Galois connection, abstract interpretation ensures that the resulting abstraction of the
neural network correctly over-approximates its true reachable sets. As such, PyRAT is said to be correct,
i.e. PyRAT will only say that a safety property is verified or falsified if it is mathematically sure of the
result, otherwise if the abstraction is not tight enough the result will be unknown.

3.4 Soundness w.r.t. real arithmetic


In this paper, we consider the mathematical definition of the neural network operations, i.e. a semantics
based on real numbers. However, computers use floating-point arithmetic, which introduces, usually small,
rounding errors. In order to be sound with respect to the real arithmetic, PyRAT captures the rounding
error by choosing the most pessimistic rounding for each operation. This means that if the result of a real
number computation is x, then PyRAT will compute a range [x, x] where x and x are two floating-point
numbers such as x ≤ x ≤ x. We refer to Goubault [36] (resp. [37]) for a presentation of an abstract domain
sound w.r.t real arithmetic based on Zonotope (resp. Polyhedra). Note that the sound mode of PyRAT is
only available in CPU and can be disabled for faster analysis.

4 Abstract domains & abstract transformers


In this section, we present several domains to abstract P(Rd ) and the operation commonly found in neural
networks.

4.1 Boxes and Interval Arithmetic


One of the easiest ways to abstract any set X ∈ P(Rd ) is to abstract each dimension individually (we consider
the projection of X on dimension i: Xi = proji (X)). We thus obtain a box α(X) = (x1 , . . . , xd ) where, for
each dimension i, xi = [xi , xi ] is an interval with xi = min Xi and xi = max Xi . This is the first abstract
domain implemented in PyRAT.
For each dimension, we can then do interval arithmetic to compute the result of different operations1
[40]:
addition [a, b] + [c, d] [a + c, b + d]
opposite − [a, b] [−b, −a]
λ × a, λ × b if λ ≥ 0
  
scalar multiplication λ × [a, b]
λ × b, λ × a if not
 

general multiplication [a, b] × [c, d] min(ac, ad, bc, bd), max(ac, ad, bc, bd)
 

 if c ≤ 0 ≤ d

 ⊤
if 0 < c ≤ d

inverse 1 / [c, d] 1/d, 1/c
if c ≤ d < 0
  
1/c, 1/d
1 in this section, we only present intervals with finite (real number) bounds and refer to Hickey et al. [38], Dawood [39] for

the extension to intervals with floating-point or infinite bounds

5
The subtraction (resp. division) operators are obtained by adding the opposite (resp. by multiplying by
the inverse) of the interval given as a second operand. We can combine the three first operations (addition,
opposite and scalar multiplication) to compute the multiplication of an abstraction X ∈ P(Rd ) by a weight
matrix W ∈ Rp×d by decomposing W = W + +W − between positive and negative weights and then computing
W X = W + X + W − X. In PyRAT, this is used to optimise the computation done on the Box domain for
large matrix multiplication, often found in neural networks.
Moreover, note that the abstraction of any increasing function (resp. decreasing function) f is simply
f ([a, b]) = [f (a), f (b)] (resp. f ([a, b]) = [f (b), f (a)]). For example, the ReLU function defined as f (x) =
max(0, x) is increasing, therefore its abstraction for an interval [a,b] is:

ReLU([a, b]) = max(0, [a, b]) = [max(0, a), max(0, b)]

Nevertheless, it is also possible to abstract periodic functions or non monotonic functions such as cos, sin or
sigmoid using interval arithmetic.
Overall, the Box domain is very efficient as it allows to compute reachable bounds for any operation with
the same complexity as that operation (on average only 2 operations are needed, one for the lower bound
and one for the upper bound of the interval). However, this domain often lacks precision, e.g. if x ∈ [−1, 1]
then a simple operation like x − x = [−1, 1] − [−1, 1] = [−2, 2] ̸= 0 will produce a wider result than expected.
Due to the nature of the Box domain, they do not capture the relations between the different variables or
inputs of a problem and so may fail to simplify such equations. In the context of neural networks, there are
numerous relations between the inputs due to the fully connected or convolutional layers. Therefore, more
complex abstract domains that capture the relations between variables such as zonotopes will be presented.

4.2 Zonotopes
A zonotope x in Rd , with d ∈ N the dimension of the zonotope, is a special case of a convex polytope [41];
with the particularity of having a central symmetry. It is formally defined as the weighted Minkwoski sum
over a set of m noise symbols (ϵ1 , . . . ϵm ) ∈ [−1, 1]m , in other words, for every dimension 1 ≤ i ≤ d we have :
 
X m 
xi = αi,j ϵj + βi ϵ ∈ [−1, 1]m
 
j=1

where αi,j are real numbers, called generators, and βi is a real number, called center. This sum can be
represented in vector form αϵ + β where α is a matrix of dimension d × m and ϵ is a vector of dimension
m; β is also a vector of dimension d. Since the zonotope x is solely characterized by the matrix α and the
vector β, we use ⟨α, β⟩ to represent it.
Unlike boxes, there is no general algorithm to abstract any set X ∈ P(Rd ) directly into a zonotope.
However, if anything else fails, there is always the possibility to first abstract X by a box and then create a
zonotope from the box. For each dimension i, given a box [li , ui ], we can create the zonotope as such:
ui − li ui + li
xi = ϵi +
2 2
with ϵ ∈ [−1, 1]d , d new noise symbols.
Moreover, it is easy to obtain a box containing all the possible values taken by the zonotope, through a
concretisation operation. For each dimension i, the bounds of the box will be given by:
m
X
xi , xi = min xi , max xi = βi − |αi |, βi + |αi | with |αi | =
     
|αi,j |
j=1

Concretising the zonotope into a box allows to compare the box obtained to the safe space defined by the
property and thus check whether it holds or not.
For most operations present in a neural network, the arithmetic of zonotope is simple as it relies on affine
arithmetic [42]. Indeed most of the neural network operations are affine functions for which a zonotope can
perfectly capture their results. The table below shows the addition, the opposite and the scalar multiplication
on a zonotope.

6
addition ⟨α, β⟩ + ⟨α′ , β ′ ⟩ ⟨α + α′ , β + β ′ ⟩
opposite − ⟨α, β⟩ ⟨−α, −β⟩
scalar multiplication λ × ⟨α, β⟩ ⟨λ × α, λ × β⟩
For non-linear operations, such as the activation functions or multiplication or division, we need to use
an abstraction of the operation to handle them as the affine arithmetic cannot represent these operations
otherwise. Thus we are constructing an over-approximation of the reachable output of the layer. We present
quickly the abstraction for the ReLU, Sigmoid and Cast functions as shown in Figure 3. The abstraction are
constructed neuron-wise, i.e. independently for each element or neuron of a given layer. In that sense, we can
consider a single dimension input zonotope x living in [l, u] ⊂ R for which we want to compute the output
zonotope y through a non-linear activation function. In turn, this can be generalised to a multi-dimension
zonotopes.

ReLU
ReLU (x) = max(x, 0)
The ReLU function is thus a piecewise linear function, if x < 0 then ReLU(x) = 0 otherwise ReLU(x) = x.
As such, it is not a linear function for all values of l and u but is linear if x is not in the interval [l, u]. Thus
we consider three cases for the ReLU abstraction. The first two cases consider a stable or linear ReLU, where
the interval is either fully positive or fully negative. If u < 0, then we have y = 0 and if l > 0, the zonotope
is fully positive and y = x.
The last case is the general case if l < 0 < u where the ReLU is said to be unstable. As it not linear, we
need to provide a linear abstraction of the ReLU as in DeepZ [43]. This abstraction ReLU♯ is parameterised
by a = u−lu
and b = − u−l
ul
and requires adding a new noise symbol to the zonotope ϵm+1 :
  
b b a×α b
y = ReLU ♯ (x) = ax + + ϵm+1 = b ,a × β +
2 2 2 2

Sigmoid
1
Sigmoid(x) =
1 + e−x
For the Sigmoid function, we build on the work presented in [44] to construct our abstraction. First, we
compute x− , (respectively x+ ) the point where the Sigmoid is the furthest below (respectively above) the
slope between Sigmoid(l) and Sigmoid(u). If l > 0, we define x− := l and, similarly, if u < 0, x+ := u.
With these points, we can obtain the distance from the slope to the Sigmoid and thus create our zonotope
abstraction:   
b δ a×α b
y = sigmoid (x) = ax + + ϵm+1 =

b ,a × β +
2 2 2 2
with
Sigmoid(u) − Sigmoid(l)
a =
u−l
Sigmoid(x+ ) + Sigmoid(x− ) − a(x+ + x− )
b = −al + Sigmoid(l) +
2
Sigmoid(x+ ) − Sigmoid(x− ) − a(x+ − x− )
δ =
2
Here a new noise symbol ϵm+1 will always be introduced for all values of l and u as the function is not
piece-wise linear.

Cast The Cast layer and its variants, such as Ceil or Floor, are operations used to change the data type
from float32 to int8 for example or to replace the activation function in some networks. These operations
are not linear and thus need an abstraction. We only detail the abstraction of the Cast operation here but
Floor and Ceil are similar. Here, we have two cases for this abstraction due to its piece-wise linear nature:

7
if l and u are in the same linear part, then y = cast(l); otherwise we add a new noise symbol ϵm+1 with a
linear abstraction of the floor function.
  
α
y = cast♯ (x) = x + 0.5ϵm+1 = , β + 0.5
0.5

Figure 3: Activation functions and their zonotope abstraction (Relu, Sigmoid, Cast)

Complexity reduction for Zonotopes As the number of abstraction tend to grow in large networks, the
number of noise symbols introduced in the network can also grow significantly in the analysis. These symbols
can in turn slow down the analysis a great deal as the complexity of the operations on zonotopes is at least
O(md) with d the number of dimension of the zonotope and m the number of noise symbol in the zonotope.
Multiple heuristics to reduce the number of noise symbols have been implemented in PyRAT; for example
merging some noise symbols in order not to exceed a set number of noise symbols. Criteria such as the one
presented in Kashiwagi [45] can be used to choose the noise symbols to merge. Moreover as we merge these
noise symbols, we create an over-approximation of their values so that the analysis with PyRAT remains
correct although it does loose some precision. Different parameters allow the user to tune this reduction
efficiently depending on the network used.

4.3 Constrained Zonotopes


Building on the work in Scott et al. [46], we extend our zonotope domain by adding linear constraints
shared over the d dimensions of the zonotope. These constraints allow for more precise abstractions of non-
linear functions, reducing the over-approximation introduced during an analysis. At the same time, these
constraints can be used to split functions into multiple parts for greater precision in a branch and bound
approach as this will be detailed in section 5.3. Instead of the equality constraints used in Scott et al. [46], we
use inequality constraints as the equality constraints are not meaningful in the context of our abstractions.
More formally a constrained zonotope x = (x1 , ..., xd ) ∈ Rd with K constraints and m noise symbols is
defined by:  
m
[−1, 1]m
ϵ ∈P
X 
xi = αi,j ϵj + βi m
 ∀k ∈ {1, ..., K}, j=1 Ak,j ϵj + bk ≥ 0 
j=1

where Ak,j ∈ R and bk ∈ R. Note that A and b are not indexed by i, implying that the constraint are shared
for all i. At the start of the analysis, the constrained zonotope is unconstrained with K = 0 and we can add
constraints at different layers.
An example of this is shown in Figure 4 with the ReLU abstraction when using the constrained zonotope
domain. The linear abstraction remains unchanged (in green) but two constraints (in purple) are added
y ≥ x and y ≥ 0 reducing the approximation introduced. These constraints are added neuron-wise for each
unstable neuron, as such for each ReLU layer we add 2p constraints with 0 ≤ p ≤ d the number of unstable
ReLU neurons.

8
Figure 4: Abstraction of the ReLU with the constrained zonotope domain

While the zonotope could easily be concretised into a Box through a simple linear operation, for a
tight concretisation from constrained zonotope to Box, we have to make use of the linear constraints of
the constrained zonotope. For this, we need to solve a system of linear equations with linear inequality
constraints. The lower bound for all i ∈ {1, ..., d} is given by:
m
X
min αi,j ϵj + βi
ϵ∈[−1,1]m
Pm j=1
( j=1 Ak,j ϵj +bk ≥0)
k=1,...,K

and the upper bound by:


m
X
max m αi,j ϵj + βi
ϵ∈[−1,1]
Pm j=1
( j=1 Ak,j ϵj +bk ≥0) k=1,...,K

To solve these optimisation problems, a classical solution is to use a Constraint Programming solver. How-
ever, as the size of the neural networks grows and more constraints are introduced, these solvers have
difficulties scaling up. On top of this, solvers are often unsound w.r.t. real arithmetic and do not match
with the verification approach in PyRAT to have a sound approximation of the neural network’s output.
With this in mind, we chose to build a sound method to solve this. Thus, we first rewrite the equation into
its Lagrangian counterpart before solving for the dual which will always yields sound bounds. Then we can
make use of the techniques developed in optimization theory to solve our problem. More formally, to obtain
the lower bound, we need to solve for i ∈ {1, ..., d}
m
X
min αi,j ϵj + βi
ϵ∈[−1,1]m
Pm j=1
( j=1 Ak,j ϵj +bk ≥0)
k=1,...,K

which is equivalent to solving the following min-max problem by introducing a Lagrange multiplier λk with
0 <= k <= K for each constraint of the constrained zonotope:
 
Xm K
X Xm
min m max α j ϵj + β − λk  Ak,j ϵj + bk 
ϵ∈[−1,1] λ∈RK
+ j=1 k=1 j=1

where we dropped the subscript i for readability. Since we have strong duality because of the linearity of
the objective and the compactness of the optimization variables, we can reverse the order of the min and

9
the max and obtain the same bound. Thus, yielding:
 
Xm K
X Xm m
X K
X K
X
max min m α j ϵj + β − λk  Ak,j ϵj + bk  = max − αj − λk Ak,j − λk bk + β
λ∈RK
+
ϵ∈[−1,1] λ∈RK
+
j=1 k=1 j=1 j=1 k=1 k=1

Hence, we have
m
X m
X K
X K
X
min αj ϵj + β = max − αj − λk Ak,j − λk bk + β
ϵ∈[−1,1]m λ∈RK
+
Pm j=1 j=1 k=1 k=1
( j=1 Ak,j ϵj +bk ≥0)
k=1,...,K

One can observe that taking any λ ∈ RK + will yield a valid lower bound of the original problem. Thus, we
can use classical gradient ascent algorithm like SGD [47] or Adam [48] to find the optimum. This approach
allows us to have a good quality bound in an efficient manner.

4.4 Hybrid Zonotopes


Introduced in the field of closed loop dynamical systems by Bird et al. [49] and then extended to neural
networks by Zhang et al. [50] and Ortiz et al. [51], an hybrid zonotope can be seen as the union of constrained
zonotopes. Over d dimensions, we define an hybrid zonotope xi for all dimension i ∈ {1, ..., d} with K
constraints similarly to other zonotopes with mc continuous noise symbols and mb binary noise symbols:

ϵc ∈ [−1, 1]mc
 
X mc mb
X 
xi = αi,j ϵcj + γi,j ϵbj + βi ϵ
Pmc
b
∈ {−1, 1} mb
Pmb
∀k ∈ {1, ..., K}, j=1 ak,j ϵcj + j=1 bk,j ϵbj + ck = 0
 
j=1 j=1

where γi,j ∈ R is called binary generators and ak,j , bk,j , ck ∈ R.


The union of constrained zonotopes is here defined through the binary noise symbols that can only take
the discrete values of −1 or 1. The hybrid zonotope is thus the union of 2mb constrained zonotopes. The
constraints used here are equality constraints as they allow to define the intersection between two constrained
zonotope (defined by [46]) which is needed for the hybrid zonotopes.
The arithmetic is the same as classical zonotope. Nevertheless, an hybrid zonotope has the advantage of
being able to exactly represent any piece-wise linear function such as the ReLU or the cast function. With
them, we can thus obtain an exact representation of any neural network for which the activation function
are piece-wise linear (which comprise a good part of the currently used networks).

Figure 5: Hybrid zonotope abstraction of the ReLU function. H = Z1 ∩(Z2 ∪Z3 ) [51] on the left. H = Z1 ∪Z2
from Zhang et al. [50] on the right.

We will now briefly introduce how to exactly represent the ReLU function with an hybrid zonotope.
For this, two representations of the ReLU function are proposed by Ortiz et al. [51] and Zhang et al. [50].
Figure 5 shows both representation. We refer the reader to the respective paper for the exact equation of their

10
representations. With these representations, we obtain an hybrid zonotope representing the input-output set
of the ReLU function for each dimension 0 ≤ i ≤ d:

Hi = {(xi , yi ) ∈ R2 | yi = ReLU (xi ), xi ∈ [li , ui ]}


The cartesian product over all dimensions returns the exact input-output set of the ReLU for all dimension
×
d
as an hybrid zonotope H = i=1 Hi . Then, the intersection of this hybrid zonotope H with X the input
of the ReLU also represented by an hybrid zonotope will give us the exact output of the ReLU function
Y = ReLU♯ (X) = H ∩[Id 0] X with the intersection defined in Bird et al. [49].
Considering the input dimension at the ReLU layer is d, this abstraction will add 4d new continuous
noise symbols, d binary noise symbols and 3d constraints. At this step, we avoid an exponential increase of
the constraints and noise symbols by remaining linear in our representation.
Repeating this for all the ReLU in the network will give an exact representation of the network. This
can allow one to reuse this representation over multiple inputs instead of doing an analysis for each input.
Nevertheless, we need to be able to find the bounds of the output, i.e. concretise it into a box, or prove
that its intersection with the negation of the safe space is empty in order to assess whether the property
holds. Considering the binary noise symbols in the constraints, this means solving a mixed integer linear
programming (MILP) problem. Overall, the hybrid zonotope approach will provide a MILP encoding of
the network and thus transfer the complexity to a MILP solver. Given the difficulty of solving this MILP
problem, the hybrid zonotope abstraction only scale on small networks for now but various solutions are
considered for future improvements.

4.5 Other domains


PyRAT also implements a variety of other domains such as polytopes from Singh et al. [22], polynomial
zonotopes [52] or small sets. PyRAT also provides interfaces to easily extend its support to new domains
either for quick proof of concept or full implementation.

5 PyRAT
In this section, we will present the main functionalities of PyRAT as a tool to verify neural networks. More
specifically, we will detail how PyRAT applies abstract domains to perform a reachability analysis and
proves safety properties.

5.1 Network and property parsing


The first step performed in PyRAT before any analysis is the parsing of the inputs, i.e. the neural network
to analyse and the property to verify. For the neural network, PyRAT supports the ONNX standard as its
main input but it can also parse, though with a more limited support, Keras/Tensorflow or PyTorch models.
These networks are transformed in PyRAT’s own internal representation which will then serve to apply all
the operations in the network on our abstract domains.
For the parsing of the property, PyRAT supports the VNN-LIB2 property specification language which
is itself based on the SMT-LIB format. At the same time, PyRAT supports its own specification format in
textual mode or through a Python API as shown in Figure 6.

Figure 6: Equivalent properties for PyRAT in Python (left) and in a text file (right).
2 https://www.vnnlib.org/

11
The simplest way to define properties in PyRAT is through scalar constraints on each input, i.e. an
interval for each input, and a condition to be respected on the outputs which can be conjunctions, disjunctions
or negations. More complex properties can be defined using both inputs and outputs at the same time but
they remain limited in expressivity. Outputs and inputs are numbered according to their order in a flattened
array, if the inputs are multidimensional.
Once both the property and the network are parsed, PyRAT can perform some simplifications on the
network to facilitate the verification. These may include removing redundant transpose layers, removing
unneeded softmax, fusing matmul and add, etc. These operations are made for the sake of the analysis and
remain conservative in the sense that they do not alter the result of the network inference w.r.t. the property.
Two simplifications are further presented here:

Adding a layer When using relational domains such as the Zonotope domain, PyRAT is able to add an
additional layer at the end of the network to make full use of the relations for certain properties. For example,
consider the property "y0 ≤ y1 ", and the output to be over-approximated by:

y0♯ = 3ϵ0 + 2ϵ1 − 2


y1♯ = ϵ0 + 2ϵ1 .

Evaluated separately with intervals, we have y0 ∈ [−7, 3] and y1 ∈ [−3, 3], thus we cannot conclude on the
property. However, y0 ≤ y1 ⇔ y0 − y1 ≤ 0 and if we compute the difference between the zonotopes we have
z0 = y0♯ − y1♯ = 2ϵ0 − 2 ∈ [−4, 0] which verifies the property.
As such, PyRAT will add a matrix multiplication at the end of the network to create the new variables
zi and rewrite the property in terms of these zi .

Maxpool to ReLU As introduced in DNNV [53], PyRAT is able to simplify the neural network for veri-
fication by transforming maxpool layers which are harder to verify into convolutions and ReLU layers. On
top of the initial simplification by DNNV, we propose an improvement to the simplification. Indeed, the
initial simplification relies on the fact that:

max (a, b) = ReLU(a − b) + ReLU(b) − ReLU(−b)

Thus any maxpool layer can be transformed in a succession of Convolution (to extract the a − b, b and −b)
and of ReLU. For a maxpool of kernel k this results in approximately 2log2 (k) convolutions followed by ReLU
layers. While it allows to verify more precisely a maxpool layer this simplification is not without issues: we
must compute additional convolutions, and do the abstraction of three new non-linear ReLU functions each
time. For the latter, when using the zonotope domain this will introduce new noise symbols if γ(b) = [l, u]
contains 0 twice, once for ReLU(b) and once for ReLU(−b). An improvement on this simplification is instead
to rely on
max (a, b) = ReLU(a − b) + b
and use a convolution to obtain a − b and b followed by a ReLU only on a − b and an Add layer to obtain
the output. This will reduce the convolution size by a third and avoid unnecessary over-approximation.

Figure 7: Maxpool layer transformed in PyRAT.

5.2 Reachability analysis in PyRAT


After these steps of parsing and preprocessing, PyRAT will create the abstract domains from the input
box specified in the property. It will then apply all the operations of the network on the input box and the

12
abstract domains following the graph execution order. These operations will be handled according to the
specific arithmetic of each domain and abstracted as detailed in the previous section for non-linear functions.
For each analysis, PyRAT is able to use multiple domains at the same time in order to benefit from
the strengths of each of them and complement their approaches. By default, the Box domain which has a
relatively low cost will always be used. It is useful for layers such as the ReLU layer where the Box domain
bounds are exact and thus, we can compute the intersection of the Box domain with other domains that
require over-approximating the output of the ReLU to improve the precision of our result.
In order to provide a fast implementation on the highly dimensional neural networks, PyRAT implements
its abstract domains using NumPy arrays or PyTorch tensors. A choice between NumPy or PyTorch can
be done automatically in function of the input dimension. Experimentally, NumPy is faster on smaller
size matrices than PyTorch. On large networks, one can also leverage GPU computation through PyTorch
with PyRAT, thus scaling to much larger networks. Nevertheless, using GPU will lose the soundness of
PyRAT w.r.t. real arithmetic as it will not be able to change the rounding mode of floating point variables.
Additional computation libraries can also be implemented in PyRAT through predefined APIs.
While the primary objective of PyRAT remains to prove that the property holds, it can happen that the
property does not hold. Such cases tend to be hard to handle with abstract interpretation alone as it over-
approximates the outputs. To handle this, PyRAT will look for counter example on top of the reachability
analysis performed. Random points and adversarial attacks will be generated before the analysis to try to
falsify the property by launching inferences with the original model. If a counter-example is found, PyRAT
will return it and conclude the analysis as the property is falsified. On the other side, if our reachability
analysis manages to prove that a certain input Box is false, a counter example will be selected in the input
Box to return to the user. This counter-example search is tunable with the adversarial attack parameters
and can be deactivated if needed.
At the end of its forward pass on the network, PyRAT will obtain output abstract domains over-
approximating the true possible outputs of the network and will evaluate the property on these outputs.
As PyRAT is always over-approximating the possible solutions, it is correct and will not say a property is
verified or falsified if this is not true. However, the abstractions used may be too rough and PyRAT will
not be able to conclude on them leading to an unknown output. As such the possible outputs for PyRAT
are:

• "True", the property is verified.

• "False", the property is false and a counter-example is provided.


• "Unknown", PyRAT could not verify nor falsify the property with the options and abstractions used.
• "Timeout", the property could not be verified nor falsified under the given time.

When using branch and bound methods on ReLU neural networks, as described by Section 5.3, PyRAT
will be complete in the sense that provided an infinite timeout it will always give an answer either "True"
or "False".

5.3 Branch and bound


As described in Section 5.2, individual analysis in PyRAT can result in "Unknown" evaluation. This
happens because the abstract domains are too loose to represent precisely the output, for this reason, we
use branch and bound techniques which rely on multiple analyses to increase the precision.
This method comes from the field of mathematical optimization [54] which consists of breaking the original
optimization problem into smaller optimization problems and offers the possibility to prune sub-problems if
they cannot improve the optimal bound. In the context of neural network verification, if the analysis cannot
prove or falsify the property with the chosen abstract domain, PyRAT offers the possibility to perform
multiple analyses to give more accurate results by partitioning the verification problem. There are multiple
ways to perform this partitioning and all can be described in the branch and bound framework [55]. PyRAT
implements two well known branch and bound techniques: branch and bound on the inputs or on the ReLU
nodes.

13
Branch and bound on input An intuitive way to reduce the over-approximation on the output is to par-
tition the input problem recursively until every sub space is proven or until one of the sub space invalidates
the property (see for example [14, 56]). This is best described in algorithm 1.

Algorithm 1 branch and bound on input in PyRAT


Require: neural network N , output property p, input lower bound x, input upper bound x, partitioning
number k.
1: v ← [(x, x)] ▷ v contains the bounds of each unknown problem
2: while v is not empty do
3: x, x ← v.pop()
4: r, α ← Analyse(N, p, x, x)
5: if r is False then return False
6: if r is unknown then
7: j ← ChooseDim(α)
8: splits ← Split(x, x, j, k) ▷ Split into k parts the input bound at dimension j
9: Append splits to v
10: return True

First, the algorithm creates a list that will store each input problem that has to be verified (line 1). One
of the problems is selected and analysed by PyRAT (line 4). If the analysis result is unknown, it chooses
the input dimension that is deemed the most important to split upon. PyRAT estimates this importance
according to the influence, α, of the input dimension on the property (line 7). Then, the input bounds
are partitioned in k equal parts along the chosen dimension j (line 8). These new input bounds define k
sub-problems that are then added to the list of sub-problem to verify.
This algorithm ends either when all of the sub-problem are verified (line 10) or when a counter-example is
found for a sub-problem (line 5). Since each sub-problem is a subset of the original problem, the algorithm
yields True when the property is verified and False when it is not.
The core of the algorithm lies in the choice of the dimension to split, and therefore, the quality of the
estimation α. To estimate the importance of the input w.r.t. property, PyRAT essentially relies on the
relation of the available abstract domain between the input and the output αi,j and the width of the input
xi ∈ [xi , xi ]:
X
αi := αi,j (xi − xi )
j∈output dim

For a more in-depth explanation of the heuristic, refer to Durand et al. [57].
This branch and bound approach on the input performs well on low dimensional input problem but has
a low efficiency when the input space get bigger, e.g. for images. Indeed, with more inputs, the number
of division to perform grows exponentially to achieve an increase in precision as the input are often very
interconnected and multiple inputs need to be split. To resolve this issue, branch and bound on ReLU
function has been developed.

Branch and bound on ReLU The idea of branching on ReLU comes from Reluplex [11] and is now imple-
mented in every state-of-the-art verifier such as alpha-beta-CROWN [20] and MN-BaB [21]. Branching
on a ReLU means we enforce the input of a specific ReLU to be either positive or negative which transform
the ReLU operation in a simple linear operation: the 0-function if the input is negative and the identity
function if the input is positive. Providing enough time for the analysis, this approach is complete since our
abstract domains represent exactly linear functions. In addition, branching on ReLU nodes allows to split
multiple input dimensions at once since ReLU nodes are often dependant on multiple inputs. Contrary to
branching on inputs this allows to address higher dimension problems.
Conceptually, the algorithm for splitting on ReLU is quite similar to the one for input splitting: it chooses
an unstable ReLU in the neural network, it forces the aforementioned ReLU to be negative and then analyses
the neural network, it then analyses the network with the chosen ReLU being positive. Since ReLU (x) is
the 0 function for x ∈] − ∞, 0] and the identity for x ∈ [0, +∞[, partitioning at x = 0 allows us to represent

14
it exactly with 2 linear functions. The main differences between splitting on input and splitting on ReLU
are highlighted in Algorithm 2.

Algorithm 2 branch and bound on ReLU in PyRAT


Require: the neural network N , the output property p, the input lower bound x, the input upper bound x.
1: v ← [ ] ▷ v contains the indices of the ReLU to split
along with their sign
2: while v is not empty do
3: split, ← v.pop()
4: r, γ ← AnalyseSplit(N, p, x, x, split)
5: if r is False then return False
6: if r is unknown then
7: l, j ← ChooseReLU(γ, ¬ split) ▷ Chooses ReLU j at layer l to split on
8: split_n ← CopyAndAdd(split, (l, j) ≤ 0)
9: split_p ← CopyAndAdd(split, (l, j) > 0)
10: Append split_n and split_p to v
11: return True

The main difference is the replacement of the function Analyse by the function AnalyseSplit. This
function takes into account the splits constraints on the ReLU and then applies these constraints to the
different abstract domains.
For example with the constrained zonotope domain, if the algorithm has to split ReLU i at layer l, denote
by xli the pre-ReLU constrained zonotope abstraction and by yil the ReLU abstraction of xli . yil with negative
split is represented in the constrained zonotope domain as 0 with the constraint yil < 0 added to the existing
constraints of the constrained zonotope. yil with positive split is represented by xli with the constraint yil ≥ 0
added.
Similarly to branch and bound on inputs, the choice of the ReLU node to split upon is crucial. The
more unstable ReLU node there is, the greater the importance of this choice to avoid an exponential increase
in the number of sub-problems. Here we want to estimate the influence of the node i of layer l on the outputs.

PyRAT relies on the relation αReLUi ,l between the output layer and the ReLUli nodes to estimate this
influence. For the zonotope domains, this stems from the noise symbols added at each unstable ReLU
node, while we can compute a gradient to the ReLU nodes for polyhedra domains. This coefficient is then
multiplied by the impact δ+ on the ReLU when fixing it to be positive and δ− when the ReLU is fixed
negative.
γil := (δ+ + δ− ) × αReLUi,l
This estimate does not take into account the influence of fixing a ReLU node on later ReLU, e.g. it can
also fix a ReLU node to be positive in the next layer, nor does computes the influence of the newly added
constraint as it can also reduce the imprecision on another ReLU node of the same layer or of another
layer. Nevertheless, this allows a fast estimation of the influence of a ReLU node with a good efficiency.
Improvements on this estimate will be subject of future works.

6 Applications
Safety properties The first application of PyRAT is to verify functional safety properties on structured
data. On top of the usual ACAS-Xu application and the properties defined by Katz et al. [33], PyRAT was
used in a collaboration with Technip Energies on the verification of artificial neural networks for mooring
line failure detection [58] on which formal safety properties were defined and verified.

Local robustness Through multiple projects like the Confiance.ai program3 , PyRAT was used in multiple
industrial use cases, including large image classification neural networks with inputs up to 224x224x3 pixels
3 https://confiance.ai/

15
for which PyRAT provided local robustness guarantees. It is also used in the TRUMPET project [59] to
verify local robustness properties on privacy enhanced neural networks.

Embedded AI PyRAT was also used to verify embedded neural networks and their Operational Design
Domain (ODD) on the ACAS-Xu use case in a collaboration with Airbus. Following the methodology
described in Damour et al. [34] the verification was performed on quantized neural networks, using int8 and
QGemm operator. New abstractions were implemented and PyRAT was used to correctly abstract the
networks w.r.t. integer and floating points. Splitting mechanisms [57] were also employed by PyRAT to
improve the ODD verification and thus providing a wider guarantee. Through this work, Airbus is able to
produce an hybrid system using neural network when PyRAT validates the ODD and using the look up
tables in the other cases as well as a certification approach for this system following the ED-324/ARP6983
standard [60].

CAISAR [61] is an open-source platform to assess the trustworthiness of an AI through a unified entry
point and high-level specification language using WhyML. PyRAT is closely integrated with CAISAR in
order to provide a wider range of properties beyond classical safety or local robustness.

VNN-Comp PyRAT participated in 2023 and 2024 to the international neural network verification com-
petition on a wide variety of benchmarks [4] reaching respectively the 3rd and 2nd place.

7 Future works
This paper presents the PyRAT tool, a neural network verification tool based on abstract interpretation
with multiple domains such as zonotopes. Currently, PyRAT can verify the safety of a large range of neural
networks of different sizes leveraging CPU and GPU as evidenced by its second place at the VNN-Comp
2024. Future work in PyRAT includes extending the support of PyRAT with more precise abstractions
for transformers or recurrent architectures while improving the representation of neural network non-linear
functions. With the development of edge AI and the proliferation of IoT devices, PyRAT will thrive to
extend its support to embedded AI systems with their specific problems, e.g. mixing integer and floating-
point numbers, new rounding functions, etc. This work will be the subject of collaboration with the Aidge
platform4 . Finally, PyRAT will also improve and extend its branch and bound techniques to new layers
and with new improved heuristics to guide the branch and bound.

Acknowledgment This work has been supported by the French government under the ”France 2030” pro-
gram, as part of the SystemX Technological Research Institute, and as part of the DeepGreen project with
grant ANR-23-DEGR-0001. PyRAT has also been funded under the Horizon Europe SPARTA project
grant no. 830892 and TRUMPET project grant no. 101070038 as well as the European Defence Fund
AINCEPTION project grant no. 101103385.

Main contributors Augustin Lemesle, Julien Lehmann, Serge Durand (past), Tristan Le Gall, Samuel
Akinwande (past).

References
[1] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić, Pavel Laskov, Giorgio
Giacinto, and Fabio Roli. Evasion Attacks against Machine Learning at Test Time, page 387–402.
Springer Berlin Heidelberg, 2013. ISBN 9783642387098. doi: 10.1007/978-3-642-40994-3_25. URL
http://dx.doi.org/10.1007/978-3-642-40994-3_25.
[2] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and
Rob Fergus. Intriguing properties of neural networks, 2014. URL https://arxiv.org/abs/1312.6199.
4 https://eclipse.dev/aidge/

16
[3] Augustin Lemesle, Aymeric Varasse, Zakaria Chihani, and Dominique Tachet. AIMOS: Metamor-
phic testing of AI - An industrial application. Lecture Notes in Computer Science, 14182:328–340,
2023. doi: 10.1007/978-3-031-40953-0\_27. URL https://cea.hal.science/cea-04228011. The
42nd Computer Safety, Reliability, and Security, (SAFECOMP 2023 Workshops) , Toulouse, France,
19-22 september 2023.
[4] Christopher Brix, Stanley Bak, Changliu Liu, and Taylor T Johnson. The fourth international ver-
ification of neural networks competition (vnn-comp 2023): Summary and results. arXiv preprint
arXiv:2312.16760, 2023.
[5] Edmund M. Clarke and E. Allen Emerson. Design and synthesis of synchronization skeletons using
branching time temporal logic. In Dexter Kozen, editor, Logics of Programs, pages 52–71, Berlin,
Heidelberg, 1982. Springer Berlin Heidelberg. ISBN 978-3-540-39047-3.
[6] Edmund M. Clarke, E. Allen Emerson, and Joseph Sifakis. Model checking: algorithmic verification
and debugging. Commun. ACM, 52(11):74–84, November 2009. ISSN 0001-0782. doi: 10.1145/1592761.
1592781. URL https://doi.org/10.1145/1592761.1592781.
[7] Radhia Cousot, Patrick. Abstract interpretation: A unified lattice model for static analysis of programs
by construction or approximation of fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN
Symposium on Principles of Programming Languages, POPL ’77, 1977.
[8] Patrick Cousot. Principles of Abstract Interpretation. MIT Press, 2022.
[9] Luca Pulina and Armando Tacchella. Challenging smt solvers to verify neural networks. Ai Communi-
cations, 25(2):117–135, 2012.
[10] Xiaowei Huang, Marta Kwiatkowska, Sen Wang, and Min Wu. Safety verification of deep neural net-
works. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Ger-
many, July 24-28, 2017, Proceedings, Part I 30, pages 3–29. Springer, 2017.
[11] Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient
smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International
Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, pages 97–117.
Springer, 2017.
[12] Vincent Tjeng, Kai Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed
integer programming. arXiv preprint arXiv:1711.07356, 2017.
[13] Souradeep Dutta, Susmit Jha, Sriram Sanakaranarayanan, and Ashish Tiwari. Output range analysis
for deep neural networks. arXiv preprint arXiv:1709.09130, 2017.
[14] Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Formal security analysis
of neural networks using symbolic intervals. In 27th USENIX Security Symposium (USENIX Security
18), pages 1599–1614, 2018.
[15] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin
Vechev. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In
2018 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2018.
[16] Denis Mazzucato and Caterina Urban. Reduced products of abstract domains for fairness certification
of neural networks. In Cezara Drăgoi, Suvam Mukherjee, and Kedar Namjoshi, editors, Static Analysis,
pages 308–322, Cham, 2021. Springer International Publishing. ISBN 978-3-030-88806-0.
[17] Caterina Urban and Antoine Miné. A review of formal methods applied to machine learning, 2021.
URL https://arxiv.org/abs/2104.02466.
[18] Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. Efficient neural network
robustness certification with general activation functions. Advances in Neural Information Processing
Systems, 31:4939–4948, 2018. URL https://arxiv.org/pdf/1811.00866.pdf.

17
[19] Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. Fast and
Complete: Enabling complete neural network verification with rapid and massively parallel incomplete
verifiers. In International Conference on Learning Representations, 2021. URL https://openreview.
net/forum?id=nVZtXBI6LNn.
[20] Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and Zico Kolter. Beta-
CROWN: Efficient bound propagation with per-neuron split constraints for complete and incomplete
neural network verification. arXiv preprint arXiv:2103.06624, 2021.
[21] Claudio Ferrari, Mark Niklas Muller, Nikola Jovanovic, and Martin Vechev. Complete verification via
multi-neuron relaxation guided branch-and-bound. arXiv preprint arXiv:2205.00263, 2022.

[22] Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract domain for certifying
neural networks. Proceedings of the ACM on Programming Languages, 3(POPL):1–30, 2019.
[23] Hoang-Dung Tran, Xiaodong Yang, Diego Manzanas Lopez, Patrick Musau, Luan Viet Nguyen, Weiming
Xiang, Stanley Bak, and Taylor T. Johnson. NNV: The neural network verification tool for deep neural
networks and learning-enabled cyber-physical systems. In 32nd International Conference on Computer-
Aided Verification (CAV), July 2020.
[24] Diego Manzanas Lopez, Sung Woo Choi, Hoang-Dung Tran, and Taylor T. Johnson. NNV 2.0: The
neural network verification tool. In 35th International Conference on Computer-Aided Verification
(CAV), July 2023.
[25] Stanley Bak, Hoang-Dung Tran, Kerianne Hobbs, and Taylor T. Johnson. Improved geometric path
enumeration for verifying ReLU neural networks. In 32nd International Conference on Computer-Aided
Verification (CAV), July 2020.
[26] Stanley Bak. nnenum: Verification of relu neural networks with optimized abstraction refinement. In
NASA Formal Methods Symposium, pages 19–36. Springer, 2021.

[27] Bertrand Jeannet and Antoine Miné. Apron: A library of numerical abstract domains for static anal-
ysis. In Ahmed Bouajjani and Oded Maler, editors, Computer Aided Verification, 21st International
Conference, CAV 2009, Grenoble, France, June 26 - July 2, 2009. Proceedings, volume 5643 of Lecture
Notes in Computer Science, pages 661–667. Springer, 2009. doi: 10.1007/978-3-642-02658-4\_52. URL
https://doi.org/10.1007/978-3-642-02658-4_52.
[28] Jianlin Li, Jiangchao Liu, Pengfei Yang, Liqian Chen, Xiaowei Huang, and Lijun Zhang. Analyzing deep
neural networks with symbolic propagation: Towards higher precision and faster verification. In Bor-
Yuh Evan Chang, editor, Static Analysis, pages 296–319, Cham, 2019. Springer International Publishing.
ISBN 978-3-030-32304-2.
[29] Caterina Urban, Maria Christakis, Valentin Wüstholz, and Fuyuan Zhang. Perfectly parallel fairness
certification of neural networks. Proc. ACM Program. Lang., 4(OOPSLA), November 2020. doi: 10.
1145/3428253. URL https://doi.org/10.1145/3428253.
[30] Abhinandan Pal, Francesco Ranzato, Caterina Urban, and Marco Zanella. Abstract interpretation-based
feature importance for support vector machines. In Rayna Dimitrova, Ori Lahav, and Sebastian Wolff,
editors, Verification, Model Checking, and Abstract Interpretation - 25th International Conference, VM-
CAI 2024, London, United Kingdom, January 15-16, 2024, Proceedings, Part I, volume 14499 of Lecture
Notes in Computer Science, pages 27–49. Springer, 2024. doi: 10.1007/978-3-031-50524-9\_2. URL
https://doi.org/10.1007/978-3-031-50524-9_2.
[31] Francesco Ranzato and Marco Zanella. Robustness verification of support vector machines. In Bor-
Yuh Evan Chang, editor, Static Analysis - 26th International Symposium, SAS 2019, Porto, Por-
tugal, October 8-11, 2019, Proceedings, volume 11822 of Lecture Notes in Computer Science, pages
271–295. Springer, 2019. doi: 10.1007/978-3-030-32304-2\_14. URL https://doi.org/10.1007/
978-3-030-32304-2_14.

18
[32] Guido Manfredi and Yannick Jestin. An Introduction to ACAS Xu and the Challenges Ahead. In
DASC, 2016 IEEE/AIAA 35th Digital Avionics Systems Conference, Digital Avionics Systems Confer-
ence (DASC), 2016 IEEE/AIAA 35th, pages .ISBN: 978–1–5090–2524–4, Sacramento, United States,
September 2016. URL https://enac.hal.science/hal-01638049.
[33] Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient smt
solver for verifying deep neural networks, 2017. URL https://arxiv.org/abs/1702.01135.
[34] Mathieu Damour, Florence de Grancey, Christophe Gabreau, Adrien Gauffriau, Jean-Brice Ginestet,
Alexandre Hervieu, Thomas Huraux, Claire Pagetti, Ludovic Ponsolle, and Arthur Clavière. To-
wards Certification of a Reduced Footprint ACAS-Xu System: a Hybrid ML-based Solution. In
SAFECOMP 2021: Computer Safety, Reliability, and Security, pages 34–48, August 2021. doi:
10.1007/978-3-030-83903-1\_3. URL https://hal.science/hal-03355299.
[35] Julien Girard-Satabin, Guillaume Charpiat, Zakaria Chihani, and Marc Schoenauer. Camus: A frame-
work to build formal specifications for deep perception systems using simulators. In ECAI 2020, pages
2497–2504. IOS Press, 2020.
[36] Eric Goubault. Static analyses of the precision of floating-point operations. In Patrick Cousot, editor,
Static Analysis, 8th International Symposium, SAS 2001, Paris, France, July 16-18, 2001, Proceedings,
volume 2126 of Lecture Notes in Computer Science, pages 234–259. Springer, 2001. doi: 10.1007/
3-540-47764-0\_14. URL https://doi.org/10.1007/3-540-47764-0_14.
[37] Liqian Chen, Antoine Miné, and Patrick Cousot. A sound floating-point polyhedra abstract domain.
In G. Ramalingam, editor, Programming Languages and Systems, pages 3–18, Berlin, Heidelberg, 2008.
Springer Berlin Heidelberg.
[38] T. Hickey, Q. Ju, and M. H. Van Emden. Interval arithmetic: From principles to implementation.
J. ACM, 48(5):1038–1068, sep 2001. ISSN 0004-5411. doi: 10.1145/502102.502106. URL https:
//doi.org/10.1145/502102.502106.

[39] Hend Dawood. Theories of interval arithmetic: mathematical foundations and applications. LAP Lam-
bert Academic Publishing, 2011.
[40] Ramon E Moore. Interval analysis, volume 4. prentice-Hall Englewood Cliffs, 1966.
[41] Eric Goubault and Sylvie Putot. A zonotopic framework for functional abstractions, 2009. URL https:
//arxiv.org/abs/0910.1763.

[42] João Luiz Dihl Comba and Jorge Stol. Affine arithmetic and its applications to computer graphics.
In Proceedings of VI SIBGRAPI (Brazilian Symposium on Computer Graphics and Image Processing),
pages 9–18, 1993.
[43] Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin Vechev. Fast and effec-
tive robustness certification. In Proceedings of the 32nd International Conference on Neural Information
Processing Systems, NIPS’18, page 10825–10836, Red Hook, NY, USA, 2018. Curran Associates Inc.
[44] Prapeepat Uewichitrapochana and Athasit Surarerks. Signed-symmetric function approximation in
affine arithmetic. pages 1–6, 05 2013. ISBN 978-1-4799-0546-1. doi: 10.1109/ECTICon.2013.6559630.
[45] Masahide Kashiwagi. An algorithm to reduce the number of dummy variables in affine arithmetic.

[46] Joseph K. Scott, Davide M. Raimondo, Giuseppe Roberto Marseglia, and Richard D. Braatz. Con-
strained zonotopes: A new tool for set-based estimation and fault detection. Automatica, 69:126–
136, July 2016. ISSN 00051098. doi: 10.1016/j.automatica.2016.02.036. URL https://linkinghub.
elsevier.com/retrieve/pii/S0005109816300772.

[47] Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical
statistics, pages 400–407, 1951.

19
[48] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,
2014.
[49] Trevor J Bird, Herschel C Pangborn, Neera Jain, and Justin P Koeln. Hybrid zonotopes: A new set
representation for reachability analysis of mixed logical dynamical systems. Automatica, 154:111107,
2023.

[50] Yuhao Zhang, Hang Zhang, and Xiangru Xu. Backward reachability analysis of neural feedback systems
using hybrid zonotopes. IEEE Control Systems Letters, 2023.
[51] Joshua Ortiz, Alyssa Vellucci, Justin Koeln, and Justin Ruths. Hybrid zonotopes exactly represent relu
neural networks. In 2023 62nd IEEE Conference on Decision and Control (CDC), pages 5351–5357.
IEEE, 2023.
[52] Niklas Kochdumper, Christian Schilling, Matthias Althoff, and Stanley Bak. Open- and Closed-Loop
Neural Network Verification Using Polynomial Zonotopes, page 16–36. Springer Nature Switzerland,
2023. ISBN 9783031331701. doi: 10.1007/978-3-031-33170-1_2. URL http://dx.doi.org/10.1007/
978-3-031-33170-1_2.

[53] David Shriver, Sebastian Elbaum, and Matthew B. Dwyer. DNNV: A Framework for Deep Neural
Network Verification, page 137–150. Springer International Publishing, 2021. ISBN 9783030816858.
doi: 10.1007/978-3-030-81685-8_6. URL http://dx.doi.org/10.1007/978-3-030-81685-8_6.
[54] A. H. Land and A. G. Doig. An automatic method for solving discrete programming problems. Springer,
1960.

[55] Rudy Bunel, Jingyue Lu, Ilker Turkaslan, Philip HS Torr, Pushmeet Kohli, and M Pawan Kumar.
Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Re-
search, 21(42):1–39, 2020.
[56] Guy Katz, Derek A. Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth
Shah, Shantanu Thakoor, Haoze Wu, Aleksandar Zeljic, David L. Dill, Mykel J. Kochenderfer, and
Clark W. Barrett. The marabou framework for verification and analysis of deep neural networks. In
Isil Dillig and Serdar Tasiran, editors, Computer Aided Verification - 31st International Conference,
CAV 2019, New York City, NY, USA, July 15-18, 2019, Proceedings, Part I, volume 11561 of Lecture
Notes in Computer Science, pages 443–452. Springer, 2019. doi: 10.1007/978-3-030-25540-4\_26. URL
https://doi.org/10.1007/978-3-030-25540-4_26.

[57] Serge Durand, Augustin Lemesle, Zakaria Chihani, Caterina Urban, and François Terrier. Reciph: Re-
lational coefficients for input partitioning heuristic. In 1st Workshop on Formal Verification of Machine
Learning (WFVML 2022), 2022.
[58] Djoni E. Sidarta, Jim O’Sullivan, and Ho-Joon Lim. Damage Detection of Offshore Platform Moor-
ing Line Using Artificial Neural Network. volume Volume 1: Offshore Technology of Interna-
tional Conference on Offshore Mechanics and Arctic Engineering, page V001T01A058, 06 2018. doi:
10.1115/OMAE2018-77084. URL https://doi.org/10.1115/OMAE2018-77084.
[59] Alberto Pedrouzo-Ulloa, Jan Ramon, Fernando Péerez-González, Siyanna Lilova, Patrick Duflot, Zakaria
Chihani, Nicola Gentili, Paola Ulivi, Mohammad Ashadul Hoque, Twaha Mukammel, Zeev Pritzker,
Augustin Lemesle, Jaime Loureiro-Acuña, Xavier Martínez, and Gonzalo Jiménez-Balsa. Introducing
the trumpet project: Trustworthy multi-site privacy enhancing technologies. In 2023 IEEE International
Conference on Cyber Security and Resilience (CSR), pages 604–611, 2023. doi: 10.1109/CSR57506.2023.
10224961.
[60] Christophe Gabreau, Marie-Charlotte Teulières, Eric Jenn, Augustin Lemesle, Dumitru Potop-
Butucaru, Floris Thiant, Lucas Fischer, and Mariem Turki. A study of an ACAS-Xu exact imple-
mentation using ED-324/ARP6983. In 12th European Congress Embedded Real Time Systems - ERTS
2024, Toulouse (31000), France, June 2024. URL https://hal.science/hal-04584782.

20
[61] Julien Girard-Satabin, Michele Alberti, François Bobot, Zakaria Chihani, and Augustin Lemesle.
CAISAR: A platform for Characterizing Artificial Intelligence Safety and Robustness. In
AISafety, CEUR-Workshop Proceedings, Vienne, Austria, July 2022. URL https://hal.science/
hal-03687211.

21

You might also like