TL;DR

Layer-Wise Relevance Propagation (LRP) is a propagation method that produces relevances for a given input with regard to a target output. Technically, the computation happens using a single back-propagation pass, similarly to deconvolution. I propose to illustrate this method on an Alpha-Zero network trained to play Othello.

Table of content

• LRP Framework
  • Formulation
  • Different Rules for Different Layers
  • Technical Details
• Interpreting Othello Zero
  • Playing Othello
  • Network Decomposition
  • Interpretation
  • Evaluate an Explanation
• Resources

# LRP Framework

Formulation

LRP @1 is a local interpretability method which attributes relevance to any neuron activation inside a neural network with regard to a target output and a given input. The target output can be non-terminal (e.g. an intermediate layer neuron activation), multi-dimensional (e.g. the final vector of logits) or uni-dimensional (e.g. a single class logit). Its formulation is similar to the classical Gradient$\times$Input @2 as it is computed using a single modified backward pass. Yet, it doesn’t suffer from the noisiness of the gradient.

Notation

The activation of the $j$-th neuron in the layer $l$ is noted $a_j^{[l]}$ and its associated relevance is noted $R_j^{[l]}$ and by convention $a_j^{[0]}$ represents the input and $R_j^{[L]}$ the target output. Weights are indexed similarly such that if the layer $l+1$ is linear its weights are noted $w_{kj}^{[l+1]}$ (torch convention) and $b_k^{[l+1]}$ performing the mapping, to the $k$-th neuron of the layer $l+1$, given by equation $\ref{eq:linear_mapping}$ where $g$ is the activation function.

Figure 1 illustrates the process of relevance propagation, which can be intuited as a redistribution flow. The relevance $R_k^{[l+1]}$ is decomposed in “messages” sent to the previous layer. The message, $R_{j\leftarrow k}^{[l\leftarrow l+1]}$, from the $j$-th neuron of the layer $l+1$ to the $i$-th layer of the layer $l$ is then given by the equation $\ref{eq:message_decomposition}$, where $\Omega_{jk}^{[l+1]}$ is the decomposition rule (see bellow). The new relevance $R_j^{[l]}$ is then obtained by summing each “message” sent, as described by equation $\ref{eq:message_aggregation}$.

Decomposition Rules

A rule $\Omega_{jk}^{[l+1]}$ defines how to redistribute the relevance $R_k^{[l+1]}$ of the layer $l+1$ into the previous layer $l$. The ideal rule depends on the nature of the layer $l+1$ and is an active topic of research @3@4@5 (e.g. for new or more complex architecture like Transformers), more on this in the rule and evaluation sections.

Figure 1: Relevance back-propagation of the dog logit, @5.

The most basic LRP rule @1 is given by the equation $\ref{eq:original_lrp_rule}$, which is a simple pre-activation weighting. One drawback of this rule is that it is not conservative, i.e. $\sum_jR_{j}^{[l]}\neq\sum_kR_{k}^{[l+1]}$ $\Leftrightarrow$ $\sum_j\Omega_{jk}^{[l+1]}\neq1$, and thus relevance get lost or created by the bias along the propagation. Therefore, this rule was revisited by @5 as LRP-$0$ by setting the biases to $0$ (a trick).

Generalisation Gotcha

The layer superscript is sometimes omitted in the literature because it doesn’t cover the general formulation. For example, to account for residual connections, the propagation should be spread across multiple layers. The general formulation is thus simply given by a causally weighted sum $R_{j}=\sum_{k}\Omega_{jk}R_k$ where the numbering is across all the neurons (causally means $\Omega_{jk}=0$ if $k<=j$, messages are only sent backwards). It is less informative about the practical implementation but still carries the idea of tracking the actual computation.

Different Rules for Different Layers

Intuitively, the relevance propagation rules should be dependent on the layer’s nature, as is the forward flow. Fundamentally, this idea comes from the ambition of tracking the model’s various kinds of actual computation into each relevance. The classical framework for deriving these rules is Deep Taylor’s series Decomposition (DTD) @6, yet it should be justified with care @7. I’ll now present the necessary rules needed for the experiments I conducted, whose derivation can be found in the various linked resources (most of them assume ReLU, which is the case here).

Notation

The layer superscript is dropped in this section for readability and since all the rules will be applied on consecutive layers. The index $j$ still refers to layer $l$ and the index $k$ to layer $l+1$.

LRP-$\epsilon$ @1: Defined by the equation $\ref{eq:lrpe_rule}$, it introduces $\epsilon$, a numerical stabilizer and the $z$ coefficients are set accordingly to the equation $\ref{eq:original_lrp_rule}$ ($z_{jk}=w_{kj}a_j$ and $z_k=\sum_{j}w_{kj}a_{j}+b_k$). A variant of this rule can be computed by setting the biases to $0$, but this is not enough for the propagation to be conservative. The stabiliser $\epsilon$ will absorb some relevance and should, therefore, be kept as small as possible. In practice, this rule is made for linear mappings (Linear & BatchNorm).

$z^+$ @6: Defined by the equation $\ref{eq:zplus_rule}$, where $w_{kj}^+$ stands for the positive part of $w_{kj}$, i.e. $w_{kj}^+$ if $w_{kj}<0$. This rule is conservative and positive (and thus consistent @6). In practice, this rule is used for convolution.

$w^2$ @4: Defined by the equation $\ref{eq:wsquare_rule}$, this rule is meant for early layers. This rule is conservative and positive (and thus consistent). In practice, it is used for the first layer, e.g. in order to propagate relevance to dead input neurons.

Flat @4: Defined by the equation $\ref{eq:flat_rule}$, this rule is similar to $w^2$ but assuming a constant weighting. It, therefore, redistributes equally the relevance across every preceding neuron. This rule is conservative and positive (and thus consistent). In practice, it is used for the first layer, e.g. in order to propagate relevance to dead input neurons.

Technical Details

All the LRP computation can be done using the original authors’ library Zennit @8. In practice, the graph and back-propagation mechanism of torch.autograd is used using backward hooks; see the snippet below. The models need to implement the forward pass only using proper modules (child of the model instance) for them to be detected by Zennit and hooked. And since it relies on full back-propagation, every module of the graph should be hooked (even activation functions).

Pass rule: This is a practical rule necessary regarding Zennit implementation. In practice, even activation functions should be hooked because otherwise, the classical gradient will be computed during the backward pass. And since the actual relevance propagation is carried by other module hooks (Linear, Conv, etc.), no modification should be done (it’s a pass-through). It is typically used for activation functions.

This important snippet explains how backward hooks are coded in Zennit, which is fundamental to designing new rules. Besides Rules, derived from the BasicHook, other fundamental objects are available:

• Stabilizers: $\epsilon$ term in the LRP-$\epsilon$, which make the computation numerically stable. All rules are used in practice.
• Canonizer: Needed when using BatchNorm layers @3, to reorder the computation.
• Composites: To easily combine rules.

To illustrate rules, below is a snippet of the $z^+$ rule implementation. The first element is param_modifiers, which is used to modify the weights of the module, like here to separate positive and negative weights or like in LRP-$0$ to set the biases to zero. Then there are the input_modifiers and the output_modifiers, which enable lightweight modification of the forward pass (using the modified weights). In the rule below, they are used to separate positive from negative inputs. Finally, the gradient_mapper is used to compute the final form of the rule ($\Omega_{jk}$) here, one for positive and one for negative contribution, and the reducer computes the module relevance ($R_j$).

# Interpreting Othello Zero

Playing Othello

Before digging into the actual interpretation of the network I borrowed from Alpha Zero General @9, it is important to understand how it is used in practice and how it was trained. I highly recommend checking their code on Github or the associated blog post as well as the original Alpha Zero paper @10.

Tree representation of game (Min-Max, Alpha-Beta, MCTS, …) is an intuitive representation of a game whose main components are the root (the current position), the nodes (board states $s$) and the edges (action chosen for a given state $(s,a)$). Regarding search, the Alpha Zero paper @10 used MCTS PUCT @11, with the upper bound confidence (UCB) given by the equation $\ref{eq:upper_confidence_boundary}$. This equation involves network predictions (heuristic) with $P_\theta(s)$ the policy vector and $Q_s$ is the average expected value over the visited children (terminal reward or intermediate network evaluation, i.e. the value $v_\theta(s)$). $c_{\rm puct}$ is a constant to balance exploitation with exploration and after multiple rollouts the action is often chosen according to the tempered visit distribution, $\pi_s$ , given by the equation $\ref{eq:visit_distribution}$, with $\tau$ the temperature.

The network is trained by combining the loss from the value and the policy predictions. It especially makes sense since these predictions share a common graph (architecture) in the model. The value output $v_\theta(s)$ should predict the ending reward of the game $z$ (-1, 0 or 1 depending on the outcome) and the policy output $P_\theta(s)$ should predict the action sampling distribution obtained after search $\pi_s$. The loss is then @10:

In my drafty accompanying notebook , I chose functional version of this algorithm, with few minor changes from the Alpha Zero General code. Also, I put below a slightly modified version of the MCTS that is compatible with the LRP framework. The idea is quite simple: you have to keep track of the gradient in order to perform a backward pass under Zennit context later. In this way, you could interpret the aggregated quantities $Q_s$ and $U_s$ in terms of relevance. These quantities carry information about the subsequent tree from node $s$ and could be used a posteriori or during inference.

Disclaimer

This last piece of work is only exploratory, and I have made no digging in this direction yet. However, I am convinced that refining and experimenting in this direction could lead to interesting tracks like tree analysis or relevance-guided search.

Network Decomposition

In order to use Zennit, it is important to remember how it is implemented and adapt the network accordingly. First, all used modules should be instantiated under the target module (even activations). Then softmax should not be used because of the exponential. Here, it can be safely removed as the output is the Soflogmax, which is a simple translation of the raw logits and doesn’t change the action sampling.

The network architecture is quite simple as it is a basic CNN mostly using ReLU activation. For the convolution layer, I’ll use the $z^+$ rule, and for the linear mapping (including batch normalisation), I’ll use LRP-$\epsilon$. These settings are recommended, but you could try various different ones. To evaluate the different combinations of rules, refer to the evaluation section.

Gotcha

It is important to acknowledge the similarity in computation for the value and the policy. This will expectedly lead to very close relevance heatmaps as only one layer differs between the two.

One practical limitation for interpreting board games concerns the empty cells. Using traditional LRP rules will attribute zero relevance to those cells. Indeed, during the computation, the model doesn’t use these pixels but rather uses biases. In order to overcome this difficulty, I propose to use the Flat or the $w^2$ rules in the first layer.

Interpretation

Disclaimer

The following experiments are highly shallow, and I don’t pretend they are highly relevant or valuable. This work is only for illustrative purposes and definitely needs more digging. If you are interested in a follow-up of this project (by yourself or by me) and/or have questions, feel free to contact me.

I’ll now study a particular board picked during a self-played game with different parameters for black and white ($c_{\rm puct}=0.1$. for black and $c_{\rm puct}=2$ for white) using 10 rollouts per move and keeping the data from the previous moves. Playing multiple games with these parameters shows that the middle game is dominated by black while the endgame is dominated by black, i.e. exploration is a long-run payoff. I picked board 31 (black to move) as it is balanced (before black domination). Using the described rules, with Flat in the first layer, the relevance heatmap obtained is plotted in Figure 2. The value relevance is localised around the best move (B6), found by the MCTS, and negative or close to 0 around illegal moves.

Figure 2: Value relevance heatmap using Flat, $z^+$ and LRP-$\epsilon$, normalised by the maximum relevance amplitude.

Yet, is it really what I think it is? Remember that the biases and the Flat rule are used here to compute the relevance. With the same rules, using an empty board yields the heatmap of Figure 3.

Figure 3: Value relevance heatmap of the empty board using Flat, $z^+$ and LRP-$\epsilon$, normalised by the maximum relevance amplitude.

If it is harder to interpret, I’ll leave it for a more comprehensive study and change the Flat rule for a $z^+$ rule (the first layer is a convolution). This then yields the value relevance in Figure [4] and the policy relevance in Figure [5]. The value relevance is still localised around the best move, but it might be a correlation. Indeed, the relevance seems to indicate that the networks attribute more value to the pieces flipped by the move A7 that put a piece on the side. The way the network perceives value in sides and corners should be dug more. The policy relevance heatmap is more difficult to interpret, and note that the sign is due to logit initialisation. Yet, it attributes more relevance to the pieces flipped by the move B6.

Figure 4: Value relevance heatmap using $z^+$ and LRP-$\epsilon$, normalised by the maximum relevance amplitude.

Figure 5: Policy relevance heatmap using $z^+$ and LRP-$\epsilon$, normalised by the maximum relevance amplitude.

The next section describes how to evaluate the explanation’s faithfulness and robustness. Yet it was not successful as the board games add an extra layer of complexity since the input space is sparse. More complex considerations are needed.

Evaluate an Explanation

It is important to keep in mind that producing a heatmap is easy, but interpreting it in a faithful way is much harder. What’s more, you also have to be careful about what actually is what you are visualising. There is sometimes a big difference between what you want to measure, what you think you’re measuring and what you actually measure. In this meaning the produced heatmaps should be interpreted with care @7@12.

Measuring the faithfulness and robustness of the XAI method is an active topic of research, and I’ll only present and use one here. The idea of @13 is quite intuitive:

1. Assuming that you have computed a pixel relevance heatmap, you start by corrupting the most relevant pixel (using mean, black, white or random noise).
2. You observe the decrease of the target, e.g. a logit, and you compute your new heatmap.
3. You iterate on 1.
4. Finally, you plot/measure the evolution of the target.

The best XAI method would lead to the highest drop in the target. In practice, this can be used to compare XAI methods, and in particular, it can be compared to random pixel corruption. For example, it could be used to verify that the rules derived using DTD are actually the best fit for each layer type.

# Resources

A drafty notebook that self-contains all the practical experiments presented here and more is available on Colab: . I first explored the network capabilities like the policy prediction and the games play. Below is a list of references containing the papers and code mentioned in this post.

In particular, I think that the method described in @5 could be the perfect match for a follow-up. It extends the LRP framework to discover concepts, i.e. global explanation. Basically, LRP serves as a means to discover locally relevant neurons and paths. Then, concepts are discovered using an activation maximisation on these neurons.

References

1. Bach, Sebastian, et al. “On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation.” PLOS ONE, vol. 10, no. 7, 2015.
2. Shrikumar, Avanti, et al. “Not Just a Black Box: Learning Important Features Through Propagating Activation Differences.” ArXiv, 2016.
3. Binder, Alexander, et al. “Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers.” ArXiv, 2016.
4. Lapuschkin, Sebastian, et al. “Unmasking Clever Hans Predictors and Assessing What Machines Really Learn.” Nature Communications, vol. 10, no. 1, 2019.
5. Achtibat, Reduan, et al. “From Attribution Maps to Human-understandable Explanations through Concept Relevance Propagation.” Nature Machine Intelligence, vol. 5, no. 9, 2023.
6. Montavon, Grégoire, et al. “Explaining Nonlinear Classification Decisions with Deep Taylor Decomposition.” Pattern Recognition, vol. 65, 2017.
7. Sixt, Leon, and Tim Landgraf. “A Rigorous Study Of The Deep Taylor Decomposition.” ArXiv, 2022.
8. Anders, Christopher J., et al. “Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAy.” ArXiv, 2021.
9. Thakoor, Shantanu, et al. “Learning to play othello without human knowledge.” Stanford University, 2016.
10. Silver, David, et al. “Mastering the Game of Go Without Human Knowledge.” Nature, vol. 550, no. 7676, 2017.
11. Rosin, Christopher D. “Multi-armed bandits with episode context,” Annals of Mathematics and Artificial Intelligence, vol. 61, pp. 203–230, 09 2010.
12. Sixt, Leon, et al. “When Explanations Lie: Why Many Modified BP Attributions Fail.” ArXiv, 2019.
13. Hedström, Anna, et al. “Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond.” Journal of Machine Learning Research, vol. 24, no. 34, 2023.

TL;DR

Thanks to recent developments, FHE can now be applied easily and scalably to deep neural networks. I think, like many, that these advancements are a real opportunity to improve AI safety. I thus outline possible applications of FHE in model evaluation and interpretability, the most mature tools in safety as of today in my opinion.

Table of content

• Context
• What is FHE?
  • The FHE scheme
  • FHE Applied to ML
• Hiding the Test Set in Public
  • The Test Set Game
  • Zero Trust for Better Safety
• Privately Inspecting Model Biases
  • I/O Interpretability Methods
  • What’s Missing?
• Resources

# Context

I recently participated in the Privacy Preserving AI Hackathon organised by Entrepreneur First, Hugging Face and Zama. With my team we focused on privately matching patients with clinic trials that have specific requirements. Even though we didn’t win, I learned a lot, and I want to continue building with this amazing technology!

Epistemic Status

I am still beginner in FHE and definitely not a crypto-guy. I know the basics and really love the field, The Code Book @1 inspired me to focus on cybersecurity back in my Bachelor. In comparison I would say that I am well versed on the subject of AI evaluation and interpretability. The following is thus an article from an AI-guy trying to apply FHE for the sake of improving AI safety.

# What is FHE?

The FHE scheme

First, FHE stands for “fully homomorphic encryption”, a framework for cryptosystems supporting computation on cyphertexts. As the name “fully homomorphic” suggests, the beauty of the method is enabling a correspondence between computation in the encrypted space and the clear space. Indeed, if $\varepsilon$ is the encryption mechanism, $\varepsilon(\lambda \cdot A + B)=\lambda\circ\varepsilon(A)\star\varepsilon(B)$, such that a series of computations can be performed while encrypted, only needing to decrypt the result.

Why does it matter? It matters because it means that the computation can be performed by an untrusted entity w.r.t. the input data. For the entity performing the computation, it also means that you don’t need to give away your “secret sauce”, i.e. the series of computations. This scheme adheres to the spirit of zero trust by design. This enables the building of privacy-preserving applications that might leverage highly tailored features based on your personal data without any leak. Such applications could range from content recommendation, based on your age, sex or localisation, to health checks.

Privacy Checklist

Here is a simple checklist for your privacy-preserving application:

• Who wants to protect what?
• What is encrypted?
• Where is the compute executed?
• Who decrypts?
• Who needs the computed results?
• Is FHE really necessary here?
• Is FHE realistic here?

Bear in mind that computing on cyphertexts is expensive, so you might want to avoid using FHE as much as possible! For example, you should prefer running the non-critical parts of your system locally, on the client machine, to the extent of the power needed. You’ll then keep the encryption scheme for most sensitive business-related computations.

FHE Applied to ML

FHE was thought of at the end of the last century and was practically discovered in 2009 @2, but it still needed to be scalable. Fortunately, the last decade has shown great improvement in the various implementations of the FHE scheme, like with TFHE @3, enabling efficient addition and multiplication over cyphertexts. What’s more, with the adoption and ecosystem growth, private companies like Zama were created and participated in improving FHE.

What particularly interests us here is how to perform non-linear computation. This step is crucial to handle neural networks’ activation functions and other ML algorithms. This was mostly made possible by fast bootstrapping @4, the evolution of the original bootstrapping trick @2, which was then improved by Zama, who made it scalable @5.

Going Deeper

If you want to go deeper and learn about the actual algorithms implemented have a look at Zama’s blog. I especially recommend their 101 on FHE, which clearly explains bootstrapping and its application to neural networks. In general, they have great resources, tutorials, and stories for crypto experts, but also for less technical wanderers.

In practice, you need to remember that all the computations are done with integers and are slower. With Zama’s library concrete-ml, you don’t need to handle or write any crypto code. You’ll be able to manipulate and customise your torch model at will, providing that you use supported operations. Indeed, when you compile your model, it will be translated under the hood into FHE-friendly operations. concrete-ml provides native APIs to create an FHE model from a regular torch or scikit-learn model. They made it so easy, it would be a shame not to use it!

Universal ML

As software/languages/libraries go, there is a need to abstract and universalise certain concepts. As such, ONNX is commonly used to share or convert models, e.g. from Tensorflow to PyTorch. On the other side MLIR is used by vendors as a middle-layer between their hardware and high-level programming, natively integrated in new ML-oriented languages like MOJO, worth checking!

In privacy-preserving ML applications, the common scheme is often based on:

• A model owner who wants to keep its secret sauce (mostly the model).
• A data owner who wants to keep their privacy while accessing a tailored feature.

An application example of such a scheme could be a method to classify people based on their private data (bank loans, insurance policies, etc.) or to enable a feature for known users (facial recognition, DNA testing, etc.). In the end, what matters is that the user has the same service but takes back control over their own data.

I argue that AI safety shares the same goals as described above. You want to be sure that all models are safe, not only the open-source ones. You might want to regulate or inspect models without letting the model owner know, as in many other domains. The current state of affairs could be more satisfying, and I will describe in the next sections what directions could be net beneficial for AI safety.

# Hiding the Test Set in Public

The Test Set Game

In ML, it is well known that you should always keep a subset of data hidden from the model, the test set. Consequently, this dataset provides a faithful model evaluation, should it be big enough and randomly sampled from the original dataset. It should be not mistaken for the validation set, used for hyperparameters search, which doesn’t provide a good evaluation metric due to the Goodhart’s law similarily to the train set.

But what if you train on the test set? In frugal ML, per se using scikit-learn, it only happens willingly in scarce data problems like health predictions. Except for this particular case, should you encounter a data leakage problem, you can often re-process the datasets and re-train from scratch. Yet, in the era of big pre-trained models, such a solution would be far too expensive. It is thus a real problem, well illustrated by this satirical article @6. You can also think of using machine unlearning for such a scale and diversity of data; you would dramatically hurt the performances.

This major problem is primarily due to an aggressive data harvest that scrapes the web and, willingly or not, incorporates a test set of different known benchmarks. In order to avoid or control this leak, the two best SOTA approaches are:

• Don’t publish the test set.
• Contaminate the test set.

Yet, since the evaluations are sent in clear to the models, they could be saved and later be reused. As for the contamination, it could be overcome by additional training, adversarial attacks, or surgical machine unlearning.

Policies

Fortunately, certain model providers, like Azure, adopted privacy policies stating that no data sent to the model would be collected. This is often encouraged by treaties like the GDPR.

Zero Trust for Better Safety

It’s thus clear that you shouldn’t trust model owners with your evaluation data. However, model owners could also mistrust the evaluations. To solve this, I propose a simple scheme. First, provide an encrypted test set and then run the evaluation publicly, e.g., on an HF space like MTEB.

We can push the distrust scheme even further. What if model owners didn’t want to leak on which inputs their model made a mistake? We would process similarly but using a common public key with a segmented private key. First, the model owners would provide the encrypted output, which is impossible to decrypt without their agreement. Then, we would run a circuit computing the accuracy. And finally, together, we would decrypt the metric result. In order to illustrate this process we can simply write a torch model computing the accuracy.

Gotcha

This problem would require the creation of an accuracy circuit for each dataset, generating a new key for each model and encrypting the test set on the fly every time. While not practicable it illustrates the power of FHE to operate in scenarios where there is absolutely no trust.

What kind of governance would it enable? The last detail I previously didn’t mention for the sake of illustration is about trusting the data. In order to overcome this issue, we’ll need an additional entity, a regulator, that would, in the end, own the data. As such, they would be the ones to approve the dataset and deliver the associated certificate. The business organisation would be based on:

• A law enforcing requiring model evaluation certificates.
• An evaluator entity, entitled by the legislator to deliver certificates.
• A model company paying the evaluator to abide by the law and look good with the certificate.

Obviously, legal enforcement might not be enough. Indeed, some companies are willing to pay fines, even more when benefits largely outweighs the fines. Yet, pushing technologies like FHE might enable new use cases for AI safety.

# Privately Inspecting Model Biases

Can we say more about a model than just its performances with only limited access?

I/O Interpretability Methods

In the era of sparse autoencoders or activation manipulation, heavily relying on model internals, let’s shed some light on interpretability methods only requiring input/output mapping, widely used by the non-DL folks.

One of the first naive approaches to analyse a true black-box would be to recreate it. Depending on the model, e.g. ChatGPT, it might be too expensive but still a good start if done locally. In the field of XAI, LIME @7 was proposed using a much simpler surrogate model trained locally. Linear models, like logistic regression, are chosen for their straightforward interpretation and can be seen as a local Taylor approximation.

Another very common framework to explain models with only a black box access is the Shapley Values @8. Put simply, the Shapley Values, originating from game theory, disentangle the contribution of each player to a common team reward. In XAI we might want to disentangle the contribution from each input feature (player) to the model prediction (reward). Feature importance was efficiently implemented in SHAP @9, and remains model agnostic.

Last but not least, it is possible to create explanations centred on input data. Like the intuitive methods of occlusion @10, which answers “What happens if I remove this part of the data?”, or anchors @11, which answers “What are parts of the data must remain to keep the same prediction?”. These methods, along with LIME and SHAP, were unified as a set of “Explaining by Removing” methods @12. More sophisticated methods could make use of adversarial samples, like using semantically equivalent adversaries @13. In general, finding adversarial examples is possible, even with black box access, by training a surrogate model and using transfer @14.

Transferability

Certain adversarial attacks are transferable by nature, as related by meaningful features @15. Indeed, the link between model robustness and adversarial attacks is still an active topic, especially for the latest models @16. It aligns with the natural abstraction hypothesis, which states that dominant concepts will eventually be learned (with enough time and computation) despite the learning methodology.

In conclusion, there exists a vast panel of methods to inspect a model only based on I/O. They would provide a deeper understanding of the model, e.g. biases, clever hans or condensed knowledge, while being FHE friendly.

What’s Missing?

Model internals. Modern post-hoc interpretability methods studying deep neural networks use the model’s internals, such as activations, embeddings, individual neurons, weights, gradients, etc. Yet this data is way more sensitive for the model owner than its output since it would drastically lower the replication barrier. It would also be illusory to ask them to compute the interpretability methods themselves, as they could impair the results.

Speed As previously said FHE suffer from performance issues compare to clear evaluation. And while evaluation only require one forward pass interpretability often require many, especially in with black-box access. So speed would be primordial to improve, but obviously Zama, and others, are working on it as it would drastically improve FHE usability.

Model support. While it is true that concrete-ml makes it super easy to rewrite a torch model, an auto-converter would increase development speed. Imagine if you could to compile any model to FHE, HuggingFace would the perfect ecosystem (it already is). More generally improving model support, that could be helped by private companies, would speed and widen adoption.

While it is true that such adjuncts would be terrific, I think that you can already build solid and useful software with it.

# Resources

Below is a short list of the references cited throughout this blog post.

References

1. Singh, Simon. “The Code Book: The Evolution of Secrecy from Mary, Queen of Scots, to Quantum Cryptography” (1st. ed.). Doubleday, USA (1999).
2. Craig Gentry. “A fully homomorphic encryption scheme.” PhD thesis, Stanford University, (2009).
3. Chillotti, Ilaria et al. “TFHE: Fast Fully Homomorphic Encryption over the Torus.” (2016).
4. Chillotti, Ilaria et al. “Programmable Bootstrapping Enables Efficient Homomorphic Inference of Deep Neural Networks.” (2021).
5. Stoian, Andrei et al. “Deep Neural Networks for Encrypted Inference with TFHE.” (2023).
6. Schaeffer, Rylan. “Pretraining on the test set is all you need.” arXiv preprint arXiv:2309.08632 (2023).
7. Ribeiro, Marco Tulio et al. ““Why Should I Trust You?”: Explaining the Predictions of Any Classifier.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016).
8. Shapley, Lloyd S. ”Notes on the n-Person Game – II: The Value of an n-Person Game.” Santa Monica, Calif.: RAND Corporation (1951).
9. Lundberg, Scott M. and Su-In Lee. “A Unified Approach to Interpreting Model Predictions.” Neural Information Processing Systems (2017).
10. Zeiler, Matthew D. and Rob Fergus. “Visualizing and Understanding Convolutional Networks.” ArXiv abs/1311.2901 (2013).
11. Ribeiro, Marco Tulio et al. “Anchors: High-Precision Model-Agnostic Explanations.” AAAI Conference on Artificial Intelligence (2018).
12. Covert, Ian et al. “Explaining by Removing: A Unified Framework for Model Explanation.” J. Mach. Learn. Res. 22 (2020): 209:1-209:90.
13. Ribeiro, Marco Tulio et al. “Semantically Equivalent Adversarial Rules for Debugging NLP models.” Annual Meeting of the Association for Computational Linguistics (2018).
14. Shi, Yi et al. “Active Deep Learning Attacks under Strict Rate Limitations for Online API Calls.” 2018 IEEE International Symposium on Technologies for Homeland Security (HST) (2018).
15. Ilyas, Andrew et al. “Adversarial Examples Are Not Bugs, They Are Features.” Neural Information Processing Systems (2019).
16. Schlarmann, Christian and Matthias Hein. “On the Adversarial Robustness of Multi-Modal Foundation Models.” 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) (2023).