A Transformer visualization system for visual analysis on Attention Mechanism.

Before you run all code, install some requirements defined in requirements.txt.

python -r requirements.txt

Firstly unzip the embedding file in the ./train/embeddings folder (or use your own).

unzip ./train/embeddings/google.zip

Then run the following command:

cd train
python san.py -emb embeddings/google.txt

The model will be stored in ./train/model folder. Or you can download our pretrained model at google drive.

The code are modefied based on SSAN-self-attention-sentiment-analysis-classification. To change which self-attention architecture, you can go through this repository.

Put the model from the previous part into ./web/static/model/

Then run the command to start Django.

cd web
python manage.py runserver

Now you can use Transformers-Vis at http://127.0.0.1:8000/.

We use D3 to complish the visualization.

All visualization codes can be found at https://observablehq.com/@wmx567?tab=notebooks

Max matrix is used to detect outlier points. It will help users to find the most contributed words or dimensions in model.

It shows softmax values which interprets how each word contributes to a word vector.

Try it to find more interesting charts! 🥳🥳

• Yuan Li
• Mengxi Wu