Emotion Analysis for African Languages

COS 760 — Group 12 · University of Pretoria · S1 2026

A comparative study of TF-IDF and multilingual transformer models (mBERT, XLM-R) for multi-label emotion classification in three Bantu languages using the BRIGHTER dataset and BRIGHTER dataset HF.

---

Research Questions

1. How do multilingual transformer models compare to a TF-IDF baseline for multi-label emotion classification in Bantu languages?
2. How does performance vary across isiZulu, isiXhosa, and Swahili, and what role does training data volume play?
3. What common prediction errors arise, and what do they reveal about class imbalance and model limitations?

---

Key Results

Model	isiZulu (F1-mac)	isiXhosa (F1-mac)	Swahili (F1-mac)
TF-IDF + Logistic Regression	0.257	0.265	0.250
mBERT (fine-tuned)	0.177	0.240	0.193
XLM-R (fine-tuned)	0.135	0.222	0.223

The TF-IDF baseline outperforms both transformers on macro F1 under these data constraints. Transformers achieve substantially higher recall (0.53–0.77 vs 0.29–0.32 baseline) thanks to per-label positive-class weighting (pos_weight).

Full results and discussion: documentation/Group12_Report.pdf

---

Project Structure

.
├── main.py                          # Single entry point — set toggles here
├── requirements.txt                 # Python dependencies
├── README.md
│
├── data/
│   └── BRIGHTER/
│       ├── zul/  (dev.parquet, test.parquet)
│       ├── xho/  (dev.parquet, test.parquet)
│       └── swa/  (train.parquet, dev.parquet, test.parquet)
│
├── src/
│   ├── data/
│   │   └── dataset_loader.py        # Parquet loading, schema detection, standardisation
│   ├── baseline/
│   │   ├── model.py                 # TF-IDF + OneVsRest LogReg
│   │   └── run_baseline.py          # Trains + evaluates all three languages
│   ├── analysis/
│   │   ├── exploratory_analysis.py  # Weak labels, class distribution, error examples
│   │   └── feature_analysis.py      # TF-IDF feature visualiser (standalone tool)
│   ├── transformers_pipeline/
│   │   ├── transformer_train.py     # WeightedBCETrainer, pos_weight computation
│   │   ├── transformer_dataset.py   # PyTorch Dataset wrapper
│   │   ├── transformer_metrics.py   # Sigmoid + threshold evaluation
│   │   ├── run_mbert.py             # Fine-tune bert-base-multilingual-cased
│   │   └── run_xlmr.py              # Fine-tune xlm-roberta-base
│   └── metrics.py                   # Shared evaluation utilities
│
├── results/
│   ├── results.py                   # Load, compare, and export result tables
│   ├── baseline_metrics.csv
│   ├── mbert_metrics.csv
│   ├── xlmr_metrics.csv
│   ├── transformer_summary.csv
│   ├── weak_labels.csv
│   ├── class_distribution.csv
│   ├── error_examples.csv
│   ├── cross_lingual_comparison.csv # Generated by Week 4
│   └── paper_tables.txt             # LaTeX-ready tables for the report
│
├── models/
│   └── baseline/
│       ├── zul_baseline.pkl
│       ├── xho_baseline.pkl
│       └── swa_baseline.pkl
│
└── documentation/
    ├── Group12_Report.pdf           # Compiled final report (ACL format)
    └── Group12_Proposal.pdf         # Research proposal submitted earlier

---

Installation

Requires Python 3.9+. It is recommended to use a virtual environment:

python3 -m venv venv
source venv/bin/activate      # Windows: venv\Scripts\activate
pip install -r requirements.txt

Note: Run all commands from the project root directory with the virtual environment active.

---

Running Experiments

All experiments are controlled from a single file. Open main.py and set the phase toggles at the top:

RUN_WEEK2 = True    # Baseline + exploratory analysis
RUN_WEEK3 = False   # mBERT + XLM-R fine-tuning (slow — CPU only)
RUN_WEEK4 = False   # Cross-lingual evaluation + final summary

Then run:

python main.py

Run all phases

RUN_WEEK2 = True
RUN_WEEK3 = True
RUN_WEEK4 = True

Transformer fine-tuning (offline mode)

The transformer models (bert-base-multilingual-cased, xlm-roberta-base) must be cached locally before running Week 3. Once cached, use offline mode to avoid network calls:

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python main.py

Or run individual model scripts directly:

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python src/transformers_pipeline/run_mbert.py
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 python src/transformers_pipeline/run_xlmr.py

---

Expected Outputs per Phase

Week 2 (Baseline)

File	Description
results/baseline_metrics.csv	Macro F1, micro F1, precision, recall per language
results/weak_labels.csv	Labels with F1 < 0.15
results/class_distribution.csv	Per-label counts per language
results/error_examples.csv	False negatives, over-predictions, partial matches
results/baseline_observations.md	Auto-generated observation report
models/baseline/*.pkl	Saved TF-IDF + classifier per language

Week 3 (Transformers)

File	Description
results/mbert_metrics.csv	mBERT per-language macro/micro F1, precision, recall
results/xlmr_metrics.csv	XLM-R per-language metrics
results/mbert_per_label_metrics.csv	Per-emotion-label breakdown for mBERT
results/xlmr_per_label_metrics.csv	Per-emotion-label breakdown for XLM-R
results/transformer_summary.csv	Combined mBERT + XLM-R summary

Week 4 (Evaluation)

File	Description
results/cross_lingual_comparison.csv	All three models × 4 metrics pivot table

---

Configuration

Baseline hyperparameters — src/baseline/model.py

TFIDF_CONFIG  = {"max_features": 5000, "ngram_range": (1, 2), "min_df": 2, ...}
LOGREG_CONFIG = {"max_iter": 1000, "class_weight": "balanced", "random_state": 42}

Transformer hyperparameters — src/transformers_pipeline/transformer_train.py

NUM_EPOCHS    = 3
BATCH_SIZE    = 8
LEARNING_RATE = 2e-5
MAX_LENGTH    = 128
THRESHOLD     = 0.3   # sigmoid binarisation threshold
POS_WEIGHT_CAP = 10.0 # cap on per-label positive-class weight
SEED          = 42

---

Reproducibility

All experiments use fixed seeds (random_state=42, seed=42). Transformer experiments were run on CPU (Apple Silicon, macOS). Results may vary slightly across platforms due to floating-point differences in scikit-learn's L-BFGS solver and PyTorch CPU kernels.

---

Troubleshooting

Problem	Fix
ModuleNotFoundError: No module named 'src'	Run from project root: python main.py
Dataset not found	Check data/BRIGHTER/{zul,xho,swa}/ folder names and parquet filenames
Transformer produces all-zero predictions	Increase epochs to ≥3, lower threshold to 0.3, ensure pos_weight is active
CUDA / memory errors	Reduce BATCH_SIZE in transformer_train.py
HuggingFace download errors	Use HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 once models are cached