SpamGuard is a complete, end-to-end machine learning application designed for real-time spam detection. It utilizes a sophisticated hybrid classification architecture, combining the speed of classical NLP models with the semantic power of modern transformer-based vector search.

The project is built as a full-stack MLOps platform, featuring a decoupled FastAPI backend for model serving and a Streamlit dashboard for real-time analysis, model management, continuous learning, and evaluation. The system is designed to be adaptive, allowing it to be retrained and improved with new data over time to combat evolving spam tactics.

Core Technology Stack:

• Backend: FastAPI, Uvicorn
• Machine Learning: Scikit-Learn, PyTorch, Transformers, FAISS
• Data Handling: Pandas, SQLite, NLTK
• Frontend: Streamlit

SpamGuard is more than just a classifier; it's a complete toolkit for managing and improving a production-grade spam detection model.

The core of SpamGuard is its two-stage classification process, designed to balance speed and accuracy:

• Stage 1: High-Speed Triage: A fine-tuned Multinomial Naive Bayes classifier, using a TF-IDF feature representation with N-grams, provides an initial, lightning-fast prediction. It confidently handles the vast majority of "easy" cases.
• Stage 2: Deep Semantic Analysis: For messages that the Naive Bayes model finds ambiguous, the task is escalated to a powerful k-Nearest Neighbors (k-NN) search. This stage uses intfloat/multilingual-e5-base sentence embeddings to find the most semantically similar messages in a high-speed FAISS vector index, making a final, context-aware prediction.

The Streamlit frontend provides a comprehensive suite of tools for managing the entire model lifecycle:

• Real-time Classification: A simple interface to test the live model with any message.
• Model Management & Registry:
  • Model Versioning: Every time the model is retrained, a new, timestamped version is created and logged in a central registry.
  • Model Activation: Users can seamlessly "hot-swap" the active production model to any historical version with a single click, allowing for instant rollbacks if a new model underperforms.
  • Traceability: An interactive table allows users to add and edit descriptive notes for each model version and dataset, ensuring perfect traceability of what each model was trained on.
• Operational Configuration: Users have full control over the classifier's behavior, with the ability to switch between Hybrid, Naive Bayes Only, or k-NN Only modes on the fly. The dataset used for k-NN indexing is also fully configurable via the UI.

SpamGuard is designed to improve over time through a robust feedback loop:

• Multi-Source Feedback: The system collects new training data from three sources: single-message corrections from the real-time classifier, bulk uploads of labeled .txt or .csv files, and LLM-generated synthetic data.
• Data Staging Area: All new data is sent to a "staging" database, not directly into the training set.
• Interactive Review: A powerful UI featuring an interactive st.data_editor table allows users to search, filter, and review all pending feedback. Users can select which messages to keep or discard before committing them to the main dataset.
• Non-Blocking Retraining: The entire retraining process is offloaded to a background task, allowing the application to remain fully operational while a new model version is being trained.

To build trust and provide insight into the model's decision-making process, the dashboard includes a Model Interpretation feature. It analyzes the active MultinomialNB model and displays the top 20 keywords that it has learned are the most powerful indicators for both the spam and ham classes.

You can also check out the "explaination" for results k-NN Search, in the section 1 of SpamGuard(if k-NN is used)

The system features a non-blocking, asynchronous module for generating synthetic training data using various local or cloud-based LLMs (Ollama, LM Studio, OpenRouter).

• Continuous Generation: Users can start a continuous generation task that runs in the background.
• Live Review: Generated messages appear in the UI in real-time for review.
• Curated Addition: Users have full control to select which synthetic messages are valuable and send only those to the staging area for retraining, preventing low-quality LLM outputs from polluting the dataset.

The SpamGuard application was architected from the ground up to handle the challenges of serving large, in-memory machine learning models in a responsive and stable way.

• Problem: The initial model load, especially the creation of the FAISS index from thousands of embeddings, can take several minutes. A standard "eager loading" approach would freeze the web server on startup, making it slow to launch and frustrating to develop with.
• Solution: We implemented a truly asynchronous startup process.
• Mechanism: When the FastAPI server starts, its startup event immediately spawns a dedicated, long-running background thread (threading.Thread). This background thread is solely responsible for performing the entire slow, blocking model-loading process. The main server thread, which handles user requests, is not blocked and starts instantly.
• Communication: A simple "flag file" (_ready.flag) is used to communicate the state. The background thread creates this file only after all models are successfully loaded into memory. The Streamlit UI polls a lightweight /status endpoint, which in turn checks for the existence of this flag file. This completely isolates the blocking I/O from the web server's main event loop, ensuring instant startups and reloads.

• Problem: When a user activates a new model or changes the k-NN dataset configuration, the application needs to load a new set of large models, which is a slow process. A simple reload would cause downtime.
• Solution: The backend uses a "hot-swapping" pattern with a ClassifierManager that holds two instances of the classifier: one for production and one for staging.
• Mechanism: When a configuration change is requested, a background task is initiated to load the new models into the staging instance. During this entire time, the production instance continues to serve all incoming /classify requests without any interruption. Once the staging instance is fully loaded, an atomic, thread-safe swap occurs, and the new model instantly becomes the production model. This provides a seamless, zero-downtime experience for the end-user.

• Problem: Long-running tasks initiated by a user, such as retraining or LLM data generation, would block the API and make the UI unresponsive.
• Solution: All potentially long-running API endpoints (/retrain, /llm/start_generation, etc.) are implemented using FastAPI's BackgroundTasks.
• Mechanism: When a user clicks "Retrain," the API endpoint immediately adds the entire retraining and reloading sequence as a background task and returns an instant "Task Started" response. The user can continue to use the application with the old model while the new one trains in the background. The UI polls the /status endpoint to track the progress and updates automatically when the new model is ready.

• Problem: Re-calculating sentence embeddings and rebuilding the FAISS index is the most time-consuming part of the loading process. Doing this on every model load is extremely inefficient.
• Solution: The SpamGuardClassifier implements an intelligent caching mechanism for the FAISS index.
• Mechanism: After an index is built for a specific dataset (e.g., before_enron.csv), it is saved to a uniquely named file on disk (e.g., faiss_index_before_enron_csv.bin). On subsequent loads that require the same dataset, the classifier checks the modification time of the cache file against the modification time of its source .csv data file (os.path.getmtime).
• Behavior: If the source data has not changed, the pre-built index is loaded directly from disk, which is orders of magnitude faster than rebuilding it. The index is only re-computed from scratch if the underlying data has been modified (i.e., after a retraining cycle has enriched that specific file).

• Problem: Loading multiple instances of the large classifier models would consume excessive memory and lead to inconsistent states.
• Solution: The backend uses a single, global "manager" object (manager = AppStateManager()) that holds the one and only instance of the production classifier and the LLM generation state. All API endpoints interact with this single, shared object, ensuring a consistent and memory-efficient state across the entire application. This singleton is made thread-safe using threading.Lock() to prevent race conditions during critical operations like model hot-swapping.

Version 1 using GaussianNB is here: https://github.com/alberttrann/SpamGuard

To use SpamGuard, from the root directory, type uvicorn backend.main:app --reload, then in a new terminal, streamlit run dashboard/app.py

If there is any model-related issue, consider deleting the models dir, and type python -m backend.train_nb for retraining and creating a fresh model dir, before running SpamGuard

The first iteration of SpamGuard was conceived as a two-tier hybrid system, combining a classical machine learning model for rapid triage with a modern vector search for nuanced analysis. While sound in theory, a retrospective analysis reveals that the specific choices for the triage component—Gaussian Naive Bayes paired with a Bag-of-Words feature representation—were fundamentally misaligned with the nature of the text classification task, leading to systemic performance degradation.

At the core of any Naive Bayes classifier is Bayes' Theorem, which allows us to calculate the posterior probability P(y | X) (the probability of a class y given a set of features X) based on the likelihood P(X | y) and the prior probability P(y). The "naive" assumption posits that all features x_i in X are conditionally independent, simplifying the likelihood calculation to:

P(X | y) = P(x_1 | y) * P(x_2 | y) * ... * P(x_n | y)

The critical differentiator between Naive Bayes variants lies in how they model the individual feature likelihood, P(x_i | y). GaussianNB assumes that for any given class y, the values of a feature x_i are drawn from a continuous Gaussian (Normal) distribution. To model this, the algorithm first calculates the mean (μ) and standard deviation (σ) of each feature x_i for each class y from the training data.

When a new data point arrives, GaussianNB calculates the likelihood P(x_i | y) using the Probability Density Function (PDF) of the normal distribution:

P(x_i | y) = (1 / (sqrt(2 * π * σ²))) * exp(-((x_i - μ)² / (2 * σ²)))

This is the central flaw. Our features, derived from a Bag-of-Words model, are discrete integer counts of word occurrences. The distribution of these counts is anything but normal; it is a sparse, zero-inflated distribution. For any given word (feature x_i), its count across the vast majority of documents (messages) will be 0. Applying a model that expects a continuous, bell-shaped curve to this type of data leads to several severe consequences:

1. Invalid Probability Estimates: The PDF calculation for a count of 0 or 1 on a distribution whose mean might be 0.05 and standard deviation is 0.2 is mathematically valid but semantically meaningless. It does not accurately represent the probability of observing that word count.
2. Extreme Sensitivity to Outliers: A word that appears an unusually high number of times can drastically skew the calculated mean and standard deviation, making the model's parameters for that feature highly unstable and unreliable.
3. Systemic Overconfidence: The mathematical nature of the Gaussian PDF, when applied to sparse, discrete data, tends to produce probability estimates that are pushed towards the extremes of 0.0 or 1.0. The model rarely expresses uncertainty, a critical failure for a triage system designed to identify ambiguous cases.

The Bag-of-Words (BoW) model was used to convert raw text into numerical vectors for GaussianNB. This process involves:

1. Tokenization: Splitting text into individual words.
2. Vocabulary Building: Creating a master dictionary of all unique words across the entire corpus.
3. Vectorization: Representing each document as a vector where each element corresponds to a word in the vocabulary, and its value is the raw count of that word's appearances in the document.

While simple and fast, BoW has two primary weaknesses in the context of spam detection:

1. It Ignores Semantic Importance: BoW treats every word equally. The word "the" is given the same initial consideration as the word "lottery". It has no mechanism to understand that certain words are far more discriminative than others for identifying spam. This places the entire burden of discerning importance on the classifier, a task for which the flawed GaussianNB is ill-equipped.
2. It Loses All Context: By treating each document as an unordered "bag" of words, all syntactic information and word collocations are lost. The model cannot distinguish between the phrases "you are free to go" (ham) and "get a free iPhone" (spam).

When this context-free, non-discriminative feature set is fed into a GaussianNB model that fundamentally misunderstands the data's distribution, the performance degradation is compounded. The model is forced to make predictions based on flawed probability estimates of features that lack the necessary semantic weight and context.

The original dataset exhibited a severe class imbalance, with ham messages outnumbering spam messages by a ratio of approximately 6.5 to 1. In a Naive Bayes framework, this directly influences the prior probability, P(y). The model learns from the data that P(ham) is approximately 0.87, while P(spam) is only 0.13.

When classifying a new message, this prior acts as a powerful weighting factor. The final posterior probability is proportional to P(X | y) * P(y). Even if the feature likelihood P(X | spam) is moderately high, it is multiplied by a very small prior P(spam), making it difficult to overcome the initial bias.

This problem becomes acute when paired with GaussianNB's weaknesses:

• The model's tendency to be overconfident means it rarely finds a message "ambiguous".
• This overconfidence, when combined with the strong ham prior, creates a system that is heavily predisposed to classify any message that isn't blatantly spam as ham with very high confidence, effectively silencing the Stage 2 classifier.

Our initial strategy to use LLM-based data augmentation was a logical step to address this imbalance by synthetically increasing the spam prior. However, as the experiments later proved, this was akin to putting a larger engine in a car with misshapen wheels. While it addressed one problem (data imbalance), it could not fix the more fundamental issue: the core incompatibility between the GaussianNB model and the BoW text features.

The intended role of the Stage 2 classifier was to act as a "deep analysis" expert for cases the Stage 1 triage found difficult. Its mechanism is fundamentally different from Naive Bayes:

1. Embedding: It uses a pre-trained sentence-transformer model (intfloat/multilingual-e5-base) to convert the entire meaning of a message into a dense, 768-dimensional vector. Unlike BoW, this embedding captures semantic relationships, syntax, and context.
2. Indexing: The entire training corpus is converted into these embeddings and stored in a FAISS index, a highly efficient library for similarity search in high-dimensional spaces.
3. Search (k-NN): When a new message arrives, it is converted into a query embedding. FAISS then performs a k-Nearest Neighbors search, rapidly finding the k messages in its index whose embeddings are closest (most similar in meaning) to the query embedding.
4. Prediction: The final prediction is made via a simple majority vote among the labels of these k neighbors.

This is a powerful but computationally expensive process. The initial architecture's critical failure was that the conditions for this fallback—an uncertain prediction from Stage 1—were never met due to the flawed and overconfident nature of the GaussianNB classifier. The "expert" was never consulted.

The empirical failure of the V1 architecture served as a powerful diagnostic tool, revealing that the system's bottleneck was not the data, but the fundamental incompatibility between the chosen triage model and the nature of text-based features. The transition to the V2 architecture was a deliberate, multi-faceted overhaul of the Stage 1 classifier, designed to replace the flawed components with tools mathematically and technically suited for the task. This involved three targeted modifications: a new Naive Bayes classifier, a more intelligent feature representation, and a more robust method for handling class imbalance.

The most critical modification was replacing GaussianNB with MultinomialNB. This decision stemmed directly from analyzing the mismatch between the Gaussian assumption and the discrete, high-dimensional nature of text data.

The Multinomial Distribution: A Model for Counts The MultinomialNB classifier is built upon the assumption that the features are generated from a multinomial distribution. This distribution models the probability of observing a certain number of outcomes in a fixed number of trials, where each outcome has a known probability. In the context of text classification, this translates perfectly:

• A document is considered the result of a series of "trials."
• Each trial is the event of "drawing a word" from the vocabulary.
• The features x_i are the counts of how many times each word w_i from the vocabulary was drawn for that document.

The Mathematical Difference in Likelihood Calculation Unlike GaussianNB's reliance on the normal distribution's PDF, MultinomialNB calculates the likelihood P(x_i | y) using a smoothed version of Maximum Likelihood Estimation. The core parameter it learns for each feature x_i (representing word w_i) and class y is θ_yi:

θ_yi = P(x_i | y) = (N_yi + α) / (N_y + α * n)

Let's break down this formula:

• N_yi is the total count of word w_i across all documents belonging to class y.
• N_y is the total count of all words in all documents belonging to class y.
• n is the total number of unique words in the vocabulary.
• α (alpha) is the smoothing parameter, typically set to a small value like 1.0 (Laplace smoothing) or 0.1.

The Role of Additive Smoothing (Alpha) The α parameter is crucial. Without it (α=0), if a word w_i never appeared in any spam message during training, N_spam,i would be 0. Consequently, P(w_i | spam) would be 0. If this word then appeared in a new message, the entire product for P(X | spam) would become zero, regardless of any other strong spam indicators in the message. This "zero-frequency problem" makes the model brittle.

By adding α, we are artificially adding a small "pseudo-count" to every word in the vocabulary. This ensures that no word ever has a zero probability, making the model far more robust to unseen words or rare word-class combinations.

By adopting MultinomialNB, we are using a model whose internal mathematics directly mirrors the generative process of creating a text document as a collection of word counts. This alignment results in more accurate, stable, and realistically calibrated probability estimates, which is essential for a functional triage system.

While MultinomialNB can operate on raw Bag-of-Words counts, its performance is significantly enhanced by providing it with more informative features. The switch from simple BoW to TfidfVectorizer with N-grams was designed to inject semantic weight and local context into the feature set.

Term Frequency-Inverse Document Frequency (TF-IDF) transforms raw word counts into a score that reflects a word's importance to a document within a corpus. The score for a term t in a document d is:

TF-IDF(t, d) = TF(t, d) * IDF(t)

• Term Frequency (TF): This measures how often a term appears in the document. To prevent longer documents from having an unfair advantage, this is often represented as a logarithmically scaled frequency: TF(t, d) = 1 + log(f_td) where f_td is the raw count. This is the sublinear_tf parameter.
• Inverse Document Frequency (IDF): This is the key component for weighting. It measures how rare a term is across the entire corpus, penalizing common words: IDF(t) = log( (N) / (df_t) ) where N is the total number of documents and df_t is the number of documents containing the term t.

A word like "the" will have a very high df_t, making its IDF score close to zero. A specific spam-related word like "unsubscribe" will have a low df_t, yielding a high IDF score. By multiplying TF and IDF, the final feature vector represents not just word counts, but a weighted measure of each word's discriminative power.

Incorporating N-grams: By setting ngram_range=(1, 2), we instruct the vectorizer to treat both individual words (unigrams) and two-word sequences (bigrams) as distinct terms. This is a crucial step for capturing local context. The model can now learn a high TF-IDF score for the token "free gift", distinguishing it from the token "free" which might appear in a legitimate context.

This improved feature set allows the MultinomialNB classifier to base its decisions on features that are inherently more predictive, significantly improving its ability to separate spam from ham.

While LLM-based data augmentation improved the overall class ratio in the dataset, this is a form of static, pre-training augmentation. SMOTE (Synthetic Minority Over-sampling Technique) offers a form of dynamic, in-training balancing that provides a distinct and complementary benefit.

When integrated into a scikit-learn Pipeline, SMOTE is applied only during the .fit() (training) process. It does not affect the .predict() or .transform() methods, meaning it never introduces synthetic data into the validation or test sets.

The Geometric Mechanism of SMOTE: SMOTE operates in the high-dimensional feature space created by the TfidfVectorizer. Its algorithm is as follows:

1. For each sample x_i in the minority class (spam), find its k nearest neighbors from the same minority class.
2. Randomly select one of these neighbors, x_j.
3. Generate a new synthetic sample x_new by interpolating between the two points: x_new = x_i + λ * (x_j - x_i), where λ is a random number between 0 and 1.

Geometrically, this is equivalent to drawing a line between two similar spam messages in the feature space and creating a new, plausible spam message at a random point along that line.

Why SMOTE is still effective with LLM-Augmented Data: The LLM augmentation provides a diverse set of real-world-like examples. However, within the feature space, there may still be "sparse" regions where spam examples are underrepresented. SMOTE's role is to densify these sparse regions. It ensures that the decision boundary learned by the MultinomialNB classifier is informed by a smooth and continuous distribution of minority class examples, preventing the classifier from overfitting to the specific (though now more numerous) examples provided by the LLM and the original data. It acts as a final "smoothing" step on the training data distribution, making the resulting classifier more generalized and robust.

The culmination of these three changes results in a new Stage 1 triage system that is not only more accurate but also more "cautious" or self-aware. The MultinomialNB classifier, trained on balanced, high-quality TF-IDF features, produces far more reliable and well-calibrated probability estimates.

This new reliability is what makes the hybrid architecture functional. The triage thresholds—classifying with NB if P(spam) < 0.15 or P(spam) > 0.85—are no longer arbitrary.

• When the new model produces a probability of 0.95, it is a high-confidence prediction backed by a robust mathematical model and strong feature evidence.
• Crucially, when it encounters a truly ambiguous message, it is now capable of producing a "middle-ground" probability like 0.60, correctly identifying that it is uncertain.

This act of "knowing what it doesn't know" is the key. By correctly escalating these genuinely difficult cases to the semantically powerful but computationally expensive k-NN Vector Search, the system achieves a synergistic effect. It combines the efficiency of the MultinomialNB model (which, as benchmarks show, handles the majority of cases) with the peak accuracy of the Vector Search, resulting in a final system that approaches the accuracy of the costly k-NN model at a fraction of the average computational cost.

This phase of the project involved a comprehensive suite of experiments designed to quantitatively measure the performance and computational efficiency of each architectural iteration. By testing three distinct classifier architectures (MultinomialNB-only, k-NN Vector Search-only, and the final Hybrid System) on both the original biased dataset and the LLM-augmented dataset, we can dissect the specific contributions of model selection, data quality, and system design to the final outcome.

The following table summarizes the performance of all evaluated models on a consistent hold-out test set. It includes key metrics for both ham (legitimate messages) and spam classes.

When trained on the limited and imbalanced original data, all models exhibited a distinct, conservative behavior.

• MultinomialNB Only (Model 1): This model was a stellar performer at correctly identifying ham. With a Ham Recall of 0.96, it almost never made the critical user-facing error of misclassifying a legitimate message as spam (only 2 out of 46 times). This is reflected in its extremely high Spam Precision of 0.94; if a message landed in the spam folder, users could be very confident it was indeed spam. However, this safety came at the cost of a poor Spam Recall of 0.67, allowing a third of all spam to slip into the inbox.

• k-NN Vector Search Only (Model 2): The semantic model performed better on all fronts, achieving a higher Spam Recall of 0.78 while maintaining an excellent Ham Recall of 0.98. This demonstrates the transformer's superior ability to generalize from limited data. However, it was the slowest model by a significant margin (21.56 ms/msg).

• Hybrid System (Model 3): This system produced the most interesting results on the biased data. It achieved a perfect Ham Recall of 1.00 and Spam Precision of 1.00. This means it never once misclassified a legitimate message as spam. The internal metrics show the MultinomialNB triage (with its strong ham bias) handled the majority of cases, while the k-NN escalation was used for 28% of messages. The system was exceptionally safe for users but still suffered from a mediocre Spam Recall of 0.74, failing to catch a quarter of all spam.

Training on the high-quality, LLM-augmented data transformed the behavior of all classifiers, shifting them from a conservative "ham-first" stance to a more aggressive and effective "spam-catching" stance.

• MultinomialNB Only (Model 4): The impact of the new data is clear. Spam Recall skyrocketed from 0.67 to 0.91, indicating a vastly more effective filter. This came at the cost of a lower Ham Recall (0.85), meaning it made more false positive errors than before (7 vs. 2). However, the balanced F1-scores for both classes (0.88) show this is a much more well-rounded and effective classifier overall.

• k-NN Vector Search Only (Model 5): This combination represents the "gold standard" for accuracy. It achieved a perfect Spam Recall of 1.00, catching every single spam message in the test set. Its Ham Recall of 0.93 is also excellent, with only 3 false positives. This demonstrates the immense power of providing a rich semantic database for similarity search. At 16.85 ms/msg, it remains the computational benchmark to beat.

• Hybrid System (Model 6): This is the champion architecture. It matches the gold standard's perfect Spam Recall of 1.00, ensuring maximum filter effectiveness. Its Ham Recall of 0.91 is also excellent and very close to the pure k-NN's performance. The system successfully blocks all spam while only misclassifying 4 legitimate messages.

  • Efficiency: The timing data proves the success of the hybrid design. At 7.56 ms/message, it is 2.2 times faster than the pure k-NN model.
  • Intelligent Triage: The system's internal metrics confirm its effectiveness. The MultinomialNB triage, with its own 96.83% accuracy, correctly handled 68.5% of cases, allowing the system to achieve its speed. The remaining 31.5% of "hard" cases were escalated to the k-NN expert, which itself was 93.10% accurate on this difficult subset.

Analyzing the error types for models trained on the original, biased dataset reveals a distinct pattern of conservative, low-confidence behavior. The models are forced to make significant trade-offs that either harm filter effectiveness or, in the worst cases, erode user trust.

Conclusion from Confusion Matrix Analysis (Original Dataset): When trained on poor, biased data, all architectures prioritize user safety (minimizing False Positives) at the direct expense of filter effectiveness (high number of False Negatives). The semantic power of the k-NN model makes it the best of the three, but all are fundamentally handicapped. This analysis proves that data quality is a prerequisite for achieving a balance between user trust and filter effectiveness. Without a rich and balanced dataset, even a sophisticated architecture is forced to make unacceptable compromises.

While overall accuracy provides a good summary, a deeper analysis of the confusion matrices is essential to understand the practical, user-facing implications of each model. For a spam filter, the two types of errors have vastly different consequences:

• False Positive (Type I Error): A legitimate message (ham) is incorrectly classified as spam. This is the most severe type of error. It can cause a user to miss critical information, such as a job offer, a security alert, or a personal message. The primary goal of a production spam filter is to minimize this value. This corresponds to a high Ham Recall.
• False Negative (Type II Error): A spam message is incorrectly allowed into the inbox. This is a minor annoyance for the user but is far less damaging than a False Positive. A robust system should minimize this, but not at the great expense of increasing False Positives. This corresponds to a high Spam Recall.

Let's analyze the confusion matrices ([[TN, FP], [FN, TP]]) for the three final models trained on the Augmented Dataset:

Conclusion from Confusion Matrix Analysis: The analysis confirms that both the k-NN Only and the Hybrid System are exceptional performers when user experience (minimizing False Positives) is the top priority. The Hybrid System, however, stands out. It successfully delivers a user experience that is almost identical to the computationally expensive "gold standard" model—catching all spam while only misclassifying 4 legitimate messages—at a fraction of the operational cost. It proves that the triage system is not just faster, but also "smart" enough to escalate cases in a way that preserves the most critical performance characteristics of the expert model.

To contextualize the performance of our specialized SpamGuard Hybrid System, a final suite of benchmarks was conducted against a diverse range of general-purpose Large Language Models (LLMs). These models, varying in parameter count from 500 million to 671 billion, were evaluated on the same hold-out test set in a zero-shot classification task. The objective was to determine the trade-offs between a small, fine-tuned, specialized system and the raw inferential power of modern foundation models.

Master Benchmark Table: All Architectures

The results from the FPT AI Cloud API are unequivocal: state-of-the-art foundation models like DeepSeek-V3 (671B) and Qwen2.5-7B-Instruct achieve perfect 100% accuracy on our test set. Their immense scale and sophisticated reasoning capabilities allow them to correctly classify every message in a zero-shot setting. This establishes the theoretical "perfect score" for this specific task.

However, this perfection comes at the highest operational cost. With average latencies of ~180-420 ms, they are 23x to 55x slower than our specialized Hybrid System. This makes them unsuitable for applications requiring real-time, low-latency processing of high-volume message streams.

The locally-hosted models tested via LM Studio reveal a fascinating trend. A clear performance tier emerges in the 2B to 4B parameter range. Models like phi-4-mini-3.8b, exaone-3.5-2.4b, qwen2.5-3b, and gemma-2/3-it consistently achieve accuracy in the 97-99% range, nearly matching the large-scale cloud models.

• Key Insight: These models are powerful enough to have robust instruction-following capabilities and a strong grasp of the nuances of spam language. They correctly balance Ham Recall and Spam Recall, making very few errors of either type.
• Latency Consideration: While highly accurate, their performance cost is still significant. Even the fastest of this tier (qwen2.5-3b at 133 ms) is 17x slower than our Hybrid System. The gemma-3-4b-it model, despite its high accuracy, was exceptionally slow in this test, highlighting that parameter count is not the only factor in performance; architecture and quantization also play a major role.

A dramatic performance collapse is observed in models with fewer than ~2 billion parameters. Models like llama-3.2-1b, qwen2.5-1.5b, exaone-1.2b, and smollm2-1.7b perform poorly, with accuracies ranging from 50% to 64%.

• Analysis of Failure Mode: Their confusion matrices reveal a consistent pattern: an extremely low Ham Recall (e.g., smollm2 at 0%, llama-3.2-1b at 7%). This is not a classification failure; it is an instruction-following failure. These models are not sophisticated enough to reliably adhere to the system prompt: "Respond with ONLY the single word 'spam' or 'ham'." Instead, they tend to default to a single response (in this case, "spam") for almost every input. Their poor accuracy is a result of this "mode collapse," not a nuanced misjudgment of the text's content. The gemma-3-1b-it model is a notable exception, achieving a respectable 75% accuracy, suggesting it has a stronger instruction-following capability for its size.

This comprehensive benchmark analysis provides the definitive argument for the SpamGuard Hybrid System. While massive, state-of-the-art LLMs can achieve perfect accuracy, they do so at an operational cost that is orders of magnitude higher.

Our SpamGuard Hybrid System, at 95.65% accuracy, successfully outperforms every tested LLM under the 2B parameter mark and performs competitively with many models in the 2-4B range.

Most critically, it delivers this high-tier accuracy with an average latency of just 7.56 milliseconds. This is:

• 23x faster than the 100% accurate Qwen2.5-7B-Instruct.
• 17x faster than the 98% accurate qwen2.5-3b-instruct.
• An astonishing 355x faster than the 98% accurate gemma-3-4b-it.

The SpamGuard project successfully demonstrates that a well-architected, specialized system leveraging domain-specific data and a hybrid of classical and modern techniques can achieve performance comparable to general-purpose models that are billions of parameters larger, while operating at a tiny fraction of the computational cost. It is a testament to the enduring value of efficient system design in the era of large-scale AI.

This document presents the definitive analysis of the SpamGuard project, charting its evolution from a flawed initial concept to a highly-optimized, specialized classification system. Through a comprehensive suite of benchmarks, we will dissect the performance of the in-house SpamGuard architectures and contextualize their capabilities against a wide array of general-purpose Large Language Models (LLMs), as well as specialized pre-trained BERT and roBERTA models. The analysis provides a clear, data-driven justification for the final architectural decisions and demonstrates the profound impact of targeted fine-tuning.

The SpamGuard project began with a hybrid architecture (V1) utilizing a Gaussian Naive Bayes classifier for rapid triage. This initial version, trained on the original, highly imbalanced dataset, serves as the crucial baseline against which all subsequent improvements are measured. Evaluation on a simple 92-message test set revealed a systemic failure.

Master Benchmark Table: Section 1

¹Timing for V1 is not applicable as the hybrid logic was non-functional due to the classifier's overconfidence.

Analysis of the V1 Architecture

The initial architecture was a categorical failure. With an accuracy of 59.78%, it performed only marginally better than random chance. The system produced an unacceptable number of both False Positives (19) and False Negatives (18), making it simultaneously unsafe for users and ineffective as a filter. The root cause was the use of GaussianNB, a model whose mathematical assumptions are fundamentally incompatible with sparse, discrete text data, leading to a complete breakdown in performance. This baseline serves as the definitive justification for the architectural pivot to a more appropriate model.

The catastrophic failure of the V1 architecture prompted a complete redesign (V2) centered on the mathematically appropriate MultinomialNB model, a more sophisticated TF-IDF vectorizer, and the use of SMOTE for in-training data balancing. This section documents the performance of the three V2 architectures (MultinomialNBOnly, k-NN Search Only, and the final Hybrid System) across a four-stage evaluation process. This methodology allows us to isolate the impact of:

1. The initial architectural uplift.
2. The effect of general data augmentation.
3. The system's performance under adversarial stress-testing.
4. The final, dramatic improvements from targeted fine-tuning.

Master Benchmark Table: Section 2 - Specialized Architectures (Complete & Final)

before_270.csv is the csv file before 270 new tricky ham messages were added in. HOWEVER, this is not the original biased csv, this is after the llm-augmentation, and this is also the dataset used to train the models that we did eval on the easy 92-message hold-out testset 2cls_spam_text_cls.csv is the current most latest csv with 270 new tricky ham messages already in, and it's also the dataset that we use to retrain our architectures to deal with the new mixed_test_set and only_tricky_ham_test_set 222542 is the model before re-trained on new 270 tricky ham messages. This is also the model that we did eval on the easy 92-message hold-out test set 075009 is the model that has been re-trained on 270 new tricky ham messages to deal with new mixed_test_set and only_tricky_ham_test_set, which beats the LLMs with SIMPLE prompt, not the later versions that are buffed heavily with few-shot examples and CoT prompting the local qwen2.5-7B-Instruct that's been applied advanced prompting techniques is the same as the cloud version, it's just that i decided to run it myself to save money

²Spam-related metrics are not applicable (---) for the Tricky Ham test set as it contains no spam samples.

Analysis of V2 Architectural Evolution

1. Foundational Improvements (Models 2a-f): The switch to the V2 architecture provided an immediate, massive uplift over the V1 baseline. On the simple Easy 92-Msg test set, the Hybrid System proved highly effective, achieving 95.65% accuracy after general LLM augmentation (Model 2f). This version demonstrated a perfect Spam Recall of 1.00, proving its effectiveness as a filter, while also maintaining a strong Ham Recall of 0.91, ensuring a good user experience. This established that a sound architecture paired with a generally balanced dataset creates a powerful baseline.
2. The "Great Filter" - Exposing Context Blindness (Models 2g-l): Subjecting the generally-augmented models to the difficult and adversarial test sets revealed their critical weakness. The performance of all three architectures collapsed.
   • The Hybrid System (Model 2i), previously the champion, saw its accuracy plummet to a disastrous 38.00% on the mixed set. Its Ham Recall (Safety) fell to just 0.07, meaning it incorrectly flagged 92 out of 99 legitimate messages as spam. On the purely adversarial Tricky Ham set (Model 2l), the accuracy was a mere 7.69%.
   • This is a crucial finding. It proves that even with a sophisticated hybrid architecture and a generally large and balanced dataset, the model is fundamentally a keyword-based system. Without explicit training on legitimate messages that contain spam-like keywords, its learned associations are brittle and fail completely when faced with adversarial, context-heavy examples.
3. The Triumph of Targeted Fine-Tuning (Models 2m-r): The final stage of retraining on the current dataset, which included 270 hand-crafted "tricky ham" examples, represents the most significant breakthrough of the project.
   • Performance Surge: The Hybrid System's accuracy on the difficult mixed_test_set (Model 2o) skyrocketed from 38.00% to 95.33%. On the purely adversarial tricky_ham_test_set (Model 2r), its accuracy leaped from 7.69% to 94.23%.
   • Solving the Core Problem: The most important metric, Ham Recall (Safety), was fundamentally repaired. On the mixed set, it jumped from a catastrophic 0.07 to an excellent 0.96. This demonstrates the profound efficiency of fine-tuning: a small, targeted dataset was sufficient to teach our specialized model the critical contextual nuances that thousands of general examples could not.
   • The Optimal System: The final retrained Hybrid System (2o, 2r) emerges as the clear winner. It achieves accuracy in the 94-95% range on the most difficult data, maintains an elite Ham Recall of 94-96%, and does so at an extremely efficient average speed of ~5 milliseconds. It has been successfully hardened against the specific contextual failures identified in the previous stage, proving the value of a data-centric approach to model improvement.

To provide a powerful external benchmark, the specialized SpamGuard models were compared against a wide array of general-purpose LLMs, ranging from 500 million to 671 billion parameters. These models were evaluated in two distinct modes to test both their raw and guided intelligence:

1. Zero-Shot Prompting: A simple, direct prompt to test the models' out-of-the-box ability to classify spam.

system_prompt = "You are an expert spam detection classifier. Analyze the user's message. Respond with ONLY the single word 'spam' or 'ham'. Do not add explanations or punctuation."

2. Advanced Prompting: A more complex prompt combining Chain-of-Thought (CoT) and Few-Shot examples to explicitly guide the models toward a more nuanced, context-aware analysis.

def construct_advanced_prompt(message: str) -> list:
    system_prompt = (
        "You are an expert spam detection classifier. Your task is to analyze the user's message. "
        "First, you will perform a step-by-step analysis to determine the message's intent. "
        "Consider if it is a transactional notification, a security alert, a marketing offer, or a phishing attempt. "
        "After your analysis, on a new line, state your final classification as ONLY the single word 'spam' or 'ham'."
    )
    few_shot_examples = [
        {"role": "user", "content": "Action required: Your account has been flagged for unusual login activity from a new device. Please verify your identity immediately."},
        {"role": "assistant", "content": ("Analysis: The message uses urgent keywords like 'Action required' and 'verify your identity immediately'. However, it describes a standard security procedure (flagging unusual login). This is a typical, legitimate security notification.\nham")},
        {"role": "user", "content": "Congratulations! You've won a $1000 Walmart gift card. Click here to claim now!"},
        {"role": "assistant", "content": ("Analysis: The message claims the user has won a high-value prize for no reason. It creates a sense of urgency ('claim now!') and requires a click. This is a classic promotional scam.\nspam")}
    ]
    final_user_message = {"role": "user", "content": message}
    return [{"role": "system", "content": system_prompt}] + few_shot_examples + [final_user_message]

This section documents the performance of these LLMs across all three test sets: the Easy 92-Msgset, the difficult Mixed (Hard) set, and the adversarial Tricky Ham set.

Master Benchmark Table: Section 3 - LLM Performance

Analysis of LLM Performance

1. Success on Simple Data: The initial benchmarks on the Easy 92-Msg test set (3a-c) demonstrate that modern, large-scale LLMs are exceptionally powerful out-of-the-box. Both cloud-hosted models achieved 100% accuracy, and even locally-hosted mid-sized models were nearly perfect. This confirms their vast pre-trained knowledge is sufficient for simple classification tasks.
2. The "Keyword Trigger" Failure in Zero-Shot: The evaluation on the more difficult Mixed (Hard)and Tricky Ham test sets reveals a critical and consistent failure mode for all zero-shot LLMs, regardless of size (3d-k).
   • Catastrophic Ham Recall: The confusion matrices show a consistent pattern: a perfect Spam Recall of 1.00 but a catastrophic Ham Recall (Safety), often falling into the 30-60% range. For instance, gemma-2-2b-it incorrectly flagged 68 out of 99 legitimate messages as spam. This renders the models unusable for this task in a zero-shot setting, as they would flood a user's spam folder with critical alerts.
   • The Root Cause: This is a classic instruction-following failure driven by powerful keyword associations. The models have learned that words like "account," "verify," and "free" are overwhelmingly associated with spam. In a zero-shot context, this powerful prior association overrides the more nuanced task of contextual analysis. They are not "reasoning" about the message's intent; they are reacting to keyword triggers.
3. "Guided Reasoning" through Advanced Prompting: The introduction of Chain-of-Thought (CoT) and Few-Shot examples in the prompt (3l-q) had a profound effect. For a model like qwen2.5-7B(3l), accuracy on the Mixed (Hard) set leaped from a failing 66% to a stellar 96.67%. On the adversarial Tricky Ham set, phi-4-mini's accuracy (3o) surged from 55.77% to 92.31%.
   • The Mechanism: This proves that the models possess the latent reasoning capability to understand context, but it must be explicitly activated. The CoT prompt forces an intermediate reasoning step, analyzing the message's intent before making a final classification. The Few-Shot examples provide a direct template for how to handle the exact type of ambiguous messages that caused the zero-shot prompt to fail.
   • The Cost: This guided reasoning, however, comes at a significant cost. The average prediction times for advanced prompts were 5x to 10x higher than their zero-shot counterparts due to the vastly larger input and output token counts required for the reasoning step.

This comparison pits our best specialized SpamGuard Hybrid System against the top-performing, expertly-prompted LLMs on the most challenging test sets. Master Benchmark Table: Section 4 - The Final Showdown

Analysis of the Showdown

1. A Contest of Titans: Accuracy and Safety: The results are extraordinary. Our specialized SpamGuard Hybrid System, after being fine-tuned on just 270 targeted examples, performs at a level that is statistically on par with the best-prompted, multi-billion parameter LLMs.

   • On the Mixed (Hard) set, the Hybrid System's 95.33% accuracy and 0.96 Ham Recall are effectively identical to the qwen2.5-7B's performance, indicating equal levels of effectiveness and user safety.
   • On the purely adversarial Tricky Ham set, our Hybrid System's 94.23% accuracy and 0.94 Ham Recall are superior to the best-performing LLM (phi-4-mini). This proves that for this highly specific and adversarial task, targeted fine-tuning is more effective than even advanced prompting on a massive general-purpose model.

2. The Decisive Factor: A Colossal Gulf in Efficiency: The final verdict is delivered by the performance metrics. The average time per message for our SpamGuard Hybrid System was benchmarked at ~5 milliseconds.

   • This is ~900x faster than the qwen2.5-7B model.
   • This is an astonishing ~5,500x faster than the phi-4-mini model.

To provide a robust benchmark, the SpamGuard system was evaluated against a selection of publicly available, pre-trained spam detection models from the Hugging Face Hub. This comparison on our most challenging test sets reveals the true performance of our specialized system against other fine-tuned models and highlights the critical importance of the training data's quality and diversity.

Master Benchmark Table: A Showdown

This table compares the performance of our final, retrained SpamGuard Hybrid System against the public BERT models on the challenging Mixed (Hard) and adversarial Tricky Ham test sets.

²Spam-related metrics are not applicable (---) for the Tricky Ham test set as it contains no spam samples.

Analysis of Pre-trained BERT Model Performance

There are eval scripts and raw bench results related to this part that you can check out in this repo

The evaluation reveals extreme variance in the quality of publicly available models, underscoring the principle that a model's performance is dictated by its training data.

1. The "Brittle" Models (5c, 5d): mrm8488/bert-tiny and AntiSpamInstitute/spam-detector-bert-MoE-v2.2 exhibit a critical failure known as mode collapse. On any test set containing spam, they achieve 0% Spam Recall, classifying every single message as ham. Their confusion matrices ([[99, 0], [51, 0]]) confirm this. This makes them completely non-functional as spam filters. Their perfect scores on the Tricky Ham set are merely a side effect of this "always say ham" behavior. These models are demonstrably broken.
2. The Context-Blind Model (5b, 5h): The AventIQ-AI/bert-spam-detection model displays the opposite failure. On the Mixed (Hard) set, it achieves a perfect 1.00 Spam Recall but a catastrophic 0.41 Ham Recall. It incorrectly flagged 58 legitimate messages as spam. This model is a pure "keyword trigger" filter, unable to distinguish the context of words like "account" or "verify," and is therefore dangerously unusable due to its extremely high number of False Positives.
3. The High-Quality Competitor (5a, 5e): The mshenoda/roberta-spam model is the only public model that demonstrates robust, high-quality performance. It was clearly fine-tuned on a diverse dataset that included contextually ambiguous messages.
   • On the Mixed (Hard) set, its 95.33% accuracy is exceptional. Its strength is a perfect 1.00 Spam Recall, meaning it caught every spam message, though at the cost of 7 False Positives.
   • On the adversarial Tricky Ham set, its 94.23% accuracy is excellent, making only 3 False Positive errors.

A Competitive, Transparent, and Superior Engineering Solution

This benchmark against other specialized models provides the ultimate context for the SpamGuard project.

1. SpamGuard is a Top-Tier Performer: Our final, retrained SpamGuard Hybrid Systemdemonstrates performance that is directly competitive with the best publicly available spam detection model. Achieving 94.00% accuracy on the Mixed (Hard) test set and 94.23% on the Tricky Ham test set places it in the top echelon of specialized classifiers.
2. A Strategic Advantage in User Safety: The most important comparison is against the best competitor, mshenoda/roberta-spam, on the Mixed (Hard) set.
   • mshenoda/roberta-spam: 7 False Positives, 0 False Negatives.
   • SpamGuard Hybrid: 3 False Positives, 6 False Negatives.
   • This is a critical distinction. Our SpamGuard system makes less than half the number of critical False Positive errors, proving to be the safer and more trustworthy model for users. It prioritizes ensuring that important messages are not lost to the spam folder, a crucial business logic decision.
3. Efficiency, Transparency, and Adaptability: While both SpamGuard and mshenoda/roberta-spamare highly efficient, the SpamGuard system offers significant advantages in a production environment:
   • Explainability: Our system includes a built-in XAI module to interpret the MultinomialNBmodel, providing transparency into its decision-making.
   • Adaptability: The entire continuous learning pipeline (data ingestion, retraining, versioning) is an integral part of the project, allowing for rapid adaptation to new threats.

This final analysis consolidates all previous findings into a direct, "best-of-the-best" comparison across our three distinct evaluation environments. We pit the top-performing specialized model (mshenoda/roberta-spam), the top-performing LLM with the best prompting strategy, and our final, retrained SpamGuard Hybrid system against each other.

Arena 1: The Baseline - Performance on Easy 92-Msg Test Set

This test establishes the baseline performance on a simple, traditional spam detection task using the evaluation_data.txt file. This arena compares the raw accuracy of different approaches before they are challenged by adversarially crafted, context-heavy messages.

Analysis: On simple, non-adversarial data, the large-scale LLMs are flawless, achieving a perfect score and setting the theoretical performance ceiling. The most insightful comparison, however, is between our own specialized models. The SpamGuard k-NN Only system, relying purely on semantic search, emerges as the top-performing specialized model in this arena with 96.74% accuracy. It successfully catches every spam message (1.00 Spam Recall) while making only 3 False Positive errors.

Our SpamGuard Hybrid system is extremely competitive, achieving 95.65% accuracy and also catching every spam message. The crucial takeaway is the performance trade-off: the Hybrid system is more than twice as fast as the pure k-NN approach (~7.6 ms vs. ~16.9 ms). This demonstrates the success of the triage architecture: it sacrifices a single percentage point of accuracy for a massive gain in efficiency.

Both SpamGuard models significantly outperform the best publicly available BERT model on this test set in terms of filter effectiveness (Spam Recall). This initial benchmark validates that our data augmentation and architectural choices have produced a highly effective classifier, with the Hybrid system representing the optimal balance of speed and accuracy for this simple task.

Arena 2: The Real-World Challenge - Performance on mixed_test_set.txt

This test introduces contextually ambiguous "tricky ham," representing a more realistic and challenging environment.

Analysis: The top-tier LLM with advanced prompting (qwen2.5-7B) achieves the highest accuracy. However, our SpamGuard Hybrid system proves to be the safest for the user, making the fewest False Positive errors (3). This is a critical win. While the mshenoda BERT is slightly more accurate and effective at catching every single spam, it does so at the cost of more than double the number of critical user-facing errors. Once again, our system delivers elite, safety-focused performance at a fraction of the LLM's latency (~600x faster).

Arena 3: The Adversarial Gauntlet - Performance on only_tricky_ham_test_set.txt

This is the ultimate stress test, composed entirely of legitimate messages designed to be misclassified. The only goal is to maximize Ham Recall (Safety).

Analysis: our SpamGuard Hybrid System is tied for first place with the best pre-trained BERT model, both achieving 94.23% accuracy and making only 3 False Positive errors. Both specialized systems outperform the best-prompted LLM (phi-4-mini) in this adversarial environment. This definitively proves that for handling highly specific, nuanced, and adversarial data, a fine-tuned specialized model is superior to even a guided general-purpose model. The fact that our system achieves this while being ~3,700x faster than the LLM is the ultimate testament to its superior engineering.

The final phase of this project was to perform a rigorous evaluation of the SpamGuard system's ability to generalize to new, unseen data from entirely different domains. This analysis benchmarks several key models:

• The final, fine-tuned SpamGuard Hybrid model (ID ...075009), trained on our corpus of SMS and "tricky" security alerts(before_deysi.csv)
• A "Deysi-Trained" model(ID ...214414, using our Hybrid architecture but retrained on the clean, public deysi_train.txt dataset(before_ẻnon.csv)
• An "Incremental" model(ID...065804), created by taking the "Deysi-Trained" model and retrain on the enron_train.txt email dataset.(the current 2cls_spam_text_cls.csv)
• A high-quality, pre-trained mshenoda/roberta-spam model.
• A powerful, instruct-tuned LLM, qwen3-4b-instruct-2507, which serves as a proxy for other top-performing mid-sized LLMs like phi-4-mini-instruct(since the performance of the 2 model is really similar, and the qwen3 model has way tighter of the guardrail compared to the phi4 model, which helps it not refusing to classifying messages containing sensitive words)

This analysis provides a clear picture of the model's real-world robustness and the critical trade-offs between specialized fine-tuning, cross-domain training, and general-purpose intelligence.

This section analyzes performance on datasets that are structurally similar to our primary training data (short-form text) but originate from different sources.

This dataset is the closest analogue to our training data, consisting of short-form messages from Telegram. The evaluation was performed on a balanced 2,000-sample subset.

The roberta-spam model was pre-trained on this dataset, so its results are omitted for a fair test of generalization.

Analysis: On this structurally similar dataset, the prompted qwen3 LLM demonstrates strong generalization with 80.15% accuracy. Our original SpamGuard Hybrid is less accurate at 70.75%, primarily due to a low Spam Recall of 0.55, indicating its specialized knowledge of "tricky ham" did not translate well to the different spam tactics on Telegram. However, its Ham Recall of 0.86 was significantly better than the LLM's (0.68), making it a safer, more reliable choice for users.

The most fascinating result is the Incremental (Deysi+Enron) model. By training on long-form emails, it developed a more generalized understanding of language, which unexpectedly boosted its performance on this short-form dataset to 84.40% accuracy, surpassing even the LLM. This demonstrates that cross-domain training can create a more robust feature set. The ultimate trade-off remains efficiency: our specialized models are all orders of magnitude faster than the LLM.

This dataset is noted for being cleaner and containing emojis, representing a different style of short-form text.

Analysis: On this dataset, the models trained on the deysi data (Deysi-Trained and Incremental) are the clear champions, achieving a near-perfect ~99.4% accuracy. This is a perfect demonstration of in-domain performance. Critically, the Incremental model's performance was not diluted by its subsequent training on Enron emails, proving that it did not suffer from "catastrophic forgetting." The roberta-spam and qwen3 models are also exceptional performers at ~96% accuracy. Our original SpamGuard Hybrid, at 77.54%, again shows its hyper-specialization, as its knowledge did not generalize as effectively to this cleaner data distribution.

This section represents the ultimate stress test: evaluating our SMS-trained models on long-form emails.

This classic dataset consists of real, often messy, corporate emails.

Analysis: The domain shift to long-form email is where the Incremental (Deysi+Enron) model truly shines. Its accuracy skyrocketed to 93.06%, drastically outperforming our other specialized models and even surpassing the generalist qwen3 LLM. This is the most powerful evidence of successful cross-domain training. By learning from the Enron train split, it became an expert on that domain. The original SpamGuard Hybrid and the Deysi-Trained models both failed catastrophically on Spam Recall (0.24 and 0.22), proving that short-form knowledge is not transferable to this context.

This dataset is cleaner but contains longer, more traditional spam emails.

Analysis: This dataset reveals the limitations of our training strategies. All specialized models, including the Incremental model, struggled. The Incremental model's performance decreased here compared to the Deysi-Trained model (53.57% vs. 63.10%). This suggests that fine-tuning on the messy Enron data created a new bias that was not helpful for this cleaner, different style of email spam. The qwen3 LLM is the winner again, its broad knowledge allowing it to achieve the highest accuracy (69.05%) and the best user safety (0.79 Ham Recall).

1. The Hyper-Specialization of Fine-Tuning: Our original SpamGuard Hybrid and the Deysi-Trained model are brilliant specialists but poor generalists. Their knowledge is highly coupled to their specific training data.

2. The Power of Cross-Domain Training: The Incremental (Deysi+Enron) model proves that training on a second, different domain can massively boost generalization to a third, unseen domain (Telegram) and provide mastery in the new domain (Enron), all without forgetting its original specialty (Deysi).

3. The Limits of Cross-Domain Training: The Kaggle email benchmark shows that this generalization is not a silver bullet. Training on a new domain can create new biases that are detrimental when applied to a fourth, dissimilar domain. The model is always a product of the data it has seen.

4. LLMs as the Ultimate Generalists: The qwen3 LLM, when properly prompted, consistently demonstrated the strongest baseline performance across the widest range of unseen domains, proving the power of its vast pre-training.

5. The Un-winnable Trade-Off: The price for the LLM's superior generalization is a colossal performance cost. In every single test, our specialized models were orders of magnitude faster.

This extensive out-of-domain evaluation provides the project's final and most nuanced conclusion. The SpamGuard Hybrid System was successfully optimized to become a state-of-the-art classifier for its specific, chosen domain: SMS-style messages, including adversarial "tricky ham" security alerts.

This specialization, however, limits its ability to generalize. We have demonstrated that incremental, cross-domain training is a powerful technique to create a more robust "general specialist" model that performs well across multiple domains. Yet, this is not a universal solution, as new biases can be introduced.

Ultimately, the choice of model is a question of "best for the task." For a dedicated, high-throughput application focused on a specific domain, a highly efficient and fine-tuned system like SpamGuard is the superior engineering choice. For a low-volume, multi-domain analysis tool where accuracy across diverse formats is the only consideration, a prompted LLM remains the most capable generalist.

The SpamGuard project has been a comprehensive and iterative journey through the practical challenges and triumphs of developing a real-world machine learning system. From initial architectural design and rigorous comparative benchmarking to out-of-domain generalization analysis, this project culminates in a powerful set of insights applicable across diverse ML endeavors.

Our initial foray with a GaussianNB classifier, despite a logical hybrid design, revealed a fundamental mismatch between model assumptions and the discrete, sparse nature of text data. This crucial diagnostic step, evidenced by the catastrophic 59.78% accuracy on the original test set, underscored that model selection is not arbitrary; mathematical compatibility is paramount. The subsequent pivot to a MultinomialNB classifier, combined with TF-IDF feature engineering, represented the single most impactful architectural shift, transforming a non-functional system into a genuinely effective one. This highlights the critical importance of selecting the right tool for the right job and the power of iterative refinement in engineering ML solutions.

The project unequivocally demonstrates that data is not merely fuel for models, but a strategic asset whose quality, balance, and specificity dictate ultimate performance.

• Broad Augmentation for General Competence: The initial use of LLM-based data augmentation to mitigate the severe 6.5:1 ham bias was a critical first step. This successfully improved the class balance, enabling our models to achieve high baseline accuracy (~95-97%) on the original test set.
• Targeted Fine-Tuning for Contextual Mastery: The most profound insight emerged from the struggle with "tricky ham" messages. General augmentation proved insufficient; models trained only on it failed catastrophically on these adversarial cases. It was the strategic injection of just 270 meticulously crafted, targeted "tricky ham" examples that fundamentally re-educated our specialized models. This small, high-quality dataset enabled a leap in performance on challenging data, and the improvement in performance was significant on the mixed test set, increasing accuracy from a failing ~75% to a stellar ~94% and proving that precision in data curation can yield exponential returns.
• Systematic Balancing with SMOTE: The integration of SMOTE within the training pipeline complemented the data augmentation efforts. By operating in the feature space, it served to "densify" the minority class, creating smoother decision boundaries and improving the model's ability to generalize from the provided examples.

The SpamGuard Hybrid System is the project's flagship achievement in system design. It elegantly combines the efficiency of a fast, statistically-driven MultinomialNB triage with the deep semantic understanding of a k-NN Vector Search.

• Optimized Performance: The hybrid design ensures that the majority of messages are processed in milliseconds by the lightweight Naive Bayes model. Only the most ambiguous cases are escalated to the more computationally intensive semantic search.
• Elite Accuracy: The Hybrid System achieved 94.00% accuracy on the mixed_test_set with a 97% Ham Recall. The retrained Hybrid System, with its superior choice of architecture and training data, is a remarkable success.

The extensive benchmarking against both public specialized models and diverse LLMs provided crucial insights into the "No Free Lunch" theorem of machine learning and the often complex realities of model generalization. The results show a stark contrast.

• Best In-Domain Performance with the hybrid model: This model achieves over 94% on both test sets and the telegram-spam-ham dataset, and is particularly notable for a Spam Recall of 100% on the augmented data from its training, guaranteeing high filter effectiveness.

• High Accuracy & Low Latency: The Cost-Effective Solution: Compared to the advanced prompting LLM (e.g., qwen2.5-7B), this Hybrid model also demonstrates significant advantage in computational efficiency and therefore overall deployment cost, with its average time is ~7.6ms.

• Specialization vs. Generalization Tradeoffs: The final out-of-domain tests on Telegram and Enron email datasets revealed that while the 92% + performance level is exceptional. The SpamGuard Hybrid System demonstrated strong performance in similar text structures, particularly the Telegram data. It was, however, less effective on long-form email data and data that was very unlike the original training data, a key example being the Email Spam (Kaggle) which shows that our system struggled, with an accuracy of 45.24%, and an overall ham recall of 0.43 and precision of 0.28. These tests underscored that the choice of model is therefore not a question of which is "best" overall, but which is best for the task, and the best approach could vary significantly based on both the type of data and resource constraints.

• The Comparative Performance of Large Language Models: The evaluation of LLMs revealed their dual nature. The most accurate large language model (such as qwen3-4b), when using advanced prompting, achieves a compelling 88.00% accuracy and a high spam precision. Yet, it comes with significant performance and cost tradeoffs.

• Strong Model Generalization, Limited by Data Diversity: The SpamGuard system was able to generalize to telegram-spam-ham due to the structural similarity of the training data. Yet its accuracy did not perform well on the Enron email due to vastly different formatting.

• A Multi-Faceted Approach for a Comprehensive Conclusion: Your chosen methodology of creating multiple benchmarks against LLM, plus the incorporation of multiple test-split datasets, created a complete overview of performance that allows you to measure the effectiveness of the project, and understand the underlying trade offs.

• Practicality: Despite the impressive performance of the top-tier models, their compute time is incredibly high. Your architecture is not only accurate but delivers elite performance at a fraction of the cost compared to the alternative. The hybrid approach has been successful.

• An Adaptive, Production-Ready Solution: Beyond raw metrics, the project's focus on explainability (XAI), model versioning and management, and an interactive evaluation UI elevates it from a mere classifier to a production-ready demonstration. These MLOps features empower practitioners to understand model behavior, manage deployments safely, and adapt to evolving challenges in a data-centric manner.