Hello everyone!
I developed a spam detection that classifies emails as spam or legitimate (“ham”) using machine learning. It processes email content and provides predictions with confidence scores.
This post is to know your opinion about it and if we should adopt it in our infra
How it works?
I used the Naive Bayes method, which is a probabilistic classification algorithm based on Bayes’ Theorem. It’s called “naive” because it assumes that all features (words in the email) are independent of each other - which isn’t entirely true in practice, but works surprisingly well for spam detection.
Why Naive Bayes is Better for Spam Detection
The state of art to detect spams would be the use of Natural Language Processing (NLP), but because it needs bigger datasets and more hardware, I started with Naive Bayes to have a prototype and it outperforms other machine learning methods, that don’t require too much data or hardware, for spam detection.
In several key aspects that make it ideal for initial implementation. Its fundamental advantage lies in its exceptional efficiency with text-based data - it processes thousands of word features with remarkable speed, enabling real-time email classification. Unlike Random Forests or SVMs that struggle with high-dimensional text data, Naive Bayes handles large vocabularies effortlessly, treating each word as an independent feature without performance degradation.
The algorithm’s probabilistic nature provides inherent confidence scores for predictions, eliminating the need for additional calibration required by other methods. Most importantly, Naive Bayes achieves excellent results with relatively small training datasets and learns spam patterns rapidly, allowing for quick implementation and iteration. This combination of speed, efficiency, and effectiveness makes it the optimal starting point for spam detection systems.
Datasets Sources
These were the sources used in this project so far to get ham and spam emails samples:
Project structure
After installing python dependencies, I downloaded the dataset in data/raw/ and divided in two classes spam and ham with src/utils.py. Then trained the model with src/train_model.py which also calls src/data_processing.py, that creates the model in data/models and it can be used with src/predict.py to check if the files in emails are spam or ham.
This is the project structure so far.
spam_detector/
├── data/
│ ├── raw/ # Downloaded raw emails
│ │ ├── spam/ # Spam emails (*.txt)
│ │ └── ham/ # Legitimate emails (ham/*.txt)
│ └── models/ # Trained models (auto-generated)
├── emails/ # Folder for new emails to be checked
│ └── mail.txt # Individual email
├── src/
│ ├── data_processing.py # Data processing and cleaning
│ ├── train_model.py # ML model training
│ ├── predict.py # New email prediction
│ └── utils.py # Utility functions
├── requirements.txt # Python dependencies
└── README.md
Let me know what you think!
Let me know what you think about it, if we should adopt it in our infra, create a repo for it or even if you think we shouldnt use an AI for it. I also can provide more info if you have questions
also will do this ROC curve and then get back to you people