Model card
How Sift scores sentiment
A tf-idf (words and word pairs) + logistic regression trained on 3,000 labelled sentences from Amazon, IMDb and Yelp. It is retrained in GitHub Actions whenever the training script changes, and the numbers below come straight from that run on 600 sentences it never saw.
Test accuracy
82.7%
Test F1
82.8%
Cross validated F1
82.7%
Gain over VADER
+10.7 pts
Against the rule based baseline
| Model | Accuracy | Negative recall | Positive recall | Amazon | IMDb | Yelp |
|---|---|---|---|---|---|---|
| Sift classifier | 82.7% | 81.7% | 83.7% | 84.5% | 79.0% | 84.5% |
| VADER (rule based) | 72.0% | 51.0% | 93.0% | 74.5% | 70.5% | 71.0% |
VADER finds only 51.0% of negative sentences. For a tool meant to surface complaints, that is the number that matters, and the trained model finds 81.7%.
Confusion matrix
Training
- Version
- 20261005
- Regularisation C
- 16, picked by 5 fold grid search
- Split
- 2400 train, 600 test, stratified
- Training time
- 14.9s on a GitHub runner
Where it goes wrong
The test sentences it was most confident about and still got wrong.
- positive“I went to Bachi Burger on a friend's recommendation and was not disappointed.” was called negative with 96.1% confidence
- positive“They know how to make them here.” was called negative with 94.3% confidence
- positive“It was just not a fun experience.” was called negative with 92.8% confidence
- negative“And the accents are absolutely abysmal!” was called positive with 90.9% confidence
- positive“I did not have any problem with this item and would order it again if needed.” was called negative with 89.5% confidence
- negative“This is infuriating.” was called positive with 89.4% confidence
Limitations
- Trained on short single sentences; long reviews mixing praise and complaints get an averaged score.
- English only.
- Learned from 2,400 sentences, so rare words and new slang carry little signal.
- Labels are positive or negative; Sift calls scores near the middle neutral.
Data
Sentiment Labelled Sentences, UCI Machine Learning Repository, licensed CC BY 4.0. Kotzias, D., Denil, M., de Freitas, N. and Smyth, P. (2015). From Group to Individual Labels using Deep Features. KDD. Dataset page
The rest of the pipeline
Meaning comes from the all MiniLM L6 v2 sentence model run through ONNX, with a TF IDF and SVD fallback. Topics come from HDBSCAN, or KMeans picked by silhouette score, with near duplicate topics merged by centroid similarity and named by class based TF IDF. Spikes are weeks more than 2.5 standard deviations above the previous eight. Summaries are written by Claude when a key is set and every citation is checked against the reviews it was shown.