Robustness of Health Text Classification

Clinical NLP demos often look strong on clean symptom text. Real users type typos, synonyms, and incomplete phrases — and models can collapse without anyone noticing until deployment.

Health text systems have to stay useful under messy language. Comparing clean accuracy alone hides brittleness that matters for triage-style or decision-support tooling.

• gretelai/symptom_to_diagnosis — natural-language symptom descriptions mapped to diagnosis labels.
• Restricted to the top-10 diagnosis classes so classical, recurrent, and Transformer baselines compete on the same label set.
• Held-out evaluation under clean text plus controlled synonym, character-typo, and word-drop perturbations.

Train TF-IDF + logistic regression, a bidirectional LSTM, and fine-tuned DistilBERT on the same splits. Score Accuracy and Macro F1 on clean text, then re-evaluate under synonym swaps, typos, and dropped words. Add a typo-intensity stress curve to see when each model breaks.

• Designed the clean-vs-perturbed protocol so every model sees identical splits and noise settings.
• Implemented classical (TF-IDF + LR), recurrent (LSTM), and Transformer (DistilBERT) baselines end to end.
• Built a typo intensity sweep that plots degradation as character noise rises — not only a single “typo” flag.
• Reported Macro F1 alongside accuracy so majority-class wins cannot hide weak rare diagnoses.

• Multi-class diagnosis classification on top-10 labels for fair head-to-head comparison.
• Perturbations applied at evaluation time: synonym substitution, character-level typos, and random word deletion.
• DistilBERT fine-tuned as a sequence classifier; TF-IDF uses sparse bag-of-words features with logistic regression.
• Stress curve varies typo rate so robustness is a curve, not a single point estimate.

• LSTM underperformed with limited labeled data — sequence capacity without enough examples.
• Transformers led on clean text but degraded sharply under heavy character noise.
• Synonym perturbations can be semantically valid yet still shift lexical features classical models rely on.

• Clean: DistilBERT Macro F1 ≈ 0.86; TF-IDF accuracy ≈ 0.97 on the restricted label set.
• Typos: TF-IDF accuracy drops from ~0.97 → ~0.79; DistilBERT Macro F1 falls from ~0.86 → ~0.44 under the same noise.
• Stress curve: classical lexical features stay surprisingly stable at mild typo rates; Transformer gains erode once character noise intensifies.
• Takeaway: the best clean model is not automatically the most reliable under realistic input noise.

Robustness evaluation belongs next to accuracy. Clean leaderboards can hide failure modes that show up the moment users type imperfectly.