DSL-13-SRMAP/Te-BERT_WOR

Telugu-BERT_WOR

Model Description

Telugu-BERT_WOR is a Telugu sentiment classification model built on Telugu-BERT (L3Cube-Telugu-BERT), a Transformer-based BERT model pretrained exclusively on Telugu text by the L3Cube Pune research group.

The base model is pretrained on Telugu OSCAR, Wikipedia, and news corpora using the Masked Language Modeling (MLM) objective. Unlike multilingual models such as mBERT or XLM-R, Telugu-BERT is specifically designed to capture Telugu vocabulary, syntax, and semantics, enabling better modeling of language-specific nuances.

The suffix WOR denotes Without Rationale supervision. This model is fine-tuned using only sentiment labels and serves as a label-only baseline for Telugu sentiment classification.

---

Pretraining Details

• Pretraining corpora:
  • Telugu OSCAR
  • Telugu Wikipedia
  • Telugu news data
• Training objective:
  • Masked Language Modeling (MLM)
• Language coverage: Telugu only

---

Training Data

• Fine-tuning dataset: Telugu-Dataset
• Task: Sentiment classification
• Supervision type: Label-only (no rationale supervision)

---

Intended Use

This model is intended for:

• Telugu sentiment classification
• Pure monolingual Telugu NLP tasks
• Benchmarking Telugu-specific vs. multilingual models
• Baseline comparisons in explainability and rationale-supervision studies

Telugu-BERT_WOR is particularly effective when sufficient Telugu-labeled data is available for fine-tuning.

---

Performance Characteristics

Due to its Telugu-only pretraining, Telugu-BERT excels at capturing fine-grained linguistic details, including idiomatic expressions and sentiment-bearing constructions that are often missed by multilingual models.

Strengths

• Strong understanding of Telugu vocabulary and syntax
• Superior modeling of nuanced sentiments and idiomatic expressions
• Well-suited for monolingual Telugu sentiment analysis

Limitations

• Not designed for cross-lingual transfer learning
• Requires sufficient Telugu-labeled data for optimal performance
• Lacks rationale supervision

---

Use as a Baseline

Telugu-BERT_WOR serves as a strong Telugu-specific baseline for:

• Comparing multilingual vs. language-specific architectures
• Evaluating the impact of rationale supervision (WOR vs. WR)
• High-quality sentiment analysis on pure Telugu text

---

References

• Joshi et al. (2022). Telugu-BERT. EMNLP.