Create README.md

README.md

@@ -0,0 +1,206 @@
+---
+license: mit
+language:
+- en
+pipeline_tag: fill-mask
+---
+# Ettin: Open Suite of Paired Encoders and Decoders
+
+📄 [Paper](https://arxiv.org/abs/XXXX.XXXXX) | 🚀 [GitHub Repository](https://github.com/jhu-clsp/ettin-encoder-vs-decoder)
+
+This model is part of the Ettin suite - the first collection of paired encoder-only and decoder-only models trained with identical data, architecture, and training recipes. Ettin enables fair comparisons between encoder and decoder architectures across multiple scales, providing state-of-the-art performance for open-data models in their respective size categories.
+
+## Model Description
+
+Ettin models are designed to provide a foundation for comparing encoder-only and decoder-only architectures. Unlike previous comparisons that were limited by different training data, architectures, and recipes, Ettin models use:
+
+1. **Identical training data** - Same high-quality mixture across all models
+2. **Open Training Data** - Data is available now with batch-level training data for each of the 250+ checkpoints
+3. **Matched architectures** - Only differing in attention patterns (bidirectional vs causal) and training objectives (MLM vs CLM)
+4. **Consistent training recipe** - Three-phase training with 2T tokens
+5. **Multiple scales** - From 17M to 1B parameters
+
+This approach allows for true apples-to-apples comparisons between encoder and decoder models, revealing the inherent strengths of each architecture.
+
+## Training Data
+
+The training data is publicly available and split across different phases:
+
+- **Pre-training Data**: [jhu-clsp/ettin-pretraining-data](https://huggingface.co/datasets/jhu-clsp/ettin-pretraining-data) - 1.7T tokens of diverse data mixture
+- **Mid-training/Extension Data**: [jhu-clsp/ettin-extension-data](https://huggingface.co/datasets/jhu-clsp/ettin-extension-data) - 250B tokens of higher-quality filtered data
+- **Decay Phase Data**: [jhu-clsp/ettin-decay-data](https://huggingface.co/datasets/jhu-clsp/ettin-decay-data) - 100B tokens of premium data sources
+- **Training Data Order**: [jhu-clsp/ettin-data-order](https://huggingface.co/datasets/jhu-clsp/ettin-data-order) - Batch-level training order (columns: input_ids, step)
+
+## Model Family
+
+### Encoder Models
+| Model | Parameters | Description |
+|:------|:-----------|:------------|
+| [ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) | 17M | Extra extra small encoder model |
+| [ettin-encoder-32m](https://huggingface.co/jhu-clsp/ettin-encoder-32m) | 32M | Extra small encoder model |
+| [ettin-encoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-68m) | 68M | Small encoder model |
+| [ettin-encoder-150m](https://huggingface.co/jhu-clsp/ettin-encoder-150m) | 150M | Base encoder model |
+| [ettin-encoder-400m](https://huggingface.co/jhu-clsp/ettin-encoder-400m) | 400M | Large encoder model |
+| [ettin-encoder-1b](https://huggingface.co/jhu-clsp/ettin-encoder-1b) | 1B | Extra large encoder model |
+
+### Decoder Models
+| Model | Parameters | Description |
+|:------|:-----------|:------------|
+| [ettin-decoder-17m](https://huggingface.co/jhu-clsp/ettin-decoder-17m) | 17M | Extra extra small decoder model |
+| [ettin-decoder-32m](https://huggingface.co/jhu-clsp/ettin-decoder-32m) | 32M | Extra small decoder model |
+| [ettin-decoder-68m](https://huggingface.co/jhu-clsp/ettin-decoder-68m) | 68M | Small decoder model |
+| [ettin-decoder-150m](https://huggingface.co/jhu-clsp/ettin-decoder-150m) | 150M | Base decoder model |
+| [ettin-decoder-400m](https://huggingface.co/jhu-clsp/ettin-decoder-400m) | 400M | Large decoder model |
+| [ettin-decoder-1b](https://huggingface.co/jhu-clsp/ettin-decoder-1b) | 1B | Extra large decoder model |
+
+### Cross-Objective Models
+
+#### Encoders Trained from Decoders (Decoder → MLM)
+| Model | Parameters | Description |
+|:------|:-----------|:------------|
+| [ettin-encoder-from-decoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-17m) | 17M | Decoder continued trained with MLM |
+| [ettin-encoder-from-decoder-32m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-32m) | 32M | Decoder continued trained with MLM |
+| [ettin-encoder-from-decoder-68m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-68m) | 68M | Decoder continued trained with MLM |
+| [ettin-encoder-from-decoder-150m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-150m) | 150M | Decoder continued trained with MLM |
+| [ettin-encoder-from-decoder-400m](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-400m) | 400M | Decoder continued trained with MLM |
+| [ettin-encoder-from-decoder-1b](https://huggingface.co/jhu-clsp/ettin-encoder-from-decoder-1b) | 1B | Decoder continued trained with MLM |
+
+#### Decoders Trained from Encoders (Encoder → CLM)
+| Model | Parameters | Description |
+|:------|:-----------|:------------|
+| [ettin-decoder-from-encoder-17m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-17m) | 17M | Encoder continued trained with CLM |
+| [ettin-decoder-from-encoder-32m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-32m) | 32M | Encoder continued trained with CLM |
+| [ettin-decoder-from-encoder-68m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-68m) | 68M | Encoder continued trained with CLM |
+| [ettin-decoder-from-encoder-150m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-150m) | 150M | Encoder continued trained with CLM |
+| [ettin-decoder-from-encoder-400m](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-400m) | 400M | Encoder continued trained with CLM |
+| [ettin-decoder-from-encoder-1b](https://huggingface.co/jhu-clsp/ettin-decoder-from-encoder-1b) | 1B | Encoder continued trained with CLM |
+
+## Usage
+
+### Encoder Models (Classification/Retrieval/MLM)
+
+```python
+from transformers import AutoTokenizer, AutoModel, AutoModelForMaskedLM
+import torch
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/{MODEL_NAME}")
+model = AutoModel.from_pretrained("jhu-clsp/{MODEL_NAME}")
+
+# Example: Text classification/embeddings
+def encode_text(text):
+    inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
+    with torch.no_grad():
+        outputs = model(**inputs)
+        # Use [CLS] token representation
+        embeddings = outputs.last_hidden_state[:, 0, :]
+    return embeddings
+
+# Example: Masked Language Modeling
+mlm_model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/{MODEL_NAME}")
+
+def predict_masked_token(text):
+    # Text should contain [MASK] token
+    inputs = tokenizer(text, return_tensors="pt")
+    
+    with torch.no_grad():
+        outputs = mlm_model(**inputs)
+        predictions = outputs.logits
+    
+    # Get predictions for masked tokens
+    masked_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)
+    masked_predictions = predictions[masked_indices]
+    
+    # Get top predictions
+    top_predictions = torch.topk(masked_predictions, 5, dim=-1)
+    predicted_tokens = [tokenizer.decode(token_id) for token_id in top_predictions.indices[0]]
+    
+    return predicted_tokens
+
+# Example usage
+text = "This is a sample text for encoding."
+embeddings = encode_text(text)
+print(f"Embedding shape: {embeddings.shape}")
+
+# MLM example
+masked_text = "The capital of France is [MASK]."
+predictions = predict_masked_token(masked_text)
+print(f"Predictions: {predictions}")
+```
+
+### Decoder Models (Text Generation)
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+# Load model and tokenizer  
+tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/{MODEL_NAME}")
+model = AutoModelForCausalLM.from_pretrained("jhu-clsp/{MODEL_NAME}")
+
+# Set pad token if not already set
+if tokenizer.pad_token is None:
+    tokenizer.pad_token = tokenizer.eos_token
+
+# Generate text
+def generate_text(prompt, max_length=100):
+    inputs = tokenizer(prompt, return_tensors="pt")
+    
+    with torch.no_grad():
+        outputs = model.generate(
+            inputs.input_ids,
+            max_length=max_length,
+            num_return_sequences=1,
+            temperature=0.7,
+            do_sample=True,
+            pad_token_id=tokenizer.eos_token_id
+        )
+    
+    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    return generated_text
+
+# Example usage
+prompt = "The future of artificial intelligence is"
+generated = generate_text(prompt)
+print(generated)
+```
+
+## Training Details
+
+**Data:** High-quality mixture including DCLM, Dolma v1.7, scientific papers, code, and curated sources totaling 2T+ tokens
+
+**Architecture:** Transformer with RoPE, GLU activations, and prenorm layers
+
+**Training Phases:**
+- **Pre-training**: 1.7T tokens with diverse data mixture
+- **Mid-training**: 250B tokens with higher-quality filtered data and context extension to 8K
+- **Decay phase**: 100B tokens with premium data sources
+
+**Key Features:**
+- Context length: Up to 8K tokens
+- Vocabulary: 50,368 tokens (ModernBERT tokenizer)
+- Deep but efficient architectures following MobileLLM principles
+
+## Model Architecture
+
+| Parameter | 17M | 32M | 68M | 150M | 400M | 1B |
+|:----------|:----|:----|:----|:-----|:-----|:---|
+| Layers | 7 | 10 | 19 | 22 | 28 | 28 |
+| Hidden Size | 256 | 384 | 512 | 768 | 1024 | 1792 |
+| Intermediate Size | 384 | 576 | 768 | 1152 | 2624 | 3840 |
+| Attention Heads | 4 | 6 | 8 | 12 | 16 | 28 |
+
+## Citation
+
+If you use Ettin models in your research, please cite our work:
+
+```bibtex
+@misc{weller2025seqvsseq,
+      title={Seq vs Seq: An Open Suite of Paired Encoders and Decoders}, 
+      author={Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
+      year={2025},
+      eprint={XXXX.XXXXX},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/XXXX.XXXXX}, 
+}
+```