Kian Kyars PRO

I didn't understand the explanation

This technique is not a better lesson pilled at all. Waste of time when the model will just learn to do this anyways.

thanks!

makes sense

For the reasoning mid-training we used the ChatML template (so no system prompt), for SFT we use SmolLM3's final chat template.

ChatML does have a system prompt, can you please elaborate?

is it not possible to do on-policy distillation?

nice thoughts

tied embeddings were just used because llama uses it or was there an ablation?

Let's do this with Muon.