Add text preprocessing
#1
by
miraclemind
- opened
Adds input text pre-processing to improve the output when text has unusual punctuation. e.g. “This isn’t all for me, is it?” he’d asked converts to "This isn't all for me, is it?" he'd asked.
Based on the helpers that pre-process in the supertonic repo.
miraclemind
changed pull request status to
open
I ran into this one too. So when it was trying to say, "I'm a helpful assistant, I'll be glad to help" it sounded like "Imm a helpful assistant. ill be glad to help"
The Qwen3-VL-8B LLM was using fancy unicode apostrophes instead of simple ASCII ones.