Default system prompt can hinder thinking-mode performance
#31
by
ShengnanAn
- opened
I noticed that DS configured a default system prompt in the PR for sglang use_dpsk_v32_encoding in SGLang. But it appears that the official API does not apply this system prompt by default.
It is merely a simple and commonly-used system prompt, but my preliminary evaluation on GPQA-Diamond suggests it may somehow harm model performance under the thinking mode.
| Official API | SGLang Deployment | |
|---|---|---|
| w/ system prompt | 80.3 | 79.3 |
| w/o system prompt | 86.7 | 85.4 |
I'm not sure whether this is due to certain characteristics of GPQA-Diamond or a more widespread issue.