I donāt think itās nearly as impressive as it is hyped to be:
They only ever evaluate on 100 prompts, and the error bars on their headline metrics are +-4-5.
The headline result (65%) only applies to Qwen3-0.6B chat; for GPT-4o, itās 11%, and for Llama-2, 17%.
Itās fine-tuned on 400 ShareGPT prompts, and I suspect most of the gains are due to learning the ShareGPT prompt format. Indeed, if fine-tuned on a different dataset, it only achieves 11%.
The paper also has several misleading claims, like calling it āblack-boxā (needs the tokenizer) or ādata-freeā (only during pretraining, but see their own appendix).
Imagine if you could take the AI marketing slop, or political smear campaigns, texts, and Images, and see the misanthropic sludge that was used as a prompt to generate it.
I donāt think itās nearly as impressive as it is hyped to be:
They only ever evaluate on 100 prompts, and the error bars on their headline metrics are +-4-5. The headline result (65%) only applies to Qwen3-0.6B chat; for GPT-4o, itās 11%, and for Llama-2, 17%. Itās fine-tuned on 400 ShareGPT prompts, and I suspect most of the gains are due to learning the ShareGPT prompt format. Indeed, if fine-tuned on a different dataset, it only achieves 11%.
The paper also has several misleading claims, like calling it āblack-boxā (needs the tokenizer) or ādata-freeā (only during pretraining, but see their own appendix).
Imagine if you could take the AI marketing slop, or political smear campaigns, texts, and Images, and see the misanthropic sludge that was used as a prompt to generate it.
Or what if you could take something not slopped and get back the prompt that would most closely result in it
There is that. https://huggingface.co/spaces/pharmapsychotic/CLIP-Interroga...
Sadly we donāt really have that for T5XXL and other yuge prompt encoder based models
That's just a summary.