A Watermark for Large Language Models

6 Experiments

1. How well did the soft watermark work with OPT-1.3B on C4 news prompts?
   With \(\gamma = 0.5\), \(\delta = 2\) and \(z = 4\) on about 200 generated tokens, 98.4% of watermarked texts were detected with multinomial sampling (99.6% with 4-way beam search), and essentially no human text was flagged.
   • Kirchenbauer et al., 2023, A Watermark for Large Language Models (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §6
2. How does beam search interact with the soft watermark?
   It “irons in” the watermark: searching among likely continuations finds ones dense in green tokens, giving much higher z-scores at little perplexity cost; with 8 beams and \(\delta = 2\), average \(z > 5\) was reached from about 35 tokens.
   • Kirchenbauer et al., 2023, A Watermark for Large Language Models (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §6
3. Why can the detector stay the same when the watermarking rule changes?
   The z-statistic depends only on \(\gamma\) and the hash function, not on \(\delta\) or how green tokens are favoured, so \(\delta\) can vary by context (e.g. prose vs code) or the generator can change without updating the detector.
   • Kirchenbauer et al., 2023, A Watermark for Large Language Models (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §6