A Watermark for Large Language Models
Kirchenbauer et al., 2023, A Watermark for Large Language Models (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
2 cards in “3 A more sophisticated watermark” · whole deck
3 A more sophisticated watermark
- How does the soft watermark change generation?Instead of forbidding red tokens, it adds a constant \(\delta\) to the logits of green-list tokens (a fraction \(\gamma\) of the vocabulary) before the softmax. Near-certain tokens stay likely even if red, while in high-entropy steps the bias strongly favours green tokens.
Green tokens get a logit bonus δ; the watermark is strong only where many tokens are plausible (invented logits) - Kirchenbauer et al., 2023, A Watermark for Large Language Models (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3
- What do the parameters \(\gamma\) and \(\delta\) control?\(\gamma\) is the green-list fraction of the vocabulary, \(\delta\) the logit bias (hardness). Smaller \(\gamma\) and larger \(\delta\) give a stronger watermark from fewer tokens but distort the text more (higher perplexity); \(\delta \to \infty\) recovers the hard rule.
- Kirchenbauer et al., 2023, A Watermark for Large Language Models (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3
Source
- John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers and Tom Goldstein. 2023. A Watermark for Large Language Models. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 17061–17084. https://proceedings.mlr.press/v202/kirchenbauer23a.html