learn.ifs.name

LLaMA: Open and Efficient Foundation Language Models

Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971)

Subscriber deck — free during early access

3 cards in “5 Bias, Toxicity and Misinformation” · whole deck

5 Bias, Toxicity and Misinformation

  1. What did LLaMA’s toxicity evaluation show about model size?
    On RealToxicityPrompts, toxicity grew with model size, especially for “respectful” prompts asking for polite completions, consistent with earlier findings (Chinchilla vs Gopher being an exception).
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §5
  2. How did WinoGender reveal gender bias in LLaMA?
    Co-reference was much more accurate for “their/them/someone” than for “her/she” or “his/he”, and errors rose on “gotcha” sentences where the pronoun doesn’t match the occupation’s majority gender: the model uses occupational gender stereotypes rather than the sentence’s evidence.
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §5
  3. What did TruthfulQA show for LLaMA?
    LLaMA scored higher than GPT-3 on truthful and truthful-and-informative answers, but the rate of correct answers was still low, so the model is likely to hallucinate wrong answers.
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §5

Source

  • Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. In arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971