Metfive fresh 2024 questions, 400 draws each
The test at a glance
- What it tests
- Accuracy against reality: whether a model's estimate can stand in for part of a survey.
- Data
- General Social Survey 2024 (Release 3a); five questions; 400 draws per question of real respondents per party-and-age group.
- Held out
- The full 2024 survey's group shares, used only as the truth.
- Compared against
- 75 and 100 real respondents per group, alone.
- Bar
- The model counted as 50 respondents plus 50 real ones: lower mean six-cell error than 75 real alone, pooled over five questions.
- Result
- 3.15 against 3.46: met. 100 real alone came closer (3.01).
A model's estimate stands in for part of a survey. Counted as 50 respondents and combined with 50 real ones per party-and-age group, Gemini 3.8 Flash's estimates split five fresh 2024 questions with a mean error of 3.15 points, closer than 75 real respondents per group alone (3.46). Registered before the model was asked; 400 draws per question. 100 real respondents alone came closer still (3.01), and on police use of force the model added nothing.
Protocol, result and scorer