31 questions total.
8 questions linear algebra, 11 questions calculus, 12 questions physics. All questions are
undergraduate level. Problems were collected when LLMs were flagged to consistently get
those problems incorrect.
Parameters:
Temperature = 0
All other samplers disabled.
No system prefill or instruct formatting.
Zero-shot prompting only.
For local models:
Using kobold.cpp.
Using self-made GGUF quants.
All models were allotted 3 rerolls/question.
Example Question:
This question was removed from the dataset to be here.
Subject: Linear Algebra
$\text{(i) If } u \cdot (v \times w) = 4 \text{ then } w \cdot (u \times v) = -4$
$\text{(ii) } (2u + v) \times (u - 5v) = -11(u \times v)$
$\text{(iii) If } u \text{ is orthogonal to } v \text{ and } w \text{ then } u \text{ is also orthogonal to } ||w||v + ||v||w$
${\color{gray} \textit{Expected solution: (ii), (iii)}}$