A small model takes the ACT Draft
Gemma 4 E4B, running on a laptop, scored a 32 composite on this year's ACT practice test.
The problem
Most claims about small open models come from benchmarks the model may have trained on. I wanted a result I could defend: a model small enough to run on a laptop, a practice test published in 2026, and scoring done with the test form's own raw-to-scale tables.
What I built
The model is Gemma 4 E4B, running through Ollama on an M1 MacBook with 16 GB of RAM. It has no internet access and no retrieval.
A builder script turns the 70-page test PDF into one packet per question. In the English section the underlined words are what the question asks about, and the PDF's text layer doesn't record underlines, so the builder finds them from the pixels. Figures and tables are cropped and sent to the model as images. About 23 items that the text layer couldn't represent are transcribed by hand, and the write-up lists which ones.
The runner sends one question per request. Math questions get a calculator tool that does exact arithmetic with fractions. A separate grader scores the answers offline. The answer key exists only in the grader, and a test fails if the key can reach the runner, the question packets or any prompt.
What happened
The model scored 27 in English, 35 in Math and 35 in Reading, for a composite of 32. Science is reported separately on this form, and it scored 30 there. It missed 14 scored questions. It rated its confidence between 90 and 100 on most answers, including most of the ones it got wrong. My first, simpler runners scored a composite of about 16, so most of the gain came from the harness.
The first grade came out as 29. My parser broke on LaTeX braces in the model's explanations and recorded 12 valid answers as malformed. I fixed the parser, re-read the saved replies without calling the model again, and kept the original results next to the corrected ones.
On my laptop the model generates about 11 tokens a second, so a full test takes hours. On a gaming GPU or hosted inference like Groq it would take minutes, which is fast enough for an agent to run the test, read the misses, change the harness and run it again. That's the next step.
Try it: you vs. Gemma
Five math questions from the test the model took. Pick an answer to see what Gemma chose, how confident it was and why.
Question 4. It takes Collin 24 minutes to walk to school in the morning. What fraction of his 24-hour day is spent walking to school in the morning?
The answer is G. Gemma answered G, 100% confident: “The total minutes in a 24-hour day is 1440, so the fraction is 24/1440, which simplifies to 1/60.”
Question 3. Given b = 40 and c = −16, b + c is equal to the product of −4 and what number?
The answer is B. Gemma answered B, 100% confident: “First, calculate b + c: 40 + (-16) = 24. Setting this equal to the product of -4 and the unknown number (x) gives 24 = -4x, which solves to x = -6.”
Question 14. The lengths of corresponding sides of 2 similar right triangles are in the ratio 4:5. The hypotenuse of the smaller triangle is 20 inches long. How many inches long is the hypotenuse of the larger triangle?
The answer is J. Gemma answered J, 100% confident: “Since the triangles are similar, the ratio of corresponding sides (including the hypotenuses) is 4:5. Setting up the proportion (20/H_L = 4/5) and solving for H_L yields 25 inches.”
Question 5. A certain triangle has interior angle measures of (6x)°, (2x)°, and x°. What is the value of x?
The answer is C. Gemma answered C, 100% confident: “The sum of the interior angles of a triangle is 180 degrees. Setting up the equation (6x + 2x + x = 180) and solving for x gives 9x = 180, so x = 20.”
Question 9. Given the function f(x) = 4x² − 14x + 12, which of the following expressions is equivalent to f(x)?
The answer is B. Gemma answered C, 100% confident: “Factoring the given quadratic f(x) = 4x² − 14x + 12 by first dividing by 2 yields 2(2x² − 7x + 6), and factoring the trinomial gives 2(2x − 3)(x − 2), which matches (2x − 4)(2x + 3) if we check the product.”
Its factoring is right: 2(2x − 3)(x − 2) is choice B. Then it picked C, at 100% confidence. This is one of the two math questions it missed on the whole test, and its confidence rating gave no warning.