The previous benchmarks in this series (Haskell, OCaml, Python, ReScript, Ruby, Elixir, Java, Elm) all used cloud API models — big, frontier-class LLMs served by Anthropic, OpenAI, and others. But what about a local model? Can a 14-billion-parameter model running on a single machine solve the same puzzles?
This post answers that question using qwen2.5-coder:14b via
Ollama, tested on AoC 2025 Day 1 in both Python and Haskell.
Following up on the Haskell benchmark, the OCaml benchmark, the Python benchmark, the ReScript benchmark, and the Ruby benchmark, I ran the same AoC 2025 Days 1–5 setup in Elixir.
Elixir is dynamic like Ruby/Python, but with its own ecosystem and idioms that models don't always handle cleanly.
Following up on the Haskell benchmark, the OCaml benchmark, the Python benchmark, the ReScript benchmark, the Ruby benchmark, the Elixir benchmark, and the Java benchmark, I ran the same AoC 2025 Days 1–5 setup in Elm.
Elm is the most niche language in this series. It's a pure functional language that compiles
to JavaScript, has no native CLI story, and sees relatively little use outside its frontend
niche. Each model received a pre-built scaffold — run.mjs, elm.json, and a
Day00.elm template — that compiles and runs Elm modules via Node.js. The question was
whether models would handle Elm's strict type system, lack of escape hatches, and unfamiliar
idioms (e.g. Debug.log for output, Platform.worker for headless programs).
The answer: every single one of them did.
Following up on the Haskell benchmark, the OCaml benchmark, the Python benchmark, the ReScript benchmark, the Ruby benchmark, and the Elixir benchmark, I ran the same AoC 2025 Days 1–5 setup in Java.
Following up on the Haskell benchmark, the OCaml benchmark, and the Python benchmark, I ran AoC 2025 Days 1–5 in ReScript — a typed functional language that compiles to JavaScript with a lean standard library, a distinct syntax, and very limited LLM training data.
This post covers three runs of the same benchmark, each adding a different intervention to see what helps models cope with an unfamiliar language:
Following up on the Haskell benchmark, the OCaml benchmark, the Python benchmark, and the ReScript benchmark, I ran the same AoC 2025 Days 1–5 puzzles in Ruby.
Same setup as before — the question is whether the leaderboard reshuffles when the target language changes.
Following up on the Haskell benchmark, I ran the same
orchestration setup on the same AoC 2025 Days 1–5 puzzles — this time requiring solutions in
OCaml. The methodology is identical: each model gets an isolated directory, a puzzle
description, and must write its final answer to ANSWER.txt. Wrong answer or no answer = ejection.
Following up on the Haskell benchmark and the OCaml benchmark, I ran the same orchestration setup on the same AoC 2025 Days 1–5 puzzles — this time in Python.
This is also the first run with full token usage and API cost tracking per part, which adds a new angle beyond raw wall-clock time.
I benchmarked 11 LLMs on Advent of Code 2025 Days 1–5, each solving independently in Haskell. The goal: see which models can reliably produce correct, working solutions — and how fast. One additional model (claude-haiku-4-5) was tested retroactively and has been added to the results.