Notes

I took the eleven top models on the Vellum leaderboard and asked them 46 spatial questions with machine-computed answers: GeoJSON axis order, great circle distance and bearing, UTM zones including the Norway and Svalbard exceptions, RFC 7946 winding order, and point-in-polygon on a concave ring. One run, temperature 0, through OpenRouter.

The correlation between general capability and spatial score is r = 0.19.

Model	Score	Answered	Of answered	Mean tokens	Vellum
GPT-5.6 Sol	97.6	97.6	100.0	711	47.2
Claude Opus 5	95.7	100.0	95.7	532	64.7
Claude Fable 5	91.3	100.0	91.3	415	—
Claude Sonnet 5	91.3	91.3	100.0	844	57.4
Claude Opus 4.8	69.6	100.0	69.6	37	57.9
DeepSeek V4 Flash	69.6	100.0	69.6	166	51.6
GLM 5.2	63.0	69.6	90.6	1530	54.7
DeepSeek V4 Pro	56.5	60.9	92.9	2104	48.2
Kimi K3	45.7	67.4	67.7	1715	56.0
Kimi K2.6	28.3	30.4	92.9	3128	54.0
Gemini 3.1 Pro	26.1	100.0	26.1	195	—

The lowest-ranked model on general capability came first. The highest-ranked came second. Below that the ordering scrambles entirely.

Three columns matter more than the score. Answered is the share that produced anything within a 4,000-token budget. Of answered is accuracy on those. Mean tokens shows why they differ: Kimi K2.6 spends 3,128 tokens per question and answers 30% of them, but is right 93% of the time when it finishes. It is not bad at geodesy, it runs out of room.

Reporting those as one number is a trap I fell into first time round. Capped at 200 tokens, ten of eleven models scored zero on distance. They had not failed, they had been truncated, and an empty response was being scored as a wrong answer. The resulting leaderboard ranked models by terseness and looked entirely plausible.

Axis order is near-solved: nine of eleven score 100. Bearing is not: four score zero. Nobody should be computing geodesy in a language model anyway, but plenty of agent frameworks now do. It is also why I assess reading code.

46 tasks, one run, $5.20. Ground truth computed, not written. Single run, so no variance estimate.

This site used to live at jwilliams.science. It now lives at jameswil.com, because a domain that some readers cannot open is not a domain.

The reason is the TLD, not the site. .science was cheap at launch and got used accordingly, so it carries a poor reputation with the blocklists that corporate proxies, university filters and mail gateways buy in wholesale. Those lists work at the TLD level. Nothing on my site was ever assessed. Readers on several Canadian networks simply got a block page, and a CV link that returns a block page is worse than no link.

Mail is the sharper edge. A .science address in a From header is enough to score a message into a junk folder before anything else about it is read, which matters when the message is an application.

The lesson is dull and worth stating: a novelty TLD is not a saving, it is a deliverability decision made once and paid for repeatedly. Old links redirect, and the five references to the old domain still sitting in project pages are next.

A first-year student can now produce a working CRUD app in an afternoon. Assessing the artefact stopped being informative the moment that became true, because the artefact no longer evidences anything about the student.

What has not become cheap is diagnosis. Generated code fails in ordinary ways: a fetch that never handles the rejected promise, an auth check enforced in the client and nowhere else, an N+1 query that is invisible at 20 rows and fatal at 20,000. A student who cannot read the code cannot see any of it, and the model will not volunteer it. It is the same failure I measure elsewhere: confident output, no signal about whether it is right.

So the marks moved. On CMU422 and CMU529 the submission is still an application, but it arrives as a repository: commit history, a README that justifies the dependencies and the architecture, and a short written reflection on what broke, what the student changed their mind about, and what they still do not understand. A repo you cannot explain is a repo you did not build.

This is not a restriction on tool use, and policing that would fail anyway. It is a change in what counts as the work. Reading code critically was always the harder skill and we could previously get away with assuming it followed from writing enough of it. It does not follow any more, so it has to be taught directly.

Two POIs sit 30 m apart with similar names. Same shop, or two shops? Every conflation pipeline answers that question and almost none of them publish the answer.

AnythingPOI, which fuses OpenStreetMap and Overture Maps into 22.7M points across six countries, uses three gates, all of which must hold: 50 m Haversine, agreement on Tier-1 category, and Jaro-Winkler name similarity of 0.85 on normalised names. Candidates are blocked first by H3 at resolution 11, which is the same parameter choice in a different costume. Across six countries the gates match 1.4% to 3.2% of records.

The number looks like failure. Loosening it is trivial: take the gate to 150 m, drop the category check, and the merged count climbs to something that reads well in a table. What you have actually done is fuse the barber and the newsagent that share a parade and a family name, and you cannot tell which of your merges are real.

A cross-source match is worth something in the confidence score precisely because it is rare. It means two independently built datasets agree. Inflate the rate and you destroy the signal that made the field worth adding.

Recall is what this costs. Transliterated names go unmatched, and so does a shopping centre whose centroid sits 200 m from its entrance. That is a deliberate trade, and it belongs in the documentation rather than in a footnote.

A table of metres cannot say what a neighbourhood is shaped like. The k-ring that overshoots 400 m reaches furthest along six axes, so a shop 390 m away is inside or outside depending on its bearing. Inline SVG draws that at the size of the text, inside the sentence making the claim. Resolution 9 gives you for a 400 m walk, and resolution 11 gives you . Same dashed 400 m target, same 15 px, and only the second is the neighbourhood you asked for. Overshoot falls from 514% to nothing.

At 64 px there is room for the boundary. Real H3 disks at 51.48 degrees north, each scaled to its own footprint:

Res	Footprint	Reach	Overshoot	Cells
7		2,457 m	+514%	7
8		922 m	+130%	7
9		702 m	+75%	19
10		527 m	+32%	61
11		401 m	+0%	217
12		414 m	+3%	1,519

Read the dashed circle, not the outline. At resolution 7 the target is a speck inside the one cell you were given. By 11 it nearly fills a footprint that has stopped being a hexagon, which is what 217 cells bought.

Each is under a kilobyte, needs no charting library, and has fixed width and height, so it cannot shift the layout. currentColor serves both themes from one file. Best of all, the script that computes the table emits the glyphs, so the picture cannot drift from the numbers beside it.

A figure does not have to be a page wide to be a figure.

Ask for a 400 m walkable neighbourhood, the standard five-minute walk, and H3 hands you something else. At resolution 9 you get 702 m. That is not a rounding error, it is a neighbourhood 76% too big.

The cause is that k is an integer and the grid step is fixed. A disk grows in whole rings, so its radius is wherever the grid happens to land, never the number you asked for. Coarse resolutions land badly.

Grown rather than estimated, at 51.48°N:

k = 1
while disk_reach(cell, k) < 400:
    k += 1

Resolution 9 reaches 702 m for 19 cells, resolution 10 reaches 527 m for 61, and only resolution 11, the resolution AnythingPOI blocks on, lands at 401 m, for 217. Measuring the disk a second way, as the radius of the circle of equal area, agrees: 769 m, 521 m, 371 m.

So each step costs roughly 3.5× the cells, and what it buys is boundary accuracy. It is not buying finer cells, which mostly encode precision your input data does not have.

Pick the resolution from the error you will accept on the boundary, then check the cell count you can afford. Choosing on cell size alone is how a 400 m study quietly becomes a 700 m one.

Cells are not equal-area, so these are numbers for 51°N. Rerun them at yours.