From a Single-City Pilot to a Six-City Benchmark: Full Results from GeoSpatialBench#

Our earlier pilot test in Stockholm left an open question: if a model can correctly identify spatial relations, can it also translate that understanding into actual map edits? Building on those initial findings, we expanded our evaluation to six cities—Stockholm, Paris, Nairobi, Hong Kong, Wuhan, and Detroit. We constructed 8,400 benchmark tasks across 28 classic spatial relations, comparing nine models under three input modalities: exact coordinates, map images, and a combination of both.

The expanded results bring the gap between answering spatial questions and correctly modifying a map into sharp focus. We have compiled the full experimental design, key findings, and failure cases into an English post, complete with interactive charts that allow readers to explore the results by model, input modality, and spatial relation.