OpenAI’s newly released Astra model has achieved a generational leap across multiple dimensions, including complex reasoning and computer use. Even more intriguingly, someone recently used it to build a virtual Manhattan neighborhood. This sparked our curiosity: just how far have GPT-6 Astra’s spatial capabilities come? When AI truly steps into "space," how much can it perceive, and how deeply does it understand? Recently, the creator of SpatialBench also remarked on social media that Astra had "saturated" their spatial reasoning benchmark—making us even more eager to see how it performs on real-world urban maps.

To find out, we fed spatial urban data from Stockholm to GPT-6 Astra in two formats—exact coordinates and annotated map images—asking it to answer spatial relation questions and subsequently move, rotate, or modify geographic features as instructed. We also compared its performance against GPT-5.6 Sol and Kimi K3. The results show that GPT-6 Astra outperforms both GPT-5.6 Sol and Kimi K3 across multiple task categories, with an especially pronounced lead in editing tasks. However, input modality still significantly impacts performance: given exact coordinates, the model can reliably judge spatial relations and execute geometric modifications; when switched to map images, accuracy drops across both question-answering (QA) and editing tasks.

1. Overall Performance: Strong on GeoJSON, Improving on Map Images#

Our test suite covers 28 classic spatial relations. The tasks include both spatial relation queries—such as “Is Feature 1 located east of Feature 2?” or “Do the two buildings overlap?”—and spatial editing instructions, such as “Move the object in the specified direction while maintaining a given distance.”

Categorized by input modality and task type, the evaluation is divided into four tracks: Coordinate QA, Coordinate Editing, Map Image QA, and Map Image Editing, with 140 tasks per track, totaling 560 recorded trials for GPT-6 Astra.

QA tasks are scored based on answer correctness; for editing tasks, we execute the modifications generated by the model and verify whether the resulting geometry truly satisfies the spatial constraints specified in the prompt.

Task Category	GPT-6 Astra	GPT-5.6 Sol	Kimi K3
Coordinate QA	95.7%	75.7%	85.0%
Coordinate Editing	87.1%	62.9%	62.9%
Map Image QA	65.0%	55.7%	49.4%*
Map Image Editing	48.6%	26.4%	33.3%*

Note: Accuracy differences are expressed in percentage points. All tracks for Astra and Sol, as well as the coordinate tracks for Kimi K3, consist of 140 tasks each. * Kimi K3’s image evaluation is still in progress; QA and Editing scores reflect 85 and 60 valid completed samples, respectively.

The four tracks reveal a consistent pattern across models.

In Coordinate QA, GPT-6 Astra achieves an accuracy of 95.7% compared to 75.7% for GPT-5.6 Sol, a lead of 20.0 percentage points; in Map Image QA, the scores are 65.0% and 55.7%, respectively, narrowing the gap to 9.3 percentage points.

The divergence is even more striking in editing tasks: Coordinate Editing accuracy jumps from 62.9% to 87.1% (+24.3 percentage points), while Map Image Editing climbs from 26.4% to 48.6% (+22.1 percentage points).

Including Kimi K3 makes the three-way comparison even clearer. In Coordinate QA, Kimi K3 achieves 85.0%, outperforming Sol (75.7%) but trailing Astra (95.7%); in Coordinate Editing, Kimi K3 ties with Sol at 62.9%, compared to Astra’s 87.1%.

Across the shared subset of completed image samples, QA accuracy for Astra, Sol, and Kimi K3 stands at 65.9%, 52.9%, and 49.4%, respectively; meanwhile, Editing accuracy reaches 36.7%, 26.7%, and 33.3%, placing Kimi K3 much closer to Astra on visual editing.

Looking at GPT-6 Astra’s own performance, the gap between input modalities is equally pronounced. While GeoJSON Coordinate QA and Editing reach 95.7% and 87.1%, Map Image QA and Editing sit at just 65.0% and 48.6%.

This indicates that under current testing conditions, models are already highly capable of processing structured geometric data, whereas visual spatial comprehension and precise manipulation remain significantly more challenging.

2. Three Visual QA Tasks: A Closer Look at Model Responses#

For the following three tasks, Astra, Sol, and Kimi K3 all provided valid responses; each model was given the exact same map image and prompt.

First, let’s examine two easily confused topological relations: do two regions partially overlap, or do they occupy the exact same space?

Map test cases for partial overlap vs. exact equality — Figure 1 | Original input maps. Left: Partial overlap (STO-L1F-T08-002); Right: Exact equality (STO-L1F-T01-003)

Question 1: Partial Overlap, or Just Touching Boundaries?#

In the left panel of Figure 1, the red and blue regions share a common interior area, making the ground-truth answer “overlaps”.

Both Astra and Sol answered “overlaps” (correct), whereas Kimi K3 answered “touches” (incorrect).

The distinction is straightforward: overlapping features share an interior area, whereas touching features only meet at their boundaries.

Question 2: Exact Equality, Yet Three Different Answers#

In the right panel of Figure 1, the two polygons occupy the exact same footprint, so the ground-truth answer is “equals”.

Astra answered “equals”, Sol answered “contains”, and Kimi K3 answered “disjoint”—making Astra the only model to get it right.

Because the polygon drawn second visually masks the first, reading the map correctly requires noticing that both labels point to the exact same feature.

Question 3: Zooming Out to the City Scale#

This task restricts choices to the four cardinal directions (North, South, East, West) and asks for the direction of the blue point (F02) relative to the red point (F01).

Because the two points do not lie on the same horizontal line, the key is to apply the four-way directional partitioning rule and determine which cardinal sector F02 falls into.

Four-way cardinal direction judgment on an urban map — Figure 2 | Original input map for the four-way directional task (STO-L1F-D01-005), with a zoomed-in inset in the corner

The ground-truth answer is “East (E)”. Both Astra and Kimi K3 correctly answered “East”, while Sol answered “South (S)”. Here, Kimi K3 matched Astra’s correct judgment.

Taken together, these three examples highlight the nuances in our benchmark: Astra answered all three correctly, while Sol and Kimi K3 each got one right—but succeeded on different questions.

3. From “Judging Relations” to “Modifying Space”: Why Is Editing So Hard?#

While both QA and editing tasks revolve around spatial relations, they place fundamentally different demands on a model.

QA only requires determining whether an existing relation holds; editing requires actively altering an object’s position or shape so that the modified geometry satisfies one or more spatial constraints.

In other words, a model must not only know the right answer in principle, but also generate executable, verifiable geometric outputs.

Having seen how the models judge relations in the three QA examples above, let’s examine how they actually modify spatial objects.

We selected two editing tasks—“partial overlap” and “move east”—which mirror the spatial relations tested in the QA section above, applied to dedicated editing scenarios. All three models received identical prompts for each task, and their outputs were evaluated based on the saved geometries and our verification logs.

Editing Case 1: Moving Two Regions to Partially Overlap#

The task requires moving F0 so that it partially overlaps with the fixed feature F1.

Both Astra and Kimi K3 shifted F0 onto the edge of F1—creating a configuration where part of F0 lies inside F1 and part lies outside—successfully passing the constraint check. Sol’s output remained near the original position without overlapping F1, failing the test.

Notably, the two passing models placed F0 in different locations, illustrating that spatial editing tasks often admit multiple valid solutions.

Original map before partial overlap editing — Figure 3a | Original map before editing. F0 is the target object to be moved, and F1 is the fixed reference object; the task is to make them partially overlap

Comparison of partial overlap editing results across three models — Figure 3b | Partial overlap editing (STO-ED-T08-001). Plotted from actual geometric coordinates; all three panels share the same spatial extent and scale

Editing Case 2: Instructed to Move East—Did It Actually Go the Right Way?#

This task requires moving F0 eastward by 8 to 200 meters while maintaining a minimum distance of at least 8 meters from F1.

According to our directional and distance constraint evaluation logs:

Astra: Displacement classified as East, distance of 69.93 m — Pass.
Kimi K3: Displacement classified as East, distance of 89.84 m — Pass.
Sol: Displacement classified as North, distance of 15.71 m (failed directional constraint) — Fail.

Crucially, this evaluation measures the actual geometric output generated by the model, not merely the direction it claims to move in its text response.

Original map before eastward movement editing — Figure 4a | Original map before editing. F0 is the target object to be moved, and F1 is the fixed reference object; the task is to move F0 east while maintaining the specified distance

Comparison of eastward movement editing results across three models — Figure 4b | Eastward movement editing (STO-ED-D01-005). Plotted from actual geometric coordinates, with North at the top and East to the right

In both editing cases, Astra and Kimi K3 passed while Sol fell short. Pairing these results with the earlier QA examples offers a side-by-side view of how models perform at “judging relations” versus “modifying objects on command.”

The next two cases from Astra further illustrate how editing outcomes are shaped by boundary precision and distance tolerances.

Editing Case 3: Strict Topological Equality vs. Interval Distance#

Editing cases for topological equality and Hausdorff distance — Figure 5 | Two real-world spatial editing cases. Left: Topological `equals` requires strict boundary alignment; Right: Hausdorff distance only needs to fall within a specified interval, allowing multiple valid positions

Figure 5 (Left): Strict Topological Equality (equals)

This task requires snapping building F0 to F1 so that the two are topologically equal.

In GeoJSON mode, Astra directly returns:

{"op":"snap","to":"F1"}

Upon execution, the output cleanly passes the equals check.

When switched to the map image modality, the polygon generated by the model is visually very close to the target, but its pixel Hausdorff distance is 6.54, exceeding the 4-pixel tolerance; once projected back to real-world coordinates, the two polygons fail to achieve topological equality.

For strict relations like equals, even a tiny boundary offset causes the output to fail the constraint check.

Editing benchmarks thus directly expose subtle geometric errors that are otherwise difficult to spot with the naked eye.

Figure 5 (Right): Hausdorff Distance Interval Constraint

This case requires the Hausdorff distance between F0 and F1 to fall within a specified interval.

In map image mode, Astra’s generated geometry deviates from the reference geometry by 12.66 pixels, but once converted back to real-world coordinates, the actual Hausdorff distance is approximately 37.47 meters—well within the required range—so it is marked as correct.

For interval-based distance tasks, the model’s output does not need to align identically with a single reference geometry; as long as the final result satisfies the given spatial constraints, it counts as a valid solution.

Conclusion: What This Pilot Reveals About GPT-6#

These pilot results bring the boundaries of GPT-6 Astra’s spatial capabilities into much sharper focus.

Given structured GeoJSON inputs, Astra can already perform spatial relation judgments and geometric edits with high reliability; compared to GPT-5.6 Sol, its improvements are evident not only in QA accuracy, but also in substantial gains across both editing tracks.

With Kimi K3 added to the comparison, Astra still holds a clear lead on coordinate-based tasks; meanwhile, across the shared subset of completed image samples, Kimi K3 outperforms Sol in visual editing and comes much closer to Astra.

However, higher overall scores do not mean all spatial relations have been solved equally well.

When broken down by relation category, GeoJSON QA reaches 100% across topological, directional, and metric relations alike—yet directional accuracy in GeoJSON Editing drops to just 40.0%. Similarly, in Map Image QA, the model answers only 8 out of 40 metric relation tasks correctly.

In other words, recognizing a directional relationship does not automatically mean a model can compute the exact translation vector or rotation angle required to satisfy a directional edit; likewise, visually identifying an object on a map does not mean it can recover precise metric scale from an image lacking a scale bar.

It is worth emphasizing that this pilot test uses a subset of data from Stockholm—5 tasks per spatial relation and 140 tasks per track—reflecting model performance solely under the current sample set and evaluation protocol.

We will soon release our full benchmark featuring expanded task types and multi-city samples to compare different model families under a unified protocol. Stay tuned!