Select the maneuver.
Choose the requested local action among feasible alternatives.
“Turn left at the junction.”
Can a driving VLA
follow your
instructions?
Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?
Keep the scene fixed. Change the instruction.
Test whether the model completes both feasible alternatives.
01 / PAIRED COUNTERFACTUAL UNITExplore the pair
Two rollouts. One shared initial state.Only language changes ↗
1THU2SEU3SJTU4Adelaide5BISTU6Amazon7BJTU8WHU9HUST10USTC11NIO12UT Austin13CAS14HiThink15HKUST(GZ)16AMD17HKU18EPFL19Li Auto
5evaluation tracks
4instruction scopes
2matched alternatives
Driving quality and
language control are measured separately.
THE BENCHMARK IDEADriving quality.
Driving quality.
Language control.
Measured together.
Language2Drive turns instruction following into a reproducible paired evaluation: matched conditions, mutually exclusive requests, and native execution evidence.
Explore the interactive framework →2×
feasible alternatives
in one matched world
Language2Drive in motionSame starting scene.
Same starting scene.
Two feasible instructions.
A 2:13 introduction to Language2Drive: paired counterfactual testing, four instruction scopes, CARLA task examples, benchmark coverage and main findings.
01 / BENCHMARK OVERVIEWFrom a maneuver
From a maneuver
to a driving program.
Four scopes ask progressively richer questions about instruction execution. Five tracks test those questions in complementary environments.
Maintain the constraint.
Preserve a requested behavior over an interval, rather than one instant.
“Keep a safe following distance.”
Resolve the destination.
Ground a physical place or reference in the observed driving scene.
“Drive to the campus gate.”
Complete the sequence.
Track stages, references and execution order across a longer drive.
“Reach the gate, then the visitor lot.”
Purpose-designed & real-world
Same sceneSame seedSame initial state
Paired Counterfactual Unit
One scene. Two possible futures.
A
B
Paired trajectories
Driving competence ≠ language controllability
Three open-loop backends01 / Shared world
See the paired evidence →Hold the non-language inputs fixed.
A pair starts from the same scene, seed and initial state. Each instruction requests a different, feasible outcome.
View the original author framework ↗
Five tracks. Complementary evidence.
Read the evaluation protocol ↗01CLOSED LOOPA / B / C / D64 paper PCUs↗02CLOSED LOOPA / B30 PCUs↗03OPEN LOOP3 s / 12 points75 release PCUs↗04OPEN LOOP4 s / 8 points38 PCUs↗05OPEN LOOP5 s / 20 points241 scored PCUs↗
CARLA-L2D
Purpose-designed driving simulation · six maps
RealSim
AlpaSim · 23 reconstructed real-world scenes
nuScenes
Frozen test release · 15 dev / 60 held-out
NAVSIM
19 recordings · four instruction wordings
WOD-E2E
STOP/GO and speed-choice contracts · P0
Four instruction scopes × five evaluation tracks. Cohort sizes follow the paper’s evaluation protocol.
CARLA-LANGUAGE2DRIVE / EXPLORE THE SIMULATORSix worlds.
Six worlds.
Built for language.
Explore our CARLA-based simulator through six custom maps. Follow the road network, open a landmark, and watch the scene from above or at road level.
CARLA-Language2Drive connects purpose-built driving environments with scene-grounded instructions: named streets, port destinations, parking bays and multi-stage programs. The atlas brings their geometry and captured scenes together.
6CUSTOM MAPS54KEY LOCATIONS24EXPLORATION CLIPS
INTERACTIVE MAP ATLAS
Loading the six worlds…
Select a map to explore its key locations.
1× · drag to explore
Key locationCamera route
CARLA WORLD · X → / Y ↓Key location
The exploration video could not be loaded.
02 / THE PAIRED COUNTERFACTUAL UNITChange the words.
Change the words.
Hold the world fixed.
A PCU pairs two feasible, mutually exclusive instructions under matched non-language inputs. Both arms must succeed to demonstrate paired instruction following.
SHARED INPUTSame scene
Same scene
Same seed
Same initial state
One public observation envelope.
ARM A
Instruction A
Model → actual trajectory A
ARM B
Instruction B
Same model → actual trajectory B
PAIRED VERIFICATIONBoth alternatives
Both alternatives
must complete.
Requested contract ✓
Opposite contract rejected ✓
Full execution requirements ✓
Individual → paired
Strict single-instruction success versus successful execution of both alternatives.
Graded completion
Measure compliance and the weaker arm, beyond a single binary verdict.
Stable across wording
Equivalent requests should preserve successful behavior across rewrites.
What reaches the model?
Observed cameras, public ego history, the active instruction and a trajectory output specification. Evaluator targets, arm identities, opposing instructions and semantic contracts stay outside the model request. CARLA standardized evaluation also withholds route identity and simulator seed.
Inspect the public adapter contract ↗03 / MAIN RESULTS & LEADERBOARDMeasure instruction following,
Measure instruction following,
one pair at a time.
Training-free baselines across five evaluation tracks. Explore driving quality, instruction success and paired compliance for each model.
Click a metric heading to explore its meaning. Click a model name to read its paper.Explore the metrics ↓
Read driving quality and instruction following together.
Native driving metrics, strict paired success and graded compliance offer complementary views of each model.
Metric definitions ↗Leaderboard protocol ↗All reported baselines are training-free on Language2Drive.
04 / THE CASE LIBRARYWatch both sides
Watch both sides
of the instruction.
Explore actual model results and benchmark targets across five backends. Switch scenes, models and task families; compare both sides of each instruction pair.
Model result
What the model actually produced: executed drives or predicted trajectories.
Benchmark target
What the instruction asks for: reference trajectories or reference-controller examples.
Model resultPaired evidence
Video unavailable. Restore the local media bundle using the showcase build instructions.
ALPAMAYO 1.5 / RECORDED MODEL OUTPUT
Chain of thought, alongside the drive.
Follow both instruction arms as the video plays. Open a transcript to explore every recorded output.
Loading the evidence library…
Camera view
Instruction A · requested goal
Instruction B · requested goal
Solid: model result · dashed: benchmark target · blue: A · orange: B
Case details & download
Find another perspective.
No cases match these filters. Try another model or reset the filters.
129 clips · five backends · diverse scenes, task families and model comparisons.
View the manuscript visualizations +
BENCHMARK CONSTRUCTIONGround the scene.
Ground the scene.
Verify the alternatives.
Source curation, geometric grounding, paired specification, compilation and validation produce the frozen closed-loop cohorts.
05 / EVALUATE YOUR MODELOne adapter.
One adapter.
All five tracks.
Keep your model in its own environment. The SDK handles public requests, trajectory validation, native exports and track-specific scoring.
Install the SDK.
Install from PyPI, the source checkout or a release wheel. Choose Open-loop, CARLA or RealSim dependencies to match your evaluation.
Implement predict(request).
Return ego-frame XY waypoints on the requested time grid. Your factory creates independent state for each rollout.
Configure once. Evaluate together.
Generate one resource configuration. Launch CARLA, RealSim and Open-loop through one adapter; each backend saves native evidence and scores.
python -m pip install language2drive
language2drive init --workspace ./l2d-workspace
cd l2d-workspace
# Probe official sources; provision selected environments
language2drive setup --tracks openloop
# Optional resource download / simulator provisioning
language2drive setup --tracks all --runtime --downloadSet up datasets and simulator environments. Installation details ↗
Training-free today. Official dataset and model next.
Every reported baseline uses an existing checkpoint without Language2Drive training or fine-tuning. We plan to release the official versioned dataset and an official SOTA model with its recipe. Follow the dataset release ↗
An open benchmark gets better with more modelsWhere does
Where does
your model stand?
Evaluate once. Share the protocol, native scores and execution coverage. Share your model with the research community.
Prepare a result submission ↗06 / CITATIONBuild on
Build on
Language2Drive.
If Language2Drive supports your research, please cite our paper.
Read the manuscript ↗@misc{language2drive,
title = {Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?},
author = {You, Zihan and
Fan, Zehua and
Lyu, Wenqi and
Wang, Aoqi and
Wang, Ni and
Yang, Yifei and
Zhao, Linge and
Li, Huan and
Ang, Sining and
Yang, Yuguang and
Chen, Jueyu and
Li, Xiangyu and
Wei, Qianshan and
Pei, Jieyuan and
Wang, Yuxin and
Chen, Zhiwen and
Li, Jiaqi and
Song, Wenxuan and
Qiao, Yanyuan and
Zhu, Benjin and
Li, Haoang and
Wang, Yan},
year = {2026},
eprint = {2610.16634},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2610.16634}
}