Co-Design Bench

A benchmark for multimodal coding agents

Co-Design Bench.

Multimodal Collaborative Coding Evaluation
for Agentic Models

Xilin Wei1,2,*, Shengyuan Ding1,*, Xinyu Fang2,3,*, Tiancheng Gu2, Yuchi Wang2,4, Zhixun Li2,4, Ziyu Liu5, Yuhang Zang6, Weixiang Yu2, Cheng Lu2, Chonggao Sun2, Tong Zhou2, Dahua Lin4,†, Hang Zhang2,†, Shuai Bai2,†, Dayiheng Liu2

Affiliations & contribution notes

1Fudan University 2Alibaba Token Hub, Alibaba Group 3Zhejiang University 4The Chinese University of Hong Kong 5Shanghai Jiao Tong University 6Nanyang Technological University

* Equal contribution† Corresponding authors

Paper
Coming soon
Coming soon
View scores Watch the agent

Inside a real task

A video brief.
Three animations.
A working page.

Watch an agent interpret a reference, edit an existing website, and check its work in the browser.

qwen3.8-max-0902
6 / 6 criteria passed
Agent run highlights28 sec
Reference Edit Check ResultEdited from an archived run
273revision tasks
2,331hidden acceptance criteria
10model endpoints
5input modalities

The benchmark

Real requests.
Working revisions.

Agents receive an existing application and a user’s request, sometimes with images, audio, video or documents. Hidden expert criteria check what changed, what works, and what should be preserved.

ME · 100 tasks

Multimodal Editing

Localized edits grounded in the page and its references.

RE · 173 tasks

Repository Editing

Coordinated changes across a multi-file application.

Benchmark details Task construction, statistics & taxonomy

Task formulation

Can a coding agent turn a user’s multimodal request into a complete, working revision of an existing application?

Each task pairs a normalized request with a runnable pre-edit application and hidden, expert-authored acceptance criteria. Requests can carry image, audio, video or document evidence. The criteria specify the requested outcomes and any preservation constraints while permitting any valid implementation — so no gold code or target screenshot is needed.

SA runnable pre-edit baseline and its containerized environment.
IThe normalized request, optional attachments and retained session history.
CHidden criteria that the agent never sees during execution.
Read the abstract

Evaluating human–AI co-design requires testing whether coding agents can translate real users’ multimodal requests into working application revisions. We introduce Co-Design Bench, a reference-free benchmark of 273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions. Its two tracks, Multimodal Editing (ME) and Repository Editing (RE), cover localized edits with optional image, audio, video, or document inputs and coordinated repository revisions. Tasks span three capability dimensions: Grounded Understanding, UI Transformation, and Functional Implementation. The tasks pair normalized requests and runnable pre-edit applications with 2,331 hidden expert-authored acceptance criteria. Our judge-then-refute protocol grades criteria using source and browser evidence, challenges positive verdicts with a separate LLM verifier, and audits visual and interaction claims without gold code or target screenshots. Under a shared agentic harness, the strongest of ten model endpoints passes an average of 69.9% of criteria per task but fully satisfies only 26.4% of tasks. All ten endpoints achieve lower criterion completion on repository edits spanning three or four change types than on those spanning one or two. Co-Design Bench provides a testbed for studying the gap between partial requirement satisfaction and complete task success under agentic execution for real-user application revisions.

Track 1 · ME

Multimodal Editing

100tasks

Localized visual, structural and content edits. The agent receives a runnable page, an annotated target region, a normalized request and optional image, audio, video or document attachments. Rubrics also check that unrelated content is preserved when requested.

Primary attachment modality100 tasks
  • 20 None
  • 4 Image
  • 27 Video
  • 32 Audio
  • 17 Document
553criteria
5.5criteria per task
79 / 21single page / bundled
Track 2 · RE

Repository Editing

173tasks

Coordinated changes across components, routes, state, data flow and content in runnable multi-file frontend repositories. Experts decompose compound requests into separately auditable criteria, leaving implementation choices open.

Edit types required (tasks can need several)173 tasks
Structure
15187%
Style
13377%
Interactive
13176%
Content
11265%
1,778criteria
10.3criteria per task
166need 2+ edit types

How the tasks were built

Recovered, validated, then judged by experts.

Every starting application must build, launch and render before it becomes a task. Requests are deduplicated, filtered, ranked by difficulty, verifiability and historical failure signals, then consolidated into bounded tasks with hidden criteria (paper, Section 3.2 and Appendix I).

(a) Recover runnable starting states

01Session corpus
39,325

production queries across 26,314 real coding sessions, with their repositories, context and attachments.

02Recover pre-edit state

Anchor each candidate to its target edit turn, then restore repository, assets and context with a source-hash snapshot.

03Build & render gate
  • dependencies & build
  • container launch
  • Playwright render
↻ repair: diagnose → fix → re-check✕ prune unrecoverable services
04Runnable package
task/
├─ baseline/
├─ env.manifest
├─ request + media/
└─ validation.log

(b) Select requests and author criteria

05Deduplicate
37,463

unique queries after exact-hash and near-duplicate removal (token Jaccard > 0.8).

06Normalize

Decode attachments, rewrite pasted-image references, optional privacy screen.

07Filter

Hard filters F1–F6 drop environment-blocked, non-actionable, unbounded and non-editing requests. Any hit → drop.

08Rank

r = d + v + f

difficulty 0–5 · verifiability 0–3 · historical failure signal 0–2.

09Consolidate

At most three queries per session become one bounded task; experts refine the request and write hidden criteria.

Co-Design Bench
273tasks2,331criteria

What the tasks test

Three dimensions, nine categories, thirty capabilities.

Each task has one primary label for its main delivery objective; additional labels record its other required capabilities — 1,172 task–capability associations in all. 265 of 273 tasks carry multiple labels and 174 span more than one dimension.

Tasks requiring each capability (labels overlap).

How it compares

Real sessions, reference-free, adversarially checked.

Existing web-coding benchmarks cover reference reconstruction, synthetic or predefined edits, version-history edits, and repository issues. Co-Design Bench combines real-user revisions, five input modalities, reference-free acceptance, agentic evaluation and adversarial verification (paper, Table 1).

Task origins and evaluation capabilities of web-coding benchmarks
reported not reported subset or metric-dependentT / I / V / A / D = text / image / video / audio / document

Real-user sessions are requests from real-user coding histories, excluding standalone issue reports (DesignBench’s session requests belong to its editing subset). Reference-free acceptance needs no gold code or target screenshot; a pre-edit baseline is allowed. Agentic evaluation actively chooses tools and actions, beyond fixed test replay. Adversarial verification separately challenges initial positive verdicts. Definitions follow the paper’s Table 1.

Leaderboard.

Ten models. One shared harness. All 273 tasks.

Best strict completion69.9%

Mean criteria passed per task.

Best full-pass rate26.4%

Only one in four tasks fully delivered.

Broader repository edits−6.3pts

Strict-completion gap: 3–4 vs. 1–2 edit types.

Leaderboard

How to read this table

Strict completionPer task, the fraction of criteria judged PASS; averaged over tasks with equal weight so long rubrics do not dominate. PARTIAL earns no strict credit.

Graded completionSame average with PARTIAL worth 0.5. On a normalized scale, Full-pass ≤ Strict ≤ Graded.

Full-pass ratePercentage of tasks where every criterion is PASS — complete delivery. The gap to Strict is criterion credit from unfinished tasks.

Turns · Len.Median solver turns per task, and mean output tokens per turn.

TokensTotal token traffic per task in millions, including retries and reported cache usage under each API’s accounting; an execution measure, not a monetary cost.

Tool mixShare of tool calls for execution (Bash), inspection (Read / Grep / Glob) and modification (Edit / Write); the remainder is auxiliary.

Explore the results Rankings, delivery gap & task groups

Rankings move

The track changes the order.

Strict-completion rank on ME (left) and RE (right). Rankings correlate (Spearman ρ = 0.818), yet qwen3.8-27b climbs from fifth to second while kimi-k3 drops from fourth to sixth.

Endpoint means span 28.7 points on ME and 13.0 on RE; a combined ranking can conceal setting-specific strengths.

The delivery gap

Criterion credit outruns delivery.

Strict completion versus full-pass rate across all 273 tasks. The distance between the two is criterion credit earned on tasks that still miss at least one requirement.

Strict completionFull-pass rate

Look beyond the average

Different tasks,
different leaders.

Pick a capability, an attachment modality or an edit breadth to compare all ten endpoints.

Task group

Mean strict completion (%)

Effort is not completion

More effort did not mean more credit.

32turns · 1.24M tokens
gpt-6-astra, ME strict 75.4%
69turns · 7.29M tokens
glm-5.2, ME strict 57.2%

On RE, qwen3.8-27b uses 57 turns and 6.72M tokens for 64.5% strict completion, versus 48 turns and 2.33M tokens for gpt-6-astra at 66.7%.

Media in the loop

Agents build their own perception.

68.5%of video runs
call OpenCV
31.5%of video runs
call ffmpeg

ffmpeg also appears in 25.0% of audio runs and faster-whisper in 4.4%. Some agents download and run pretrained perception models — AST, CLAP, Whisper, Qwen2-Audio, CLIP, DINOv2, OWLv2 — to ground their edits.

Execution profiles

Rollouts differ across tracks.

44median turns
ME video tasks
27median turns
ME text-only tasks

Image inspections average 14.47 per run on ME and 2.78 on RE, while the Edit/Write share of tool calls rises from 7.3% to 27.6%. These profiles describe effort; they do not isolate causal effects.

Why another benchmark

Agentic execution
changes the score.

On the same 150 WebCompass-edit tasks, switching qwen3.7-plus from single-shot to agentic execution raises mean graded feature completion from 73.6% to 95.1%, with the feature rubric, judge and render pipeline held fixed. With agentic execution, opus-4-8 reaches 99.4%.

Co-Design Bench evaluates real users’ compound revisions under iterative tool access, and reports both criterion satisfaction and complete task success.

The cross-benchmark bars use different task pools and evaluation protocols; the paired WebCompass comparison concerns execution within a fixed task set (paper, Appendix A).

Mean graded completion (%)

Figure 1(a), redrawn
The Co-Design Bench bar (n = 273) matches opus-4-8’s graded completion over all tasks in Table 5 (0.649).
The tasks

Three requests, three kinds of edits.

Open a task to inspect its inputs, model output and acceptance criteria.

Read the agent’s run Actual tool calls & browser observations

A six-second reference clip shows three different text animations. The agent must map them to the hero, section title and quotation, loop each effect, and preserve the wording and page layout. This timeline follows the archived qwen3.8-max-0902 run, including the frames and screenshots the model actually looked at.

qwen3.8-max-0902 passes all six hidden criteria: 100% strict completion in the frozen paper run.

    The timeline shows selected steps; the counts include every call in the archived run.

    The evaluation

    Judge, then refute.

    Test the application, then challenge every positive verdict.

    01 / Build

    Run the candidate

    Build and launch the final application in an isolated sandbox.

    02 / Gather

    Evidence per claim

    Inspect source, screenshots and browser interactions for each requirement.

    03 / Judge

    Verdict per criterion

    Grade each hidden criterion PASS, PARTIAL or FAIL against the starting app.

    04 / Refute

    Challenge every credit

    A separate verifier challenges credit for stubs, hardcoding and unreachable features.

    Scoring definitions
    Strict completion

    Rt = # PASSnt

    fully satisfied criteria in task t
    Graded completion

    Gt = # PASS + ½ # PARTIALnt

    credits partial progress
    Full-pass rate

    F = share of tasks with Rt = 1

    every criterion delivered
    PASS = 1PARTIAL = 0.5 graded · 0 strictFAIL = 0

    Task means weight all 273 tasks equally, so long rubrics do not dominate. On a common 0–1 scale F ≤ R̄ ≤ Ḡ, and a high R̄ can coexist with a low F when unmet requirements are spread across many tasks.

    See the full evaluation pipeline
    Evaluation pipeline. (A) Evidence extraction and primary evaluation: query, expert rubric, baseline and output feed sandbox execution (copy, Vite build, serve), evidence extraction (code diff, DOM body text, screenshot, Playwright actions) and a primary evaluator that issues Pass, Partial or Fail. (B) Adversarial verification and attribution: an independent verifier checks stubs, hardcoding, data-flip tests, entry-to-UI reachability and semantic criteria; claim audits check renders and browser logs and abort on violations; scores and report aggregate raw scores; conditional attribution runs a baseline render and solvability agent for low-scoring cases without changing raw verdicts.
    Figure 3 from the paper. For low-scoring cases (Gt < 0.8) with a startable baseline, a parallel solvability probe attributes failures to model limitations, environment blockage, mixed causes or starting-artifact defects. Raw scores and denominators never change; environment-adjusted scores are retained separately as a supplementary analysis.
    Paper & page data

    Project resources