Midscene AppControlBench Benchmark Report
This is Midscene's benchmark report for AppControlBench. In this evaluation, each model ran the same set of 60 tasks, with the following results:
| Model | Completion | Time / run | Cost / run | Task outcomes |
|---|---|---|---|---|
| Doubao Seed 2.1 Turbo | 98% | 52s | $0.0099/¥0.0670 | |
| Qwen3.7 Plus | 97% | 54s | $0.0097/¥0.0657 | |
| DeepSeek V4 Flash Vision Exp | 90% | 1m 26s | $0.0045/¥0.0308 |
Note: Doubao Seed 2.1 Turbo costs use Volcengine's official pricing: CNY ¥3 / M input tokens, ¥0.6 / M cached input tokens, and ¥15 / M output tokens. Other model costs use OpenRouter pricing. USD values use ¥6.8 per $1.
The results show that the Midscene + GUI vision approach is highly competitive in both cost and pass rate.
About AppControlBench
AppControlBench evaluates how well an Agent can operate real iOS apps in isolated Simulator runs. The final screenshot is graded against a task-specific solved-screen description. Each run covers 30 Bluesky tasks and 30 Element iOS tasks.
Run configuration
Case Intent Calibration
For this evaluation, the action-route description of 1 case was calibrated against the actual UI of the pinned Element iOS version. The calibration only removes a stale or unavailable UI-entry description; it does not change the final target state or judge rules of the case.
Report files
Doubao Seed 2.1 Turbo · Pass@1 96.67% · 58 PASS · 1 PARTIAL · 1 FAIL
| # | Task | Status | Model Cost | Report |
|---|---|---|---|---|
| 1 | bsky-01 | PASS | $0.0038/¥0.0259 | report |
| 2 | bsky-02 | PASS | $0.0038/¥0.0259 | report |
| 3 | bsky-03 | PASS | $0.0079/¥0.0537 | report |
| 4 | bsky-04 | PASS | $0.0060/¥0.0405 | report |
| 5 | bsky-05 | PASS | $0.0039/¥0.0263 | report |
| 6 | bsky-06 | PASS | $0.0040/¥0.0272 | report |
| 7 | bsky-07 | PASS | $0.0039/¥0.0268 | report |
| 8 | bsky-08 | PASS | $0.0130/¥0.0884 | report |
| 9 | bsky-09 | PASS | $0.0081/¥0.0552 | report |
| 10 | bsky-10 | PASS | $0.0060/¥0.0410 | report |
| 11 | bsky-11 | PASS | $0.0038/¥0.0256 | report |
| 12 | bsky-12 | PASS | $0.0038/¥0.0258 | report |
| 13 | bsky-13 | PASS | $0.0038/¥0.0259 | report |
| 14 | bsky-14 | PASS | $0.0104/¥0.0704 | report |
| 15 | bsky-15 | PASS | $0.0104/¥0.0706 | report |
| 16 | bsky-16 | PASS | $0.0240/¥0.1631 | report |
| 17 | bsky-17 | PASS | $0.0059/¥0.0399 | report |
| 18 | bsky-18 | PASS | $0.0038/¥0.0255 | report |
| 19 | bsky-19 | PASS | $0.0080/¥0.0545 | report |
| 20 | bsky-20 | PASS | $0.0080/¥0.0547 | report |
| 21 | bsky-21 | PASS | $0.0059/¥0.0399 | report |
| 22 | bsky-22 | PASS | $0.0037/¥0.0254 | report |
| 23 | bsky-23 | PASS | $0.0058/¥0.0394 | report |
| 24 | bsky-24 | PASS | $0.0082/¥0.0554 | report |
| 25 | bsky-25 | FAIL | $0.0079/¥0.0535 | report |
| 26 | bsky-26 | PASS | $0.0152/¥0.1030 | report |
| 27 | bsky-27 | PASS | $0.0174/¥0.1183 | report |
| 28 | bsky-28 | PASS | $0.0080/¥0.0542 | report |
| 29 | bsky-29 | PASS | $0.0080/¥0.0541 | report |
| 30 | bsky-30 | PASS | $0.0106/¥0.0723 | report |
| 31 | element-01 | PASS | $0.0037/¥0.0252 | report |
| 32 | element-02 | PASS | $0.0038/¥0.0259 | report |
| 33 | element-03 | PASS | $0.0101/¥0.0686 | report |
| 34 | element-04 | PASS | $0.0057/¥0.0390 | report |
| 35 | element-05 | PASS | $0.0066/¥0.0447 | report |
| 36 | element-06 | PARTIAL | $0.0988/¥0.6720 | report |
| 37 | element-07 | PASS | $0.0059/¥0.0400 | report |
| 38 | element-08 | PASS | $0.0103/¥0.0700 | report |
| 39 | element-09 | PASS | $0.0040/¥0.0275 | report |
| 40 | element-10 | PASS | $0.0187/¥0.1272 | report |
| 41 | element-11 | PASS | $0.0068/¥0.0460 | report |
| 42 | element-12 | PASS | $0.0059/¥0.0404 | report |
| 43 | element-13 | PASS | $0.0106/¥0.0724 | report |
| 44 | element-14 | PASS | $0.0060/¥0.0410 | report |
| 45 | element-15 | PASS | $0.0081/¥0.0554 | report |
| 46 | element-16 | PASS | $0.0083/¥0.0567 | report |
| 47 | element-17 | PASS | $0.0080/¥0.0543 | report |
| 48 | element-18 | PASS | $0.0105/¥0.0714 | report |
| 49 | element-19 | PASS | $0.0182/¥0.1240 | report |
| 50 | element-20 | PASS | $0.0038/¥0.0256 | report |
| 51 | element-21 | PASS | $0.0119/¥0.0810 | report |
| 52 | element-22 | PASS | $0.0156/¥0.1062 | report |
| 53 | element-23 | PASS | $0.0081/¥0.0550 | report |
| 54 | element-24 | PASS | $0.0129/¥0.0880 | report |
| 55 | element-25 | PASS | $0.0059/¥0.0404 | report |
| 56 | element-26 | PASS | $0.0038/¥0.0257 | report |
| 57 | element-27 | PASS | $0.0160/¥0.1087 | report |
| 58 | element-28 | PASS | $0.0100/¥0.0683 | report |
| 59 | element-29 | PASS | $0.0102/¥0.0693 | report |
| 60 | element-30 | PASS | $0.0100/¥0.0680 | report |
Qwen3.7 Plus · Pass@1 93.33% · 56 PASS · 4 PARTIAL · 0 FAIL
| # | Task | Status | Model Cost | Report |
|---|---|---|---|---|
| 1 | bsky-01 | PASS | $0.0073/¥0.0497 | report |
| 2 | bsky-02 | PASS | $0.0074/¥0.0501 | report |
| 3 | bsky-03 | PASS | $0.0108/¥0.0736 | report |
| 4 | bsky-04 | PASS | $0.0092/¥0.0624 | report |
| 5 | bsky-05 | PASS | $0.0053/¥0.0362 | report |
| 6 | bsky-06 | PASS | $0.0075/¥0.0507 | report |
| 7 | bsky-07 | PASS | $0.0075/¥0.0507 | report |
| 8 | bsky-08 | PASS | $0.0170/¥0.1155 | report |
| 9 | bsky-09 | PASS | $0.0110/¥0.0746 | report |
| 10 | bsky-10 | PASS | $0.0111/¥0.0756 | report |
| 11 | bsky-11 | PASS | $0.0053/¥0.0360 | report |
| 12 | bsky-12 | PASS | $0.0053/¥0.0357 | report |
| 13 | bsky-13 | PASS | $0.0054/¥0.0365 | report |
| 14 | bsky-14 | PASS | $0.0086/¥0.0585 | report |
| 15 | bsky-15 | PASS | $0.0107/¥0.0724 | report |
| 16 | bsky-16 | PARTIAL | $0.0196/¥0.1332 | report |
| 17 | bsky-17 | PASS | $0.0050/¥0.0342 | report |
| 18 | bsky-18 | PASS | $0.0033/¥0.0224 | report |
| 19 | bsky-19 | PASS | $0.0067/¥0.0457 | report |
| 20 | bsky-20 | PASS | $0.0068/¥0.0460 | report |
| 21 | bsky-21 | PASS | $0.0049/¥0.0335 | report |
| 22 | bsky-22 | PASS | $0.0032/¥0.0219 | report |
| 23 | bsky-23 | PASS | $0.0050/¥0.0340 | report |
| 24 | bsky-24 | PASS | $0.0069/¥0.0469 | report |
| 25 | bsky-25 | PARTIAL | $0.0068/¥0.0463 | report |
| 26 | bsky-26 | PASS | $0.0125/¥0.0849 | report |
| 27 | bsky-27 | PASS | $0.0145/¥0.0988 | report |
| 28 | bsky-28 | PASS | $0.0068/¥0.0461 | report |
| 29 | bsky-29 | PASS | $0.0068/¥0.0462 | report |
| 30 | bsky-30 | PASS | $0.0171/¥0.1160 | report |
| 31 | element-01 | PASS | $0.0052/¥0.0356 | report |
| 32 | element-02 | PASS | $0.0032/¥0.0214 | report |
| 33 | element-03 | PASS | $0.0087/¥0.0593 | report |
| 34 | element-04 | PASS | $0.0048/¥0.0326 | report |
| 35 | element-05 | PASS | $0.0107/¥0.0727 | report |
| 36 | element-06 | PARTIAL | $0.0640/¥0.4355 | report |
| 37 | element-07 | PASS | $0.0088/¥0.0600 | report |
| 38 | element-08 | PASS | $0.0086/¥0.0582 | report |
| 39 | element-09 | PASS | $0.0032/¥0.0217 | report |
| 40 | element-10 | PASS | $0.0234/¥0.1593 | report |
| 41 | element-11 | PASS | $0.0050/¥0.0341 | report |
| 42 | element-12 | PASS | $0.0050/¥0.0338 | report |
| 43 | element-13 | PASS | $0.0070/¥0.0474 | report |
| 44 | element-14 | PASS | $0.0051/¥0.0345 | report |
| 45 | element-15 | PASS | $0.0068/¥0.0461 | report |
| 46 | element-16 | PARTIAL | $0.0346/¥0.2352 | report |
| 47 | element-17 | PASS | $0.0206/¥0.1403 | report |
| 48 | element-18 | PASS | $0.0112/¥0.0761 | report |
| 49 | element-19 | PASS | $0.0108/¥0.0738 | report |
| 50 | element-20 | PASS | $0.0032/¥0.0219 | report |
| 51 | element-21 | PASS | $0.0033/¥0.0222 | report |
| 52 | element-22 | PASS | $0.0128/¥0.0872 | report |
| 53 | element-23 | PASS | $0.0069/¥0.0466 | report |
| 54 | element-24 | PASS | $0.0109/¥0.0739 | report |
| 55 | element-25 | PASS | $0.0050/¥0.0341 | report |
| 56 | element-26 | PASS | $0.0033/¥0.0224 | report |
| 57 | element-27 | PASS | $0.0051/¥0.0344 | report |
| 58 | element-28 | PASS | $0.0106/¥0.0723 | report |
| 59 | element-29 | PASS | $0.0087/¥0.0588 | report |
| 60 | element-30 | PASS | $0.0087/¥0.0590 | report |
DeepSeek V4 Flash Vision Exp · Pass@1 86.67% · 52 PASS · 4 PARTIAL · 4 FAIL
| # | Task | Status | Model Cost | Report |
|---|---|---|---|---|
| 1 | bsky-01 | PASS | $0.0005/¥0.0035 | report |
| 2 | bsky-02 | PASS | $0.0005/¥0.0035 | report |
| 3 | bsky-03 | PASS | $0.0015/¥0.0100 | report |
| 4 | bsky-04 | PASS | $0.0009/¥0.0062 | report |
| 5 | bsky-05 | PASS | $0.0005/¥0.0037 | report |
| 6 | bsky-06 | PASS | $0.0021/¥0.0144 | report |
| 7 | bsky-07 | PASS | $0.0010/¥0.0066 | report |
| 8 | bsky-08 | PASS | $0.0019/¥0.0128 | report |
| 9 | bsky-09 | FAIL | $0.0155/¥0.1055 | report |
| 10 | bsky-10 | PASS | $0.0018/¥0.0121 | report |
| 11 | bsky-11 | PASS | $0.0011/¥0.0077 | report |
| 12 | bsky-12 | PASS | $0.0011/¥0.0073 | report |
| 13 | bsky-13 | PASS | $0.0012/¥0.0082 | report |
| 14 | bsky-14 | PASS | $0.0029/¥0.0200 | report |
| 15 | bsky-15 | PASS | $0.0012/¥0.0081 | report |
| 16 | bsky-16 | PARTIAL | $0.0160/¥0.1085 | report |
| 17 | bsky-17 | PASS | $0.0017/¥0.0114 | report |
| 18 | bsky-18 | PASS | $0.0011/¥0.0078 | report |
| 19 | bsky-19 | PASS | $0.0025/¥0.0168 | report |
| 20 | bsky-20 | PASS | $0.0022/¥0.0152 | report |
| 21 | bsky-21 | PASS | $0.0023/¥0.0156 | report |
| 22 | bsky-22 | PASS | $0.0011/¥0.0074 | report |
| 23 | bsky-23 | PASS | $0.0017/¥0.0119 | report |
| 24 | bsky-24 | PARTIAL | $0.0072/¥0.0493 | report |
| 25 | bsky-25 | PASS | $0.0029/¥0.0199 | report |
| 26 | bsky-26 | PASS | $0.0025/¥0.0171 | report |
| 27 | bsky-27 | PASS | $0.0034/¥0.0231 | report |
| 28 | bsky-28 | PASS | $0.0023/¥0.0157 | report |
| 29 | bsky-29 | PASS | $0.0023/¥0.0159 | report |
| 30 | bsky-30 | FAIL | $0.0290/¥0.1970 | report |
| 31 | element-01 | PASS | $0.0010/¥0.0069 | report |
| 32 | element-02 | PASS | $0.0010/¥0.0071 | report |
| 33 | element-03 | PASS | $0.0030/¥0.0201 | report |
| 34 | element-04 | PASS | $0.0038/¥0.0256 | report |
| 35 | element-05 | PASS | $0.0018/¥0.0122 | report |
| 36 | element-06 | FAIL | $0.0232/¥0.1575 | report |
| 37 | element-07 | PASS | $0.0019/¥0.0127 | report |
| 38 | element-08 | PASS | $0.0028/¥0.0193 | report |
| 39 | element-09 | PASS | $0.0011/¥0.0077 | report |
| 40 | element-10 | PASS | $0.0024/¥0.0161 | report |
| 41 | element-11 | PASS | $0.0019/¥0.0131 | report |
| 42 | element-12 | PASS | $0.0019/¥0.0130 | report |
| 43 | element-13 | PASS | $0.0026/¥0.0176 | report |
| 44 | element-14 | PASS | $0.0020/¥0.0139 | report |
| 45 | element-15 | PASS | $0.0010/¥0.0067 | report |
| 46 | element-16 | PARTIAL | $0.0549/¥0.3734 | report |
| 47 | element-17 | PASS | $0.0026/¥0.0178 | report |
| 48 | element-18 | PASS | $0.0027/¥0.0182 | report |
| 49 | element-19 | PARTIAL | $0.0106/¥0.0723 | report |
| 50 | element-20 | PASS | $0.0011/¥0.0075 | report |
| 51 | element-21 | FAIL | $0.0236/¥0.1604 | report |
| 52 | element-22 | PASS | $0.0024/¥0.0160 | report |
| 53 | element-23 | PASS | $0.0014/¥0.0094 | report |
| 54 | element-24 | PASS | $0.0020/¥0.0138 | report |
| 55 | element-25 | PASS | $0.0009/¥0.0059 | report |
| 56 | element-26 | PASS | $0.0006/¥0.0039 | report |
| 57 | element-27 | PASS | $0.0010/¥0.0065 | report |
| 58 | element-28 | PASS | $0.0016/¥0.0107 | report |
| 59 | element-29 | PASS | $0.0016/¥0.0107 | report |
| 60 | element-30 | PASS | $0.0016/¥0.0107 | report |