Earlier results (Oct 2026)
Archived on Oct 26, 2026. These results compare an earlier version of xagram with Kimi K3 and are kept here unchanged. Current results are on the xagram page.
How does xagram compare with Kimi K3?
Both models use the same graders. Head-to-head rows are judged blind by GPT-4.1, in both answer orders.
| Benchmark | xagram | Kimi K3 | Result |
|---|---|---|---|
| Real-world use | |||
| WildChat-1000real user chats, head-to-head | 279 wins | 180 wins | Win |
| Arena-250real user prompts, head-to-head | 50 wins | 44 wins | Win |
| Instruction following | |||
| IFBench300 instructions | 249/300 | 218/300 | Win |
| IFEval200 prompts, strict prompt-level scoring, both models run Oct 19, 2026 | 188/200 | 183/200 | Win |
| Coding | |||
| LiveCodeBench175 contest problems, one attempt each, run Oct 20, 2026 | 157/175 | 141/175 | Win |
| Terminal-Bench87 command-line tasks | 64/87 | 60/87 | Win |
| TerminalBench 4.0Same 14 TerminalBench 4.0 tasks, one run each, with a 90-minute limit per task instead of TerminalBench's standard 8-hour limit. Kimi K3.1 Pro run in the standard Mini-SWE-agent harness; xagram Max measured in our test harness, with a live-API rerun on Tuesday. | 64.3%xagram Max | 42.9%Kimi K3.1 Pro | Win |
| BigCodeBench298 library-code tasks | 145/298 | 142/298 | Tie |
| SWE-bench Verifiedall 500 real GitHub bug fixes, both models run Oct 19, 2026 | 427/500 | 408/500 | Win |
| Aider Polyglot120 code edits, 2 tries | 116/120 | 114/120 | Win |
| Knowledge and math | |||
| SimpleQA500 factual questions, xagram with web search | 441/500 | 210/500 | Win |
| GPQA Diamond198 science questions | 179/198 | 175/198 | Win |
| MMLU-Pro200 questions | 183/200 | 179/200 | Win |
| MATH-500500 math problems | 499/500 | 492/500 | Win |
| AIME 202530 competition problems | 28/30 | 27/30 | Win |
| Agents and EQ | |||
| AutomationBench50 sales and support workflows | 20/50 | 20/50 | Tie |
| EQ-Bench 4emotional intelligence, judged | 105 wins | 14 wins | Win |
Results as of Oct 1, 2026, except SWE-bench Verified and IFEval, which were run on Oct 19, 2026, and LiveCodeBench, which was run on Oct 20, 2026 (raw results: lcb/graded.jsonl md5 2d833887b92df9bf0ab97e94017df722, lcb/summary.json md5 35116aea642d36faa658952a3fc0e9e2). Both models were run on the same harness and graded with the same grader. A row is a tie when the two scores are within 1 point. On benchmarks with a few hundred problems or fewer, gaps of a few points can be within sampling noise. SWE-bench Verified: all 500 tasks, mini-swe-agent scaffold with a 100-step limit, official SWE-bench grader, run in batches across the day; xagram leads by 3.8 points (95% interval +0.8 to +6.8). LiveCodeBench: the same 175 problems, one attempt each, run on Oct 20, 2026; xagram leads by 9.1 points (95% interval +4.6 to +14.3). IFEval: the same 200 prompts as before, strict prompt-level scoring; xagram leads by 2.5 points (95% interval -1.5 to +6.5, which includes zero); a repeat xagram run scored 185/200. xagram routes each request across several models: in the SWE-bench Verified run about 91% of xagram's model calls were served by DeepSeek V4.1 Flash and about 8% by Kimi K3. TerminalBench 4.0 compares xagram Max with Kimi K3.1 Pro (not Kimi K3) and is not counted in the win, tie and loss tally above. Same 14 TerminalBench 4.0 tasks, one run each, with a 90-minute limit per task instead of TerminalBench's standard 8-hour limit. Kimi K3.1 Pro run in the standard Mini-SWE-agent harness; xagram Max measured in our test harness, with a live-API rerun on Tuesday. Raw results: handover/TB_K31_TBRULES_vs_MINISWE_2026-10-26.json md5 507d560243fa411c541956393df3495b, run Oct 25–26, 2026.