34 lines
14 KiB
JSON
34 lines
14 KiB
JSON
{"name": "arithmetic-0", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 141, "completion_tokens": 82, "completion_tokens_details": {"reasoning_tokens": 75}}, "ttft_s": 0.07772354000007908, "elapsed_s": 1.149622212000395, "decode_tps": 75.5668442510918, "max_stream_gap_s": 0.03973271099948761, "events": 79, "content": "\n\n3973", "reasoning": "I need to compute 137 * 29 + 0.\n\n137 * 29 = 137 * 30 - 137 = 4110 - 137 = 3973\n\nAdding 0 doesn't change anything, so the answer is 3973.\n", "sha256": "d929fd317464b07ece811a135b5937af8eb36ea0b1e88ec82bc5137102d0a62c"}
|
|
{"name": "arithmetic-1", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 96, "completion_tokens": 37, "completion_tokens_details": {"reasoning_tokens": 30}}, "ttft_s": 0.07883534699976735, "elapsed_s": 0.5551211570000305, "decode_tps": 75.58486783383303, "max_stream_gap_s": 0.03967835599996761, "events": 34, "content": "\n\n5761", "reasoning": "160 * 36 = 5760. 5760 + 1 = 5761.\n", "sha256": "5cd050422245af84010bf1c71082001dcd6d880bdd6dd7002e69a8402ab6c73c"}
|
|
{"name": "arithmetic-2", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 125, "completion_tokens": 66, "completion_tokens_details": {"reasoning_tokens": 59}}, "ttft_s": 0.07831667499976902, "elapsed_s": 0.9377429689993733, "decode_tps": 75.63184935557712, "max_stream_gap_s": 0.039701828999568534, "events": 63, "content": "\n\n7871", "reasoning": "183 * 43 = 183 * 40 + 183 * 3 = 7320 + 549 = 7869. Then 7869 + 2 = 7871.\n", "sha256": "c6dabe0673abae92ebdac3d245deb7c9ca89c874d316c46b8ce09fde923a752f"}
|
|
{"name": "arithmetic-3", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 101, "completion_tokens": 42, "completion_tokens_details": {"reasoning_tokens": 34}}, "ttft_s": 0.07822468900030799, "elapsed_s": 0.6203575899999123, "decode_tps": 75.62721230237588, "max_stream_gap_s": 0.03962699600015185, "events": 39, "content": "\n\n10303", "reasoning": "206 * 50 = 10300, then 10300 + 3 = 10303.\n", "sha256": "faf8742035a51aed0549e27e042a16ddce400e4a9f6cb41b47540844a9fde0a7"}
|
|
{"name": "arithmetic-4", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 131, "completion_tokens": 72, "completion_tokens_details": {"reasoning_tokens": 64}}, "ttft_s": 0.07851739099987753, "elapsed_s": 1.017388297999787, "decode_tps": 75.62275012533416, "max_stream_gap_s": 0.039650787999562453, "events": 69, "content": "\n\n13057", "reasoning": "229 * 57 = 229 * 50 + 229 * 7 = 11450 + 1603 = 13053. Then 13053 + 4 = 13057.\n", "sha256": "5c8a761b69d32970d9e4021e9cddda2caf44a6201a9bf34e503844cd2ce1dc0c"}
|
|
{"name": "arithmetic-5", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 100, "completion_tokens": 41, "completion_tokens_details": {"reasoning_tokens": 33}}, "ttft_s": 0.07850254000004497, "elapsed_s": 0.6072512140008257, "decode_tps": 75.6503079191475, "max_stream_gap_s": 0.039656347000345704, "events": 38, "content": "\n\n16133", "reasoning": "252 * 64 = 16128. 16128 + 5 = 16133.\n", "sha256": "d7ffeebe2bcd01af72cd619ced3e33606c067dbbcee20babfbe1aded61293b92"}
|
|
{"name": "arithmetic-6", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 130, "completion_tokens": 71, "completion_tokens_details": {"reasoning_tokens": 63}}, "ttft_s": 0.07873150400064333, "elapsed_s": 1.0037764430007883, "decode_tps": 75.67199932541767, "max_stream_gap_s": 0.03964858600011212, "events": 68, "content": "\n\n19531", "reasoning": "275 * 71 = 275 * 70 + 275 * 1 = 19250 + 275 = 19525. Then 19525 + 6 = 19531.\n", "sha256": "7ac565b6322b904ff447bc9ef3ed069ebc4e48b8a51e72f9e18919bd8e3000d1"}
|
|
{"name": "arithmetic-7", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 153, "completion_tokens": 94, "completion_tokens_details": {"reasoning_tokens": 86}}, "ttft_s": 0.0784115259993996, "elapsed_s": 1.3077415259995178, "decode_tps": 75.65096434642534, "max_stream_gap_s": 0.03962907900040591, "events": 91, "content": "\n\n23251", "reasoning": "298 * 78 = 298 * 78. Let me compute this step by step.\n\n298 * 78 = 298 * 80 - 298 * 2 = 23840 - 596 = 23244\n\n23244 + 7 = 23251\n", "sha256": "e7b73eb708d2c4c7d2fc5dca2aed40a571c8b0445cd1867d916b47f57ec4cac0"}
|
|
{"name": "arithmetic-8", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 144, "completion_tokens": 85, "completion_tokens_details": {"reasoning_tokens": 77}}, "ttft_s": 0.07822253599988471, "elapsed_s": 1.1883135919997585, "decode_tps": 75.6694683251367, "max_stream_gap_s": 0.03964279800038639, "events": 82, "content": "\n\n27293", "reasoning": "Let me compute 321 * 85 + 8.\n\n321 * 85 = 321 * 80 + 321 * 5 = 25680 + 1605 = 27285\n\n27285 + 8 = 27293\n", "sha256": "9d93671c15eac2005a27c4fe38e5d4285b3feca77fa1e90f138c71a2b5164db8"}
|
|
{"name": "arithmetic-9", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 59, "total_tokens": 143, "completion_tokens": 84, "completion_tokens_details": {"reasoning_tokens": 76}}, "ttft_s": 0.0801680779995877, "elapsed_s": 1.1772930710003493, "decode_tps": 75.65227346884657, "max_stream_gap_s": 0.03965692699966894, "events": 81, "content": "\n\n31657", "reasoning": "Let me compute 344 * 92 + 9.\n\n344 * 92 = 344 * 90 + 344 * 2 = 30960 + 688 = 31648\n\n31648 + 9 = 31657\n", "sha256": "6b95a8b82a4ba5a8335b5f10899663aeebe71fed7e5673836b82ec74cfa1cbee"}
|
|
{"name": "arithmetic-10", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 60, "total_tokens": 143, "completion_tokens": 83, "completion_tokens_details": {"reasoning_tokens": 75}}, "ttft_s": 0.07851070200013055, "elapsed_s": 1.1620557369997186, "decode_tps": 75.67751902442262, "max_stream_gap_s": 0.039518157000202336, "events": 80, "content": "\n\n36343", "reasoning": "Let me compute 367 * 99 + 10.\n\n367 * 99 = 367 * (100 - 1) = 36700 - 367 = 36333\n\n36333 + 10 = 36343\n", "sha256": "662f32768d742ea67e45b8109190fc2ba55c429346f4be9b9492ff457cdacc48"}
|
|
{"name": "arithmetic-11", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 61, "total_tokens": 151, "completion_tokens": 90, "completion_tokens_details": {"reasoning_tokens": 82}}, "ttft_s": 0.07955242300067766, "elapsed_s": 1.2561235120001584, "decode_tps": 75.64353810162275, "max_stream_gap_s": 0.03971186299986584, "events": 87, "content": "\n\n41351", "reasoning": "Let me compute 390 * 106 + 11.\n\n390 * 106 = 390 * 100 + 390 * 6 = 39000 + 2340 = 41340\n\n41340 + 11 = 41351\n", "sha256": "cdbf70494a3484ed891137af4bad12b7417009befd921930fc1fa5e5e3907e27"}
|
|
{"name": "json", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 66, "total_tokens": 109, "completion_tokens": 43, "completion_tokens_details": {"reasoning_tokens": 26}}, "ttft_s": 0.09644218499943236, "elapsed_s": 0.651459989999239, "decode_tps": 75.6732479960974, "max_stream_gap_s": 0.03967342000032659, "events": 40, "content": "\n\n{\"name\":\"设备A\",\"count\":7,\"enabled\":true}", "reasoning": "The user wants me to return exactly a specific JSON object with no markdown formatting. Let me provide exactly what they asked for.\n", "sha256": "2acba5868b21fa90ff7d281b66898b4ed66b6ff2f42a3d5a1d5bd11569806949"}
|
|
{"name": "multiturn", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 82, "total_tokens": 146, "completion_tokens": 64, "completion_tokens_details": {"reasoning_tokens": 52}}, "ttft_s": 0.07780361999994057, "elapsed_s": 0.9104519810007332, "decode_tps": 75.66219180961076, "max_stream_gap_s": 0.039605606999430165, "events": 61, "content": "\n\nAQUA-58321", "reasoning": "The user is asking me to recall the code they previously told me to remember. They said \"Remember the code AQUA-58321.\" and now they're asking what the code was. They want me to reply only with the code.\n", "sha256": "5a11c72c78b0b7d9d0ec352a11aa21a4fa1ea34ff915277ee160bbb83bbb1dd6"}
|
|
{"name": "image-ocr", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 3135, "total_tokens": 3315, "completion_tokens": 180, "completion_tokens_details": {"reasoning_tokens": 138}}, "ttft_s": 0.7455413950001457, "elapsed_s": 3.1262464499995986, "decode_tps": 75.18781027666661, "max_stream_gap_s": 0.03989871899921127, "events": 177, "content": "\n\nVerification code: **SPARK-82741**\n\nSum of amounts: 125 + 478 + 765 = **1368**", "reasoning": "The user wants the verification code and the sum of three amounts from the invoice image. Reading the image: Verification: SPARK-82741. Amount A: 125, Amount B: 478, Amount C: 765. Sum: 125 + 478 = 603; 603 + 765 = 1368. I need to provide a self-contained reply with the code and the sum. Reasoning should show entropy reduction—confirming each value and the arithmetic. Final answer: SPARK-82741 and 1368.\n", "sha256": "64d438bdfb6d81cbaa352b6ee35b7a538b4e46438117f85e1d529ff210e9a90d"}
|
|
{"name": "video-120s", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 12069, "total_tokens": 12183, "completion_tokens": 114, "completion_tokens_details": {"reasoning_tokens": 100}}, "ttft_s": 2.4627599759996883, "elapsed_s": 3.9740706529992167, "decode_tps": 74.76953727630898, "max_stream_gap_s": 0.040032412999607914, "events": 111, "content": "\n\nRed, Green, Blue, Yellow, Purple, White", "reasoning": "The user wants the six background colors in chronological order from the video frames. Looking at the timestamps: 0-19s RED, 20-39s GREEN, 40-59s BLUE, 60-79s YELLOW, 80-99s PURPLE, 100-119s WHITE. That's six colors. I need to reply only with the English color names, in order. No extra text.\n", "sha256": "53ddddc25a28dbc73b210512f71d35f5b4daee20506cee8470192ce5d98d945d"}
|
|
{"name": "multi-image", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 12356, "total_tokens": 12427, "completion_tokens": 71, "completion_tokens_details": {"reasoning_tokens": 60}}, "ttft_s": 1.4047450600000957, "elapsed_s": 2.3313891259995216, "decode_tps": 75.54141074059753, "max_stream_gap_s": 0.03995001000021148, "events": 68, "content": "\n\nSPARK-82741", "reasoning": "The user wants the verification code from the invoice. All four copies show the same line: \"Verification: SPARK-82741\". The instruction is to reply only with the code. So the answer is SPARK-82741. No need for elaboration.\n", "sha256": "c4d4ac93f59330deca1f3c5fe9842a60a2ac31170fdbc40f311385c571ed9024"}
|
|
{"name": "warmup-code", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.005225435999818728}
|
|
{"name": "perf-code-0", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0015374110007542185}
|
|
{"name": "perf-code-1", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0014719609998792293}
|
|
{"name": "perf-code-2", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0015831439995963592}
|
|
{"name": "warmup-chinese", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0012888200008092099}
|
|
{"name": "perf-chinese-0", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0015359890003310284}
|
|
{"name": "perf-chinese-1", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0013069549995634588}
|
|
{"name": "perf-chinese-2", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0015163429998210631}
|
|
{"name": "warmup-reasoning", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0012579570002344553}
|
|
{"name": "perf-reasoning-0", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.001492990000770078}
|
|
{"name": "perf-reasoning-1", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0012681409998549498}
|
|
{"name": "perf-reasoning-2", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0014675349993922282}
|
|
{"name": "long-output-16384", "passed": false, "error": "HTTP Error 400: Bad Request", "elapsed_s": 0.0012407340000208933}
|
|
{"name": "long-context-0", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 125995, "total_tokens": 126124, "completion_tokens": 129, "completion_tokens_details": {"reasoning_tokens": 109}}, "ttft_s": 13.898595239000315, "elapsed_s": 15.572495153999625, "decode_tps": 76.46813220613176, "max_stream_gap_s": 0.03968665800039162, "events": 126, "content": "\n\nNORTH-49273\nSOUTH-18362", "reasoning": "The user is asking me to provide the northern and southern access codes that are embedded in the data. Let me find them:\n\n1. \"The northern access code is NORTH-49273.\" - found between Item 000019 and Item 000021\n2. \"The southern access code is SOUTH-18362.\" - found between Item 002499 and Item 002501\n\nThe user wants only these two codes.\n", "sha256": "120d58d463e19b76220422b4b9e9ae6449cf6f89f796335bd1cbe86ec55dcc05"}
|
|
{"name": "long-context-1", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 125995, "total_tokens": 126112, "completion_tokens": 117, "completion_tokens_details": {"reasoning_tokens": 97}}, "ttft_s": 0.24017905600067024, "elapsed_s": 1.7552992750006524, "decode_tps": 76.56158141468335, "max_stream_gap_s": 0.03971610900043743, "events": 114, "content": "\n\nNORTH-49273\nSOUTH-18362", "reasoning": "The user is asking me to provide both the northern and southern access codes from the data provided.\n\nLooking through the data, I can find:\n- \"The northern access code is NORTH-49273.\" (appears after Item 000019)\n- \"The southern access code is SOUTH-18362.\" (appears after Item 002499)\n\nThe user wants only these two codes.\n", "sha256": "7f8768048dbe7257ce2b8a8b9aa144900f50a7a3a2eae9d79f0120f2323e42d7"}
|
|
{"name": "long-context-2", "passed": true, "finish": "stop", "usage": {"prompt_tokens": 125995, "total_tokens": 126117, "completion_tokens": 122, "completion_tokens_details": {"reasoning_tokens": 102}}, "ttft_s": 0.22724735799965856, "elapsed_s": 1.8088994879999518, "decode_tps": 76.50228372276625, "max_stream_gap_s": 0.03969127400068828, "events": 119, "content": "\n\nNORTH-49273\nSOUTH-18362", "reasoning": "The user is asking me to provide both the northern and southern access codes that are embedded in the data.\n\nLooking through the data:\n- \"The northern access code is NORTH-49273.\" (found between Item 000019 and Item 000021)\n- \"The southern access code is SOUTH-18362.\" (found between Item 002499 and Item 002501)\n", "sha256": "27006fc14e3dcb44599f4eaf64b2b32126e28e70c7dbca114c947205a75d4094"}
|