Hugging Face's logo

NameONEStudios
/
hypernet-n1-sdc

English
code-generation
multi-model
routing
humaneval
constellation
orchestration
Eval Results
Model card Files
xet
Community
NameONEStudios commited on
Commit
1db22f0
·
verified ·
1 Parent(s): 6f8cacd

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. README.md +167 -3
  2. humaneval_6lane_123525.json +1834 -0
  3. humaneval_results_105027.json +1498 -0
  4. run_6lane.py +90 -0
  5. run_full_benchmark.py +101 -0
README.md CHANGED
@@ -1,3 +1,167 @@
1
- ---
2
- license: apache-2.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: en
3
+ license: mit
4
+ tags:
5
+ - code-generation
6
+ - multi-model
7
+ - routing
8
+ - humaneval
9
+ - constellation
10
+ - orchestration
11
+ datasets:
12
+ - openai/openai_humaneval
13
+ metrics:
14
+ - pass@1
15
+ model-index:
16
+ - name: HyperNet N1 SDC
17
+ results:
18
+ - task:
19
+ type: code-generation
20
+ name: Code Generation
21
+ dataset:
22
+ type: openai/openai_humaneval
23
+ name: HumanEval
24
+ metrics:
25
+ - type: pass@1
26
+ value: 98.2
27
+ name: Constellation (At Least One Correct)
28
+ - type: pass@1
29
+ value: 97.0
30
+ name: Claude (claude-sonnet-4)
31
+ - type: pass@1
32
+ value: 87.8
33
+ name: Lola (GPT-4o)
34
+ - type: pass@1
35
+ value: 87.8
36
+ name: Kimi (Moonshot)
37
+ - type: pass@1
38
+ value: 85.4
39
+ name: Grok (grok-2)
40
+ - type: pass@1
41
+ value: 83.5
42
+ name: Deep (Llama-4)
43
+ ---
44
+
45
+ # HyperNet N1 SDC
46
+
47
+ **Multi-model routing architecture for AI constellation orchestration.**
48
+
49
+ HyperNet N1 SDC (Secure Discovery Channel) is not a model — it is a routing layer that orchestrates multiple AI models under human governance, achieving higher effective accuracy than any single model alone.
50
+
51
+ ## Official HumanEval Benchmark Results
52
+
53
+ **Date:** November 29, 2025
54
+ **Dataset:** Official OpenAI HumanEval (164 problems)
55
+ **Source:** huggingface.co/datasets/openai/openai_humaneval
56
+
57
+ ### Individual Lane Performance (pass@1)
58
+
59
+ | Lane | Model | Pass | Score |
60
+ |------|-------|------|-------|
61
+ | Claude | claude-sonnet-4 | 159/164 | **97.0%** |
62
+ | Lola | GPT-4o | 144/164 | **87.8%** |
63
+ | Kimi | Moonshot kimi-latest | 144/164 | **87.8%** |
64
+ | Grok | grok-2-1212 | 140/164 | **85.4%** |
65
+ | Deep | Llama-4-Maverick-17B | 137/164 | **83.5%** |
66
+
67
+ ### Constellation Consensus Metrics (5 Lanes)
68
+
69
+ | Metric | Count | Rate |
70
+ |--------|-------|------|
71
+ | **Unanimous Pass (5/5)** | 118/164 | 72.0% |
72
+ | **Majority Pass (3+/5)** | 147/164 | 89.6% |
73
+ | **At Least One Correct (1+/5)** | 161/164 | **98.2%** |
74
+ | Unanimous Fail (0/5) | 3/164 | 1.8% |
75
+ | Lane Independence | — | 26.2% disagreement |
76
+
77
+ ### Key Finding
78
+
79
+ | Metric | Best Single Model | Constellation |
80
+ |--------|-------------------|---------------|
81
+ | Accuracy | 97.0% (Claude) | **98.2%** |
82
+ | Problems Unsolved | 5 | **3** |
83
+
84
+ The constellation achieves higher coverage than any individual model.
85
+
86
+ ## Infrastructure
87
+
88
+ | Spec | Value |
89
+ |------|-------|
90
+ | **Instance** | AWS t3.small |
91
+ | **vCPUs** | 2 |
92
+ | **RAM** | 2 GB |
93
+ | **GPU** | None |
94
+ | **Training** | None required |
95
+ | **Setup Time** | < 1 hour |
96
+ | **Benchmark Cost** | < $20 |
97
+
98
+ ## Methodology
99
+
100
+ - **Dataset:** Official OpenAI HumanEval from HuggingFace (`openai/openai_humaneval`)
101
+ - **Problems:** 164 (full benchmark, no sampling)
102
+ - **Evaluation:** pass@1 (single attempt per problem)
103
+ - **Grading:** Automated code execution against official unit tests
104
+ - **Execution:** Python subprocess with 10-second timeout
105
+ - **No cherry-picking:** Every problem, every lane, logged
106
+
107
+ ## Architecture
108
+ ```
109
+ ┌─────────────────┐
110
+ │ CPN (Human) │
111
+ │ Steve Kawa │
112
+ └────────┬────────┘
113
+ │
114
+ ┌────────▼────────┐
115
+ │ HyperNet N1 │
116
+ │ SDC Router │
117
+ └────────┬────────┘
118
+ │
119
+ ┌──────────┬─────────┼─────────┬──────────┐
120
+ ▼ ▼ ▼ ▼ ▼
121
+ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐
122
+ │ Lola │ │Claude│ │ Grok │ │ Deep │ │ Kimi │
123
+ │GPT-4o│ │Sonnet│ │grok-2│ │Llama4│ │ Moon │
124
+ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘
125
+ ```
126
+
127
+ ## Reproduce
128
+ ```bash
129
+ # Clone this repo
130
+ git clone https://huggingface.co/NameONEStudios/hypernet-n1-sdc
131
+
132
+ # Install dependencies
133
+ pip install datasets requests
134
+
135
+ # Start the router (requires API keys)
136
+ python N1_Router.py
137
+
138
+ # Run benchmark
139
+ python run_6lane.py
140
+ ```
141
+
142
+ ## Files
143
+
144
+ - `humaneval_6lane_123525.json` — Raw results (5-lane run)
145
+ - `humaneval_results_105027.json` — Raw results (4-lane run)
146
+ - `run_6lane.py` — Benchmark script
147
+ - `run_full_benchmark.py` — Alternative benchmark script
148
+
149
+ ## Citation
150
+ ```bibtex
151
+ @misc{hypernet2025,
152
+ author = {Kawa, Steve},
153
+ title = {HyperNet N1 SDC: Multi-Model Routing Architecture},
154
+ year = {2025},
155
+ publisher = {NameONE Studios Inc.},
156
+ howpublished = {\url{https://huggingface.co/NameONEStudios/hypernet-n1-sdc}}
157
+ }
158
+ ```
159
+
160
+ ## License
161
+
162
+ MIT License — NameONE Studios Inc.
163
+
164
+ ## Contact
165
+
166
+ Steve Kawa — CPN (Central Processing Node)
167
+ NameONE Studios Inc.
humaneval_6lane_123525.json ADDED
@@ -0,0 +1,1834 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "lanes": {
3
+ "lola": {
4
+ "pass": 144,
5
+ "fail": 20
6
+ },
7
+ "claude": {
8
+ "pass": 159,
9
+ "fail": 5
10
+ },
11
+ "grok": {
12
+ "pass": 140,
13
+ "fail": 24
14
+ },
15
+ "deep": {
16
+ "pass": 137,
17
+ "fail": 27
18
+ },
19
+ "gemini": {
20
+ "pass": 0,
21
+ "fail": 164
22
+ },
23
+ "kimi": {
24
+ "pass": 144,
25
+ "fail": 20
26
+ }
27
+ },
28
+ "problems": [
29
+ {
30
+ "task_id": "HumanEval/0",
31
+ "lanes": {
32
+ "lola": true,
33
+ "claude": true,
34
+ "grok": true,
35
+ "deep": true,
36
+ "gemini": false,
37
+ "kimi": true
38
+ }
39
+ },
40
+ {
41
+ "task_id": "HumanEval/1",
42
+ "lanes": {
43
+ "lola": true,
44
+ "claude": true,
45
+ "grok": true,
46
+ "deep": true,
47
+ "gemini": false,
48
+ "kimi": true
49
+ }
50
+ },
51
+ {
52
+ "task_id": "HumanEval/2",
53
+ "lanes": {
54
+ "lola": true,
55
+ "claude": true,
56
+ "grok": true,
57
+ "deep": true,
58
+ "gemini": false,
59
+ "kimi": true
60
+ }
61
+ },
62
+ {
63
+ "task_id": "HumanEval/3",
64
+ "lanes": {
65
+ "lola": true,
66
+ "claude": true,
67
+ "grok": true,
68
+ "deep": true,
69
+ "gemini": false,
70
+ "kimi": true
71
+ }
72
+ },
73
+ {
74
+ "task_id": "HumanEval/4",
75
+ "lanes": {
76
+ "lola": false,
77
+ "claude": true,
78
+ "grok": true,
79
+ "deep": true,
80
+ "gemini": false,
81
+ "kimi": true
82
+ }
83
+ },
84
+ {
85
+ "task_id": "HumanEval/5",
86
+ "lanes": {
87
+ "lola": true,
88
+ "claude": true,
89
+ "grok": true,
90
+ "deep": true,
91
+ "gemini": false,
92
+ "kimi": true
93
+ }
94
+ },
95
+ {
96
+ "task_id": "HumanEval/6",
97
+ "lanes": {
98
+ "lola": true,
99
+ "claude": true,
100
+ "grok": true,
101
+ "deep": true,
102
+ "gemini": false,
103
+ "kimi": true
104
+ }
105
+ },
106
+ {
107
+ "task_id": "HumanEval/7",
108
+ "lanes": {
109
+ "lola": false,
110
+ "claude": true,
111
+ "grok": true,
112
+ "deep": true,
113
+ "gemini": false,
114
+ "kimi": true
115
+ }
116
+ },
117
+ {
118
+ "task_id": "HumanEval/8",
119
+ "lanes": {
120
+ "lola": true,
121
+ "claude": true,
122
+ "grok": true,
123
+ "deep": true,
124
+ "gemini": false,
125
+ "kimi": true
126
+ }
127
+ },
128
+ {
129
+ "task_id": "HumanEval/9",
130
+ "lanes": {
131
+ "lola": true,
132
+ "claude": true,
133
+ "grok": true,
134
+ "deep": true,
135
+ "gemini": false,
136
+ "kimi": true
137
+ }
138
+ },
139
+ {
140
+ "task_id": "HumanEval/10",
141
+ "lanes": {
142
+ "lola": true,
143
+ "claude": true,
144
+ "grok": true,
145
+ "deep": false,
146
+ "gemini": false,
147
+ "kimi": true
148
+ }
149
+ },
150
+ {
151
+ "task_id": "HumanEval/11",
152
+ "lanes": {
153
+ "lola": true,
154
+ "claude": true,
155
+ "grok": true,
156
+ "deep": true,
157
+ "gemini": false,
158
+ "kimi": true
159
+ }
160
+ },
161
+ {
162
+ "task_id": "HumanEval/12",
163
+ "lanes": {
164
+ "lola": true,
165
+ "claude": true,
166
+ "grok": true,
167
+ "deep": true,
168
+ "gemini": false,
169
+ "kimi": true
170
+ }
171
+ },
172
+ {
173
+ "task_id": "HumanEval/13",
174
+ "lanes": {
175
+ "lola": true,
176
+ "claude": true,
177
+ "grok": true,
178
+ "deep": true,
179
+ "gemini": false,
180
+ "kimi": true
181
+ }
182
+ },
183
+ {
184
+ "task_id": "HumanEval/14",
185
+ "lanes": {
186
+ "lola": true,
187
+ "claude": true,
188
+ "grok": true,
189
+ "deep": true,
190
+ "gemini": false,
191
+ "kimi": true
192
+ }
193
+ },
194
+ {
195
+ "task_id": "HumanEval/15",
196
+ "lanes": {
197
+ "lola": true,
198
+ "claude": true,
199
+ "grok": true,
200
+ "deep": true,
201
+ "gemini": false,
202
+ "kimi": true
203
+ }
204
+ },
205
+ {
206
+ "task_id": "HumanEval/16",
207
+ "lanes": {
208
+ "lola": true,
209
+ "claude": true,
210
+ "grok": true,
211
+ "deep": true,
212
+ "gemini": false,
213
+ "kimi": true
214
+ }
215
+ },
216
+ {
217
+ "task_id": "HumanEval/17",
218
+ "lanes": {
219
+ "lola": true,
220
+ "claude": true,
221
+ "grok": true,
222
+ "deep": true,
223
+ "gemini": false,
224
+ "kimi": true
225
+ }
226
+ },
227
+ {
228
+ "task_id": "HumanEval/18",
229
+ "lanes": {
230
+ "lola": true,
231
+ "claude": true,
232
+ "grok": true,
233
+ "deep": true,
234
+ "gemini": false,
235
+ "kimi": true
236
+ }
237
+ },
238
+ {
239
+ "task_id": "HumanEval/19",
240
+ "lanes": {
241
+ "lola": true,
242
+ "claude": true,
243
+ "grok": true,
244
+ "deep": true,
245
+ "gemini": false,
246
+ "kimi": true
247
+ }
248
+ },
249
+ {
250
+ "task_id": "HumanEval/20",
251
+ "lanes": {
252
+ "lola": true,
253
+ "claude": true,
254
+ "grok": true,
255
+ "deep": true,
256
+ "gemini": false,
257
+ "kimi": true
258
+ }
259
+ },
260
+ {
261
+ "task_id": "HumanEval/21",
262
+ "lanes": {
263
+ "lola": true,
264
+ "claude": true,
265
+ "grok": true,
266
+ "deep": true,
267
+ "gemini": false,
268
+ "kimi": true
269
+ }
270
+ },
271
+ {
272
+ "task_id": "HumanEval/22",
273
+ "lanes": {
274
+ "lola": true,
275
+ "claude": true,
276
+ "grok": true,
277
+ "deep": true,
278
+ "gemini": false,
279
+ "kimi": true
280
+ }
281
+ },
282
+ {
283
+ "task_id": "HumanEval/23",
284
+ "lanes": {
285
+ "lola": true,
286
+ "claude": true,
287
+ "grok": true,
288
+ "deep": true,
289
+ "gemini": false,
290
+ "kimi": true
291
+ }
292
+ },
293
+ {
294
+ "task_id": "HumanEval/24",
295
+ "lanes": {
296
+ "lola": true,
297
+ "claude": true,
298
+ "grok": true,
299
+ "deep": true,
300
+ "gemini": false,
301
+ "kimi": true
302
+ }
303
+ },
304
+ {
305
+ "task_id": "HumanEval/25",
306
+ "lanes": {
307
+ "lola": true,
308
+ "claude": true,
309
+ "grok": true,
310
+ "deep": true,
311
+ "gemini": false,
312
+ "kimi": true
313
+ }
314
+ },
315
+ {
316
+ "task_id": "HumanEval/26",
317
+ "lanes": {
318
+ "lola": true,
319
+ "claude": true,
320
+ "grok": true,
321
+ "deep": false,
322
+ "gemini": false,
323
+ "kimi": true
324
+ }
325
+ },
326
+ {
327
+ "task_id": "HumanEval/27",
328
+ "lanes": {
329
+ "lola": true,
330
+ "claude": true,
331
+ "grok": true,
332
+ "deep": true,
333
+ "gemini": false,
334
+ "kimi": true
335
+ }
336
+ },
337
+ {
338
+ "task_id": "HumanEval/28",
339
+ "lanes": {
340
+ "lola": true,
341
+ "claude": true,
342
+ "grok": true,
343
+ "deep": true,
344
+ "gemini": false,
345
+ "kimi": true
346
+ }
347
+ },
348
+ {
349
+ "task_id": "HumanEval/29",
350
+ "lanes": {
351
+ "lola": true,
352
+ "claude": true,
353
+ "grok": true,
354
+ "deep": true,
355
+ "gemini": false,
356
+ "kimi": true
357
+ }
358
+ },
359
+ {
360
+ "task_id": "HumanEval/30",
361
+ "lanes": {
362
+ "lola": true,
363
+ "claude": true,
364
+ "grok": true,
365
+ "deep": true,
366
+ "gemini": false,
367
+ "kimi": true
368
+ }
369
+ },
370
+ {
371
+ "task_id": "HumanEval/31",
372
+ "lanes": {
373
+ "lola": true,
374
+ "claude": true,
375
+ "grok": true,
376
+ "deep": true,
377
+ "gemini": false,
378
+ "kimi": true
379
+ }
380
+ },
381
+ {
382
+ "task_id": "HumanEval/32",
383
+ "lanes": {
384
+ "lola": false,
385
+ "claude": true,
386
+ "grok": true,
387
+ "deep": false,
388
+ "gemini": false,
389
+ "kimi": true
390
+ }
391
+ },
392
+ {
393
+ "task_id": "HumanEval/33",
394
+ "lanes": {
395
+ "lola": true,
396
+ "claude": true,
397
+ "grok": true,
398
+ "deep": true,
399
+ "gemini": false,
400
+ "kimi": false
401
+ }
402
+ },
403
+ {
404
+ "task_id": "HumanEval/34",
405
+ "lanes": {
406
+ "lola": true,
407
+ "claude": true,
408
+ "grok": true,
409
+ "deep": true,
410
+ "gemini": false,
411
+ "kimi": true
412
+ }
413
+ },
414
+ {
415
+ "task_id": "HumanEval/35",
416
+ "lanes": {
417
+ "lola": true,
418
+ "claude": true,
419
+ "grok": true,
420
+ "deep": true,
421
+ "gemini": false,
422
+ "kimi": true
423
+ }
424
+ },
425
+ {
426
+ "task_id": "HumanEval/36",
427
+ "lanes": {
428
+ "lola": true,
429
+ "claude": true,
430
+ "grok": true,
431
+ "deep": true,
432
+ "gemini": false,
433
+ "kimi": true
434
+ }
435
+ },
436
+ {
437
+ "task_id": "HumanEval/37",
438
+ "lanes": {
439
+ "lola": true,
440
+ "claude": true,
441
+ "grok": true,
442
+ "deep": true,
443
+ "gemini": false,
444
+ "kimi": true
445
+ }
446
+ },
447
+ {
448
+ "task_id": "HumanEval/38",
449
+ "lanes": {
450
+ "lola": true,
451
+ "claude": true,
452
+ "grok": true,
453
+ "deep": true,
454
+ "gemini": false,
455
+ "kimi": true
456
+ }
457
+ },
458
+ {
459
+ "task_id": "HumanEval/39",
460
+ "lanes": {
461
+ "lola": true,
462
+ "claude": true,
463
+ "grok": true,
464
+ "deep": true,
465
+ "gemini": false,
466
+ "kimi": true
467
+ }
468
+ },
469
+ {
470
+ "task_id": "HumanEval/40",
471
+ "lanes": {
472
+ "lola": true,
473
+ "claude": true,
474
+ "grok": true,
475
+ "deep": true,
476
+ "gemini": false,
477
+ "kimi": true
478
+ }
479
+ },
480
+ {
481
+ "task_id": "HumanEval/41",
482
+ "lanes": {
483
+ "lola": false,
484
+ "claude": true,
485
+ "grok": true,
486
+ "deep": true,
487
+ "gemini": false,
488
+ "kimi": true
489
+ }
490
+ },
491
+ {
492
+ "task_id": "HumanEval/42",
493
+ "lanes": {
494
+ "lola": true,
495
+ "claude": true,
496
+ "grok": true,
497
+ "deep": true,
498
+ "gemini": false,
499
+ "kimi": true
500
+ }
501
+ },
502
+ {
503
+ "task_id": "HumanEval/43",
504
+ "lanes": {
505
+ "lola": true,
506
+ "claude": true,
507
+ "grok": true,
508
+ "deep": true,
509
+ "gemini": false,
510
+ "kimi": true
511
+ }
512
+ },
513
+ {
514
+ "task_id": "HumanEval/44",
515
+ "lanes": {
516
+ "lola": true,
517
+ "claude": true,
518
+ "grok": true,
519
+ "deep": true,
520
+ "gemini": false,
521
+ "kimi": true
522
+ }
523
+ },
524
+ {
525
+ "task_id": "HumanEval/45",
526
+ "lanes": {
527
+ "lola": true,
528
+ "claude": true,
529
+ "grok": true,
530
+ "deep": true,
531
+ "gemini": false,
532
+ "kimi": true
533
+ }
534
+ },
535
+ {
536
+ "task_id": "HumanEval/46",
537
+ "lanes": {
538
+ "lola": true,
539
+ "claude": true,
540
+ "grok": true,
541
+ "deep": true,
542
+ "gemini": false,
543
+ "kimi": true
544
+ }
545
+ },
546
+ {
547
+ "task_id": "HumanEval/47",
548
+ "lanes": {
549
+ "lola": true,
550
+ "claude": true,
551
+ "grok": true,
552
+ "deep": true,
553
+ "gemini": false,
554
+ "kimi": true
555
+ }
556
+ },
557
+ {
558
+ "task_id": "HumanEval/48",
559
+ "lanes": {
560
+ "lola": true,
561
+ "claude": true,
562
+ "grok": true,
563
+ "deep": true,
564
+ "gemini": false,
565
+ "kimi": true
566
+ }
567
+ },
568
+ {
569
+ "task_id": "HumanEval/49",
570
+ "lanes": {
571
+ "lola": true,
572
+ "claude": true,
573
+ "grok": true,
574
+ "deep": true,
575
+ "gemini": false,
576
+ "kimi": true
577
+ }
578
+ },
579
+ {
580
+ "task_id": "HumanEval/50",
581
+ "lanes": {
582
+ "lola": true,
583
+ "claude": true,
584
+ "grok": true,
585
+ "deep": true,
586
+ "gemini": false,
587
+ "kimi": true
588
+ }
589
+ },
590
+ {
591
+ "task_id": "HumanEval/51",
592
+ "lanes": {
593
+ "lola": true,
594
+ "claude": true,
595
+ "grok": true,
596
+ "deep": true,
597
+ "gemini": false,
598
+ "kimi": true
599
+ }
600
+ },
601
+ {
602
+ "task_id": "HumanEval/52",
603
+ "lanes": {
604
+ "lola": true,
605
+ "claude": true,
606
+ "grok": true,
607
+ "deep": true,
608
+ "gemini": false,
609
+ "kimi": true
610
+ }
611
+ },
612
+ {
613
+ "task_id": "HumanEval/53",
614
+ "lanes": {
615
+ "lola": true,
616
+ "claude": true,
617
+ "grok": true,
618
+ "deep": true,
619
+ "gemini": false,
620
+ "kimi": true
621
+ }
622
+ },
623
+ {
624
+ "task_id": "HumanEval/54",
625
+ "lanes": {
626
+ "lola": true,
627
+ "claude": true,
628
+ "grok": true,
629
+ "deep": true,
630
+ "gemini": false,
631
+ "kimi": true
632
+ }
633
+ },
634
+ {
635
+ "task_id": "HumanEval/55",
636
+ "lanes": {
637
+ "lola": true,
638
+ "claude": true,
639
+ "grok": true,
640
+ "deep": true,
641
+ "gemini": false,
642
+ "kimi": true
643
+ }
644
+ },
645
+ {
646
+ "task_id": "HumanEval/56",
647
+ "lanes": {
648
+ "lola": true,
649
+ "claude": true,
650
+ "grok": true,
651
+ "deep": true,
652
+ "gemini": false,
653
+ "kimi": true
654
+ }
655
+ },
656
+ {
657
+ "task_id": "HumanEval/57",
658
+ "lanes": {
659
+ "lola": true,
660
+ "claude": true,
661
+ "grok": true,
662
+ "deep": true,
663
+ "gemini": false,
664
+ "kimi": true
665
+ }
666
+ },
667
+ {
668
+ "task_id": "HumanEval/58",
669
+ "lanes": {
670
+ "lola": true,
671
+ "claude": true,
672
+ "grok": true,
673
+ "deep": true,
674
+ "gemini": false,
675
+ "kimi": true
676
+ }
677
+ },
678
+ {
679
+ "task_id": "HumanEval/59",
680
+ "lanes": {
681
+ "lola": true,
682
+ "claude": true,
683
+ "grok": true,
684
+ "deep": true,
685
+ "gemini": false,
686
+ "kimi": true
687
+ }
688
+ },
689
+ {
690
+ "task_id": "HumanEval/60",
691
+ "lanes": {
692
+ "lola": true,
693
+ "claude": true,
694
+ "grok": true,
695
+ "deep": true,
696
+ "gemini": false,
697
+ "kimi": true
698
+ }
699
+ },
700
+ {
701
+ "task_id": "HumanEval/61",
702
+ "lanes": {
703
+ "lola": true,
704
+ "claude": true,
705
+ "grok": true,
706
+ "deep": true,
707
+ "gemini": false,
708
+ "kimi": true
709
+ }
710
+ },
711
+ {
712
+ "task_id": "HumanEval/62",
713
+ "lanes": {
714
+ "lola": true,
715
+ "claude": true,
716
+ "grok": true,
717
+ "deep": true,
718
+ "gemini": false,
719
+ "kimi": true
720
+ }
721
+ },
722
+ {
723
+ "task_id": "HumanEval/63",
724
+ "lanes": {
725
+ "lola": true,
726
+ "claude": true,
727
+ "grok": true,
728
+ "deep": true,
729
+ "gemini": false,
730
+ "kimi": true
731
+ }
732
+ },
733
+ {
734
+ "task_id": "HumanEval/64",
735
+ "lanes": {
736
+ "lola": true,
737
+ "claude": true,
738
+ "grok": true,
739
+ "deep": true,
740
+ "gemini": false,
741
+ "kimi": true
742
+ }
743
+ },
744
+ {
745
+ "task_id": "HumanEval/65",
746
+ "lanes": {
747
+ "lola": false,
748
+ "claude": true,
749
+ "grok": true,
750
+ "deep": false,
751
+ "gemini": false,
752
+ "kimi": false
753
+ }
754
+ },
755
+ {
756
+ "task_id": "HumanEval/66",
757
+ "lanes": {
758
+ "lola": true,
759
+ "claude": true,
760
+ "grok": true,
761
+ "deep": true,
762
+ "gemini": false,
763
+ "kimi": true
764
+ }
765
+ },
766
+ {
767
+ "task_id": "HumanEval/67",
768
+ "lanes": {
769
+ "lola": true,
770
+ "claude": true,
771
+ "grok": true,
772
+ "deep": true,
773
+ "gemini": false,
774
+ "kimi": true
775
+ }
776
+ },
777
+ {
778
+ "task_id": "HumanEval/68",
779
+ "lanes": {
780
+ "lola": true,
781
+ "claude": true,
782
+ "grok": true,
783
+ "deep": true,
784
+ "gemini": false,
785
+ "kimi": true
786
+ }
787
+ },
788
+ {
789
+ "task_id": "HumanEval/69",
790
+ "lanes": {
791
+ "lola": true,
792
+ "claude": true,
793
+ "grok": true,
794
+ "deep": true,
795
+ "gemini": false,
796
+ "kimi": true
797
+ }
798
+ },
799
+ {
800
+ "task_id": "HumanEval/70",
801
+ "lanes": {
802
+ "lola": true,
803
+ "claude": true,
804
+ "grok": true,
805
+ "deep": true,
806
+ "gemini": false,
807
+ "kimi": true
808
+ }
809
+ },
810
+ {
811
+ "task_id": "HumanEval/71",
812
+ "lanes": {
813
+ "lola": true,
814
+ "claude": true,
815
+ "grok": true,
816
+ "deep": true,
817
+ "gemini": false,
818
+ "kimi": true
819
+ }
820
+ },
821
+ {
822
+ "task_id": "HumanEval/72",
823
+ "lanes": {
824
+ "lola": true,
825
+ "claude": true,
826
+ "grok": true,
827
+ "deep": true,
828
+ "gemini": false,
829
+ "kimi": true
830
+ }
831
+ },
832
+ {
833
+ "task_id": "HumanEval/73",
834
+ "lanes": {
835
+ "lola": true,
836
+ "claude": true,
837
+ "grok": true,
838
+ "deep": true,
839
+ "gemini": false,
840
+ "kimi": true
841
+ }
842
+ },
843
+ {
844
+ "task_id": "HumanEval/74",
845
+ "lanes": {
846
+ "lola": true,
847
+ "claude": true,
848
+ "grok": true,
849
+ "deep": true,
850
+ "gemini": false,
851
+ "kimi": true
852
+ }
853
+ },
854
+ {
855
+ "task_id": "HumanEval/75",
856
+ "lanes": {
857
+ "lola": true,
858
+ "claude": true,
859
+ "grok": true,
860
+ "deep": true,
861
+ "gemini": false,
862
+ "kimi": false
863
+ }
864
+ },
865
+ {
866
+ "task_id": "HumanEval/76",
867
+ "lanes": {
868
+ "lola": true,
869
+ "claude": true,
870
+ "grok": true,
871
+ "deep": true,
872
+ "gemini": false,
873
+ "kimi": true
874
+ }
875
+ },
876
+ {
877
+ "task_id": "HumanEval/77",
878
+ "lanes": {
879
+ "lola": false,
880
+ "claude": true,
881
+ "grok": false,
882
+ "deep": true,
883
+ "gemini": false,
884
+ "kimi": false
885
+ }
886
+ },
887
+ {
888
+ "task_id": "HumanEval/78",
889
+ "lanes": {
890
+ "lola": true,
891
+ "claude": true,
892
+ "grok": true,
893
+ "deep": true,
894
+ "gemini": false,
895
+ "kimi": true
896
+ }
897
+ },
898
+ {
899
+ "task_id": "HumanEval/79",
900
+ "lanes": {
901
+ "lola": true,
902
+ "claude": true,
903
+ "grok": true,
904
+ "deep": true,
905
+ "gemini": false,
906
+ "kimi": true
907
+ }
908
+ },
909
+ {
910
+ "task_id": "HumanEval/80",
911
+ "lanes": {
912
+ "lola": true,
913
+ "claude": true,
914
+ "grok": true,
915
+ "deep": true,
916
+ "gemini": false,
917
+ "kimi": true
918
+ }
919
+ },
920
+ {
921
+ "task_id": "HumanEval/81",
922
+ "lanes": {
923
+ "lola": true,
924
+ "claude": true,
925
+ "grok": true,
926
+ "deep": true,
927
+ "gemini": false,
928
+ "kimi": false
929
+ }
930
+ },
931
+ {
932
+ "task_id": "HumanEval/82",
933
+ "lanes": {
934
+ "lola": true,
935
+ "claude": true,
936
+ "grok": true,
937
+ "deep": true,
938
+ "gemini": false,
939
+ "kimi": true
940
+ }
941
+ },
942
+ {
943
+ "task_id": "HumanEval/83",
944
+ "lanes": {
945
+ "lola": false,
946
+ "claude": false,
947
+ "grok": false,
948
+ "deep": true,
949
+ "gemini": false,
950
+ "kimi": false
951
+ }
952
+ },
953
+ {
954
+ "task_id": "HumanEval/84",
955
+ "lanes": {
956
+ "lola": true,
957
+ "claude": true,
958
+ "grok": true,
959
+ "deep": false,
960
+ "gemini": false,
961
+ "kimi": true
962
+ }
963
+ },
964
+ {
965
+ "task_id": "HumanEval/85",
966
+ "lanes": {
967
+ "lola": true,
968
+ "claude": true,
969
+ "grok": true,
970
+ "deep": true,
971
+ "gemini": false,
972
+ "kimi": true
973
+ }
974
+ },
975
+ {
976
+ "task_id": "HumanEval/86",
977
+ "lanes": {
978
+ "lola": true,
979
+ "claude": true,
980
+ "grok": false,
981
+ "deep": true,
982
+ "gemini": false,
983
+ "kimi": false
984
+ }
985
+ },
986
+ {
987
+ "task_id": "HumanEval/87",
988
+ "lanes": {
989
+ "lola": true,
990
+ "claude": true,
991
+ "grok": true,
992
+ "deep": true,
993
+ "gemini": false,
994
+ "kimi": true
995
+ }
996
+ },
997
+ {
998
+ "task_id": "HumanEval/88",
999
+ "lanes": {
1000
+ "lola": true,
1001
+ "claude": true,
1002
+ "grok": true,
1003
+ "deep": true,
1004
+ "gemini": false,
1005
+ "kimi": true
1006
+ }
1007
+ },
1008
+ {
1009
+ "task_id": "HumanEval/89",
1010
+ "lanes": {
1011
+ "lola": true,
1012
+ "claude": true,
1013
+ "grok": true,
1014
+ "deep": true,
1015
+ "gemini": false,
1016
+ "kimi": true
1017
+ }
1018
+ },
1019
+ {
1020
+ "task_id": "HumanEval/90",
1021
+ "lanes": {
1022
+ "lola": true,
1023
+ "claude": true,
1024
+ "grok": true,
1025
+ "deep": true,
1026
+ "gemini": false,
1027
+ "kimi": true
1028
+ }
1029
+ },
1030
+ {
1031
+ "task_id": "HumanEval/91",
1032
+ "lanes": {
1033
+ "lola": false,
1034
+ "claude": true,
1035
+ "grok": false,
1036
+ "deep": false,
1037
+ "gemini": false,
1038
+ "kimi": false
1039
+ }
1040
+ },
1041
+ {
1042
+ "task_id": "HumanEval/92",
1043
+ "lanes": {
1044
+ "lola": true,
1045
+ "claude": true,
1046
+ "grok": true,
1047
+ "deep": true,
1048
+ "gemini": false,
1049
+ "kimi": true
1050
+ }
1051
+ },
1052
+ {
1053
+ "task_id": "HumanEval/93",
1054
+ "lanes": {
1055
+ "lola": false,
1056
+ "claude": false,
1057
+ "grok": false,
1058
+ "deep": true,
1059
+ "gemini": false,
1060
+ "kimi": true
1061
+ }
1062
+ },
1063
+ {
1064
+ "task_id": "HumanEval/94",
1065
+ "lanes": {
1066
+ "lola": true,
1067
+ "claude": true,
1068
+ "grok": true,
1069
+ "deep": true,
1070
+ "gemini": false,
1071
+ "kimi": true
1072
+ }
1073
+ },
1074
+ {
1075
+ "task_id": "HumanEval/95",
1076
+ "lanes": {
1077
+ "lola": true,
1078
+ "claude": true,
1079
+ "grok": true,
1080
+ "deep": true,
1081
+ "gemini": false,
1082
+ "kimi": true
1083
+ }
1084
+ },
1085
+ {
1086
+ "task_id": "HumanEval/96",
1087
+ "lanes": {
1088
+ "lola": true,
1089
+ "claude": true,
1090
+ "grok": true,
1091
+ "deep": true,
1092
+ "gemini": false,
1093
+ "kimi": true
1094
+ }
1095
+ },
1096
+ {
1097
+ "task_id": "HumanEval/97",
1098
+ "lanes": {
1099
+ "lola": false,
1100
+ "claude": true,
1101
+ "grok": true,
1102
+ "deep": true,
1103
+ "gemini": false,
1104
+ "kimi": true
1105
+ }
1106
+ },
1107
+ {
1108
+ "task_id": "HumanEval/98",
1109
+ "lanes": {
1110
+ "lola": true,
1111
+ "claude": true,
1112
+ "grok": true,
1113
+ "deep": true,
1114
+ "gemini": false,
1115
+ "kimi": true
1116
+ }
1117
+ },
1118
+ {
1119
+ "task_id": "HumanEval/99",
1120
+ "lanes": {
1121
+ "lola": true,
1122
+ "claude": true,
1123
+ "grok": true,
1124
+ "deep": false,
1125
+ "gemini": false,
1126
+ "kimi": true
1127
+ }
1128
+ },
1129
+ {
1130
+ "task_id": "HumanEval/100",
1131
+ "lanes": {
1132
+ "lola": true,
1133
+ "claude": true,
1134
+ "grok": true,
1135
+ "deep": false,
1136
+ "gemini": false,
1137
+ "kimi": true
1138
+ }
1139
+ },
1140
+ {
1141
+ "task_id": "HumanEval/101",
1142
+ "lanes": {
1143
+ "lola": true,
1144
+ "claude": true,
1145
+ "grok": true,
1146
+ "deep": true,
1147
+ "gemini": false,
1148
+ "kimi": true
1149
+ }
1150
+ },
1151
+ {
1152
+ "task_id": "HumanEval/102",
1153
+ "lanes": {
1154
+ "lola": true,
1155
+ "claude": true,
1156
+ "grok": false,
1157
+ "deep": true,
1158
+ "gemini": false,
1159
+ "kimi": true
1160
+ }
1161
+ },
1162
+ {
1163
+ "task_id": "HumanEval/103",
1164
+ "lanes": {
1165
+ "lola": true,
1166
+ "claude": true,
1167
+ "grok": true,
1168
+ "deep": true,
1169
+ "gemini": false,
1170
+ "kimi": true
1171
+ }
1172
+ },
1173
+ {
1174
+ "task_id": "HumanEval/104",
1175
+ "lanes": {
1176
+ "lola": true,
1177
+ "claude": true,
1178
+ "grok": true,
1179
+ "deep": true,
1180
+ "gemini": false,
1181
+ "kimi": true
1182
+ }
1183
+ },
1184
+ {
1185
+ "task_id": "HumanEval/105",
1186
+ "lanes": {
1187
+ "lola": true,
1188
+ "claude": true,
1189
+ "grok": true,
1190
+ "deep": true,
1191
+ "gemini": false,
1192
+ "kimi": true
1193
+ }
1194
+ },
1195
+ {
1196
+ "task_id": "HumanEval/106",
1197
+ "lanes": {
1198
+ "lola": true,
1199
+ "claude": true,
1200
+ "grok": true,
1201
+ "deep": true,
1202
+ "gemini": false,
1203
+ "kimi": true
1204
+ }
1205
+ },
1206
+ {
1207
+ "task_id": "HumanEval/107",
1208
+ "lanes": {
1209
+ "lola": true,
1210
+ "claude": true,
1211
+ "grok": true,
1212
+ "deep": true,
1213
+ "gemini": false,
1214
+ "kimi": true
1215
+ }
1216
+ },
1217
+ {
1218
+ "task_id": "HumanEval/108",
1219
+ "lanes": {
1220
+ "lola": true,
1221
+ "claude": true,
1222
+ "grok": false,
1223
+ "deep": false,
1224
+ "gemini": false,
1225
+ "kimi": false
1226
+ }
1227
+ },
1228
+ {
1229
+ "task_id": "HumanEval/109",
1230
+ "lanes": {
1231
+ "lola": true,
1232
+ "claude": true,
1233
+ "grok": true,
1234
+ "deep": true,
1235
+ "gemini": false,
1236
+ "kimi": true
1237
+ }
1238
+ },
1239
+ {
1240
+ "task_id": "HumanEval/110",
1241
+ "lanes": {
1242
+ "lola": true,
1243
+ "claude": true,
1244
+ "grok": true,
1245
+ "deep": true,
1246
+ "gemini": false,
1247
+ "kimi": true
1248
+ }
1249
+ },
1250
+ {
1251
+ "task_id": "HumanEval/111",
1252
+ "lanes": {
1253
+ "lola": true,
1254
+ "claude": true,
1255
+ "grok": true,
1256
+ "deep": true,
1257
+ "gemini": false,
1258
+ "kimi": true
1259
+ }
1260
+ },
1261
+ {
1262
+ "task_id": "HumanEval/112",
1263
+ "lanes": {
1264
+ "lola": true,
1265
+ "claude": true,
1266
+ "grok": true,
1267
+ "deep": true,
1268
+ "gemini": false,
1269
+ "kimi": true
1270
+ }
1271
+ },
1272
+ {
1273
+ "task_id": "HumanEval/113",
1274
+ "lanes": {
1275
+ "lola": true,
1276
+ "claude": true,
1277
+ "grok": true,
1278
+ "deep": true,
1279
+ "gemini": false,
1280
+ "kimi": true
1281
+ }
1282
+ },
1283
+ {
1284
+ "task_id": "HumanEval/114",
1285
+ "lanes": {
1286
+ "lola": true,
1287
+ "claude": true,
1288
+ "grok": true,
1289
+ "deep": false,
1290
+ "gemini": false,
1291
+ "kimi": true
1292
+ }
1293
+ },
1294
+ {
1295
+ "task_id": "HumanEval/115",
1296
+ "lanes": {
1297
+ "lola": false,
1298
+ "claude": true,
1299
+ "grok": false,
1300
+ "deep": false,
1301
+ "gemini": false,
1302
+ "kimi": true
1303
+ }
1304
+ },
1305
+ {
1306
+ "task_id": "HumanEval/116",
1307
+ "lanes": {
1308
+ "lola": true,
1309
+ "claude": true,
1310
+ "grok": true,
1311
+ "deep": false,
1312
+ "gemini": false,
1313
+ "kimi": true
1314
+ }
1315
+ },
1316
+ {
1317
+ "task_id": "HumanEval/117",
1318
+ "lanes": {
1319
+ "lola": true,
1320
+ "claude": true,
1321
+ "grok": true,
1322
+ "deep": true,
1323
+ "gemini": false,
1324
+ "kimi": true
1325
+ }
1326
+ },
1327
+ {
1328
+ "task_id": "HumanEval/118",
1329
+ "lanes": {
1330
+ "lola": true,
1331
+ "claude": true,
1332
+ "grok": false,
1333
+ "deep": true,
1334
+ "gemini": false,
1335
+ "kimi": true
1336
+ }
1337
+ },
1338
+ {
1339
+ "task_id": "HumanEval/119",
1340
+ "lanes": {
1341
+ "lola": false,
1342
+ "claude": true,
1343
+ "grok": true,
1344
+ "deep": false,
1345
+ "gemini": false,
1346
+ "kimi": true
1347
+ }
1348
+ },
1349
+ {
1350
+ "task_id": "HumanEval/120",
1351
+ "lanes": {
1352
+ "lola": true,
1353
+ "claude": true,
1354
+ "grok": false,
1355
+ "deep": false,
1356
+ "gemini": false,
1357
+ "kimi": false
1358
+ }
1359
+ },
1360
+ {
1361
+ "task_id": "HumanEval/121",
1362
+ "lanes": {
1363
+ "lola": true,
1364
+ "claude": true,
1365
+ "grok": false,
1366
+ "deep": true,
1367
+ "gemini": false,
1368
+ "kimi": true
1369
+ }
1370
+ },
1371
+ {
1372
+ "task_id": "HumanEval/122",
1373
+ "lanes": {
1374
+ "lola": true,
1375
+ "claude": true,
1376
+ "grok": true,
1377
+ "deep": true,
1378
+ "gemini": false,
1379
+ "kimi": true
1380
+ }
1381
+ },
1382
+ {
1383
+ "task_id": "HumanEval/123",
1384
+ "lanes": {
1385
+ "lola": true,
1386
+ "claude": true,
1387
+ "grok": true,
1388
+ "deep": true,
1389
+ "gemini": false,
1390
+ "kimi": true
1391
+ }
1392
+ },
1393
+ {
1394
+ "task_id": "HumanEval/124",
1395
+ "lanes": {
1396
+ "lola": true,
1397
+ "claude": true,
1398
+ "grok": true,
1399
+ "deep": true,
1400
+ "gemini": false,
1401
+ "kimi": true
1402
+ }
1403
+ },
1404
+ {
1405
+ "task_id": "HumanEval/125",
1406
+ "lanes": {
1407
+ "lola": false,
1408
+ "claude": true,
1409
+ "grok": false,
1410
+ "deep": true,
1411
+ "gemini": false,
1412
+ "kimi": true
1413
+ }
1414
+ },
1415
+ {
1416
+ "task_id": "HumanEval/126",
1417
+ "lanes": {
1418
+ "lola": true,
1419
+ "claude": true,
1420
+ "grok": false,
1421
+ "deep": true,
1422
+ "gemini": false,
1423
+ "kimi": true
1424
+ }
1425
+ },
1426
+ {
1427
+ "task_id": "HumanEval/127",
1428
+ "lanes": {
1429
+ "lola": true,
1430
+ "claude": true,
1431
+ "grok": false,
1432
+ "deep": false,
1433
+ "gemini": false,
1434
+ "kimi": false
1435
+ }
1436
+ },
1437
+ {
1438
+ "task_id": "HumanEval/128",
1439
+ "lanes": {
1440
+ "lola": true,
1441
+ "claude": true,
1442
+ "grok": true,
1443
+ "deep": true,
1444
+ "gemini": false,
1445
+ "kimi": true
1446
+ }
1447
+ },
1448
+ {
1449
+ "task_id": "HumanEval/129",
1450
+ "lanes": {
1451
+ "lola": true,
1452
+ "claude": true,
1453
+ "grok": true,
1454
+ "deep": false,
1455
+ "gemini": false,
1456
+ "kimi": false
1457
+ }
1458
+ },
1459
+ {
1460
+ "task_id": "HumanEval/130",
1461
+ "lanes": {
1462
+ "lola": false,
1463
+ "claude": false,
1464
+ "grok": false,
1465
+ "deep": false,
1466
+ "gemini": false,
1467
+ "kimi": false
1468
+ }
1469
+ },
1470
+ {
1471
+ "task_id": "HumanEval/131",
1472
+ "lanes": {
1473
+ "lola": true,
1474
+ "claude": true,
1475
+ "grok": true,
1476
+ "deep": true,
1477
+ "gemini": false,
1478
+ "kimi": true
1479
+ }
1480
+ },
1481
+ {
1482
+ "task_id": "HumanEval/132",
1483
+ "lanes": {
1484
+ "lola": false,
1485
+ "claude": false,
1486
+ "grok": false,
1487
+ "deep": false,
1488
+ "gemini": false,
1489
+ "kimi": false
1490
+ }
1491
+ },
1492
+ {
1493
+ "task_id": "HumanEval/133",
1494
+ "lanes": {
1495
+ "lola": true,
1496
+ "claude": true,
1497
+ "grok": true,
1498
+ "deep": true,
1499
+ "gemini": false,
1500
+ "kimi": true
1501
+ }
1502
+ },
1503
+ {
1504
+ "task_id": "HumanEval/134",
1505
+ "lanes": {
1506
+ "lola": false,
1507
+ "claude": true,
1508
+ "grok": false,
1509
+ "deep": false,
1510
+ "gemini": false,
1511
+ "kimi": false
1512
+ }
1513
+ },
1514
+ {
1515
+ "task_id": "HumanEval/135",
1516
+ "lanes": {
1517
+ "lola": true,
1518
+ "claude": true,
1519
+ "grok": true,
1520
+ "deep": true,
1521
+ "gemini": false,
1522
+ "kimi": false
1523
+ }
1524
+ },
1525
+ {
1526
+ "task_id": "HumanEval/136",
1527
+ "lanes": {
1528
+ "lola": true,
1529
+ "claude": true,
1530
+ "grok": true,
1531
+ "deep": true,
1532
+ "gemini": false,
1533
+ "kimi": true
1534
+ }
1535
+ },
1536
+ {
1537
+ "task_id": "HumanEval/137",
1538
+ "lanes": {
1539
+ "lola": true,
1540
+ "claude": true,
1541
+ "grok": false,
1542
+ "deep": false,
1543
+ "gemini": false,
1544
+ "kimi": true
1545
+ }
1546
+ },
1547
+ {
1548
+ "task_id": "HumanEval/138",
1549
+ "lanes": {
1550
+ "lola": true,
1551
+ "claude": true,
1552
+ "grok": true,
1553
+ "deep": true,
1554
+ "gemini": false,
1555
+ "kimi": true
1556
+ }
1557
+ },
1558
+ {
1559
+ "task_id": "HumanEval/139",
1560
+ "lanes": {
1561
+ "lola": true,
1562
+ "claude": true,
1563
+ "grok": true,
1564
+ "deep": true,
1565
+ "gemini": false,
1566
+ "kimi": true
1567
+ }
1568
+ },
1569
+ {
1570
+ "task_id": "HumanEval/140",
1571
+ "lanes": {
1572
+ "lola": false,
1573
+ "claude": true,
1574
+ "grok": false,
1575
+ "deep": false,
1576
+ "gemini": false,
1577
+ "kimi": true
1578
+ }
1579
+ },
1580
+ {
1581
+ "task_id": "HumanEval/141",
1582
+ "lanes": {
1583
+ "lola": true,
1584
+ "claude": true,
1585
+ "grok": true,
1586
+ "deep": true,
1587
+ "gemini": false,
1588
+ "kimi": true
1589
+ }
1590
+ },
1591
+ {
1592
+ "task_id": "HumanEval/142",
1593
+ "lanes": {
1594
+ "lola": true,
1595
+ "claude": true,
1596
+ "grok": false,
1597
+ "deep": true,
1598
+ "gemini": false,
1599
+ "kimi": true
1600
+ }
1601
+ },
1602
+ {
1603
+ "task_id": "HumanEval/143",
1604
+ "lanes": {
1605
+ "lola": true,
1606
+ "claude": true,
1607
+ "grok": true,
1608
+ "deep": true,
1609
+ "gemini": false,
1610
+ "kimi": true
1611
+ }
1612
+ },
1613
+ {
1614
+ "task_id": "HumanEval/144",
1615
+ "lanes": {
1616
+ "lola": true,
1617
+ "claude": true,
1618
+ "grok": true,
1619
+ "deep": true,
1620
+ "gemini": false,
1621
+ "kimi": true
1622
+ }
1623
+ },
1624
+ {
1625
+ "task_id": "HumanEval/145",
1626
+ "lanes": {
1627
+ "lola": false,
1628
+ "claude": false,
1629
+ "grok": false,
1630
+ "deep": false,
1631
+ "gemini": false,
1632
+ "kimi": false
1633
+ }
1634
+ },
1635
+ {
1636
+ "task_id": "HumanEval/146",
1637
+ "lanes": {
1638
+ "lola": true,
1639
+ "claude": true,
1640
+ "grok": true,
1641
+ "deep": false,
1642
+ "gemini": false,
1643
+ "kimi": true
1644
+ }
1645
+ },
1646
+ {
1647
+ "task_id": "HumanEval/147",
1648
+ "lanes": {
1649
+ "lola": true,
1650
+ "claude": true,
1651
+ "grok": true,
1652
+ "deep": false,
1653
+ "gemini": false,
1654
+ "kimi": true
1655
+ }
1656
+ },
1657
+ {
1658
+ "task_id": "HumanEval/148",
1659
+ "lanes": {
1660
+ "lola": true,
1661
+ "claude": true,
1662
+ "grok": true,
1663
+ "deep": true,
1664
+ "gemini": false,
1665
+ "kimi": true
1666
+ }
1667
+ },
1668
+ {
1669
+ "task_id": "HumanEval/149",
1670
+ "lanes": {
1671
+ "lola": true,
1672
+ "claude": true,
1673
+ "grok": true,
1674
+ "deep": true,
1675
+ "gemini": false,
1676
+ "kimi": true
1677
+ }
1678
+ },
1679
+ {
1680
+ "task_id": "HumanEval/150",
1681
+ "lanes": {
1682
+ "lola": true,
1683
+ "claude": true,
1684
+ "grok": true,
1685
+ "deep": true,
1686
+ "gemini": false,
1687
+ "kimi": true
1688
+ }
1689
+ },
1690
+ {
1691
+ "task_id": "HumanEval/151",
1692
+ "lanes": {
1693
+ "lola": true,
1694
+ "claude": true,
1695
+ "grok": true,
1696
+ "deep": true,
1697
+ "gemini": false,
1698
+ "kimi": true
1699
+ }
1700
+ },
1701
+ {
1702
+ "task_id": "HumanEval/152",
1703
+ "lanes": {
1704
+ "lola": true,
1705
+ "claude": true,
1706
+ "grok": true,
1707
+ "deep": true,
1708
+ "gemini": false,
1709
+ "kimi": true
1710
+ }
1711
+ },
1712
+ {
1713
+ "task_id": "HumanEval/153",
1714
+ "lanes": {
1715
+ "lola": true,
1716
+ "claude": true,
1717
+ "grok": true,
1718
+ "deep": true,
1719
+ "gemini": false,
1720
+ "kimi": true
1721
+ }
1722
+ },
1723
+ {
1724
+ "task_id": "HumanEval/154",
1725
+ "lanes": {
1726
+ "lola": true,
1727
+ "claude": true,
1728
+ "grok": false,
1729
+ "deep": false,
1730
+ "gemini": false,
1731
+ "kimi": false
1732
+ }
1733
+ },
1734
+ {
1735
+ "task_id": "HumanEval/155",
1736
+ "lanes": {
1737
+ "lola": true,
1738
+ "claude": true,
1739
+ "grok": true,
1740
+ "deep": true,
1741
+ "gemini": false,
1742
+ "kimi": true
1743
+ }
1744
+ },
1745
+ {
1746
+ "task_id": "HumanEval/156",
1747
+ "lanes": {
1748
+ "lola": true,
1749
+ "claude": true,
1750
+ "grok": true,
1751
+ "deep": true,
1752
+ "gemini": false,
1753
+ "kimi": true
1754
+ }
1755
+ },
1756
+ {
1757
+ "task_id": "HumanEval/157",
1758
+ "lanes": {
1759
+ "lola": true,
1760
+ "claude": true,
1761
+ "grok": true,
1762
+ "deep": true,
1763
+ "gemini": false,
1764
+ "kimi": true
1765
+ }
1766
+ },
1767
+ {
1768
+ "task_id": "HumanEval/158",
1769
+ "lanes": {
1770
+ "lola": false,
1771
+ "claude": true,
1772
+ "grok": false,
1773
+ "deep": true,
1774
+ "gemini": false,
1775
+ "kimi": false
1776
+ }
1777
+ },
1778
+ {
1779
+ "task_id": "HumanEval/159",
1780
+ "lanes": {
1781
+ "lola": true,
1782
+ "claude": true,
1783
+ "grok": true,
1784
+ "deep": true,
1785
+ "gemini": false,
1786
+ "kimi": true
1787
+ }
1788
+ },
1789
+ {
1790
+ "task_id": "HumanEval/160",
1791
+ "lanes": {
1792
+ "lola": true,
1793
+ "claude": true,
1794
+ "grok": true,
1795
+ "deep": false,
1796
+ "gemini": false,
1797
+ "kimi": true
1798
+ }
1799
+ },
1800
+ {
1801
+ "task_id": "HumanEval/161",
1802
+ "lanes": {
1803
+ "lola": true,
1804
+ "claude": true,
1805
+ "grok": true,
1806
+ "deep": true,
1807
+ "gemini": false,
1808
+ "kimi": true
1809
+ }
1810
+ },
1811
+ {
1812
+ "task_id": "HumanEval/162",
1813
+ "lanes": {
1814
+ "lola": true,
1815
+ "claude": true,
1816
+ "grok": true,
1817
+ "deep": true,
1818
+ "gemini": false,
1819
+ "kimi": true
1820
+ }
1821
+ },
1822
+ {
1823
+ "task_id": "HumanEval/163",
1824
+ "lanes": {
1825
+ "lola": false,
1826
+ "claude": true,
1827
+ "grok": false,
1828
+ "deep": false,
1829
+ "gemini": false,
1830
+ "kimi": false
1831
+ }
1832
+ }
1833
+ ]
1834
+ }
humaneval_results_105027.json ADDED
@@ -0,0 +1,1498 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "lanes": {
3
+ "lola": {
4
+ "pass": 144,
5
+ "fail": 20
6
+ },
7
+ "claude": {
8
+ "pass": 159,
9
+ "fail": 5
10
+ },
11
+ "grok": {
12
+ "pass": 135,
13
+ "fail": 29
14
+ },
15
+ "deep": {
16
+ "pass": 137,
17
+ "fail": 27
18
+ }
19
+ },
20
+ "problems": [
21
+ {
22
+ "task_id": "HumanEval/0",
23
+ "lanes": {
24
+ "lola": false,
25
+ "claude": true,
26
+ "grok": true,
27
+ "deep": true
28
+ }
29
+ },
30
+ {
31
+ "task_id": "HumanEval/1",
32
+ "lanes": {
33
+ "lola": true,
34
+ "claude": true,
35
+ "grok": true,
36
+ "deep": true
37
+ }
38
+ },
39
+ {
40
+ "task_id": "HumanEval/2",
41
+ "lanes": {
42
+ "lola": true,
43
+ "claude": true,
44
+ "grok": true,
45
+ "deep": true
46
+ }
47
+ },
48
+ {
49
+ "task_id": "HumanEval/3",
50
+ "lanes": {
51
+ "lola": true,
52
+ "claude": true,
53
+ "grok": true,
54
+ "deep": true
55
+ }
56
+ },
57
+ {
58
+ "task_id": "HumanEval/4",
59
+ "lanes": {
60
+ "lola": true,
61
+ "claude": true,
62
+ "grok": true,
63
+ "deep": true
64
+ }
65
+ },
66
+ {
67
+ "task_id": "HumanEval/5",
68
+ "lanes": {
69
+ "lola": true,
70
+ "claude": true,
71
+ "grok": true,
72
+ "deep": true
73
+ }
74
+ },
75
+ {
76
+ "task_id": "HumanEval/6",
77
+ "lanes": {
78
+ "lola": true,
79
+ "claude": true,
80
+ "grok": true,
81
+ "deep": true
82
+ }
83
+ },
84
+ {
85
+ "task_id": "HumanEval/7",
86
+ "lanes": {
87
+ "lola": false,
88
+ "claude": true,
89
+ "grok": true,
90
+ "deep": true
91
+ }
92
+ },
93
+ {
94
+ "task_id": "HumanEval/8",
95
+ "lanes": {
96
+ "lola": true,
97
+ "claude": true,
98
+ "grok": true,
99
+ "deep": true
100
+ }
101
+ },
102
+ {
103
+ "task_id": "HumanEval/9",
104
+ "lanes": {
105
+ "lola": true,
106
+ "claude": true,
107
+ "grok": true,
108
+ "deep": true
109
+ }
110
+ },
111
+ {
112
+ "task_id": "HumanEval/10",
113
+ "lanes": {
114
+ "lola": true,
115
+ "claude": true,
116
+ "grok": true,
117
+ "deep": false
118
+ }
119
+ },
120
+ {
121
+ "task_id": "HumanEval/11",
122
+ "lanes": {
123
+ "lola": true,
124
+ "claude": true,
125
+ "grok": true,
126
+ "deep": true
127
+ }
128
+ },
129
+ {
130
+ "task_id": "HumanEval/12",
131
+ "lanes": {
132
+ "lola": true,
133
+ "claude": true,
134
+ "grok": true,
135
+ "deep": true
136
+ }
137
+ },
138
+ {
139
+ "task_id": "HumanEval/13",
140
+ "lanes": {
141
+ "lola": true,
142
+ "claude": true,
143
+ "grok": true,
144
+ "deep": true
145
+ }
146
+ },
147
+ {
148
+ "task_id": "HumanEval/14",
149
+ "lanes": {
150
+ "lola": true,
151
+ "claude": true,
152
+ "grok": true,
153
+ "deep": true
154
+ }
155
+ },
156
+ {
157
+ "task_id": "HumanEval/15",
158
+ "lanes": {
159
+ "lola": true,
160
+ "claude": true,
161
+ "grok": true,
162
+ "deep": true
163
+ }
164
+ },
165
+ {
166
+ "task_id": "HumanEval/16",
167
+ "lanes": {
168
+ "lola": true,
169
+ "claude": true,
170
+ "grok": true,
171
+ "deep": true
172
+ }
173
+ },
174
+ {
175
+ "task_id": "HumanEval/17",
176
+ "lanes": {
177
+ "lola": true,
178
+ "claude": true,
179
+ "grok": true,
180
+ "deep": true
181
+ }
182
+ },
183
+ {
184
+ "task_id": "HumanEval/18",
185
+ "lanes": {
186
+ "lola": true,
187
+ "claude": true,
188
+ "grok": true,
189
+ "deep": true
190
+ }
191
+ },
192
+ {
193
+ "task_id": "HumanEval/19",
194
+ "lanes": {
195
+ "lola": true,
196
+ "claude": true,
197
+ "grok": true,
198
+ "deep": true
199
+ }
200
+ },
201
+ {
202
+ "task_id": "HumanEval/20",
203
+ "lanes": {
204
+ "lola": false,
205
+ "claude": true,
206
+ "grok": true,
207
+ "deep": true
208
+ }
209
+ },
210
+ {
211
+ "task_id": "HumanEval/21",
212
+ "lanes": {
213
+ "lola": false,
214
+ "claude": true,
215
+ "grok": true,
216
+ "deep": true
217
+ }
218
+ },
219
+ {
220
+ "task_id": "HumanEval/22",
221
+ "lanes": {
222
+ "lola": true,
223
+ "claude": true,
224
+ "grok": true,
225
+ "deep": true
226
+ }
227
+ },
228
+ {
229
+ "task_id": "HumanEval/23",
230
+ "lanes": {
231
+ "lola": true,
232
+ "claude": true,
233
+ "grok": true,
234
+ "deep": true
235
+ }
236
+ },
237
+ {
238
+ "task_id": "HumanEval/24",
239
+ "lanes": {
240
+ "lola": true,
241
+ "claude": true,
242
+ "grok": true,
243
+ "deep": true
244
+ }
245
+ },
246
+ {
247
+ "task_id": "HumanEval/25",
248
+ "lanes": {
249
+ "lola": true,
250
+ "claude": true,
251
+ "grok": true,
252
+ "deep": true
253
+ }
254
+ },
255
+ {
256
+ "task_id": "HumanEval/26",
257
+ "lanes": {
258
+ "lola": true,
259
+ "claude": true,
260
+ "grok": false,
261
+ "deep": false
262
+ }
263
+ },
264
+ {
265
+ "task_id": "HumanEval/27",
266
+ "lanes": {
267
+ "lola": true,
268
+ "claude": true,
269
+ "grok": true,
270
+ "deep": true
271
+ }
272
+ },
273
+ {
274
+ "task_id": "HumanEval/28",
275
+ "lanes": {
276
+ "lola": true,
277
+ "claude": true,
278
+ "grok": true,
279
+ "deep": true
280
+ }
281
+ },
282
+ {
283
+ "task_id": "HumanEval/29",
284
+ "lanes": {
285
+ "lola": true,
286
+ "claude": true,
287
+ "grok": true,
288
+ "deep": true
289
+ }
290
+ },
291
+ {
292
+ "task_id": "HumanEval/30",
293
+ "lanes": {
294
+ "lola": true,
295
+ "claude": true,
296
+ "grok": true,
297
+ "deep": true
298
+ }
299
+ },
300
+ {
301
+ "task_id": "HumanEval/31",
302
+ "lanes": {
303
+ "lola": true,
304
+ "claude": true,
305
+ "grok": true,
306
+ "deep": true
307
+ }
308
+ },
309
+ {
310
+ "task_id": "HumanEval/32",
311
+ "lanes": {
312
+ "lola": false,
313
+ "claude": true,
314
+ "grok": false,
315
+ "deep": false
316
+ }
317
+ },
318
+ {
319
+ "task_id": "HumanEval/33",
320
+ "lanes": {
321
+ "lola": true,
322
+ "claude": true,
323
+ "grok": true,
324
+ "deep": true
325
+ }
326
+ },
327
+ {
328
+ "task_id": "HumanEval/34",
329
+ "lanes": {
330
+ "lola": true,
331
+ "claude": true,
332
+ "grok": true,
333
+ "deep": true
334
+ }
335
+ },
336
+ {
337
+ "task_id": "HumanEval/35",
338
+ "lanes": {
339
+ "lola": true,
340
+ "claude": true,
341
+ "grok": true,
342
+ "deep": true
343
+ }
344
+ },
345
+ {
346
+ "task_id": "HumanEval/36",
347
+ "lanes": {
348
+ "lola": true,
349
+ "claude": true,
350
+ "grok": true,
351
+ "deep": true
352
+ }
353
+ },
354
+ {
355
+ "task_id": "HumanEval/37",
356
+ "lanes": {
357
+ "lola": true,
358
+ "claude": true,
359
+ "grok": true,
360
+ "deep": true
361
+ }
362
+ },
363
+ {
364
+ "task_id": "HumanEval/38",
365
+ "lanes": {
366
+ "lola": true,
367
+ "claude": true,
368
+ "grok": true,
369
+ "deep": true
370
+ }
371
+ },
372
+ {
373
+ "task_id": "HumanEval/39",
374
+ "lanes": {
375
+ "lola": true,
376
+ "claude": true,
377
+ "grok": true,
378
+ "deep": true
379
+ }
380
+ },
381
+ {
382
+ "task_id": "HumanEval/40",
383
+ "lanes": {
384
+ "lola": true,
385
+ "claude": true,
386
+ "grok": true,
387
+ "deep": true
388
+ }
389
+ },
390
+ {
391
+ "task_id": "HumanEval/41",
392
+ "lanes": {
393
+ "lola": false,
394
+ "claude": true,
395
+ "grok": true,
396
+ "deep": true
397
+ }
398
+ },
399
+ {
400
+ "task_id": "HumanEval/42",
401
+ "lanes": {
402
+ "lola": true,
403
+ "claude": true,
404
+ "grok": true,
405
+ "deep": true
406
+ }
407
+ },
408
+ {
409
+ "task_id": "HumanEval/43",
410
+ "lanes": {
411
+ "lola": true,
412
+ "claude": true,
413
+ "grok": true,
414
+ "deep": true
415
+ }
416
+ },
417
+ {
418
+ "task_id": "HumanEval/44",
419
+ "lanes": {
420
+ "lola": true,
421
+ "claude": true,
422
+ "grok": true,
423
+ "deep": true
424
+ }
425
+ },
426
+ {
427
+ "task_id": "HumanEval/45",
428
+ "lanes": {
429
+ "lola": true,
430
+ "claude": true,
431
+ "grok": true,
432
+ "deep": true
433
+ }
434
+ },
435
+ {
436
+ "task_id": "HumanEval/46",
437
+ "lanes": {
438
+ "lola": true,
439
+ "claude": true,
440
+ "grok": true,
441
+ "deep": true
442
+ }
443
+ },
444
+ {
445
+ "task_id": "HumanEval/47",
446
+ "lanes": {
447
+ "lola": true,
448
+ "claude": true,
449
+ "grok": true,
450
+ "deep": true
451
+ }
452
+ },
453
+ {
454
+ "task_id": "HumanEval/48",
455
+ "lanes": {
456
+ "lola": true,
457
+ "claude": true,
458
+ "grok": true,
459
+ "deep": true
460
+ }
461
+ },
462
+ {
463
+ "task_id": "HumanEval/49",
464
+ "lanes": {
465
+ "lola": true,
466
+ "claude": true,
467
+ "grok": true,
468
+ "deep": true
469
+ }
470
+ },
471
+ {
472
+ "task_id": "HumanEval/50",
473
+ "lanes": {
474
+ "lola": true,
475
+ "claude": true,
476
+ "grok": true,
477
+ "deep": true
478
+ }
479
+ },
480
+ {
481
+ "task_id": "HumanEval/51",
482
+ "lanes": {
483
+ "lola": true,
484
+ "claude": true,
485
+ "grok": true,
486
+ "deep": true
487
+ }
488
+ },
489
+ {
490
+ "task_id": "HumanEval/52",
491
+ "lanes": {
492
+ "lola": true,
493
+ "claude": true,
494
+ "grok": true,
495
+ "deep": true
496
+ }
497
+ },
498
+ {
499
+ "task_id": "HumanEval/53",
500
+ "lanes": {
501
+ "lola": true,
502
+ "claude": true,
503
+ "grok": true,
504
+ "deep": true
505
+ }
506
+ },
507
+ {
508
+ "task_id": "HumanEval/54",
509
+ "lanes": {
510
+ "lola": true,
511
+ "claude": true,
512
+ "grok": true,
513
+ "deep": true
514
+ }
515
+ },
516
+ {
517
+ "task_id": "HumanEval/55",
518
+ "lanes": {
519
+ "lola": true,
520
+ "claude": true,
521
+ "grok": true,
522
+ "deep": true
523
+ }
524
+ },
525
+ {
526
+ "task_id": "HumanEval/56",
527
+ "lanes": {
528
+ "lola": true,
529
+ "claude": true,
530
+ "grok": true,
531
+ "deep": true
532
+ }
533
+ },
534
+ {
535
+ "task_id": "HumanEval/57",
536
+ "lanes": {
537
+ "lola": true,
538
+ "claude": true,
539
+ "grok": true,
540
+ "deep": true
541
+ }
542
+ },
543
+ {
544
+ "task_id": "HumanEval/58",
545
+ "lanes": {
546
+ "lola": true,
547
+ "claude": true,
548
+ "grok": true,
549
+ "deep": true
550
+ }
551
+ },
552
+ {
553
+ "task_id": "HumanEval/59",
554
+ "lanes": {
555
+ "lola": true,
556
+ "claude": true,
557
+ "grok": true,
558
+ "deep": true
559
+ }
560
+ },
561
+ {
562
+ "task_id": "HumanEval/60",
563
+ "lanes": {
564
+ "lola": true,
565
+ "claude": true,
566
+ "grok": true,
567
+ "deep": true
568
+ }
569
+ },
570
+ {
571
+ "task_id": "HumanEval/61",
572
+ "lanes": {
573
+ "lola": true,
574
+ "claude": true,
575
+ "grok": true,
576
+ "deep": true
577
+ }
578
+ },
579
+ {
580
+ "task_id": "HumanEval/62",
581
+ "lanes": {
582
+ "lola": true,
583
+ "claude": true,
584
+ "grok": true,
585
+ "deep": true
586
+ }
587
+ },
588
+ {
589
+ "task_id": "HumanEval/63",
590
+ "lanes": {
591
+ "lola": true,
592
+ "claude": true,
593
+ "grok": true,
594
+ "deep": true
595
+ }
596
+ },
597
+ {
598
+ "task_id": "HumanEval/64",
599
+ "lanes": {
600
+ "lola": true,
601
+ "claude": true,
602
+ "grok": true,
603
+ "deep": true
604
+ }
605
+ },
606
+ {
607
+ "task_id": "HumanEval/65",
608
+ "lanes": {
609
+ "lola": true,
610
+ "claude": true,
611
+ "grok": false,
612
+ "deep": false
613
+ }
614
+ },
615
+ {
616
+ "task_id": "HumanEval/66",
617
+ "lanes": {
618
+ "lola": true,
619
+ "claude": true,
620
+ "grok": true,
621
+ "deep": true
622
+ }
623
+ },
624
+ {
625
+ "task_id": "HumanEval/67",
626
+ "lanes": {
627
+ "lola": true,
628
+ "claude": true,
629
+ "grok": false,
630
+ "deep": true
631
+ }
632
+ },
633
+ {
634
+ "task_id": "HumanEval/68",
635
+ "lanes": {
636
+ "lola": true,
637
+ "claude": true,
638
+ "grok": true,
639
+ "deep": true
640
+ }
641
+ },
642
+ {
643
+ "task_id": "HumanEval/69",
644
+ "lanes": {
645
+ "lola": true,
646
+ "claude": true,
647
+ "grok": true,
648
+ "deep": true
649
+ }
650
+ },
651
+ {
652
+ "task_id": "HumanEval/70",
653
+ "lanes": {
654
+ "lola": true,
655
+ "claude": true,
656
+ "grok": true,
657
+ "deep": true
658
+ }
659
+ },
660
+ {
661
+ "task_id": "HumanEval/71",
662
+ "lanes": {
663
+ "lola": true,
664
+ "claude": true,
665
+ "grok": true,
666
+ "deep": true
667
+ }
668
+ },
669
+ {
670
+ "task_id": "HumanEval/72",
671
+ "lanes": {
672
+ "lola": true,
673
+ "claude": true,
674
+ "grok": true,
675
+ "deep": true
676
+ }
677
+ },
678
+ {
679
+ "task_id": "HumanEval/73",
680
+ "lanes": {
681
+ "lola": true,
682
+ "claude": true,
683
+ "grok": true,
684
+ "deep": true
685
+ }
686
+ },
687
+ {
688
+ "task_id": "HumanEval/74",
689
+ "lanes": {
690
+ "lola": true,
691
+ "claude": true,
692
+ "grok": true,
693
+ "deep": true
694
+ }
695
+ },
696
+ {
697
+ "task_id": "HumanEval/75",
698
+ "lanes": {
699
+ "lola": true,
700
+ "claude": true,
701
+ "grok": true,
702
+ "deep": true
703
+ }
704
+ },
705
+ {
706
+ "task_id": "HumanEval/76",
707
+ "lanes": {
708
+ "lola": true,
709
+ "claude": false,
710
+ "grok": false,
711
+ "deep": true
712
+ }
713
+ },
714
+ {
715
+ "task_id": "HumanEval/77",
716
+ "lanes": {
717
+ "lola": false,
718
+ "claude": true,
719
+ "grok": true,
720
+ "deep": false
721
+ }
722
+ },
723
+ {
724
+ "task_id": "HumanEval/78",
725
+ "lanes": {
726
+ "lola": true,
727
+ "claude": true,
728
+ "grok": true,
729
+ "deep": true
730
+ }
731
+ },
732
+ {
733
+ "task_id": "HumanEval/79",
734
+ "lanes": {
735
+ "lola": true,
736
+ "claude": true,
737
+ "grok": true,
738
+ "deep": true
739
+ }
740
+ },
741
+ {
742
+ "task_id": "HumanEval/80",
743
+ "lanes": {
744
+ "lola": true,
745
+ "claude": true,
746
+ "grok": true,
747
+ "deep": true
748
+ }
749
+ },
750
+ {
751
+ "task_id": "HumanEval/81",
752
+ "lanes": {
753
+ "lola": true,
754
+ "claude": true,
755
+ "grok": true,
756
+ "deep": false
757
+ }
758
+ },
759
+ {
760
+ "task_id": "HumanEval/82",
761
+ "lanes": {
762
+ "lola": true,
763
+ "claude": true,
764
+ "grok": true,
765
+ "deep": true
766
+ }
767
+ },
768
+ {
769
+ "task_id": "HumanEval/83",
770
+ "lanes": {
771
+ "lola": true,
772
+ "claude": false,
773
+ "grok": false,
774
+ "deep": true
775
+ }
776
+ },
777
+ {
778
+ "task_id": "HumanEval/84",
779
+ "lanes": {
780
+ "lola": true,
781
+ "claude": true,
782
+ "grok": true,
783
+ "deep": false
784
+ }
785
+ },
786
+ {
787
+ "task_id": "HumanEval/85",
788
+ "lanes": {
789
+ "lola": true,
790
+ "claude": true,
791
+ "grok": true,
792
+ "deep": true
793
+ }
794
+ },
795
+ {
796
+ "task_id": "HumanEval/86",
797
+ "lanes": {
798
+ "lola": true,
799
+ "claude": true,
800
+ "grok": false,
801
+ "deep": true
802
+ }
803
+ },
804
+ {
805
+ "task_id": "HumanEval/87",
806
+ "lanes": {
807
+ "lola": true,
808
+ "claude": true,
809
+ "grok": true,
810
+ "deep": true
811
+ }
812
+ },
813
+ {
814
+ "task_id": "HumanEval/88",
815
+ "lanes": {
816
+ "lola": true,
817
+ "claude": true,
818
+ "grok": true,
819
+ "deep": true
820
+ }
821
+ },
822
+ {
823
+ "task_id": "HumanEval/89",
824
+ "lanes": {
825
+ "lola": true,
826
+ "claude": true,
827
+ "grok": true,
828
+ "deep": true
829
+ }
830
+ },
831
+ {
832
+ "task_id": "HumanEval/90",
833
+ "lanes": {
834
+ "lola": true,
835
+ "claude": true,
836
+ "grok": true,
837
+ "deep": true
838
+ }
839
+ },
840
+ {
841
+ "task_id": "HumanEval/91",
842
+ "lanes": {
843
+ "lola": false,
844
+ "claude": true,
845
+ "grok": false,
846
+ "deep": false
847
+ }
848
+ },
849
+ {
850
+ "task_id": "HumanEval/92",
851
+ "lanes": {
852
+ "lola": true,
853
+ "claude": true,
854
+ "grok": true,
855
+ "deep": true
856
+ }
857
+ },
858
+ {
859
+ "task_id": "HumanEval/93",
860
+ "lanes": {
861
+ "lola": true,
862
+ "claude": false,
863
+ "grok": false,
864
+ "deep": false
865
+ }
866
+ },
867
+ {
868
+ "task_id": "HumanEval/94",
869
+ "lanes": {
870
+ "lola": true,
871
+ "claude": true,
872
+ "grok": true,
873
+ "deep": true
874
+ }
875
+ },
876
+ {
877
+ "task_id": "HumanEval/95",
878
+ "lanes": {
879
+ "lola": true,
880
+ "claude": true,
881
+ "grok": false,
882
+ "deep": true
883
+ }
884
+ },
885
+ {
886
+ "task_id": "HumanEval/96",
887
+ "lanes": {
888
+ "lola": true,
889
+ "claude": true,
890
+ "grok": true,
891
+ "deep": true
892
+ }
893
+ },
894
+ {
895
+ "task_id": "HumanEval/97",
896
+ "lanes": {
897
+ "lola": false,
898
+ "claude": true,
899
+ "grok": true,
900
+ "deep": false
901
+ }
902
+ },
903
+ {
904
+ "task_id": "HumanEval/98",
905
+ "lanes": {
906
+ "lola": true,
907
+ "claude": true,
908
+ "grok": true,
909
+ "deep": true
910
+ }
911
+ },
912
+ {
913
+ "task_id": "HumanEval/99",
914
+ "lanes": {
915
+ "lola": true,
916
+ "claude": true,
917
+ "grok": true,
918
+ "deep": true
919
+ }
920
+ },
921
+ {
922
+ "task_id": "HumanEval/100",
923
+ "lanes": {
924
+ "lola": true,
925
+ "claude": true,
926
+ "grok": false,
927
+ "deep": true
928
+ }
929
+ },
930
+ {
931
+ "task_id": "HumanEval/101",
932
+ "lanes": {
933
+ "lola": true,
934
+ "claude": true,
935
+ "grok": true,
936
+ "deep": true
937
+ }
938
+ },
939
+ {
940
+ "task_id": "HumanEval/102",
941
+ "lanes": {
942
+ "lola": true,
943
+ "claude": true,
944
+ "grok": false,
945
+ "deep": true
946
+ }
947
+ },
948
+ {
949
+ "task_id": "HumanEval/103",
950
+ "lanes": {
951
+ "lola": true,
952
+ "claude": true,
953
+ "grok": true,
954
+ "deep": true
955
+ }
956
+ },
957
+ {
958
+ "task_id": "HumanEval/104",
959
+ "lanes": {
960
+ "lola": true,
961
+ "claude": true,
962
+ "grok": true,
963
+ "deep": true
964
+ }
965
+ },
966
+ {
967
+ "task_id": "HumanEval/105",
968
+ "lanes": {
969
+ "lola": true,
970
+ "claude": true,
971
+ "grok": true,
972
+ "deep": true
973
+ }
974
+ },
975
+ {
976
+ "task_id": "HumanEval/106",
977
+ "lanes": {
978
+ "lola": true,
979
+ "claude": true,
980
+ "grok": true,
981
+ "deep": true
982
+ }
983
+ },
984
+ {
985
+ "task_id": "HumanEval/107",
986
+ "lanes": {
987
+ "lola": true,
988
+ "claude": true,
989
+ "grok": true,
990
+ "deep": true
991
+ }
992
+ },
993
+ {
994
+ "task_id": "HumanEval/108",
995
+ "lanes": {
996
+ "lola": false,
997
+ "claude": true,
998
+ "grok": false,
999
+ "deep": false
1000
+ }
1001
+ },
1002
+ {
1003
+ "task_id": "HumanEval/109",
1004
+ "lanes": {
1005
+ "lola": true,
1006
+ "claude": true,
1007
+ "grok": true,
1008
+ "deep": true
1009
+ }
1010
+ },
1011
+ {
1012
+ "task_id": "HumanEval/110",
1013
+ "lanes": {
1014
+ "lola": true,
1015
+ "claude": true,
1016
+ "grok": true,
1017
+ "deep": true
1018
+ }
1019
+ },
1020
+ {
1021
+ "task_id": "HumanEval/111",
1022
+ "lanes": {
1023
+ "lola": true,
1024
+ "claude": true,
1025
+ "grok": true,
1026
+ "deep": true
1027
+ }
1028
+ },
1029
+ {
1030
+ "task_id": "HumanEval/112",
1031
+ "lanes": {
1032
+ "lola": true,
1033
+ "claude": true,
1034
+ "grok": true,
1035
+ "deep": true
1036
+ }
1037
+ },
1038
+ {
1039
+ "task_id": "HumanEval/113",
1040
+ "lanes": {
1041
+ "lola": true,
1042
+ "claude": true,
1043
+ "grok": true,
1044
+ "deep": true
1045
+ }
1046
+ },
1047
+ {
1048
+ "task_id": "HumanEval/114",
1049
+ "lanes": {
1050
+ "lola": true,
1051
+ "claude": true,
1052
+ "grok": true,
1053
+ "deep": false
1054
+ }
1055
+ },
1056
+ {
1057
+ "task_id": "HumanEval/115",
1058
+ "lanes": {
1059
+ "lola": false,
1060
+ "claude": true,
1061
+ "grok": false,
1062
+ "deep": false
1063
+ }
1064
+ },
1065
+ {
1066
+ "task_id": "HumanEval/116",
1067
+ "lanes": {
1068
+ "lola": true,
1069
+ "claude": true,
1070
+ "grok": true,
1071
+ "deep": false
1072
+ }
1073
+ },
1074
+ {
1075
+ "task_id": "HumanEval/117",
1076
+ "lanes": {
1077
+ "lola": true,
1078
+ "claude": true,
1079
+ "grok": true,
1080
+ "deep": true
1081
+ }
1082
+ },
1083
+ {
1084
+ "task_id": "HumanEval/118",
1085
+ "lanes": {
1086
+ "lola": true,
1087
+ "claude": true,
1088
+ "grok": false,
1089
+ "deep": true
1090
+ }
1091
+ },
1092
+ {
1093
+ "task_id": "HumanEval/119",
1094
+ "lanes": {
1095
+ "lola": true,
1096
+ "claude": true,
1097
+ "grok": true,
1098
+ "deep": true
1099
+ }
1100
+ },
1101
+ {
1102
+ "task_id": "HumanEval/120",
1103
+ "lanes": {
1104
+ "lola": false,
1105
+ "claude": true,
1106
+ "grok": false,
1107
+ "deep": true
1108
+ }
1109
+ },
1110
+ {
1111
+ "task_id": "HumanEval/121",
1112
+ "lanes": {
1113
+ "lola": true,
1114
+ "claude": true,
1115
+ "grok": false,
1116
+ "deep": true
1117
+ }
1118
+ },
1119
+ {
1120
+ "task_id": "HumanEval/122",
1121
+ "lanes": {
1122
+ "lola": true,
1123
+ "claude": true,
1124
+ "grok": true,
1125
+ "deep": false
1126
+ }
1127
+ },
1128
+ {
1129
+ "task_id": "HumanEval/123",
1130
+ "lanes": {
1131
+ "lola": true,
1132
+ "claude": true,
1133
+ "grok": true,
1134
+ "deep": true
1135
+ }
1136
+ },
1137
+ {
1138
+ "task_id": "HumanEval/124",
1139
+ "lanes": {
1140
+ "lola": true,
1141
+ "claude": true,
1142
+ "grok": true,
1143
+ "deep": true
1144
+ }
1145
+ },
1146
+ {
1147
+ "task_id": "HumanEval/125",
1148
+ "lanes": {
1149
+ "lola": true,
1150
+ "claude": true,
1151
+ "grok": false,
1152
+ "deep": true
1153
+ }
1154
+ },
1155
+ {
1156
+ "task_id": "HumanEval/126",
1157
+ "lanes": {
1158
+ "lola": true,
1159
+ "claude": true,
1160
+ "grok": false,
1161
+ "deep": true
1162
+ }
1163
+ },
1164
+ {
1165
+ "task_id": "HumanEval/127",
1166
+ "lanes": {
1167
+ "lola": true,
1168
+ "claude": true,
1169
+ "grok": true,
1170
+ "deep": false
1171
+ }
1172
+ },
1173
+ {
1174
+ "task_id": "HumanEval/128",
1175
+ "lanes": {
1176
+ "lola": true,
1177
+ "claude": true,
1178
+ "grok": true,
1179
+ "deep": true
1180
+ }
1181
+ },
1182
+ {
1183
+ "task_id": "HumanEval/129",
1184
+ "lanes": {
1185
+ "lola": true,
1186
+ "claude": true,
1187
+ "grok": false,
1188
+ "deep": true
1189
+ }
1190
+ },
1191
+ {
1192
+ "task_id": "HumanEval/130",
1193
+ "lanes": {
1194
+ "lola": false,
1195
+ "claude": false,
1196
+ "grok": false,
1197
+ "deep": false
1198
+ }
1199
+ },
1200
+ {
1201
+ "task_id": "HumanEval/131",
1202
+ "lanes": {
1203
+ "lola": true,
1204
+ "claude": true,
1205
+ "grok": true,
1206
+ "deep": true
1207
+ }
1208
+ },
1209
+ {
1210
+ "task_id": "HumanEval/132",
1211
+ "lanes": {
1212
+ "lola": false,
1213
+ "claude": false,
1214
+ "grok": false,
1215
+ "deep": false
1216
+ }
1217
+ },
1218
+ {
1219
+ "task_id": "HumanEval/133",
1220
+ "lanes": {
1221
+ "lola": true,
1222
+ "claude": true,
1223
+ "grok": true,
1224
+ "deep": true
1225
+ }
1226
+ },
1227
+ {
1228
+ "task_id": "HumanEval/134",
1229
+ "lanes": {
1230
+ "lola": false,
1231
+ "claude": true,
1232
+ "grok": false,
1233
+ "deep": false
1234
+ }
1235
+ },
1236
+ {
1237
+ "task_id": "HumanEval/135",
1238
+ "lanes": {
1239
+ "lola": true,
1240
+ "claude": true,
1241
+ "grok": false,
1242
+ "deep": true
1243
+ }
1244
+ },
1245
+ {
1246
+ "task_id": "HumanEval/136",
1247
+ "lanes": {
1248
+ "lola": true,
1249
+ "claude": true,
1250
+ "grok": true,
1251
+ "deep": true
1252
+ }
1253
+ },
1254
+ {
1255
+ "task_id": "HumanEval/137",
1256
+ "lanes": {
1257
+ "lola": true,
1258
+ "claude": true,
1259
+ "grok": true,
1260
+ "deep": false
1261
+ }
1262
+ },
1263
+ {
1264
+ "task_id": "HumanEval/138",
1265
+ "lanes": {
1266
+ "lola": true,
1267
+ "claude": true,
1268
+ "grok": false,
1269
+ "deep": true
1270
+ }
1271
+ },
1272
+ {
1273
+ "task_id": "HumanEval/139",
1274
+ "lanes": {
1275
+ "lola": true,
1276
+ "claude": true,
1277
+ "grok": true,
1278
+ "deep": true
1279
+ }
1280
+ },
1281
+ {
1282
+ "task_id": "HumanEval/140",
1283
+ "lanes": {
1284
+ "lola": false,
1285
+ "claude": true,
1286
+ "grok": true,
1287
+ "deep": true
1288
+ }
1289
+ },
1290
+ {
1291
+ "task_id": "HumanEval/141",
1292
+ "lanes": {
1293
+ "lola": true,
1294
+ "claude": true,
1295
+ "grok": true,
1296
+ "deep": true
1297
+ }
1298
+ },
1299
+ {
1300
+ "task_id": "HumanEval/142",
1301
+ "lanes": {
1302
+ "lola": true,
1303
+ "claude": true,
1304
+ "grok": false,
1305
+ "deep": true
1306
+ }
1307
+ },
1308
+ {
1309
+ "task_id": "HumanEval/143",
1310
+ "lanes": {
1311
+ "lola": true,
1312
+ "claude": true,
1313
+ "grok": true,
1314
+ "deep": true
1315
+ }
1316
+ },
1317
+ {
1318
+ "task_id": "HumanEval/144",
1319
+ "lanes": {
1320
+ "lola": true,
1321
+ "claude": true,
1322
+ "grok": true,
1323
+ "deep": true
1324
+ }
1325
+ },
1326
+ {
1327
+ "task_id": "HumanEval/145",
1328
+ "lanes": {
1329
+ "lola": false,
1330
+ "claude": true,
1331
+ "grok": false,
1332
+ "deep": false
1333
+ }
1334
+ },
1335
+ {
1336
+ "task_id": "HumanEval/146",
1337
+ "lanes": {
1338
+ "lola": true,
1339
+ "claude": true,
1340
+ "grok": true,
1341
+ "deep": false
1342
+ }
1343
+ },
1344
+ {
1345
+ "task_id": "HumanEval/147",
1346
+ "lanes": {
1347
+ "lola": true,
1348
+ "claude": true,
1349
+ "grok": true,
1350
+ "deep": false
1351
+ }
1352
+ },
1353
+ {
1354
+ "task_id": "HumanEval/148",
1355
+ "lanes": {
1356
+ "lola": true,
1357
+ "claude": true,
1358
+ "grok": true,
1359
+ "deep": true
1360
+ }
1361
+ },
1362
+ {
1363
+ "task_id": "HumanEval/149",
1364
+ "lanes": {
1365
+ "lola": true,
1366
+ "claude": true,
1367
+ "grok": true,
1368
+ "deep": true
1369
+ }
1370
+ },
1371
+ {
1372
+ "task_id": "HumanEval/150",
1373
+ "lanes": {
1374
+ "lola": true,
1375
+ "claude": true,
1376
+ "grok": true,
1377
+ "deep": true
1378
+ }
1379
+ },
1380
+ {
1381
+ "task_id": "HumanEval/151",
1382
+ "lanes": {
1383
+ "lola": true,
1384
+ "claude": true,
1385
+ "grok": true,
1386
+ "deep": true
1387
+ }
1388
+ },
1389
+ {
1390
+ "task_id": "HumanEval/152",
1391
+ "lanes": {
1392
+ "lola": true,
1393
+ "claude": true,
1394
+ "grok": true,
1395
+ "deep": true
1396
+ }
1397
+ },
1398
+ {
1399
+ "task_id": "HumanEval/153",
1400
+ "lanes": {
1401
+ "lola": true,
1402
+ "claude": true,
1403
+ "grok": true,
1404
+ "deep": true
1405
+ }
1406
+ },
1407
+ {
1408
+ "task_id": "HumanEval/154",
1409
+ "lanes": {
1410
+ "lola": true,
1411
+ "claude": true,
1412
+ "grok": false,
1413
+ "deep": false
1414
+ }
1415
+ },
1416
+ {
1417
+ "task_id": "HumanEval/155",
1418
+ "lanes": {
1419
+ "lola": true,
1420
+ "claude": true,
1421
+ "grok": true,
1422
+ "deep": true
1423
+ }
1424
+ },
1425
+ {
1426
+ "task_id": "HumanEval/156",
1427
+ "lanes": {
1428
+ "lola": true,
1429
+ "claude": true,
1430
+ "grok": true,
1431
+ "deep": true
1432
+ }
1433
+ },
1434
+ {
1435
+ "task_id": "HumanEval/157",
1436
+ "lanes": {
1437
+ "lola": true,
1438
+ "claude": true,
1439
+ "grok": true,
1440
+ "deep": true
1441
+ }
1442
+ },
1443
+ {
1444
+ "task_id": "HumanEval/158",
1445
+ "lanes": {
1446
+ "lola": false,
1447
+ "claude": true,
1448
+ "grok": true,
1449
+ "deep": false
1450
+ }
1451
+ },
1452
+ {
1453
+ "task_id": "HumanEval/159",
1454
+ "lanes": {
1455
+ "lola": false,
1456
+ "claude": true,
1457
+ "grok": true,
1458
+ "deep": true
1459
+ }
1460
+ },
1461
+ {
1462
+ "task_id": "HumanEval/160",
1463
+ "lanes": {
1464
+ "lola": true,
1465
+ "claude": true,
1466
+ "grok": true,
1467
+ "deep": false
1468
+ }
1469
+ },
1470
+ {
1471
+ "task_id": "HumanEval/161",
1472
+ "lanes": {
1473
+ "lola": true,
1474
+ "claude": true,
1475
+ "grok": true,
1476
+ "deep": true
1477
+ }
1478
+ },
1479
+ {
1480
+ "task_id": "HumanEval/162",
1481
+ "lanes": {
1482
+ "lola": true,
1483
+ "claude": true,
1484
+ "grok": true,
1485
+ "deep": true
1486
+ }
1487
+ },
1488
+ {
1489
+ "task_id": "HumanEval/163",
1490
+ "lanes": {
1491
+ "lola": false,
1492
+ "claude": true,
1493
+ "grok": false,
1494
+ "deep": false
1495
+ }
1496
+ }
1497
+ ]
1498
+ }
run_6lane.py ADDED
@@ -0,0 +1,90 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import json, sys, time, requests, subprocess
2
+ from datetime import datetime
3
+
4
+ HYPERNET_URL = "http://localhost:5000"
5
+ AUTH_TOKEN = "cpn-steve-kawa-hypernet-alpha"
6
+ LANES = ["lola", "claude", "grok", "deep", "gemini", "kimi"]
7
+
8
+ def call_lane(query, lane):
9
+ try:
10
+ r = requests.post(f"{HYPERNET_URL}/api/v1/run",
11
+ headers={"Authorization": f"Bearer {AUTH_TOKEN}"},
12
+ json={"query": query, "lane": lane}, timeout=120)
13
+ if r.status_code == 200:
14
+ return r.json()
15
+ except: pass
16
+ return {"error": "failed"}
17
+
18
+ def extract_code(response):
19
+ code = response
20
+ if "```python" in code:
21
+ code = code.split("```python")[1].split("```")[0]
22
+ elif "```" in code:
23
+ code = code.split("```")[1].split("```")[0]
24
+ return code.strip()
25
+
26
+ def test_solution(problem, solution):
27
+ code = extract_code(solution)
28
+ test_code = f'''{problem["prompt"]}
29
+ {code}
30
+
31
+ {problem["test"]}
32
+ check({problem["entry_point"]})
33
+ print("PASS")
34
+ '''
35
+ try:
36
+ result = subprocess.run([sys.executable, "-c", test_code],
37
+ capture_output=True, text=True, timeout=10)
38
+ return result.returncode == 0 and "PASS" in result.stdout
39
+ except: return False
40
+
41
+ def run_benchmark(problems):
42
+ results = {"lanes": {l: {"pass": 0, "fail": 0} for l in LANES}, "problems": []}
43
+
44
+ print(f"\n{'='*60}")
45
+ print(f"FULL 6-LANE BENCHMARK - 164 PROBLEMS")
46
+ print(f"{'='*60}\n")
47
+
48
+ for i, p in enumerate(problems):
49
+ print(f"[{i+1}/164] {p['task_id']}")
50
+ prob_result = {"task_id": p["task_id"], "lanes": {}}
51
+
52
+ for lane in LANES:
53
+ prompt = f"Solve this Python function. Return ONLY the implementation, no explanation.\n\n{p['prompt']}"
54
+ resp = call_lane(prompt, lane)
55
+
56
+ if resp.get("response_text"):
57
+ passed = test_solution(p, resp["response_text"])
58
+ prob_result["lanes"][lane] = passed
59
+ results["lanes"][lane]["pass" if passed else "fail"] += 1
60
+ print(f" {lane}: {'PASS' if passed else 'FAIL'}")
61
+ else:
62
+ prob_result["lanes"][lane] = False
63
+ results["lanes"][lane]["fail"] += 1
64
+ print(f" {lane}: ERROR")
65
+
66
+ results["problems"].append(prob_result)
67
+ print()
68
+
69
+ print(f"{'='*60}")
70
+ print("RESULTS (pass@1) - ALL 6 LANES")
71
+ print(f"{'='*60}")
72
+ for lane, stats in results["lanes"].items():
73
+ total = stats["pass"] + stats["fail"]
74
+ pct = (stats["pass"]/total*100) if total > 0 else 0
75
+ print(f" {lane:10s}: {stats['pass']:3d}/{total:3d} ({pct:.1f}%)")
76
+
77
+ return results
78
+
79
+ if __name__ == "__main__":
80
+ from datasets import load_dataset
81
+ print("Loading official HumanEval...")
82
+ ds = load_dataset("openai/openai_humaneval")
83
+ problems = [dict(item) for item in ds["test"]]
84
+ print(f"Loaded {len(problems)} problems\n")
85
+
86
+ results = run_benchmark(problems)
87
+
88
+ with open(f"humaneval_6lane_{datetime.now().strftime('%H%M%S')}.json", "w") as f:
89
+ json.dump(results, f, indent=2)
90
+ print("\nResults saved!")
run_full_benchmark.py ADDED
@@ -0,0 +1,101 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """HYPERNET N1 - OFFICIAL HUMANEVAL WITH CODE EXECUTION"""
3
+ import os, sys, json, time, requests, subprocess
4
+ from datetime import datetime
5
+
6
+ HYPERNET_URL = "http://localhost:5000"
7
+ AUTH_TOKEN = "cpn-steve-kawa-hypernet-alpha"
8
+ LANES = ["lola", "claude", "grok", "deep"]
9
+
10
+ def call_lane(query, lane):
11
+ try:
12
+ r = requests.post(f"{HYPERNET_URL}/api/v1/run",
13
+ headers={"Authorization": f"Bearer {AUTH_TOKEN}"},
14
+ json={"query": query, "lane": lane}, timeout=120)
15
+ if r.status_code == 200:
16
+ return r.json()
17
+ except: pass
18
+ return {"error": "failed"}
19
+
20
+ def extract_code(response):
21
+ code = response
22
+ if "```python" in code:
23
+ code = code.split("```python")[1].split("```")[0]
24
+ elif "```" in code:
25
+ code = code.split("```")[1].split("```")[0]
26
+ return code.strip()
27
+
28
+ def test_solution(problem, solution):
29
+ code = extract_code(solution)
30
+ test_code = f'''{problem["prompt"]}
31
+ {code}
32
+
33
+ {problem["test"]}
34
+ check({problem["entry_point"]})
35
+ print("PASS")
36
+ '''
37
+ try:
38
+ result = subprocess.run([sys.executable, "-c", test_code],
39
+ capture_output=True, text=True, timeout=10)
40
+ return result.returncode == 0 and "PASS" in result.stdout
41
+ except: return False
42
+
43
+ def run_benchmark(problems, limit=10):
44
+ results = {"lanes": {l: {"pass": 0, "fail": 0} for l in LANES}, "problems": []}
45
+
46
+ print(f"\n{'='*60}")
47
+ print(f"OFFICIAL HUMANEVAL - {limit} PROBLEMS - CODE EXECUTION")
48
+ print(f"{'='*60}\n")
49
+
50
+ for i, p in enumerate(problems[:limit]):
51
+ print(f"[{i+1}/{limit}] {p['task_id']}")
52
+ prob_result = {"task_id": p["task_id"], "lanes": {}}
53
+
54
+ for lane in LANES:
55
+ prompt = f"Solve this Python function. Return ONLY the implementation, no explanation.\n\n{p['prompt']}"
56
+ resp = call_lane(prompt, lane)
57
+
58
+ if resp.get("response_text"):
59
+ passed = test_solution(p, resp["response_text"])
60
+ prob_result["lanes"][lane] = passed
61
+ results["lanes"][lane]["pass" if passed else "fail"] += 1
62
+ print(f" {lane}: {'PASS' if passed else 'FAIL'}")
63
+ else:
64
+ prob_result["lanes"][lane] = False
65
+ results["lanes"][lane]["fail"] += 1
66
+ print(f" {lane}: ERROR")
67
+
68
+ results["problems"].append(prob_result)
69
+ print()
70
+
71
+ # Summary
72
+ print(f"{'='*60}")
73
+ print("RESULTS (pass@1)")
74
+ print(f"{'='*60}")
75
+ for lane, stats in results["lanes"].items():
76
+ total = stats["pass"] + stats["fail"]
77
+ pct = (stats["pass"]/total*100) if total > 0 else 0
78
+ print(f" {lane:10s}: {stats['pass']:3d}/{total:3d} ({pct:.1f}%)")
79
+
80
+ return results
81
+
82
+ if __name__ == "__main__":
83
+ from datasets import load_dataset
84
+ print("Loading official HumanEval...")
85
+ ds = load_dataset("openai/openai_humaneval")
86
+ problems = [dict(item) for item in ds["test"]]
87
+ print(f"Loaded {len(problems)} problems\n")
88
+
89
+ print("Options:")
90
+ print(" 1. Run 10 problems (test)")
91
+ print(" 2. Run 50 problems")
92
+ print(" 3. Run ALL 164 problems")
93
+ choice = input("Choice (1/2/3): ").strip()
94
+
95
+ limit = {1: 10, 2: 50, 3: 164}.get(int(choice), 10)
96
+ results = run_benchmark(problems, limit)
97
+
98
+ # Save
99
+ with open(f"humaneval_results_{datetime.now().strftime('%H%M%S')}.json", "w") as f:
100
+ json.dump(results, f, indent=2)
101
+ print("\nResults saved!")