Published studies · compare
How different models answered the same question
The same research question and the same data, run across three model backends with two repeats each. The comparison shows where the runs made the same design choices and where they diverged.
These runs are versions of the SAME research question — coverage of the solution space, not a contest. No run is promoted here. Any decision to carry one run's estimates forward must be pre-committed in the pre-registration, never chosen after seeing this comparison.
Question. Did the January 2024 approval of US spot Bitcoin ETFs change the co-movement between Bitcoin returns and US equity returns?
Design-choice matrix
Agreement is the share of runs sharing the modal value, or the mean pairwise Jaccard index for fixed effects and controls. ⚠ marks a field where the runs diverged.
| Field | claude_code-1 | claude_code-2 | codex-1 | codex-2 | gemini-1 | gemini-2 | Agreement |
|---|---|---|---|---|---|---|---|
| ⚠ estimator | two-way fixed-effects DiD | two-way fixed-effects DiD | two-way fixed-effects DiD | event study | two-way fixed-effects DiD | two-way fixed-effects DiD | 0.83 |
| unit_of_analysis | coin-month | coin-month | coin-month | coin-month | coin-month | coin-month | 1.00 |
| outcome | Fisher-z correlation with S&P 500 | Fisher-z correlation with S&P 500 | Fisher-z correlation with S&P 500 | Fisher-z correlation with S&P 500 | Fisher-z correlation with S&P 500 | Fisher-z correlation with S&P 500 | 1.00 |
| treatment | d_etf_listed | d_etf_listed | d_etf_listed | d_etf_listed | d_etf_listed | d_etf_listed | 1.00 |
| ⚠ fixed_effects | coin, year-month | coin, year-month | coin, year-month | coin, year-month | coin, month | coin, year-month | 0.78 |
| ⚠ controls | VIX, DXY | VIX, DXY, 10y yield | VIX | VIX | VIX, DXY | VIX, DXY | 0.64 |
| ⚠ cluster_level | coin | coin | coin | coin | coin and month | coin | 0.83 |
| identifying_assumption | parallel trends in co-movement | parallel trends in co-movement | parallel trends in co-movement | parallel trends in co-movement | parallel trends in co-movement | parallel trends in co-movement | 1.00 |
| ⚠ n_observations | 540 | 540 | 540 | 540 | 520 | 540 | 0.83 |
| coef_term | d_etf_listed | d_etf_listed | d_etf_listed | d_etf_listed | d_etf_listed | d_etf_listed | 1.00 |
| coef_estimate | -0.021 | -0.018 | -0.034 | -0.029 | -0.009 | -0.015 | · |
| coef_se | 0.017 | 0.018 | 0.016 | 0.019 | 0.021 | 0.018 | · |
| coef_p_value | 0.24 | 0.33 | 0.05 | 0.14 | 0.67 | 0.41 | · |
Divergent fields: estimator, fixed_effects, controls, cluster_level, n_observations
Coefficient of interest: dispersion
Descriptive variance of the point estimate; no significance tests.
- Estimates
- -0.021, -0.018, -0.034, -0.029, -0.009, -0.015
- Overall variance
- 7.03e-5
- Within backends
- 5.83e-6
- Between backends
- 6.45e-5
Run it on your question
The subscription backends cost nothing per token, so the default matrix runs three of them.
e2er run-matrix "<your research question>" --backends claude_code,codex,gemini --repeats 3e2er compare matrix.json