Reading the leaderboard
Snapshot date: 29 July 2026. Matbench Discovery is a live, PR-driven leaderboard; every number below was read from the model registry on that date and several of them will have moved by the time you read this. Where a figure is soft, or where the benchmark itself warns that a metric is unstable, it is flagged in place.
Matbench Discovery (Riebesell, Goodall, Benner, Chiang, Deng, Ceder, Asta, Lee, Jain and Persson; arXiv:2308.14920; Nature Machine Intelligence 7, 836–847, 2025, doi:10.1038/s42256-025-01055-1) is the benchmark that the machine-learned interatomic potential field currently argues about. It is worth understanding in detail, because almost every claim of the form “model X is the best universal potential” resolves, on inspection, to a claim about one of its subscores — and the subscores disagree with each other in ways that are not accidents.
The benchmark’s founding move is to refuse the obvious framing. It does not ask a model to regress formation energies on a random hold-out split. It asks: if you were spending a fixed DFT budget searching for new stable inorganic crystals, and you used this model as the pre-filter that decides which candidates get relaxed, how much better off would you be? Everything downstream — the test set, the metrics, the aggregate score — follows from taking that question literally.
1. The pipeline, and what a model is actually asked to do
A submission receives an unrelaxed candidate structure — a prototype generated by a substitution procedure described in §2, not a DFT-relaxed geometry — and must produce the energy of the structure after DFT relaxation. Most modern entries do this by running their own machine-learned relaxation first and then evaluating the energy at their own relaxed geometry; the benchmark calls this test task IS2RE-SR (initial structure to relaxed energy, with structure relaxation). The alternatives are IS2RE (predict the relaxed energy directly from the unrelaxed structure, no relaxation) and RS2RE (given the DFT-relaxed structure, predict its energy — a much easier and less honest task, since in a real campaign you do not have the relaxed structure). Training tasks are coded symmetrically: S2E, S2EF, S2EFS, S2EFSM for structure to energy, plus forces, plus stress, plus magnetic moments.
The predicted energy is converted to a formation energy, then to a distance above the Materials Project convex hull, then thresholded to a binary stable/unstable decision. The chain matters, because errors enter at the energy and are judged at the decision, and the map between them is not Lipschitz in any useful sense.
2. WBM: a test set built by elemental substitution
The test set is not held out from the training distribution by random sampling. It is a separate dataset, WBM, built by Hai-Chen Wang (Halle), Silvana Botti (Jena) and Miguel Marques (Halle) (doi:10.1038/s41524-020-00481-6) for a different purpose and adopted wholesale.
WBM was generated by single-element substitution on Materials Project source structures. For each
source prototype, one element is swapped for a chemically similar one, with the replacement drawn at
random with weights taken from a chemical-similarity matrix data-mined from the ICSD. Every generated
candidate was then relaxed with VASP using MP-compatible settings — MPRelaxSet, the same POTCARs,
PBE+U, ISYM=2, ISIF=3, IBRION=2 conjugate-gradient relaxation — and MP2020 anion/GGA+U
compatibility corrections were applied post hoc so that WBM energies sit on the same scale as the MP
hull. Hull distances are computed against the MP PatchedPhaseDiagram built from the 2022.10.28
release, queried 2023-02-07.
That DFT-protocol identity is the design’s cleverest feature. Because the electronic-structure settings match MP exactly, any degradation in model accuracy on WBM is attributable to extrapolation — the chemistry and structure have shifted — rather than to a covariate shift in the labels themselves. You are measuring the model, not the DFT.
Sizes. 257,487 raw structures reduce to 256,963 after cleaning. The cleaning filters are: six zero-volume structures, 502 entries with formation energy below $-5$ eV/atom (including an anomalous block of 500 structures sitting at exactly $-10$ eV/atom, which is a data-generation artefact rather than a chemistry result), and 22 entries above $+5$ eV/atom. Those counts sum to 530 against an observed reduction of 524, so a handful of structures fail more than one filter; the surviving total of 256,963 is the number that matters. Of these, 42,825 (16.7%) lie on or below the MP hull. WBM is roughly 75% larger than the MP training set and is enriched in transition metals, post-transition metals and metalloids relative to MP.
Unique prototypes. Raw WBM contains structures whose Aflow-style protostructure label —
stoichiometry, space group, Pearson symbol, Wyckoff positions — duplicates either an MP structure or
another WBM structure. A model would score points on those for material it effectively saw in
training. The benchmark therefore computes protostructure labels on the relaxed structures (so that
distinct initial prototypes falling into the same basin of attraction do not inflate the metrics),
drops every WBM entry whose prototype already appears in MP, and keeps only the lowest-energy
representative of each remaining prototype. That removes 41,475 candidates (11,175 of them
overlapping MP prototypes, the rest intra-WBM duplicates) and leaves 215,488 unique prototypes,
of which 32,942 (15.29%) are stable. This is the default leaderboard view. Two other subsets are
reported alongside it: the full 256,963-structure test set, and most_stable_10k, each model’s
10,000 most-stable predictions — the fixed-budget view that most closely matches how a screening
campaign actually spends money.
The five batches
The substitution procedure was iterated five times. Round 1 starts from structures on the MP
convex hull; every generated candidate is relaxed, and those that turn out to lie on the hull are fed
back into the parent pool for round 2. The batch index is encoded in the material ID
(wbm-<batch>-<n>), so a wbm-5-* structure has had five successive element swaps applied to an MP
ancestor and is, on average, chemically furthest from anything in MP.
| Batch | After cleaning | Unique prototypes |
|---|---|---|
| 1 | 61,466 | 54,209 |
| 2 | 52,755 | 45,979 |
| 3 | 79,160 | 66,528 |
| 4 | 40,314 | 34,531 |
| 5 | 23,268 | 14,241 |
| Total | 256,963 | 215,488 |
(The pre-cleaning per-batch figures quoted in the WBM documentation — 61,848 / 52,800 / 79,205 / 40,328 / 23,308 — sum to 257,489, two more than the quoted raw total of 257,487, so treat the raw per-batch numbers as approximate. The two columns above sum exactly.)
This staged structure is the single most useful diagnostic property of the test set, and it is underused. It converts an artefact of the data-generation loop into a graded out-of-distribution ladder: five nested slices of monotonically increasing distance from the training manifold, on which you can watch a model’s rolling MAE grow as it is pushed away from what it knows. The benchmark’s own finding is that “batches 4 and 5 consistently incur the highest convex hull distance errors for all models”, but with an important asymmetry — universal interatomic potentials degrade only mildly across batches, while structure-only regressors such as Wrenformer show a pronounced MAE increase with batch index. That contrast is the closest thing the benchmark has to a direct measurement of whether a model has learned physics or has memorised MP chemistry, and it is invisible in the headline F1.
Note also what the batch ladder cannot fix. Iterated substitution from an MP seed is still iterated substitution. Batch 5 is chemically far from MP in the sense of composition, but it is not structurally novel: it inhabits the same prototype space, reached by five hops instead of one. The ladder measures compositional extrapolation, not the discovery of new structural motifs.
The reading group got there first
In the 23 July 2026 session, before anyone in the room had opened the WBM documentation, one participant worked through the leaderboard’s headline classification numbers, noted the roughly 200,000-structure candidate pool, and then dismissed it: “it’s just like very individual [ionic] substitutions in it.” (The automatic transcript is poor throughout and renders ionic as economic; the surrounding sentences are too garbled to quote, but the complaint that follows is clearly that the candidate generation is too constrained to be interesting and that they wanted something else out of the table.) That is a precise, unprompted description of the WBM generation protocol, arrived at from the smell of the numbers alone by someone who did not know the protocol had a name. Another participant had, a moment earlier, made the benchmark’s own argument about accuracy — that on an unbalanced set F1 is the better score — which is exactly why Matbench Discovery ranks on F1 and treats accuracy as nearly uninformative at 15.3% prevalence.
The criticism is correct and it is worth stating plainly rather than defending: WBM’s candidates are the output of a chemical-similarity-weighted substitution engine seeded on the Materials Project. They are not proposals from a generative model exploring novel structure space, and they are not targeted at any property a materials scientist wants. The benchmark measures how well a model pre-filters that pool. Whether that generalises to a pool built some other way is an open question the benchmark does not answer.
3. Training sets, and how to read a suffix
The sanctioned corpus
To keep the benchmark prospective, the compliant training corpus is frozen at everything available in
the MP v2022.10.28 release (snapshot taken 2023-03-15): energies, forces, stresses and magnetic
moments for every MP ionic relaxation step. Energy-only models typically train on the 154,719
final relaxed ComputedStructureEntries. Universal potentials train on MPtrj, the cleaned,
subsampled trajectory set curated by Bowen Deng for CHGNet (doi:10.1038/s42256-023-00716-3),
comprising 1,580,395 structures across 145,923 materials — MP trajectories with unrealistic
energies and forces removed and roughly every tenth ionic step retained to cut redundancy. MPtrj is
the de facto control condition of the entire board. (Its predecessor MPF.2021.2.8, 188,349 frames
over 62,783 materials, is what M3GNet was trained on.)
The three non-compliant corpora
Alexandria (Schmidt, Hoffmann, Wang, Botti, Marques et al., doi:10.1002/adma.202210788) is a PBE relaxation-trajectory database of 30,497,628 structures across ~3.1 million materials. sAlex is Alexandria decontaminated against WBM: every trajectory whose structure carries a protostructure label appearing in any WBM initial or relaxed structure is deleted, plus filters ($E > 0$ eV, $|F| > 50$ eV/Å, $|\sigma| > 80$ GPa) and 10 meV/atom energy-spacing subsampling, leaving 10,447,765 training structures and 553,218 validation structures.
OMat24 (Barroso-Luque, Shuaibi, Fu, Wood, Dzamba, Gao, Rizvi, Zitnick, Ulissi — FAIR at Meta, arXiv:2410.12771) takes Alexandria structures as seeds and perturbs them hard: rattled Boltzmann sampling (500 candidates per seed from Gaussian position noise at $\sigma = 0.5$ Å plus cell deformation, of which five are drawn from Boltzmann distributions at 300, 500 and 1000 K); 50-step AIMD at 1000 K and 3000 K in NVT and NPT; and re-relaxation of mildly rattled structures ($\sigma = 0.2$ Å, 4% cell). The result is 100,824,585 training structures (from ≤3,227,606 materials) out of 118M labelled in total, deliberately concentrated off equilibrium, where MP data is empty. CC-BY-4.0.
Decoding “-MP”, “-MPA”, “-OMat24”, “-OAM”
The suffixes on leaderboard entries name the training mixture, and they are load-bearing:
| Suffix | Training mixture | Compliant? |
|---|---|---|
-MP, -MPtrj |
MPtrj only (1.58M structures, 146k materials) | yes |
-MPA |
MPtrj + sAlex (e.g. MACE-MPA-0, orb-v2-mpa) | no |
-OMat24, -OMAT |
trained or pretrained on OMat24 (101M structures) | no |
-OAM (sometimes -OMA) |
OMat24 + Alexandria(sAlex) + MPtrj — the full three-stage recipe | no |
The -OAM recipe is not a data union; it is a sequence, and the last stage is not cosmetic. The
published eSEN-30M-OAM procedure is three stages: (1) two epochs of direct pre-training on OMat24;
(2) two epochs of conservative fine-tuning on OMat24; (3) one epoch of conservative fine-tuning on
MPtrj + sAlex. Stage (3) exists because OMat24 is systematically incompatible with the Materials
Project at the DFT level: OMat24 uses VASP PAW version 54 pseudopotentials (with Yb_3 and W_sv)
where MP uses version 52, and the mean formation-energy difference between the two protocols is
13.5 meV/atom. An OMat24-only model therefore predicts energies on a shifted reference and cannot
be scored against the MP convex hull at all — which is why entries labelled purely -OMat24 are
normally reported without an F1, and why the recipe is always pretrain-then-fine-tune rather than
train-and-stop. Thirteen and a half meV/atom is not a rounding error on a benchmark whose stability
threshold is exactly zero and whose best models have an 18 meV/atom MAE.
Compliance and openness are orthogonal
Two labels govern how a submission is read. Compliance is about training-data provenance: a
compliant model trains only on data available in MP v2022.10.28. Anything trained on OMat24,
Alexandria, GNoME, MatterSim or OpenLAM is non-compliant and appears in a separate filtered view.
Openness is a four-way code — OSOD (open source, open data), OSCD, CSOD, CSCD — describing
whether code and training data are public. They are independent: eqV2 M is OSOD and non-compliant.
The mid-2026 census of the default view is 45 OSOD, 5 OSCD, 1 CSOD. Every model registry entry
carries a machine-readable training_sets, openness, train_task, test_task and targets
declaration, and every dataset in datasets.yml carries a compliant flag and a free-text
notes.Leakage block.
The maintainers’ position on the fairness objection — that comparing a 1.6M-frame MPtrj model against a 100M-structure OMat24 model as though architecture were the variable is unfair — is that leveraging more data is a genuine capability the benchmark should reflect, so both views are published rather than one.
What the data actually buys
The popular reading is that the 2024–2026 jump was a data-scale story. On the leaderboard’s own composite metric, it was not, or at least not primarily.
Holding the architecture fixed and swapping MPtrj for the OAM mixture is worth a real, consistent, and bounded amount:
| Architecture | MPtrj-only CPS | OAM CPS | Δ |
|---|---|---|---|
| GRACE-2L | 0.681 | 0.837 | +0.156 |
| NequIP-L | 0.730 | 0.870 | +0.140 |
| Allegro-L | 0.720 | 0.840 | +0.120 |
| MatRIS-10M | 0.778 | 0.877 | +0.099 |
| eSEN-30M | 0.797 | 0.888 | +0.091 |
| EquiformerV3+DeNS | 0.830 | 0.902 | +0.072 |
Roughly +0.07 to +0.16 CPS, clustering near +0.10, with the best-engineered architecture showing the smallest gain. Now compare the spread within the OAM corpus itself, where the data is fixed: from TECE-OAM-RRA-1.0 at 0.908 down to eqV2 M at 0.558 — a spread of 0.35 CPS, more than twice the data premium. And hold the data at MPtrj and let architecture and training recipe vary: MACE-MP-0 scored 0.637; DPA-4.0.1-Pro-MPtrj, on the same 1.58M structures, scores 0.840. That +0.203 is about 75% of the entire 0.637 → 0.908 climb to the current top of the board.
DPA-4.0.1-Pro-MPtrj deserves emphasis: 22.8M parameters, MPtrj only, CPS 0.840 / F1 0.857 / $\kappa_{\mathrm{SRME}}$ 0.211. It matches or beats OAM-trained models with roughly 71× more training data, and it beats EquiformerV3+DeNS-MP (0.830), the MPtrj ablation of the model that holds the board’s best F1.
Nor is the data effect primarily a scale effect. The OMat24 paper’s own ablation (arXiv:2410.12771v2, Table 2) holds the eqV2-S architecture and the MPtrj fine-tune fixed and varies only the pretraining corpus: pretraining on OC20 (134M structures) gives F1 0.837 and 33 meV/atom MAE; pretraining on OMat24 (101M structures — fewer) does better. What OMat24 bought was non-equilibrium and compositional diversity, plus the correction of a systematic softening in MP-trained potentials — not raw size.
4. Energy above the hull, and why stability is a decision
The target
Energy above the convex hull is the per-atom energy difference between a material and the lowest-energy linear combination of competing phases at the same composition:
\[E_{\text{above hull}}(x) \;=\; E_{\text{form}}(x) \;-\; E_{\text{hull}}\!\left(\text{comp}(x)\right)\]where $E_{\text{hull}}$ is evaluated on the convex hull of the formation-energy-versus-composition
diagram, here the MP PatchedPhaseDiagram. A value $\le 0$ means no combination of known phases is
more stable, so at 0 K and 0 bar the material is thermodynamically stable.
The reason the benchmark targets this rather than formation energy is that formation energy alone says nothing about stability. A compound can have a strongly negative $E_{\text{form}}$ and still sit far above the hull because its chemical system is crowded with even better phases. The paper is blunt: high-throughput DFT formation energies, “although widely used as regression targets, do not indicate thermodynamic stability or synthesizability.”
One consequence is worth internalising, because it trips people up: since the hull reference is a per-composition constant, MAE on hull distance is numerically identical to MAE on formation energy. The two are the same number. What differs is only what you do with it afterwards.
A second consequence is a reminder about what “stable” means here. The reference hull is MP’s. If you let WBM compete against itself — build the joint MP+WBM hull — the ~33k stable unique prototypes collapse to about 20k, roughly 10% of all WBM. Something like 40% of the “discoveries” are discoveries only because the other discoveries were not counted.
The threshold, and the decision framing
Every continuous prediction is thresholded at exactly 0 eV/atom above the MP hull
(STABILITY_THRESHOLD = 0 in matbench_discovery/__init__.py). classify_stable() is as literal as
it looks:
This mirrors how a screening campaign spends its budget — you either compute a candidate or you don’t — so the quantity that matters is the decision, not the residual. The threshold is a tunable parameter, so precision/recall trade-off curves and cumulative-discovery curves can be swept; but 0 is the published operating point, which is stricter than the 50 or 100 meV/atom windows many screening campaigns use in practice.
The consequences are sharp. A model can have excellent MAE and still be a poor screener if its errors are concentrated near the boundary; conversely a mediocre regressor whose errors are large but far from 0 can classify well. The benchmark’s diagnostic for this is the “triangle of peril”: plot rolling MAE against true hull distance, and shade the region where the local MAE exceeds $|E_{\text{hull}}|$. “As long as a model’s rolling MAE remains inside the triangle, the model is highly susceptible to misclassifying structures.” The empirical demonstration in the paper is that Voronoi RF, CGCNN and MEGNet score worse than a dummy predictor on regression metrics while beating the dummy on some classification metrics — and that CGCNN+P has better regression metrics than CGCNN yet is a worse classifier, because its predicted hull-distance histogram is more sharply peaked at 0, so even tiny errors invert the decision.
This repo’s own numerical experiment
(results/convex_hull_decision.json, chapter
Stability is a decision, not a regression) reproduces the inversion
under controlled conditions on a synthetic binary A–B system (260 candidates, 27 compositions, 11
stable, prevalence 4.2%). Predictor $P$ adds i.i.d. Gaussian noise of constant standard deviation 25
meV/atom; predictor $Q$ adds Gaussian noise of standard deviation $2\text{ meV/atom} +
0.35\,E_{\text{hull}}$ — accurate near the hull, sloppy far above it. $P$ wins on MAE (20.0 vs 48.5
meV/atom) and loses on F1 (0.454 vs 0.824), and the inversion holds in 96.5% of 200 trials. The
same run shows the metrics have completely different sensitivities: MAE’s log-log elasticity with
respect to the noise scale is pinned at 1.000 by construction, while F1’s ranges from 0.064 to 0.815
— and F1 falls from 0.9 to 0.3 across a window only 33.2 meV/atom wide in MAE. There is no
power-law conversion between the two.
The classification metrics
With “positive” = predicted stable:
\[\text{Precision} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\qquad \text{Recall} = \mathrm{TPR} = \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},\qquad \mathrm{F1} = \frac{2\,\text{Precision}\cdot\text{Recall}}{\text{Precision}+\text{Recall}}\]with $\mathrm{TNR} = \mathrm{TN}/(\mathrm{TN}+\mathrm{FP})$, $\mathrm{FPR} = 1 - \mathrm{TNR}$, $\mathrm{FNR} = 1 - \mathrm{TPR}$ and accuracy $= (\mathrm{TP}+\mathrm{TN})/N$.
F1 is the primary discovery ranking metric because precision and recall pull in opposite directions in a screening campaign. Precision is your DFT hit rate — what fraction of the structures you paid to compute turned out stable. Recall is your coverage — what fraction of the stable materials in the pool you actually found. F1 refuses to let a model win by being either ultra-conservative (high precision, tiny recall) or ultra-permissive (high recall, worthless precision).
Accuracy is nearly useless here, and eqV2 M is the standard demonstration. Its accuracy on unique prototypes is 0.975, which sounds superb; the underlying confusion matrix is 30,372 true positives, 2,481 false positives, 3,002 false negatives and 179,633 true negatives. At 15.3% prevalence the negative class dominates the numerator, and the 0.975 is mostly a statement about how many unstable compounds the model correctly declined.
That confusion matrix also exposes a subtlety in the DAF denominator worth recording, because it is the kind of thing that silently corrupts cross-paper comparisons. The classification counts are computed against reference hull distances rounded to three decimals, which flips 432 barely unstable unique prototypes to stable (33,374 counted stable versus 32,942 actually stable — a 1.3% inflation). The DAF denominator deliberately uses the unrounded prevalence, 0.1529, so that every published DAF stays comparable across re-evaluations.
DAF
\[\mathrm{DAF} \;=\; \frac{\text{Precision}}{\text{Prevalence}}\]The Discovery Acceleration Factor answers the only question a lab cares about: how many times more
stable materials do I find per DFT calculation if the model picks the candidates, versus picking at
random from the same pool? It is mathematically identical to “lift” in the direct-marketing
literature; the contribution is naming it, giving it a physical interpretation in DFT-hours saved,
and making its ceiling explicit. Because it is precision-derived it is bounded above by
$1/\text{prevalence}$: 6.54 on unique prototypes and 6.00 on the full test set. Current
values sit close to that ceiling — eqV2 M 6.047 and ORB v2 MPA 6.041 on unique prototypes, and on
most_stable_10k ORB v2 MPA reaches 6.348 against a 6.54 ceiling. A DAF of 6.0 on this benchmark is
near-saturation, not a score with headroom. Reporting DAF improvements at this point is close to
reporting nothing.
The demoted metrics
MAE, RMSE and $R^2$ on hull distance are reported and explicitly not ranked on. The paper’s framing is a negative result: “Improvements in regression accuracy are of limited use to materials discovery in their own right unless they also improve classification accuracy. Our results demonstrate that this is not a given.” Typical current values on unique prototypes for the leaders are MAE 0.018 eV/atom, RMSE 0.065–0.068 eV/atom, $R^2$ 0.86–0.875; a negative $R^2$, which several early entries post, means the model explains the data less well than predicting the test-set mean.
5. $\kappa_{\mathrm{SRME}}$: what a good F1 cannot certify
The metric
$\kappa_{\mathrm{SRME}}$ — symmetric relative mean error on lattice thermal conductivity — was introduced by Póta, Ahlawat, Csányi and Simoncelli (Cambridge, arXiv:2408.00755) and folded into Matbench Discovery in late 2024. It compares a model’s predicted lattice thermal conductivity against DFT-PBE references mode by mode:
\[\mathrm{SRME} \;=\; \frac{2\sum_i w_i\,\bigl|\kappa_{\text{pred},i} - \kappa_{\text{true},i}\bigr|} {\kappa_{\text{pred,tot}} + \kappa_{\text{true,tot}}}\]where $i$ runs over phonon modes (q-point, branch) and $w_i$ are mode weights. The mode resolution is the point: the scalar variant $\mathrm{SRE} = 2|\kappa_{\text{pred}} - \kappa_{\text{true}}|/(\kappa_{\text{pred}}+\kappa_{\text{true}})$ lets over- and under-predicted modes cancel, so a badly wrong potential can look right on the total. The signed variant $\mathrm{SRD} \in [-2, 2]$ records the direction. $\mathrm{SRME}$ is bounded in $[0, 2]$, with 0 perfect and 2 maximum error.
The protocol is: one simultaneous cell-and-site relaxation, then finite-displacement forces for
second- and third-order force constants, then thermal conductivity via the Wigner/BTE treatment in
phono3py. Evaluation is on 103 PhononDB-PBE crystals — binary compounds, 34 chemical species,
rocksalt/zincblende/wurtzite structures. Failures are censored to 2: imaginary frequencies after
relaxation, symmetry broken during relaxation, or an outright failed calculation all score the
maximum, with kappa_failure_rate and imaginary_mode_rate reported separately.
What it probes that nothing else on the board does is the anharmonicity of the potential energy surface — the second and third derivatives of $E$ with respect to atomic positions. Energy and hull-distance metrics are almost completely blind to these. It was added precisely because it exposes models that post great discovery numbers while carrying a physically unusable PES.
The eqV2 M case
eqV2 M (EquiformerV2-Medium, 86.6M parameters, pretrained on OMat24 and fine-tuned on sAlex+MPtrj,
targets: EFS_D) is the standard exhibit. On unique prototypes it posts:
| Metric | Value | Rank |
|---|---|---|
| F1 | 0.917 | #9 / 51 |
| DAF | 6.047 | near ceiling |
| MAE | 0.020 eV/atom | — |
| RMSE | 0.072 eV/atom | — |
| $R^2$ | 0.848 | — |
| RMSD | 0.0691 | #17 / 42 |
| $\kappa_{\mathrm{SRME}}$ | 1.7707 | #41 / 41 |
| CPS | 0.558 | #36 / 41 |
Its $\kappa$ failure rate and imaginary-mode rate are both 5.8%. A model with the ninth-best classification score on the board, whose geometry optimisation looks entirely healthy at RMSD 0.0691, is within 0.23 of the worst possible value on thermal conductivity. eSEN-30M-MP — an MPtrj-only compliant model with a worse F1 of 0.831 — scores $\kappa_{\mathrm{SRME}}$ 0.340 and CPS 0.797, beating eqV2 M by 0.24 CPS while losing the headline discovery metric by 0.086.
The conservativeness story, and why it is wrong as usually told
The usual gloss is that this is a direct-force problem: eqV2 M predicts forces from a separate network head rather than as $-\nabla_{\mathbf r} E$, so the learned field has no guaranteed conservative structure, and phonons — which depend on the Hessian — collapse.
The association is real and strong. All four direct-force (EFS_D) models on the board sit above
$\kappa_{\mathrm{SRME}}$ 1.67, and every model below 1.0 is energy-gradient-based. The cleanest
before/after on the whole leaderboard is ORB’s: ORB v2 (direct) posts F1 0.880 and
$\kappa_{\mathrm{SRME}}$ 1.7338; ORB v3 (orb-v3-conservative-inf-mpa, same lineage, conservative
head plus unlimited-neighbour graph construction) posts F1 0.905 and $\kappa_{\mathrm{SRME}}$ 0.2102.
But non-conservativeness is neither necessary nor sufficient, and the leaderboard’s own numbers say so. Seven models on the board exceed $\kappa_{\mathrm{SRME}}$ 1.0. Only four of them are direct:
| Model | $\kappa_{\mathrm{SRME}}$ | Force type | F1 | CPS |
|---|---|---|---|---|
| eqV2 M | 1.7707 | direct (EFS_D) |
0.917 | 0.558 |
| ORB v2 (MPA) | 1.7338 | direct (EFS_D) |
0.880 | 0.528 |
| ORB v2 (MPtrj) | 1.7255 | direct (EFS_D) |
0.765 | 0.470 |
| CHGNet 0.3.0 | 1.7167 | conservative (EFS_GM) |
0.613 | 0.400 |
| eqV2 S DeNS (MPtrj) | 1.6763 | direct (EFS_D) |
0.815 | 0.522 |
| M3GNet | 1.4094 | conservative (EFS_G) |
0.569 | 0.428 |
| AlphaNet-v1-MPtrj | 1.3046 | conservative (EFS_G) |
0.799 | 0.567 |
CHGNet is a fully conservative model — its forces and stresses are analytic gradients of the
predicted energy, the G in EFS_GM — and it scores 1.7167, essentially indistinguishable from
eqV2 M’s 1.7707 and worse than eqV2 S DeNS. That refutes non-conservativeness as a necessary
condition for phonon failure. M3GNet at 1.4094 and AlphaNet-v1-MPtrj at 1.3046 confirm it is not an
isolated case.
And the converse fails too. The ORB v3 paper (arXiv:2504.06231, Table 1) reports direct-force
variants with good thermal conductivity: orb-v3-direct-inf-mpa achieves F1 0.883 with
$\kappa_{\mathrm{SRME}}$ 0.348 — better than several conservative models on the board, though this
checkpoint is not the one submitted to the leaderboard. That refutes non-conservativeness as a
sufficient condition.
The quantity that actually determines $\kappa_{\mathrm{SRME}}$ is the accuracy of the second (and third) derivatives of the energy surface — the force constants that phonons are built from. Direct force training does not constrain them at all, which is why direct models fail so reliably. But a conservative model constrains only that the force field is a gradient; it says nothing about whether the curvature of that potential is right. A conservative model trained on a corpus with no off-equilibrium sampling, or with a discontinuous radial cutoff, can and does get the Hessian wrong. Póta et al. found precisely this: MP-trained potentials systematically underestimate thermal conductivity, consistent with PES softening inflating anharmonic linewidths. That failure has nothing to do with force heads.
The eSEN paper (arXiv:2502.12147) makes the constructive version of the argument — that smoothness of the learned energy surface, not expressivity and not the force head alone, is what determines whether a model can do phonons — and demonstrates the diagnostic: conservative models like eSEN and MACE give phonon band structures that converge as the finite-displacement amplitude shrinks, while eqV2-S-DeNS does not, showing missing acoustic branches and spurious imaginary frequencies at small displacements while still looking competitive at the 0.2 Å amplitude where Boltzmann-weighted integrals smooth the errors away.
The repo’s controlled version
This distinction is easy to state and hard to see, so
results/conservative_vs_direct.json isolates it on a
six-particle Morse cluster with architecture-matched surrogates (chapter
Forces from a gradient). Both models are exactly O(3)-equivariant to
machine precision — maximum relative equivariance deviation $1.5\times10^{-14}$ and
$2.6\times10^{-14}$ respectively — so equivariance is explicitly not what distinguishes them, and
the conservative model’s parameter space is a strict subspace of the direct model’s (28 versus 79
parameters).
On one-step force RMSE the direct model is the better of the two: 0.0114 versus 0.0129, a factor 0.881. Everything downstream of the first derivative then diverges:
| Quantity | Conservative | Direct | Ratio |
|---|---|---|---|
| Force RMSE (test, clean) | 0.0129 | 0.0114 | 0.88 |
| RMS relative force-Jacobian asymmetry | $1.4\times10^{-8}$ | $7.3\times10^{-3}$ | $5.2\times10^{5}$ |
| Median relative net work around a closed loop | $3.1\times10^{-16}$ | $3.6\times10^{-3}$ | $\sim10^{13}$ |
| Median MD energy-drift rate | $2.8\times10^{-8}$ | $8.0\times10^{-5}$ | 149 |
The Jacobian asymmetry row is the one that matters for $\kappa_{\mathrm{SRME}}$: it is the second-derivative defect, measured directly. A force field that is not a gradient has a non-symmetric $\partial F_i / \partial r_j$, so its force-constant matrix is not the Hessian of any scalar potential. A phonon code fed such a field will still return frequencies — it symmetrises — but they are the frequencies of an object that no energy surface generates, and the error is exactly the antisymmetric part being thrown away. 12.5% of the direct model’s MD trajectories run away over 40,000 steps. One negative result is worth reporting rather than tuning away: the direct model’s drift is not systematic heating — 15 of 24 trajectories gain energy, binomial $p = 0.31$ — so a non-conservative field is better described as pumping energy in or out depending on where in phase space it goes. What is systematic is the magnitude: every single trajectory drifts faster under the direct model, by at least 38×.
6. RMSD, and the Combined Performance Score
Geometry optimisation
Every model running the IS2RE-SR task produces a relaxed structure, so the benchmark checks whether
that structure matches the DFT-relaxed ground state. RMSD is the normalised, unitless root-mean-square
displacement returned by pymatgen’s StructureMatcher (configured stol=1.0, scale=False) after
matching the ML-relaxed and DFT-relaxed cells, averaged over the ~257k WBM relaxations (each reported
model evaluated on at least 249k of them). Structures that StructureMatcher cannot match at all are
assigned RMSD = 1.0 rather than dropped, which closes an obvious gaming loophole. It is
symprec-invariant, unlike the symmetry statistics reported alongside it at symprec $10^{-2}$ and
$10^{-5}$ — space-group match rate $\Sigma_=$, the fractions whose symmetry-operation count decreased
or increased ($\Sigma_\downarrow$, $\Sigma_\uparrow$), and $N_{\text{ops}}$ MAE, all computed with
moyopy.
The documented caveat matters: WBM’s DFT used MPRelaxSet with ISYM=2 and conjugate-gradient
relaxation, while most MLFF relaxations use FIRE. A symmetry mismatch is therefore not automatically
a model error.
Current values span roughly 0.01 to 0.25. The frontier sits at 0.0575 (TECE-OAM-RRA-1.0 and GRACE-3L-OAM-L, tied for best); eqV2 M is 0.0691, MACE-MPA-0 0.0731, eSEN-30M-MP 0.0752, ORB v2 MPA 0.0973, and M3GNet 0.1117 — the worst model at the time the CPS baseline was set.
CPS
The Combined Performance Score is a single scalar in $[0,1]$, higher better, introduced in March 2025 and versioned as CPS v1. It is a weighted mean of three normalised subscores, each mapped against a frozen reference point so that scores do not drift as new models are added:
\[\mathrm{CPS} \;=\; \frac{0.5\,s_{\mathrm{F1}} \;+\; 0.4\,s_{\kappa} \;+\; 0.1\,s_{\mathrm{RMSD}}}{0.5+0.4+0.1}\]with
\[s_{\mathrm{F1}} = \mathrm{F1}, \qquad s_{\kappa} = 1 - \frac{\kappa_{\mathrm{SRME}}}{2}, \qquad s_{\mathrm{RMSD}} = \mathrm{clamp}\!\left(1 - \frac{\mathrm{RMSD}}{0.15},\, 0,\, 1\right)\]The RMSD baseline of 0.15 is hard-coded in site/src/lib/labels.ts with the comment that it was
chosen because “the worst performing model at time of writing is M3GNet at 0.1117”. A missing
component with non-zero weight invalidates the score entirely — there is no partial credit, which is
why 41 models have a CPS while 51 have an F1.
Recomputing this formula from the raw metrics reproduces every published CPS on the board to within 0.0015. Worked examples:
| Model | $s_{\mathrm{F1}}$ | $s_{\kappa}$ | $s_{\mathrm{RMSD}}$ | CPS |
|---|---|---|---|---|
| TECE-OAM-RRA-1.0 | 0.929 | 0.954 | 0.617 | 0.908 |
| DPA-4.0.1-Pro-MPtrj | 0.857 | 0.894 | 0.542 | 0.840 |
| eSEN-30M-MP | 0.831 | 0.830 | 0.499 | 0.797 |
| MACE-MP-0 | 0.669 | 0.659 | 0.390 | 0.637 |
| eqV2 M | 0.917 | 0.115 | 0.539 | 0.558 |
| ORB v2 (MPA) | 0.880 | 0.133 | 0.351 | 0.528 |
| CHGNet 0.3.0 | 0.613 | 0.142 | 0.367 | 0.400 |
The 0.4 weight on $\kappa_{\mathrm{SRME}}$ is the single design decision that converted “this model has a soft, unphysical potential energy surface” from a curiosity into a ranking penalty, and it is what reshuffled the board in 2025. It is also, transparently, a choice: the site lets users reweight the components live, and the maintainers’ own warning is unusually candid — “This is not a stable metric… When referring to it in papers, best include the benchmark version (e.g. CPS v1).”
Two structural weaknesses in CPS are worth naming. First, the RMSD subscore is clamped and linearly normalised against a baseline chosen from the worst 2024 model, so it saturates: at the frontier, the difference between RMSD 0.0575 and 0.0691 is worth only 0.008 of final CPS, while a model with an anomalously low recorded RMSD gets a large free subscore. SevenNet-0 records RMSD 0.0193 and DPA3-v1-OpenLAM records 0.0128 — values four to five times better than the frontier models’, on architectures that are otherwise unremarkable. Those give $s_{\mathrm{RMSD}}$ of 0.871 and 0.915 respectively, and they feed straight into CPS (SevenNet-0’s 0.6967 checks out exactly with that subscore). Treat them with suspicion. Second, sibling aggregate scores for the later-added tasks use the same machinery with their own arbitrary weights: CMDS for molecular dynamics (vDOS 0.3, ADF 0.2, speed 0.2, pressure 0.3) and CDS for diatomics (accuracy 4/9, geometry 2/9, physicality 2/9, speed 1/9).
7. The standings, 29 July 2026
The default view is the unique-prototypes subset with superseded checkpoints hidden: 51 models with a discovery F1, 41 with a complete CPS. (The underlying registry carries 64 model YAMLs, the difference being superseded checkpoints such as TACE-OAM-RRA-Preview, whose CPS of 0.9054 would otherwise place it third.)
Top 15 by Combined Performance Score
| # | Model | Group | Training set | Params | Forces | Lineage | CPS | F1 | $\kappa_{\mathrm{SRME}}$ | RMSD |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | TECE-OAM-RRA-1.0 | ShanghaiTech / Nanjing | OAM | 222M | conservative | ACE → TACE (Cartesian) + SO(2) attention | 0.908 | 0.929 | 0.0927 | 0.0575 |
| 2 | EquFlashV2-45M-OAM | Samsung Electronics | OAM | 44.9M | conservative | TFN → NequIP → SevenNet → EquFlash | 0.907 | 0.929 | 0.0941 | 0.0577 |
| 3 | EquiformerV3+DeNS-OAM | MIT Atomic Architects / Mirror Physics | OAM | 30.3M | conservative | TFN → Equiformer → eSCN → EqV2 | 0.902 | 0.931 | 0.1178 | 0.0595 |
| 4 | GRACE-3L-OAM-L | ICAMS, Ruhr Univ. Bochum | OAM | 42.1M | conservative | ACE → graph ACE | 0.900 | 0.925 | 0.1211 | 0.0575 |
| 5 | PET-OAM-XL | COSMO, EPFL | OAM | 730M | conservative | unconstrained edge transformer | 0.898 | 0.924 | 0.1192 | 0.0596 |
| 6 | TACE-OAM-L | ShanghaiTech / Nanjing | OAM | 82.9M | conservative | ACE, irreducible Cartesian tensors | 0.889 | 0.910 | 0.1260 | 0.0606 |
| 7 | eSEN-30M-OAM | FAIR at Meta | OAM | 30.2M | conservative | TFN → eSCN / EquiformerV2 → eSEN | 0.888 | 0.925 | 0.1704 | 0.0608 |
| 8 | EquFlash-29M-OAM | Samsung Electronics | OAM | 28.7M | conservative | TFN → NequIP → SevenNet | 0.888 | 0.919 | 0.1583 | 0.0602 |
| 9 | NequIP-OAM-XL | MIR Group Harvard / Cambridge | OAM | 32.1M | conservative | TFN → NequIP (root) | 0.886 | 0.906 | 0.1252 | 0.0630 |
| 10 | MatRIS-10M-OAM | ICT, Chinese Academy of Sciences | OAM | 10.4M | conservative + magmoms | invariant 3-body attention (BP → SchNet → DimeNet → M3GNet/CHGNet) | 0.877 | 0.921 | 0.2183 | 0.0601 |
| 11 | SevenNet-Omni-i12 | MDIL-SNU, Seoul National Univ. | COSMOS (15 DBs) | 54.9M | conservative | TFN → NequIP → SevenNet | 0.873 | 0.906 | 0.1917 | 0.0618 |
| 12 | NequIP-OAM-L | MIR Group, Harvard | OAM | 9.6M | conservative | TFN → NequIP | 0.870 | 0.893 | 0.1657 | 0.0647 |
| 13 | GRACE-2L-OAM-L | ICAMS, Ruhr Univ. Bochum | OAM | 26.4M | conservative | ACE → graph ACE | 0.865 | 0.883 | 0.1688 | 0.0639 |
| 14 | ORB v3 | Orbital Materials | MPtrj+Alex+OMat24 | 25.5M | conservative | unconstrained GNS + diffusion pretraining | 0.861 | 0.905 | 0.2102 | 0.0750 |
| 15 | Allegro-OAM-L | MIR Group, Harvard | OAM | 9.7M | conservative | TFN → NequIP → Allegro (strictly local) | 0.840 | 0.895 | 0.3186 | 0.0651 |
Immediately below the cut, and more interesting than several entries above it: DPA-4.0.1-Pro-MPtrj (AI for Science Institute Beijing / Peking Univ. / DP Technology, 22.8M params, conservative, DeepPot → DPA lineage with an eSCN-style SO(2) convolution) at CPS 0.840, F1 0.857, $\kappa_{\mathrm{SRME}}$ 0.2114 — trained on MPtrj alone.
All fifteen top entries use conservative gradient forces. That is a 2025–2026 development, not a constant of the field: the 2024 state of the art, eqV2 M, was direct.
Top 15 by discovery F1
The two orderings are not the same ordering, and the differences are the whole point of the CPS metric.
| F1 # | Model | F1 | CPS | CPS # | Δrank | $\kappa_{\mathrm{SRME}}$ | Forces | Lineage |
|---|---|---|---|---|---|---|---|---|
| 1 | EquiformerV3+DeNS-OAM | 0.931 | 0.902 | 3 | −2 | 0.1178 | conservative | Equiformer / eSCN |
| 2 | TECE-OAM-RRA-1.0 | 0.929 | 0.908 | 1 | +1 | 0.0927 | conservative | ACE + Cartesian tensors |
| 3 | EquFlashV2-45M-OAM | 0.929 | 0.907 | 2 | +1 | 0.0941 | conservative | NequIP / SevenNet |
| 4 | GRACE-3L-OAM-L | 0.925 | 0.900 | 4 | 0 | 0.1211 | conservative | graph ACE |
| 5 | eSEN-30M-OAM | 0.925 | 0.888 | 7 | −2 | 0.1704 | conservative | eSCN / EquiformerV2 |
| 6 | PET-OAM-XL | 0.924 | 0.898 | 5 | +1 | 0.1192 | conservative | unconstrained transformer |
| 7 | MatRIS-10M-OAM | 0.921 | 0.877 | 10 | −3 | 0.2183 | conservative | invariant 3-body attention |
| 8 | EquFlash-29M-OAM | 0.919 | 0.888 | 8 | 0 | 0.1583 | conservative | NequIP / SevenNet |
| 9 | eqV2 M | 0.917 | 0.558 | 36 | −27 | 1.7707 | direct | Equiformer / eSCN |
| 10 | TACE-OAM-L | 0.910 | 0.889 | 6 | +4 | 0.1260 | conservative | ACE, Cartesian tensors |
| 11 | NequIP-OAM-XL | 0.906 | 0.886 | 9 | +2 | 0.1252 | conservative | NequIP (root) |
| 12 | SevenNet-Omni-i12 | 0.906 | 0.873 | 11 | +1 | 0.1917 | conservative | NequIP / SevenNet |
| 13 | ORB v3 | 0.905 | 0.861 | 14 | −1 | 0.2102 | conservative | unconstrained GNS |
| 14 | AlphaNet-v1-OAM | 0.901 | 0.769 | ≈22 | ≈−8 | 0.6435 | conservative | equivariant local frames |
| 15 | Allegro-OAM-L | 0.895 | 0.840 | 15 | 0 | 0.3186 | conservative | Allegro (strictly local) |
Two entries fall out of the top tier when phonons are counted. eqV2 M (FAIR at Meta, 86.6M params, OMat24 + sAlex + MPtrj, direct forces, 12.0 Å cutoff — much larger than the 6.0 Å typical of current leaders) drops 27 places on a $\kappa_{\mathrm{SRME}}$ of 1.7707. AlphaNet-v1-OAM (Tsinghua / UT Austin / DAMO Academy / Cornell, 4.65M params, equivariant local frames, 5.0 Å cutoff — the smallest cutoff among top models) falls out of the top twenty on 0.6435, despite being conservative. (Its exact CPS rank depends on which superseded checkpoints the default view is hiding on a given day, hence the approximation.) AlphaNet is a second, quieter instance of the same lesson as CHGNet: a conservative force field with an inaccurate Hessian is still a bad phonon model.
What the standings say about architecture
Half the CPS top ten descends from the Tensor Field Network / e3nn lineage — EquFlash and EquFlashV2 (via SevenNet, from NequIP), NequIP-OAM-XL itself, and the eSCN/Equiformer transformer branch (eSEN-30M-OAM, EquiformerV3+DeNS-OAM). The other half does not, and this is the correction the field most needs to absorb (see Genealogy):
- GRACE-3L-OAM-L (#4) descends from Drautz’s Atomic Cluster Expansion (Phys. Rev. B 99, 014104, 2019) through graph ACE (Bochkarev, Lysogorskiy & Drautz, Phys. Rev. X 14, 021036, 2024). Its equivariance comes from ACE’s own spherical-harmonic product basis, not from TFN-style Clebsch–Gordan convolution layers.
- TECE-OAM-RRA-1.0 (#1) and TACE-OAM-L (#6) are irreducible Cartesian-tensor ACE variants (arXiv:2509.14961, arXiv:2607.10664) that explicitly avoid spherical-tensor Clebsch–Gordan products on the feature path — though TECE does import the SO(2)/Wigner-D attention machinery that originates in eSCN (arXiv:2302.03655).
- PET-OAM-XL (#5) is a rotationally unconstrained transformer (Pozdnyakov & Ceriotti, arXiv:2305.19302; Nat. Commun., doi:10.1038/s41467-025-65662-7). Standard multi-head self-attention over per-edge tokens; equivariance is learned from data and augmentation, not imposed. At 730M parameters it is by far the largest model on the board, and its $\kappa_{\mathrm{SRME}}$ of 0.1192 is the fourth-best on the board — ahead of eSEN, NequIP-OAM-XL, GRACE-3L and TACE-OAM-L, beaten only by TECE, EquFlashV2 and EquiformerV3+DeNS. Thermal conductivity was historically the failure mode of non-equivariant models; PET does not have it.
- MatRIS-10M-OAM (#10) is explicitly invariant, recovering equivariant-level accuracy through separable linear-complexity attention over three-body terms (arXiv:2603.02002), at 10.4M parameters — the cheapest top-ten model.
- ORB v3 (#14) is an unconstrained graph network with attention, pretrained as a denoising diffusion model over structures (arXiv:2504.06231).
So the top tier is architecturally plural, and the two strongest counterexamples — PET at #5 and MatRIS at #10 — are not merely non-e3nn but non-equivariant by design. The claim that built-in equivariance is what separates the leaders was true in 2022 and is no longer true in 2026. What equivariance bought was data efficiency and a strong inductive prior; at 100M-structure training scales that prior is less decisive, and other things — smooth energy surfaces, accurate second derivatives, training-corpus diversity, and fast enough kernels to train a big model at all — now separate the top from the middle.
The parameter counts complicate the scaling story without demolishing it, and it is worth being
precise about which. Across the CPS top fifteen, $\log_{10}(\text{params})$ and CPS are positively
correlated — Pearson $r = 0.61$, Spearman $\rho = 0.73$ — so bigger does help. But the relation is
loose and the range is enormous relative to the payoff: a 76× spread in parameter count (9.6M to
730M) buys 0.068 of CPS. The best F1 on the board belongs to a 30.3M parameter model. The top
CPS belongs to a 222M model, but it leads by 0.0004 over a 44.9M one whose entire pitch is a
fused tensor-product kernel (FlashTP, ICML 2025, https://proceedings.mlr.press/v267/lee25l.html)
reported at up to 41.6× over e3nn and 60.8× over NVIDIA cuEquivariance with 6× lower memory. The
730M-parameter entry sits fifth. Whatever is separating the frontier from the middle, at the frontier
itself an order of magnitude in parameters is worth less than a better kernel or a better recipe.
8. What the leaderboard does not measure
Leakage
WBM was generated by elemental substitution on MP. Alexandria was generated by the same Halle/Jena group with the same philosophy. Consequently Alexandria shares roughly 97,000 protostructure labels with WBM, and even MPtrj — the compliant corpus — contains WBM-overlapping prototypes.
The mitigations are real: unique-prototype filtering of the test set (which is precisely why the
default view is 215,488 structures rather than 256,963), the sAlex decontamination rule (exclude any
structure whose Aflow protostructure label matches a WBM initial or relaxed structure, and any OMat24
structure generated from such an Alexandria structure), and per-dataset notes.Leakage blocks in
datasets.yml. They reduce but cannot eliminate the problem, because prototype identity is a coarse
proxy for “the model has effectively seen this”. A different Wyckoff decoration of the same lattice
with the same chemistry is not, informationally, a new test item.
The maintainers’ counter-evidence is worth knowing and is weaker than it is usually taken to be: fine-tuning ORB v2 for under one epoch on Alexandria versus sAlex gave identical metrics. That is evidence that fine-tuning on the contaminated set for a fraction of an epoch does not help, not that the ~100M-structure pretraining corpus is clean.
“Discovery”
The word is doing a lot of work. A WBM candidate is an MP structure with between one and five elements swapped, weighted by ICSD-mined chemical similarity. Correctly predicting that such a compound sits on the hull is a useful capability — it is exactly the pre-filter step in a high-throughput campaign — but it is not the discovery of a new structural motif, and the benchmark does not claim otherwise. The generative half of the problem is entirely absent.
The GNoME episode is the cautionary parallel. Merchant, Batzner, Schoenholz, Aykol, Cheon and Cubuk (Nature 624, 80–85, 2023, doi:10.1038/s41586-023-06735-9) reported 2.2M structures below the previously known hull and 381,000 new stable entries against ~48,000 previously known — an order-of-magnitude expansion, driven by substitution-based generation (including symmetry-aware partial substitutions) plus an active-learning GNN filter. Cheetham and Seshadri (Chem. Mater. 36, doi:10.1021/acs.chemmater.4c00643, 2024) argued that few of those “discoveries” are genuinely new compounds. The same question applies, in miniature, to every WBM positive.
Thermodynamic, dynamical, and synthesisable are three different things
$E_{\text{hull}} \le 0$ at 0 K and 0 bar is a statement about a static energy landscape relative to the phases you happen to have enumerated. It is necessary for stability and nowhere near sufficient.
- Thermodynamic stability is what the discovery task measures — and only against the MP hull. Against the joint MP+WBM hull, about 40% of the stable set stops being stable.
- Dynamical stability requires the Hessian to be positive semi-definite: no imaginary phonon
frequencies. Nothing in the discovery task tests this. $\kappa_{\mathrm{SRME}}$ tests it as a
side effect on 103 crystals, which is why the board reports
imaginary_mode_ratealongside it — eqV2 M’s is 5.8%. - Synthesisability depends on kinetics, precursor availability, competing metastable phases, entropy at finite temperature, and whether anyone can be bothered. It is not a function of the 0 K energy at all. The benchmark’s own framing paper concedes the point about formation energies; the same concession applies with more force to hull distance.
The benchmark has, to its credit, absorbed most of its own critiques into its design — leakage into the unique-prototype default view, the data-scale objection into the compliant/non-compliant filters, and the F1-without-physics objection into a 0.4 CPS weight on thermal conductivity. What it has not absorbed, and probably cannot, is the gap between “this crystal is on the convex hull of the Materials Project” and “someone made this.”
Sources
Primary benchmark: Riebesell et al., arXiv:2308.14920; Nat. Mach. Intell. 7, 836–847 (2025),
doi:10.1038/s42256-025-01055-1; leaderboard and model registry at
https://matbench-discovery.materialsproject.org. Test set: Wang, Botti & Marques,
doi:10.1038/s41524-020-00481-6. Training corpora: MP doi:10.1063/1.4812323; MPtrj
doi:10.1038/s42256-023-00716-3; Alexandria doi:10.1002/adma.202210788; OMat24 arXiv:2410.12771.
Metrics: $\kappa_{\mathrm{SRME}}$ from Póta, Ahlawat, Csányi & Simoncelli, arXiv:2408.00755; CPS from
site/src/lib/combined-scores.svelte.ts and site/src/lib/labels.ts; classification and DAF
definitions from matbench_discovery/metrics/discovery.py. Model-specific citations appear inline in
§7. Numerical experiments are this repository’s own:
results/convex_hull_decision.json and
results/conservative_vs_direct.json.