MLPerf

mlcommons.org · Compute · 1 record · JSON

MLPerf Compute

fails: identityinstitutionno cardDwrite-gated

mlcommons.org/benchmarks/ · cloud VM · compute/mlperf · JSON · compare

Free tier	MLPerf awards no compute at all. The programme is a measurement, and the submitter supplies the system, quoted: 'The working group defines the AI model to run, the data set against which it gets run, sets rules on what changes to the model are allowed, and measures how fast a given hardware runs the model.' The gate is membership, quoted: 'Membership is required for most benchmark working groups (e.g., Training, Inference, Mobile). There are some public benchmark working groups that have no access requirements where non-members may submit to the benchmark by first signing a Non-member Test Agreement.' The submission requirements list, quoted: 'Join an appropriate MLCommons Working Group', 'A signed Contributor License Agreement', 'Membership in MLCommons OR a signed Non-member Test Agreement', 'A signed trademark license agreement'. unit none; metered on wall-clock; none
Price	none stated
Per-object cap	not applicable
Retention	not applicable
Egress	not stated
Identity	institution not published
Limits	write-gated two gates stack: the Training and Inference working groups require organisational membership, and even a non-member submission needs a signed test agreement and a trademark licence before a result can be published, so the constraint that fails first is the membership, not the hardware. source two gates stack: the Training and Inference working groups require organisational membership, and even a non-member submission needs a signed test agreement and a trademark licence before a result can be published, so the constraint that fails first is the membership, not the hardware.
Binding	write-gated two gates stack: the Training and Inference working groups require organisational membership, and even a non-member submission needs a signed test agreement and a trademark licence before a result can be published, so the constraint that fails first is the membership, not the hardware.
Interface	unknown inferred from the tooling the provider documents; not probed
Tooling	not stated
Composes with	Nil. The useful product of the record is the exclusion: a reader who sees MLPerf results quoted as free capacity should be told that the numbers are vendor submissions, and that no MLCommons programme pays for the machine under test.
Reliability	no reliability event recorded
Notes	This is a refutation rather than a discovery, and it belongs because the brief lists MLPerf as a compute-awarding programme and it is not one. Three readings of the membership sentence were considered: (a) membership means an individual can join, (b) membership means a company pays dues, (c) non-members are fine through the test agreement. The page's own 'Membership and Affiliate' FAQ and the phrase 'if you do not think your organization would be a good fit as a Member' in the GitHub route point at (b), and (c) is true but ends in a signed agreement that a solo entrant can sign. So the honest verdict is fails-identity for the main suites and a narrow non-member path for the public groups; the hardware cost stands either way.
Evidence rung	documented
Checked	2026-10-22 (live); stale on 2026-11-21

Evidence (3)

date	source	what it proved
2026-10-22	mlcommons.org/benchmarks/	'Membership is required for most benchmark working groups (e.g., Training, Inference, Mobile)'; 'There are some public benchmark working groups that have no access requirements where non-members may submit to the benchmark by first signing a Non-member Test Agreement'; 'Membership in MLCommons OR a signed Non-member Test Agreement'; 'A signed trademark license agreement'; page 200
2026-10-22	mlcommons.org/benchmarks/inference-datacenter/	the suite page describes measurement only ('measures how fast systems can process inputs and produce results') and names no grant, credit, allowance or hardware pool; page 200
2026-10-22	mlcommons.org/about-us/policies/	the policies page offers three agreements, 'Membership Agreement', 'Membership and Affiliate FAQ', 'Non-member Test Agreement', which is the only route for an unaffiliated submitter; page 200

More fields (1)

{
 "wave2": {
  "class": "benchmark-programme",
  "free_tier_kind": "none",
  "shape": {
   "vcpu": null,
   "ram_gib": null,
   "gpu_class": null,
   "gpu_count": null
  }
 }
}

Unknown or withheld values (10)

$.egress.metered egress-unknown
$.egress.terms not-stated
$.identity_cost.detail process
$.interface interface-unknown
$.per_object_cap.notes process
$.per_object_cap.stated not-applicable
$.reliability_note none-observed
$.retention.notes process
$.retention.stated not-applicable
$.tooling not-stated