While at the 60th edition of the IT Press Tour in Palo Alto, California, David Kanter, founder and executive director of MLCommons, provided a comprehensive look at how the non-profit organization is working to advance AI through standardized benchmarking and measurement.
Setting the Standard for AI Performance Measurement
MLCommons, with over 125 members and affiliates spanning six continents, has established itself as the leading authority in AI benchmarking through its MLPerf suite of benchmarks. The organization has recorded over 56,000 MLPerf results to date, making it the industry standard for measuring AI system performance and energy efficiency.
“What gets measured, gets improved,” noted Kanter, quoting management consultant Peter Drucker to emphasize the importance of benchmarking in driving progress. The organization’s goal is to align the entire AI industry on clear objectives and metrics for improvement.
A Full System Approach to AI Benchmarking
MLCommons takes a comprehensive view of AI systems, measuring everything from silicon-level performance to algorithms, system scale, software architecture, and data handling. This holistic approach recognizes that AI workloads stress every part of modern computing infrastructure.
The MLPerf benchmark suite currently includes:
- MLPerf Training – For measuring model training performance
- MLPerf Inference – For measuring inference deployment performance
- MLPerf Client – For client-side AI applications
- MLPerf Storage – For measuring storage system performance in AI workloads
- MLPerf Mobile/Tiny – For mobile and IoT devices
- MLPerf Automotive – In development for automotive use cases
Impressive Performance Gains
One of the most striking revelations from Kanter’s presentation was the pace of performance improvement shown through MLPerf results. The benchmarks have documented performance gains significantly outpacing Moore’s Law, with some workloads showing up to 50x improvement over approximately five years.
“We’re seeing massive leaps forward that combine better silicon, better architectures, larger scale systems and improved algorithms,” explained Kanter. “All of these factors working together are driving unprecedented rates of progress in AI system capabilities.”
Focus on Real-World Applicability
A key principle for MLCommons is ensuring benchmarks reflect real production workloads and use cases. The organization works closely with industry practitioners to define representative tasks, datasets, and quality targets.
For example, MLPerf Training includes benchmarks for:
- Recommendation systems (like those used in social media and e-commerce)
- Large language model pre-training and fine-tuning
- Object detection
- Image generation
- Graph neural networks for applications like fraud detection
Power and Energy Efficiency Take Center Stage
Recognizing the growing importance of energy efficiency in AI deployments, MLCommons has integrated power measurement into several benchmarks. This addition helps organizations evaluate not just raw performance, but also the energy costs associated with AI workloads.
“In many countries, it’s becoming harder to build power plants while energy constraints increase,” Kanter noted. “MLPerf power measurements help optimize energy usage while maintaining high performance – this is critical for sustainable AI deployment at scale.”
New Developments in Storage Benchmarking
The latest addition to the MLPerf family is focused on storage systems, addressing a critical bottleneck in AI training pipelines. Kanter shared that some major tech companies found their large AI clusters were only achieving 30% utilization due to storage limitations.
MLPerf Storage helps organizations evaluate storage systems’ ability to feed data to AI training workloads efficiently. The benchmark simulates realistic training scenarios without requiring massive investments in compute hardware for testing.
Looking Ahead
As AI continues to evolve rapidly, MLCommons is expanding its scope to address new challenges and use cases. The organization recently launched MLPerf Client for measuring client-side AI performance and is developing new benchmarks for automotive applications.
“We’re seeing incredible innovation across the industry,” concluded Kanter. “Our role is to provide the trusted, standardized measurements that help drive that innovation forward while ensuring practical applicability.”
Industry Impact
The broad industry adoption of MLPerf benchmarks demonstrates their value, with major technology vendors, cloud providers, and enterprise buyers using them to evaluate AI systems. Government agencies and large organizations frequently incorporate MLPerf results into their procurement processes.
Kanter emphasized that MLCommons maintains strict rules around result validation and reproducibility. All submissions are peer-reviewed by other members, and the complete methodology and code for submissions are made publicly available.
This transparency and rigor have helped establish MLPerf as the de facto standard for AI system evaluation, providing valuable guidance for organizations investing in AI infrastructure while driving continuous improvement across the industry.
For more information about MLCommons and the MLPerf benchmarks, visit mlcommons.org.
