Overview of data sets

What's HLE?

Humanity's Last Exam (HLE) by Center for AI Safety（CAIS） and Scale AI The data set of the multi-modical academic reasoning benchmark, which was co-sponsored by nearly 1,000 leading scholars around the world. Published in January 2026 in the regular Nature.(Vol. 649, pp. 11339-1146), listed by Stanford University 2025 AI Index Report as a core new benchmark for assessing AI forward capacity.

HLE includes 2,500 difficult questions• Includes mathematics, physics, chemistry, biology, computer science, humanities, engineering, etc. 100+ Disaggregated disciplineAll the topics are masters/doctoral difficulties, which cannot be obtained through simple web searches, and are globally accepted as the "end of the big model" test.

Langhui Technology HLE Data Asset

The AIG team of scientists built a complete HLE data service based on an in-depth study of all HLE 100+ disciplines:

HLE Source Library: 2,500 complete data sets (including titles, standard answers, solvers)
HLE Phase III Expanded Question Bank: 1,000 precise purchases by discipline, validated 16 times
Training data development: High-quality SFT/RLHF fine-tuning data sets based on HLE standards
EvaluationServices: Modelling capabilities diagnostic and short-board analysis based on HLE calibre

Core data indicators

2,500

Total number of topics (public collection)

+ Private reservations collected against a possible merger

100+

Scripts covered by subdivision

Mathematical / CS / Engineering / Humanities and Society

76%

Accurate Matching Threshold

24% Multi-selection + 10-14% Multi-modular

3.6%

Final Selection Rate

70,000+scripts ⁇ 2,500 go to library

Strict multi-level screening process

70,000+
Global expert contributions

LLM difficulty prescreening
Frontline model is not correct

Question 13,000
Access to expert review

Double expert revisers
Suk/Bo/JD Review

2,500
Final Library

The selection rate was only 3.6 per cent, with expert solvers (rationale) attached to each topic

Distribution of fields of study

	Percentage of title	Number of subareas	Typical subarea
Mathematics	41.8%	8	Combination and graphics, geometry and scaling up, analysis, algebra, mathematical theory, mathematical logic, probability statistics
Humanities and Social Sciences	15.9%	9	Linguistics, Literature, History, Law, Philosophy and Logic, Economics and Finance
Interdisciplinary/other	14.5%	10	Chess and table and game, mathematical puzzles, music, video, video games, Earth sciences
CS / AI	10.4%	3	Algorithms and computational theory, programming and software engineering, artificial intelligence and machine learning
Physics	9.3%	7	Classic mechanics, quantum field theory and particle physics, electromagnetics, quantum mechanics
Engineering	3.6%	9	Electrical, electronic, communications, optical, metallurgy, civil transport, computer engineering
BiologyPharmaceuticals	2.3%	4	Molecular and cytology, ecology and evolution, epidemiology and public health
Chemistry	2.2%	4	Organic, quantum and computational chemistry, physicochemical, biochemical

Global mainstream model HLE performance

	Models	HLE accuracy rate	Milestones
2025.01	GPT-4o / Claude 3.5	<3%	Benchmark established, all models not exceeded 10%
2025.04	OpenAI o1	8.8%	Top score.
2025.06	Kimi-Researcher	26.9%	First break 25% of the smart body
2025.07	X-Masters (overturn)	32.1%	First break, 30%.
2025.07	Grok 4 Heavy	44.4%	Multi-Intelligent Versions Largely Forward
2025.12	GLM-4.7	> GPT-5.1	The national model is the first to reverse.
2026.02	Ali Qwen	HLE-Verified	Release an amended version to enhance the scientific nature of the assessment

LANG Huitech HLE 3rd DDS Construction Program NEW

NEW

HLE Source Library

2,500 | Multimodular + Text

Full HLE Open Data Set with title, standard answer and expert solution (rationale), covering 100+ subdivisions

NEW

HLE Phase III Expanded Question Bank

1,000 questions | Text

Accurately distributed by discipline, validated 16 times by head-of-the-state production model (DeepSeek/Ki/GLM), Judge calibration aligned to official

NEW

SFT Training Data Set

Custom Size | Dialogue Format

High-quality, hard-to-exposure SFT data based on HLE standards, with multiple-wheeled reasoning lines, for fine-tuning training in large models

NEW

RLHF Preference Dataset

Custom Size | Compare Format

Apply RLHF/DPO training process based on HLE difficulty alignment data, overwrite positive/negative comparisons

NEW

Area Enhancement Package

Customise by discipline | Multimodular

Depth data sets for specific disciplines, including mathematics/physics/CS/chemistry, with combined graphic reasoning topics to support the construction of a peripherical model

NEW

Model assessment services

Evaluationreports | Diagnostic analysis

Modelling capability assessment diagnostics based on HLE, short-board analysis and recommendations for targeted improvements

Example of typical title

Math, geometry and poking.

"Compute the reduced 12-th dimensional Spin bordism of the classifying space of the Lie group G2."

Micro-drives ..doctoral research.

CS/AI . Theory of computational learning

"Determine the VC dimension of the class H_{z-ones} = {h: X → {0,1}: |{x: h(x)=1}| = z} in terms of z and T."

VCV - Theory of Machine Learning

- Physical Quantum Fields

"The product γ_{μν} γ_{μ₁...μₖ} γ^{μν} is proportional to γ_{μ₁...μₖ} in d dimensions. Find the proportionality factor."

Rotation algebra .. High-energy theoretical physics

Humanities . History and Linguistics

"If the PIE root *kʷeys were inherited into English as an o-grade causative via Proto-West Germanic, what would the 3sg present verbal form of its reflex in Middle English be?"

Indo-European comparison • Doctoral expertise

HLE Core Values in Large Model Training

High-quality training data for exceptional cases

HLE is naturally "anti-retrievability" - the answer is not available through simple web search, and requires that the model truly understands the reasoning process. The difficulty of the subject is verified 16 times, and the precision is contained in the "right rate 6.25-50%" active range, which is the ideal source of RLHF/DPO training.

Base ceiling breach

The traditional benchmark (MMLU, etc.) has become saturated (an accuracy rate of > 90%) and lost its distinction. HLE maintains the challenge of the front-line model by dynamic updating the private test set + Canary GUId Discretion Mechanism + HLE-Rolling, which is an important reference rule for measuring "real-expert intelligence".

Multi-dimensional capability probe

100+Scientific coverage makes HLE a natural capability diagnostic tool - a short panel that can accurately detect the dimensions of the model in mathematical reasoning, knowledge breadth, cross-modular integration, calibration capacity, etc., and directs orientation optimization.

Quest-Screen-Train-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Screen-Spect

HLE evaluation short panel ⁇ Targeted collection difficult case ⁇ Construct SFT/RLHF training data ⁇ HLE private cluster survey validation upgrade. Convert "test" to "trainer" to achieve data-driven model iterative.

Dataset Overview

What is HLE?

Humanity's Last Exam (HLE) is a multi-modal academic reasoning benchmark co-developed by the Center for AI Safety (CAIS) and Scale AI, with contributions from nearly 1,000 subject-matter experts across 50+ countries. Published in Nature (Vol.649, pp.1139-1146, January 2026) and recognized by Stanford's 2025 AI Index Report as a key benchmark for frontier AI capabilities.

HLE comprises 2,500 challenging questions spanning 100+ academic disciplines — from advanced mathematics to medieval literature. Every question is graduate-level or above, with answers intentionally unsearchable via the web, making it the definitive "final exam" for frontier AI models.

Langhui's HLE Data Assets

Leveraging our AI+ Scientist team's deep research across all HLE disciplines, Langhui Technology offers a complete HLE data service ecosystem:

HLE Original Dataset: Complete 2,500-question set with answers and expert rationales
HLE Phase III Expanded Set: 1,000 precisely curated questions validated with 16-shot sampling
Training Data Development: High-quality SFT/RLHF datasets built to HLE standards
Evaluation Services: Model capability diagnosis and weakness analysis

Key Metrics

2,500

Public Questions

+ private held-out test set

100+

Academic Disciplines

STEM / Humanities / Engineering

~1,000

Expert Contributors

500+ institutions, 50+ countries

<10%

Initial Accuracy

All frontier models at launch

Rigorous Selection Pipeline

70,000+
Global Submissions

LLM Difficulty Filter
Models must fail

13,000 Passed
Enter expert review

Dual-Round Review
Graduate-level reviewers

2,500 Finalized
3.6% acceptance rate

Discipline Distribution

Category	Share	Subtopics	Example Fields
Mathematics	41.8%	8	Combinatorics, Geometry & Topology, Analysis, Algebra, Number Theory
Humanities / SS	15.9%	9	Linguistics, Literature, History, Law, Philosophy, Economics
Other / Interdisc.	14.5%	10	Board Games, Math Puzzles, Music, Film, Video Games, Earth Science
CS / AI	10.4%	3	Algorithms & Theory, Programming & Software Engineering, AI & ML
Physics	9.3%	7	Classical Mechanics, QFT & Particle Physics, E&M, Quantum Mechanics
Engineering	3.6%	9	Electrical, Communications, Optical, Manufacturing, Civil
Bio / Medicine	2.3%	4	Molecular & Cell Biology, Ecology & Evolution, Epidemiology
Chemistry	2.2%	4	Organic, Quantum & Computational, Physical, Biochemistry

Model Performance on HLE

Date	Model	Accuracy	Milestone
2025.01	GPT-4o / Claude 3.5	<3%	All models below 10% at launch
2025.04	OpenAI o1	8.8%	Highest at the time
2025.06	Kimi-Researcher	26.9%	First agent system above 25%
2025.07	X-Masters (SJTU)	32.1%	First above 30%
2025.07	Grok 4 Heavy	44.4%	Multi-agent variant leading
2026.02	Alibaba Qwen	HLE-Verified	Corrected version published

Sample Questions

Mathematics · Geometry & Topology

"Compute the reduced 12-th dimensional Spin bordism of the classifying space of the Lie group G2."

Differential topology · PhD-level research

CS/AI · Computational Learning Theory

"Determine the VC dimension of H_{z-ones} in terms of z and T."

VC dimension · ML theory foundation

Physics · QFT & Particle Physics

"The product of antisymmetrized gamma matrices is proportional to γ_{μ₁...μₖ}. Find the proportionality factor in d dimensions."

Spinor algebra · High-energy theory

Humanities · Historical Linguistics

"Trace the PIE root *kʷeys through Proto-West Germanic to its Middle English 3sg present verbal reflex."

Comparative IE linguistics · PhD-level

Value for LLM Training

High-Quality Hard Negatives

HLE questions are inherently unsearchable, requiring genuine reasoning. Difficulty is calibrated to the "sweet spot" (6.25%-50% accuracy), making them ideal hard negatives for contrastive learning and RLHF.

Ceiling Effect Breakthrough

With MMLU exceeding 90% saturation, HLE remains the only benchmark resisting ceiling effects. Private test set + Canary GUID + HLE-Rolling ensure enduring challenge.

Multi-Dimensional Diagnostic

100+ disciplines enable precise capability probing — identifying weakness in math reasoning, knowledge breadth, cross-modal integration, and model calibration.

Evaluate → Curate → Train → Verify

HLE diagnosis pinpoints weaknesses → targeted hard-example curation → SFT/RLHF training data construction → HLE private set re-evaluation. Transforming the "exam" into a "coach."

HLE Humanity's Last Exam