Quantifying Reasoning: From Theory to Measurement

Jack Hodges & Zach Glasser · May 2026

Polling measures what people believe. Argumentation research measures how persuasively they argue. AI benchmarks measure whether a machine can chain its way to a correct answer. Nothing measures the thing in between: how well a person forms and revises a view in proportion to the evidence in front of them. This paper sets out what that thing is, why it has never been measured, and how we'd go about scoring it.

The gap

The question of how well a person reasons is among the oldest in intellectual history, and among the least technically addressed. The absence is not accidental. Reasoning quality is a latent construct: it cannot be observed directly, only inferred from imperfect, context-bound signals. A written argument reveals something about the reasoning that produced it, but the relationship between text and thought is lossy. Premises are omitted, inferential steps are compressed, confidence is performed rather than reported, and the same conclusion can be reached through radically different chains of inference.

A convincing argument and a well-reasoned one are not the same thing, and a framework optimised for the former can be actively misleading about the latter. Any system that claims to evaluate reasoning from natural language has to confront that honestly.

What is reasoning?

Before a system can claim to measure reasoning, it has to settle what reasoning is. Every scoring dimension, every weighting decision and every architectural choice that follows rests on the definition adopted. A technically sophisticated apparatus built on a poorly specified target isn't measuring reasoning: it's measuring something else with impressive infrastructure.

The difficulty is that there is no agreed definition. Reasoning has been contested across at least 2,500 years of philosophy, logic, psychology and epistemology, and not only within the Western tradition. Thinkers working independently in ancient Greece, classical India and early China each arrived at the problem from different directions, and each isolates something the others underweight. That three civilisations with no contact each developed a formal theory of inference is itself instructive: the pressure to understand the movement from evidence to conclusion isn't culturally contingent. It's what thinking creatures run into whenever they try to organise knowledge, settle disputes or govern complex societies.

What follows isn't a comprehensive intellectual history. Each tradition below earns its place by grounding a specific design decision, and nothing is included that doesn't.

Form, warrant and framing

Aristotle's contribution is foundational in the most literal sense: without it there would be nothing to quantify. In the Prior Analytics he gives the first systematic account of valid deduction and establishes that validity depends on form, not subject matter. "All humans are mortal; Socrates is human; therefore Socrates is mortal" works because of its structure, not because of anything about Socrates. Swap in any terms and the same form guarantees the same result. If reasoning were simply a flow of impressions with no evaluable structure, there would be no property to capture. Aristotle established that there is.

The Stoics, particularly Chrysippus, extended this into propositional logic: reasoning about whole claims and their connections rather than class membership. "If p then q; p; therefore q." The modern conditional, the "if… then" that structures legal argument, scientific hypothesis and everyday planning, descends more directly from the Stoics than from the syllogism. Together they fix the principle that underwrites the whole project: reasoning has structure, and some structures are better than others.

The classical Indian tradition starts somewhere else, and for our purposes equally important. The Nyāya school embeds reasoning within a broader theory of pramāṇa, the valid means of acquiring knowledge. Inference is one such means, and the test of one isn't that it looks tidy but that it reliably gets you to what's true. Formal validity is not enough: the process has to actually track the world. That reliability condition becomes the philosophical home of our accuracy dimension.

The Mohists, early Chinese logicians working around the same period, add a third caution: the correctness of a categorisation made before inference even begins matters as much as the inference itself. How a question is framed can carry the answer inside it. That is the root of our consistency dimension, which tests whether a judgement survives a change of frame that changes nothing structural.

Bias, uncertainty, and the limits of coherence

Francis Bacon's contribution in 1620 was diagnostic rather than constructive. He catalogued the systematic ways the mind reliably destroys its own reasoning, the "idols of the mind": the idol of the cave, where personal history warps perception; the idol of the marketplace, where language misleads; the idol of the theatre, where received wisdom blinds. It is the first serious taxonomy of cognitive bias, three centuries ahead of Kahneman and Tversky. For us it is a design constraint: evaluation environments must force commitment before outcomes are revealed. If a player sees the answer before committing to a position, what is measured is recognition, not reasoning.

David Hume identified a deeper problem that remains unsolved. The sun has risen every day in recorded history, so we conclude it will rise tomorrow. But that conclusion doesn't follow logically from the premise, and justifying the inference by pointing to its past success is circular. Our confidence that the future will resemble the past rests on habit, not reason. The consequence for a scoring system: certainty is never available to someone reasoning about the empirical world, and every conclusion is held provisionally. Which is why we measure calibration rather than correctness. Correctness is partly luck. Someone who is confidently right by chance and someone who is tentatively right because they tracked the evidence carefully have produced the same output. Only calibration, over repeated judgements, separates them.

Kant's answer to Hume carries a warning we take seriously. Causal structure, he argued, isn't learned from experience but built into how the mind organises it, a precondition of coherent thought rather than a conclusion drawn from it. Hume and Kant, from opposite directions, both point at the same thing: the human mind has a deep, compulsive drive to make things cohere, to close gaps, to land on explanations that feel settled. That drive is exactly as likely to produce rationalisation as insight. A mind that feels certain it has understood something may have reasoned carefully to the truth, or may have found a story that satisfies the craving for closure. Those two states feel identical from the inside. So consistency has to be interrogated, not simply rewarded: a person who reasons consistently may be consistently rationalising, and telling the two apart requires convergent evidence across several dimensions, not consistency data alone.

Abduction, bounds, and measurable variation

The nineteenth-century formalisation of logic by Boole and Frege captured deduction with extraordinary precision and said almost nothing about the reasoning that dominates actual life. C.S. Peirce corrected this by identifying abduction, inference to the best available explanation, as a distinct and irreducible mode. The grass is wet, so it probably rained. The patient presents with these symptoms, so the diagnosis is this. These inferences aren't deductively valid, alternative explanations always exist, but they are recognisably rational, and they are open to revision when new evidence arrives. That is the epistemic structure of most real-world judgement, and the structure a pipeline has to assess. A system that measures only deductive competence measures a vanishingly small fraction of the reasoning people actually do.

Herbert Simon then established the conditions under which all reasoning occurs. No reasoner has ever had unlimited time, memory or information. Real people satisfice: they find answers good enough under the constraints they face, using heuristics that trade accuracy for speed. This isn't degraded reasoning; it is the only kind that exists. A pipeline that penalises cognitive limits is penalising a feature of human cognition, not a failure, so evaluation has to assess how well someone reasons within their constraints, not against an ideal no human could meet.

Kahneman and Tversky showed that those heuristics produce systematic, predictable errors, consistent across individuals and contexts. That consistency is the empirical licence for the entire enterprise. If reasoning errors were random noise, there would be nothing stable to measure at the level of a person. Because they are patterned, reasoning quality varies between people in ways a scoring system can in principle detect. Without that finding there would be no stable property to capture and no score worth building.

Philip Tetlock supplied the applied confirmation: some individuals are consistently better calibrated than others, the difference is measurable across many judgements over time, and it improves with practice. Calibration is a learnable, trackable, individual-level property. That underwrites both our calibration dimension and the longitudinal architecture built to measure it.

The Bayesian turn

The most consequential modern redefinition changes the question. Stop asking whether a conclusion follows necessarily from premises. Start asking: given what you know, how confident should you be, and how should that change when new information arrives? Bayesian reasoning is updating correctly, revising beliefs in proportion to what the evidence actually warrants. A doctor who puts a patient's risk at twenty percent, gets a positive test, and revises appropriately for the test's sensitivity and specificity is reasoning well, not because the conclusion is correct, which it may not be, but because the update is proportional. Two failure modes separate cleanly: under-updating, anchoring too stubbornly when evidence should move you, and over-updating, lurching in response to information that doesn't warrant it. Both are detectable by tracking how stated positions move when new evidence arrives.

The Brier score makes part of this measurable, calculating the squared distance between a stated probability and the eventual outcome, rewarding accurate confidence and penalising confident error. Across repeated forecasts it is an established measure of probabilistic accuracy and calibration. But it doesn't measure reasoning in full: it can't show whether a forecast came from sound evidence engagement, consistent judgement, or luck. So we treat it as one instrument for accuracy and calibration, sitting alongside separate measures of updating, consistency and evidence engagement.

One recent theory cuts against the tradition before it. Mercier and Sperber argued that reasoning evolved not to help individuals find truth but to win arguments in social settings. The claim is probably overstated, but the observation underneath it is real: the context in which a person reasons changes what their reasoning produces. The design implication is direct. Our structured environments deliberately remove the features of social reasoning that corrupt epistemic orientation. There is no audience to perform for, no opponent to defeat, and no reward for confidence or eloquence. What gets measured is how a person's private commitments track evidence, not how effectively they persuade.

The five dimensions

Five measurable dimensions follow directly from that definition, each grounded in a specific tradition and each operationalisable through the pipeline described below.

Not all five are equally visible everywhere. Calibration and updating behaviour require repeated judgements over time, so they are computed at the profile level rather than the individual-response level. Accuracy, consistency and evidence engagement can be assessed from a single response where the input gives enough structure.

Why existing methods fall short

The reason this capability doesn't already exist isn't a shortage of effort. It reflects three genuine technical problems that existing approaches address only partially.

1. The reconstruction problem

Reasoning is mostly invisible. What a person writes or says is the output of a process, not the process. Premises are omitted, steps are compressed, assumptions stay unstated, and one conclusion can be reached by many different chains. Any evaluator has to reconstruct the hidden structure, and Habernal and colleagues showed the scale of that job: human arguments routinely depend on unstated bridging assumptions connecting stated reasons to stated conclusions. The hard part isn't that the assumptions are hidden. It's that several plausible reconstructions exist for any given text, and the evaluator cannot know which one the author actually used.

Turpin and colleagues sharpened the concern: when asked to reconstruct reasoning chains, language models tend to produce the chain that best justifies the stated conclusion rather than the chain most likely to have been used. That is systematic bias, not noise. A pipeline that treats reconstructed chains as ground truth is evaluating a fictional version of someone's reasoning and calling it measurement. Lanham and colleagues reinforced the point: the faithfulness of model-generated explanations to the actual process can't be assumed, it has to be tested.

The problem is unsolvable in the general case but mitigable, and the pipeline mitigates it three ways: environment design that forces more of the process to the surface; treating the user's own text as the primary evidence, with reconstruction as a query layer that illuminates structure rather than replacing it; and longitudinal aggregation, since one response's hidden reasoning may be irrecoverable while patterns across many responses reveal stable characteristics of how a person reasons.

2. Calibration can't be seen in a single answer

Calibration is among the strongest indicators of reasoning quality in the forecasting and decision-science literature, and Gneiting and Raftery formalised the relationship between stated confidence and observed outcomes. But it cannot be reliably inferred from one response. A written answer carries linguistic signals of confidence, hedging, qualifying, asserting, and those are signals of expression style, not of epistemic calibration. A cautious-sounding user may look well calibrated while consistently being wrong; a confident user may sound overconfident while being right at exactly the right rate. Calibration is a property of a sequence of judgements with resolvable outcomes. Updating behaviour has the same constraint, though less severely. The architectural consequence is unavoidable: the pipeline must keep a longitudinal profile, and calibration cannot be meaningfully weighted in any single-response evaluation.

3. The evaluator is biased too

Even with perfect reconstruction, the judge is a problem. Language models used as evaluators are influenced by features of the input that have nothing to do with reasoning quality: authority effects, stylistic preferences, and order effects that shift verdicts without anything about the reasoning changing. Koo and colleagues organised these into implicit biases drawn out of single-prompt evaluations and induced biases arising from conversational context, and showed both produce systematic distortion. The instability is equally serious: presenting the same two responses in the opposite order can change which is preferred, and running the same evaluation twice on identical inputs can produce different scores. These are structural properties of probabilistic models in evaluation roles, not edge cases. Any pipeline relying on a single judge call without mitigation is producing scores with substantial evaluator noise inside them.

What each existing approach gets, and misses

• Natural language inference computes entailment, contradiction or neutrality between sentence pairs, and large benchmarks made it a central task in sentence-level NLP. But it works on isolated pairs, can't assess a reasoning process across a full response, and McCoy and colleagues showed it frequently leans on shallow lexical heuristics rather than genuine semantic inference. As a standalone route to reasoning assessment, it is largely superseded.
• Direct LLM-as-judge scoring sends a response and a rubric to a model and gets a structured score back. It is flexible, quick to deploy, needs no labelled data, and Mirzakhmedova and colleagues showed judges can reach high agreement with human annotators on structured tasks. But it inherits the bias and instability above, and it scores the surface text rather than the reasoning structure underneath. For short responses where the reasoning is fairly transparent that may be enough. For longer arguments where the critical moves are implicit, it isn't.
• Implicit premise reconstruction offers a route to the hidden structure of an argument by identifying unstated warrants and testing their robustness. Stahl and colleagues further showed that spotting a gap in an argument and correctly filling it are separate problems with different failure modes. It is analytically powerful and computationally expensive: a full reconstruction pipeline is a chain of sequential model calls, each dependent on the last, which puts evaluation depth in direct tension with feasibility at scale.

Five requirements fall out of that review, and no single existing approach satisfies all of them. The pipeline must handle reconstruction without treating generated chains as truth. It must run in at least two evaluation modes, one tuned for short, high-volume responses and one capable of structural analysis for longer argument. It must mitigate evaluator bias structurally rather than by instruction alone. It must keep a longitudinal profile, because two of the five dimensions are only assessable across a sequence. And it must balance depth against cost, because a system that produces excellent evaluations at prohibitive expense is a research artefact, not a capability.

The pipeline

Five design principles

1. Environment over reconstruction. Prefer contexts that force reasoning to the surface over post-hoc guessing about hidden chains.
2. User text as primary evidence. Reconstruction is a query layer, never ground truth. Scores attach to what the person demonstrably wrote or said.
3. Longitudinal profile over single-shot scoring. Calibration and updating aren't meaningfully assessable from one response, so the profile carries them.
4. Mode-specific weighting. A dimension that can't be reliably assessed from the available input is weighted toward zero rather than scored on thin evidence.
5. Feasibility at scale. Every decision is bounded by compute: the minimum number of model calls that still supports a defensible score.

Three input modes

Short-form pathway: one model call

This handles responses produced inside structured environments, where the input is brief, the information context is known, and game metadata can be used.

1. Input structuring. The response is packaged with its metadata into a standard evaluation object: the question asked, the information supplied, the response itself (text, selection, confidence level, prediction), any verifiable ground truth, and the dimension weightings for that game type. Deterministic, no model call.
2. Dimension evaluation. The object goes to a model with a rubric describing each dimension and the weightings for this game type. It returns constrained JSON: a numeric score per dimension in a defined range, plus a short justification for each. The rubric is concrete and anchored to observable features of the response, to shrink the room for evaluator bias. One call.
3. Output validation. The result is checked against a schema and against sanity conditions: do the justifications actually reference the response, are zero-weighted dimensions scored zero, do the numbers fall in the expected range. Malformed or anomalous output is rejected and retried or flagged. Deterministic.
4. Profile update. Validated scores are written to the player's longitudinal profile, where calibration is recomputed as the running comparison between stated confidence and actual accuracy, and updating is recomputed where the session introduced new information. Weighted toward recency and the current topic. Deterministic.

Long-form pathway: three model calls

Debate transcripts, speeches and articles are too long, too layered, and too full of implicit assumptions for a single judge pass to produce defensible scores. The full implicit premise reconstruction framework runs to eight sequential steps. We compress it to four, keeping the operations that contribute most to scoring accuracy and halving the cost, on the principle that each step must justify its compute against the scoring improvement it buys.

1. Stance and structure extraction. One call reads the whole text and pulls out the position or positions being advanced, the core reasons offered for each, and a compressed gist of every reason, as structured JSON: stance labels plus reason objects holding the original text span, the gist, and the conclusion each most naturally supports.
2. Warrant reconstruction and stress test. For each reason, a second call identifies the hidden assumption connecting reason to conclusion, then asks whether the same reason could support the opposite conclusion under a different assumption. Output per reason: the reconstructed warrant, an alternative warrant, and a robustness assessment. The compression works because that one stress-test question captures most of the analytical value of the four separate steps it replaces.
3. Dimension evaluation against structure. The scoring call receives the extracted stance, reasons, hidden assumptions, stress tests and robustness assessments rather than raw text, so scores reflect reasoning structure rather than surface language. Constrained JSON again, with weightings adjusted for the mode. The third call.
4. Validation and profile update. Identical to step four of the short-form pathway. No additional call.

Three calls per long-form evaluation, roughly half the cost of the full eight-step framework. Whether the compression costs acceptable scoring quality is an empirical question, and the validation programme below is designed to answer it.

What each mode can score

The longitudinal profile

The profile is the structure that makes calibration and updating assessable beyond single responses. It is not a permanent, ever-growing record. It is a lightweight rolling summary of recent reasoning behaviour, weighted toward the current context. After each evaluation event it updates deterministically: a recomputed calibration score from the running comparison of confidence against accuracy across all responses with resolvable outcomes, a recomputed updating score where the session introduced new information, and trend data on whether the person is improving, declining or stable on each dimension.

It is weighted by recency and topical relevance. Performance on a geopolitics story last week counts more for the current geopolitics story than performance on an economics story last month, because reasoning quality isn't a fixed trait: it varies by domain, by familiarity, and over time. The profile captures that variation instead of smoothing it away.

The cost of the update is negligible, arithmetic over a small structured object rather than a language model operation. That matters architecturally: the longitudinal intelligence of the system doesn't scale with model cost. Only per-response evaluation does, one call for short-form and three for long-form. The layer that makes the system genuinely longitudinal is effectively free.

How we'll know it works

A pipeline that produces numbers has to show those numbers mean something. Three pieces of empirical work, presented as a methodology for planned research rather than as completed results.

• Human benchmark. A dataset of responses across all three modes, spanning poor to excellent reasoning, on politically and ideologically diverse topics so content bias can be probed. Every response is scored independently by at least three trained annotators using the same rubric the pipeline uses, with inter-annotator agreement measured by Krippendorff's alpha. The target is for the pipeline to agree with human annotators about as well as the annotators agree with each other. Any dimension where it falls short of that bar needs its scoring mechanism revised.
• Bias testing. A balanced benchmark of high- and low-quality reasoning on both sides of contested political topics, built so that reasoning quality and political position are orthogonal, with the pipeline's scores analysed for systematic skew. This runs on every prompt version and every model change. Three structural mitigations sit alongside it: rubric anchoring on specific observable criteria rather than abstract quality words, position blinding where feasible for dimensions that shouldn't care which side you argue, and a standing regression suite that watches for drift across model updates and prompt changes.
• Longitudinal validation. Calibration is validated the same way it's defined: a player scored as eighty percent confident should be right roughly eighty percent of the time, testable by holding out recent responses and comparing predicted accuracy with actual accuracy. Updating is validated by events that introduce new information, where players scored as strong updaters should show larger, more proportional revisions than weak updaters.

Limitations

• The reconstruction problem is irreducible. Multiple reasoning chains can produce identical text, and no system can determine with certainty which one a person relied on. Where reconstruction is used, our scores describe the reasoning structure of the text, not the author's internal process. The distinction matters and shouldn't be obscured.
• Cold start. A new player has no history, so calibration and updating are effectively unscoreable at first. Accuracy, consistency and evidence engagement can be assessed from the first interaction, but the dimensions that separate this from a generic text evaluator need data that doesn't yet exist.
• Cost against fidelity. The compressed long-form pathway halves cost but may give up analytical nuance. Whether that drop crosses an acceptable threshold is an open empirical question.
• Model dependence. Scoring quality is bounded by the model doing the scoring. Updates, provider changes and behavioural drift all move the numbers; versioning and regression testing mitigate this without eliminating it.
• Cultural and linguistic scope. The philosophical grounding draws on several traditions, but the rubrics and prompts are built for English. Reasoning conventions, rhetorical norms and the relationship between text and thought vary across languages. The New Confucian thinkers Mou Zongsan and Tang Junyi argued that the Western scientific tradition developed its capacity for detached, propositional reasoning partly by bracketing moral and relational knowing. If reasoning quality is scored purely as accuracy and calibrated probabilistic updating, the system may quietly privilege one culturally specific epistemic mode over others. Extending beyond English needs adaptation of the framework, not translation of the prompts.
• Validation is outstanding. The benchmark hasn't been built, the annotation study hasn't been run, the bias tests haven't been run. Everything claimed here is architectural and theoretical until that programme is complete.

What it would mean

The foundational contribution is the definition. Synthesising Aristotelian formal logic, classical Indian epistemology, early Chinese philosophy of language, Enlightenment theories of bias and uncertainty, Peircean abduction, bounded rationality and Bayesian belief management produces a single formulation, and from it five dimensions that are each philosophically grounded and each scoreable by the architecture above.

The technical contribution is the hybrid pipeline: a lean single-call pathway for short-form responses, a compressed three-call pathway that extracts structure from long-form argument before scoring, and a lightweight longitudinal profile that makes calibration and updating assessable without inflating per-evaluation cost.

The honest limitation is that these are architectural claims, not empirical ones. The pipeline has been designed; it has not been validated against human judgement at the scale needed to confirm its scores are meaningful. The next phase is to run the programme above, build the benchmark, conduct the annotation study, run the bias tests, and report whether the scores reach the agreement with expert human judgement that would justify deployment.

If they do, the result is a capability that has never existed at scale: not what a person believes, not how persuasively they argue, and not whether a machine can replicate a chain of logic, but how well a human being reasons. Whether it can be built is an empirical question. This paper is the case that it is worth attempting, and the blueprint for how to try.