AI Can Make Bad Reasoning Faster

AI does not receive a problem in raw form. It receives a representation of a problem: a request written by someone, evidence selected from somewhere, categories and objectives defined in advance, and a surrounding process that already contains judgments about what matters. By the time a model begins reasoning, a substantial amount of reasoning may already have happened outside the model.

That matters because the earlier reasoning can be wrong. A user can give AI a task whose assumptions are faulty. An organization can connect the system to data that are inaccurate, stale, incomplete, poorly timed, or inappropriate for the decision. A workflow can encode a diagnosis that was never adequately established. In each case, the AI may perform the assigned work competently while making the underlying weakness easier to execute, repeat, and trust.

Much of the discussion about AI reliability focuses on whether the model hallucinates or produces an incorrect answer. Those problems matter, but they are only part of the reasoning environment. A system can also behave exactly as instructed, use the information it was given, and still magnify a mistake that entered before the model ever had a chance to reason about it.

The Problem Can Enter Through the Prompt

A 2026 study presented at ACL examined a simple but consequential question: what happens when users ask AI questions that already contain false assumptions? The researchers constructed queries across health, science, and general knowledge with incorrect presuppositions and varied how strongly those presuppositions were expressed. Reasoning models performed somewhat better than non-reasoning models, but they still failed to challenge between 26 and 42 percent of the false presuppositions they were given. Stronger wording of the false assumption also made the models more susceptible to accepting it.

A separate 2025 evaluation of fifteen language models reached a related conclusion. Most models showed limited ability to critique flawed premises unless they were explicitly prompted to look for errors, and stronger general reasoning performance did not consistently translate into stronger premise-critique ability. Earlier work on false-premise questions had already shown that language models can sometimes possess the knowledge needed to reject a false premise without reliably activating that knowledge when the question is asked.

The same issue can persist even when a system has access to an outside knowledge source. A 2025 ACL study of retrieval-augmented generation tested what happens when information supplied by the user conflicts with information retrieved from a database. Across six widely used models, the researchers identified an authority bias: the systems tended to favor the user’s information even when it departed from the factual information in the retrieved source.

The organizational implication is larger than question answering. A request to AI can already contain a diagnosis, a causal explanation, an assumed constraint, a preferred solution, or a definition of success. If the system treats those premises only as instructions to satisfy, greater capability can produce a more sophisticated answer inside a weak frame. The resulting analysis may become more persuasive precisely because the AI has organized and developed the original assumption so well.

Useful AI behavior therefore cannot be reduced to obedient task completion. The system does not need to challenge every instruction, but it should be able to recognize when a material conclusion depends on a premise that is contradicted by available evidence, weakly supported, unusually consequential, or central enough that getting it wrong would make the rest of the work irrelevant. In those situations, questioning or clarifying the assignment is part of doing the assignment well.

Grounded in Data Does Not Mean Grounded in Reality

The second weakness can enter through the data rather than the prompt. Connecting an AI system to organizational information is often described as grounding: instead of relying only on what the model learned during training, the system can retrieve internal documents, records, metrics, policies, histories, or other sources relevant to the task. That can substantially improve usefulness, but it does not turn the underlying information into truth merely because the AI can retrieve it.

A large qualitative study published in 2021 examined this problem in high-stakes AI projects. Researchers interviewed 53 AI practitioners working across health care, conservation, agriculture, and other settings in India, East and West Africa, and the United States. They described what they called data cascades: data problems that create compounding downstream effects as they move through models and operational decisions. The researchers found such cascades in 92 percent of the cases they studied and emphasized that the problems were often difficult to see until later stages of the system.

A 2025 JAMA Network Open study provides a particularly clear example of why data can be technically real and still be wrong for the reasoning task. The researchers examined prediction studies built with MIMIC, a widely used database of deidentified hospital records. They found that 37 of 92 published studies predicting outcomes during the same hospital admission used ICD diagnostic codes as input features even though those codes are finalized only after discharge. The researchers then built mortality models using only those codes and obtained extremely high performance, with AUROC values around 0.97 to 0.98. Some of the most influential codes included cardiac arrest, brain death, and palliative care - information that would often not exist at the earlier point when a useful mortality prediction would need to be made.

The problem was not simply that the database contained false facts. Much of the information was accurate as a record of what eventually happened. It was invalid as evidence for a prediction that was supposed to occur before those events were known. The model could therefore look exceptionally capable while partly benefiting from hindsight embedded in the data.

Enterprise generative AI introduces another version of the same problem. A 2025 ACL benchmark on retrieval-augmented generation found that outdated information in a knowledge base could reduce answer accuracy and mislead models even when current information was also available. NIST’s AI Risk Management Framework likewise treats data quality, representativeness, suitability, provenance, and context of use as system-level concerns rather than assuming that access to more data is inherently better.

For organizations, this means that ‘grounded in our data’ is not a sufficient quality standard. The relevant question is whether the information is fit for the reasoning being performed: accurate enough, current enough, sufficiently complete, appropriately representative, traceable to its source, and available at the point in time the decision actually requires. If AI-generated conclusions are then saved back into documents, decisions, or operating records, any weakness that survives the first pass can also become part of the environment later systems retrieve from. That possibility makes provenance and correction mechanisms increasingly important as AI becomes more integrated with ordinary work.

Automation Changes the Human System Around It

The risk is not limited to prompts and data. Automation also changes the human system in which reasoning occurs. In 1983, human-factors researcher Lisanne Bainbridge published ‘Ironies of Automation,’ examining highly automated industrial systems in which people were removed from routine control but still expected to intervene when something unusual went wrong.

Bainbridge’s concern was structural. The routine involvement that automation removed was also part of how operators maintained practical skill and an up-to-date understanding of the system. When the automated system encountered an abnormal condition, the human could be asked to take over precisely when the situation was hardest to understand and after routine participation had diminished.

Modern AI systems are not identical to the industrial control systems Bainbridge studied, but the design question carries forward. Removing a human step does more than save time or labor. It changes where observation occurs, which anomalies are noticed, how expertise is maintained, where assumptions can be challenged, and what people will still understand when the automated path stops working. The design problem is therefore not how much human involvement can be eliminated, but which interactions were unnecessary friction and which were performing a reasoning function that still has to exist somewhere in the new system.

A Correct Result Is Not the Same as Understanding

Mathematics makes another part of the problem unusually clear. Mathematical practice distinguishes verifying that a proof is valid from understanding why the proof works. Recent philosophical work on proof understanding describes that difference in terms of reconstructing the plan or structure behind a proof rather than merely checking each inference one after another.

The distinction has become more consequential as AI systems have begun producing mathematical results that people may not be able to reconstruct independently. In a 2026 essay hosted by Oxford University, computer scientist and mathematician Leslie Ann Goldberg raised concern about preserving the human infrastructure for mathematical understanding as AI-generated proofs become more capable. Her concern was not that mathematical truth stops being true when AI is involved. It was that a formally checkable result and a human-understandable route to the result do not provide the same kind of intellectual value.

For AI systems, that distinction matters in at least two ways. A conclusion that happens to be correct does not establish that the method producing it is reliable; a weak assumption or invalid intermediate relationship can occasionally land on the right answer. And even when the reasoning is sound, the path can generate reusable knowledge: a representation of the problem, an intermediate relationship, a method, a boundary condition, or a failed approach that teaches something useful for later work. If only the final answer survives, the organization may lose both a way to evaluate the result and some of the knowledge created while reaching it.

An Explanation Is Not Automatically Provenance

There is an additional complication. Asking a language model to explain an answer does not necessarily reveal the process that actually determined the answer. In a 2023 NeurIPS paper, researchers deliberately introduced biasing features into prompts and then examined the models’ chain-of-thought explanations. The models were often influenced by those features without acknowledging them, and could produce plausible reasoning that rationalized the resulting answer.

That finding puts a useful boundary around the demand for explainability. A fluent explanation can help a person inspect an answer, but it should not automatically be treated as a transparent transcript of hidden model computation. For consequential work, the more reliable target is externally inspectable reasoning provenance: the evidence supplied to the system, the assumptions that materially shaped the task, conflicting sources that were encountered, intermediate claims or relationships that need verification, relevant uncertainty, specialist inputs, and the conditions that should cause the conclusion to be reopened.

This form of provenance does not require exposing every internal computational step. It preserves the parts of the reasoning environment that people need in order to decide whether the result is warranted, understand where it came from, correct it when evidence changes, and carry useful discoveries into later work.

A Different AI Adoption Question

When an organization evaluates AI-enabled work, technical feasibility is only one part of the assessment. The surrounding reasoning environment deserves examination as well, particularly when the consequences are material or the task depends on uncertain premises and organizational data.

• What problem is the user or process asking AI to solve, and which assumptions are already embedded in that framing?

• What should cause the system to question, clarify, or reopen a premise rather than simply optimize inside it?

• Are the data accurate, current, complete, representative, and temporally appropriate for the decision being made?

• What is the provenance of important evidence, and how should conflicts among user claims, internal records, and outside sources be resolved?

• What relevant knowledge or specialist expertise exists outside the information currently available to the AI?

• What reasoning provenance needs to remain inspectable for an important conclusion to be evaluated, challenged, taught, or reused?

• Which judgments can AI support or execute, and where should authority remain explicitly human or specialist?

• How will the organization know whether the system is improving outcomes and learning rather than simply increasing throughput?

Not every implementation requires the same depth of examination. Low-consequence, reversible work can often be tested cheaply and corrected quickly. As consequences, irreversibility, uncertainty, or automation increase, the value of premise critique, data-quality controls, provenance, specialist routing, and deliberate human challenge increases with them.

Better Reasoning Is Also an AI Opportunity

The concern here is not that AI inevitably creates bad reasoning. The stronger point is that AI can amplify weaknesses introduced at several layers. A user can supply a faulty frame. Organizational data can supply inaccurate, stale, biased, or contextually invalid evidence. A capable model can develop the resulting task fluently. Automation can repeat the output at scale, and polished results can make the underlying weakness less visible rather than more visible.

The same capabilities can be used in the opposite direction. AI can compare the user’s premise with available evidence, surface conflicts in the organization’s own records, flag stale or poorly supported information, distinguish observation from assumption, seek discriminating evidence, bring specialist knowledge into the problem, preserve uncertainty, and make important reasoning provenance visible before a conclusion becomes embedded in later work.

That is a more demanding view of AI than task completion, but it is also a more useful one. The objective is not merely to make organizational work faster. It is to make sure the reasoning that becomes faster is worth amplifying, and to build systems capable of noticing when the task, the evidence, or the surrounding process should be questioned before speed turns a weak premise into a durable result.

Selected sources and further reading

Sathyanathan, R., Vasisht, K., and Pruthi, D. Evaluating Reasoning Models for Queries with Presuppositions. Findings of ACL 2026.

Li, J., Li, G., Chang, Y., and Wu, Y. Don’t Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Models. Findings of EMNLP 2025.

Hu, S. et al. Won’t Get Fooled Again: Answering Questions with False Premises. ACL 2023.

Li, Y. et al. LLMs Trust Humans More, That’s a Problem! Unveiling and Mitigating the Authority Bias in Retrieval-Augmented Generation. ACL 2025.

Sambasivan, N. et al. Everyone wants to do the model work, not the data work: Data Cascades in High-Stakes AI. CHI 2021.

Ramadan, B. et al. Diagnostic Codes in AI Prediction Models and Label Leakage of Same-Admission Clinical Outcomes. JAMA Network Open, 2025.

Ouyang, J. et al. HoH: A Dynamic Benchmark for Evaluating the Impact of Outdated Information on Retrieval-Augmented Generation. ACL 2025.

National Institute of Standards and Technology. AI Risk Management Framework Playbook - Measure.

Bainbridge, L. Ironies of Automation. Automatica 19(6), 1983.

Goldberg, L. A. Mathematics and AI. University of Oxford, 2026.

Hamami, Y., and Morris, R. L. Understanding in mathematics: The case of mathematical proofs. Noûs 58(4), 2024.

Turpin, M., Michael, J., Perez, E., and Bowman, S. Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023.