Chain-of-thought prompting works because it forces a language model to generate intermediate reasoning tokens before it commits to a final answer. Each token the model produces becomes part of the context it conditions on for the next token. Ask a model to reason step by step instead of blurting out an answer, and you change what it computes, not just what it says.
Most people treat this as a trick, a magic phrase you paste into a prompt. It is not magic. It is a direct consequence of how transformer models allocate computation, and understanding that mechanism tells you exactly when chain-of-thought prompting will help and when it will waste your tokens.
Ask a model "What is 17% of 240, minus 12?" and tell it to answer with just a number, and you are asking it to produce the correct result in a single forward pass, compressed into one token or a short burst of tokens. Ask the same question but add "show your work step by step," and the model first writes out 17% of 240 is 40.8, then 40.8 minus 12 is 28.8. The final answer is the same claim either way. The path to it is completely different.
This idea got formalized in a 2022 Google paper by Jason Wei and colleagues. On the GSM8K grade-school math benchmark, a 540-billion-parameter PaLM model jumped from about 18% accuracy to 57% accuracy just by adding worked examples that showed reasoning steps in the prompt. No retraining. No new data. Same weights, same model, different prompt structure.
The gain was not because the model suddenly understood arithmetic better. It was because the task got restructured into something the model's architecture is actually built to do well: predict the next token given a growing sequence of prior tokens.
A transformer spends a roughly fixed amount of computation per token it generates. That is a hard architectural constraint, not a design choice you can prompt away. If you demand a complex, multi-step answer squeezed into one or two output tokens, you are asking the model to do multi-step reasoning inside a single computational budget that was never sized for it.
Chain-of-thought prompting sidesteps that constraint by spreading the reasoning across many tokens instead of one. Each intermediate step the model writes gets fed back into its own context window. The model then attends over its own prior reasoning the same way it attends over your original prompt. It is effectively using its own output as scratch memory.
This is why the phrasing matters less than people assume. "Let's think step by step" works, but so does asking for a numbered list, or telling the model to explain its reasoning before stating a conclusion. What matters is that the model produces intermediate tokens it can condition on, not the exact wording used to request them.
There are two ways to trigger this behavior, and they were discovered a few months apart in 2022, and they behave differently.
| Approach | How it works | Best for |
|---|---|---|
| Few-shot chain-of-thought | Prompt includes 2-8 worked examples showing full reasoning before the answer | Domain-specific tasks, consistent formats, when accuracy matters more than prompt length |
| Zero-shot chain-of-thought | A single instruction like "let's think step by step" is added, no examples given | Quick one-off tasks, general reasoning, when you cannot supply good examples |
Kojima and colleagues showed in 2022 that zero-shot chain-of-thought alone, with no worked examples at all, roughly quadrupled performance on some arithmetic benchmarks compared to a plain zero-shot prompt. That result mattered because it proved the effect was not just pattern-matching against examples. The instruction to reason step by step was doing real work on its own.
Few-shot still tends to win on narrow, format-sensitive tasks, because the examples teach the model exactly how you want the reasoning structured. If you are building a repeatable pipeline, invest in two or three strong worked examples. If you are asking a one-off question, zero-shot chain-of-thought costs you almost nothing to try.
Chain-of-thought prompting is not a universal upgrade. It helps on tasks that genuinely decompose into steps: arithmetic, multi-hop logic, planning, code tracing. It does little or nothing for tasks that are closer to lookup or pattern recognition, like factual recall or simple classification. Asking a model to reason step by step before naming the capital of a country adds tokens and latency without adding accuracy.
Worse, on some tasks it can actively hurt. Researchers have found cases where forcing a model to verbalize reasoning about a task it handles better through fast, implicit pattern matching actually degrades accuracy, because the generated reasoning steps introduce errors that the model then treats as established fact and builds on. A wrong early step compounds. The model does not go back and check its own math the way a careful human would; it keeps generating forward from whatever it already wrote.
This is also why chain-of-thought output should never be mistaken for a transparent record of what the model actually computed internally. The written reasoning is a generated artifact, produced the same way any other text is produced. It can look coherent and still not reflect the real path to the answer. Treat it as a tool that improves accuracy on the right tasks, not as a window into the model's mind.
It is worth being precise about what chain-of-thought prompting is not, because it gets lumped in with other techniques that solve different problems entirely.
These three are complementary, not competing. A fine-tuned model can still benefit from chain-of-thought prompting on hard reasoning tasks. A RAG system that retrieves the right documents still needs the model to reason correctly over them, and chain-of-thought prompting often improves that reasoning step. None of them substitutes for the others.
The strongest version of this technique in production is self-consistency: instead of generating one chain-of-thought answer, you sample several independent reasoning paths at a higher temperature and take the majority answer. This costs more tokens and more latency, but it catches cases where one flawed reasoning chain would have locked in a wrong answer. On math and logic benchmarks, self-consistency on top of chain-of-thought regularly beats either technique alone.
Newer reasoning-tuned models, the ones trained specifically to generate long internal reasoning traces, do a version of this automatically, whether or not you ask for it. That does not make chain-of-thought prompting obsolete for older or smaller models. It just means the technique moved from something you request explicitly to something baked into how certain models operate by default.
The practical rule holds regardless of which model you are using: reserve chain-of-thought for tasks with real multi-step structure, skip it for simple lookups, and add self-consistency sampling when the cost of a wrong answer is higher than the cost of a few extra API calls.
It is a prompting technique where you ask a language model to write out its reasoning step by step before giving a final answer, instead of jumping straight to the conclusion. This gives the model more intermediate tokens to condition on, which improves accuracy on tasks that require multiple logical steps, like math word problems or multi-hop reasoning.
No. It helps most on tasks that genuinely require multiple reasoning steps, such as arithmetic or logical deduction. On simple factual recall or classification tasks it typically adds no benefit, and on some tasks it can even reduce accuracy if an early flawed reasoning step compounds into a wrong final answer.
Zero-shot chain-of-thought adds a single instruction, like telling the model to think step by step, without providing any worked examples. Few-shot chain-of-thought includes several worked examples in the prompt that demonstrate the exact reasoning format you want the model to follow. Few-shot tends to perform better on narrow, format-specific tasks, while zero-shot is faster to set up for general use.
No, they are entirely different mechanisms. Fine-tuning permanently changes a model's weights through additional training, while chain-of-thought prompting changes nothing about the model itself. It simply restructures the output at inference time so the model spreads its reasoning across more tokens, which changes how much computation it effectively applies to the problem.
Not fully. The written reasoning is generated text, produced the same way any other output is produced, and it does not necessarily reflect what the model computed internally. It can look logical and coherent while still being disconnected from the actual path to the answer, so treat it as a tool for improving accuracy, not as proof of how the model reached its conclusion.