Use a minimal reproducer, explicit semantics, intermediate evidence, and controlled measurements.
By the end: Locate failures without blaming the wrong stage and distinguish a compiler defect from undefined behavior or an invalid performance comparison.
Classify the symptom and preserve the conditions
Start with the exact source, inputs, target, compiler version, build options, libraries, and observed behavior. A lexical error, type error, unresolved symbol, runtime exception, wrong result, and slow execution are different symptoms. They may require evidence from different stages.
Reduce the example while preserving the failure. A smaller program can expose the relevant binding, alias, lifetime, arithmetic, or control-flow condition. If a change removes the symptom, do not assume it identifies the cause; it may have changed optimization opportunities, layout, timing, or whether the program was defined.
Find the first representation that disagrees
A wrong result is an observation, not yet a diagnosis. Suppose a report says that 8 - 3 - 1 returns six instead of four. First establish the source contract. SmallExpr makes these subtraction operators associate left, so the required grouping is the subtraction of one from the result of eight minus three. Both operands of each operation are integers, and these small results do not wrap. This establishes four without consulting either the tree evaluator or the bytecode machine that might contain the defect.
Now compare evidence at successive boundaries. The token spellings should be 8, -, 3, -, and 1. The syntax tree should have an outer subtraction whose left child is another subtraction. If the tree instead puts subtraction of three and one on the right, the disagreement already exists before code generation. The parser is then a useful investigation target. If the tree has the required grouping but emitted code computes the other tree, inspect emission. These are hypothetical defects to localize, not faults claimed to exist in the lab.
The same tokens can become different trees
Original schematic trees. Edges mean child relationships, not control flow or time. SmallExpr requires the left tree for 8 - 3 - 1. The right tree explains a hypothetical wrong result of six.
The required bytecode pushes eight, pushes three, subtracts, pushes one, then subtracts again. The competing stream pushes all three numbers before either subtraction. Its first subtraction consumes three and one, leaving eight below their difference. Its second subtraction therefore computes eight minus two. A trace returning six can be completely faithful to that incorrect instruction stream. Blaming the virtual machine merely because it produces the final wrong number would miss the earlier loss of source structure during translation.
At each boundary, compare meaning rather than spelling alone. A tree can discard parentheses while preserving their grouping. Constant folding can remove an entire arithmetic subtree while preserving its result. A different instruction sequence is therefore not itself evidence of a defect. Ask which required property first becomes false. In this example the property concerns operand grouping and order. In a separate language with output or exceptions, the property might instead concern an observable event that a transformation removed or moved.
Reduction should preserve that property as well as the visible symptom. Changing subtraction to addition makes both candidate groupings return twelve, which hides this particular defect. Replacing names with literals can remove a binding problem while leaving arithmetic intact. A useful reduction log records which source change preserves the disagreement and which removes it. Do not infer a cause from one disappearing symptom. The altered case may no longer distinguish the two implementations even when the same faulty rule remains present.
Practice this investigation on paper before using the editor. Write the two subtraction trees and predict each result. Then use the syntax lesson to inspect 8 - 3 - 1 and the execution lesson to inspect its instruction order. The diagnosis widget below is the folding comparison, not an instruction editor or a stage debugger. It can compare the original and folded forms of a valid expression. It cannot inject the hypothetical bad stream, step through parser internals, or establish anything about a native compiler installed on your machine.
Locate the first different stack state
Completed step
Required stream; stack
Competing stream; stack
0
Before execution; []
Before execution; []
1
PUSH 8; [8]
PUSH 8; [8]
2
PUSH 3; [8, 3]
PUSH 3; [8, 3]
3
SUB; [5]
PUSH 1; [8, 3, 1]
4
PUSH 1; [5, 1]
SUB; [8, 2]
5
SUB; [4]
SUB; [6]
Exact traces in the teaching VM, with stacks written bottom first. SUB pops the right operand first. Both streams are locally executable and leave one value, but only the required stream translates the left-associated source correctly.
Inspect representations and check equivalence
For SmallExpr, compare the parsed tree, inferred type, original code, folded code, direct evaluation, and VM result. Tests can enumerate many bounded expressions and environments and verify that all routes agree. They also need malformed-input cases, overflow cases, and independent expected results.
Agreement between two components is not conclusive if both share the same mistake. Our tests therefore include hand-calculated cases and a separate big-integer wrapping reference for arithmetic. Production compiler validation may also use differential testing, fuzzing, translation validation, formal arguments, and language conformance suites. Each supplies different evidence.
Build an expected result that does not share the suspected mistake
Comparing the direct evaluator with the VM is valuable because their control structures differ. One recursively returns child values; the other changes an operand stack. Agreement can expose neither a shared parser error nor a shared arithmetic error, however. Both consume the same tree, and both use the same arithmetic helper in this implementation. A validation plan therefore needs expectations derived independently of those components. Independence is a specific question about shared assumptions and code, not a status granted simply by naming two execution routes.
Overflow makes this distinction concrete. The accepted expression 999 * 999 * 999 * 999 has four leaves, so it fits the paired view. Left association first produces 998,001, then 997,002,999, both within the signed range. The next mathematical product is 996,005,996,001. Divide that value by 4,294,967,296, the number of possible thirty-two-bit patterns. Subtracting 231 complete multiples leaves 3,868,550,625. That unsigned remainder exceeds the largest signed value, so subtract one more modulus to obtain the signed result, negative 426,416,671.
That calculation defines an independent expected value without borrowing the implementation's multiplication routine. A checker can calculate mathematical products with arbitrary-precision integers, take a nonnegative remainder modulo the same modulus, and convert it to the signed range. It must apply the language's wrapping contract rather than assume that the host language's ordinary multiplication already has it. Merely multiplying large values in a host numeric type and comparing two rounded answers can reproduce a host limitation instead of checking SmallExpr semantics.
Derive the wrapped product independently
Quantity
Exact integer value
Reason
First product
998,001
999 × 999
Second product
997,002,999
998,001 × 999; still signed int32
Final mathematical product
996,005,996,001
997,002,999 × 999
231 complete moduli
992,137,445,376
231 × 4,294,967,296
Unsigned remainder
3,868,550,625
Subtract complete moduli
Signed result
−426,416,671
Remainder minus 4,294,967,296
All quantities are exact integers, not times or addresses. The expression is left-associated. Its first two products need no wrap; the final product does. The signed conversion uses the int32 contract, not a host floating-point approximation.
Separate acceptance tests from value tests. The input flag * 0 should fail typing even though a numerical multiplication by zero would yield zero. Here flag denotes a boolean, and SmallExpr has no boolean coercion. The source absent + 1 should instead fail binding. Neither should become an accepted integer because an optimizer erased an inconvenient operand. Such negative tests check the order and obligations of the pipeline. They are not executions with surprising numeric answers, because accepted execution is never reached.
Some tests compare related inputs rather than require a new expected number for each. Adding zero to an accepted integer expression should preserve its wrapped result, and an extra pair of grouping parentheses should preserve its meaning. These relationships are often called metamorphic tests. State their domain. Adding zero to a boolean is invalid here, while extra parentheses can exceed a nesting bound. For an otherwise accepted integer case within the editor limits, a failed relationship suggests a defect worth reducing, but a passed relationship does not prove complete correctness.
Construct a small evidence set with a normal arithmetic case, a noncommutative case, an overflow case, and rejected inputs. For each, record the required outcome before inspecting the lab. Distinguish an incorrect returned integer from an incorrect stage classification or a teaching-limit rejection. A very long identifier tests the implementation's display bound rather than the arithmetic rules. This makes the test set informative when a case fails. It also prevents a collection of passing addition examples from concealing reversed subtraction or an optimizer that skips validation.
Different tests check different obligations
Case
Required outcome
Mistake this can expose
2 + 3 * 4
Integer 14
Incorrect precedence
8 - 3 - 1
Integer 4
Wrong association or operand handling
999 * 999 * 999 * 999
Integer −426,416,671
Wrong overflow contract
flag * 0
Typing error
Boolean coercion or skipped validation
absent + 1
Binding error
Acceptance of an undeclared name
(x * (2 + 3)) + 0
Integer 20, unchanged by added zero
Broken integer identity within limits
These are distinct proof obligations for a bounded test set, not a proof for every source. The last case is a paper test or a general-view input; its five leaves exceed the paired optimization view's four-leaf display limit.
Work through the mechanism
Compare before and after folding
Presets replace the whole source. Editing replaces the current calculation and resets any execution trace. Names: x = 4, y = 7, flag = true. Use ASCII - for subtraction and negation. Arithmetic is signed 32-bit wrapping; booleans are not numbers. See lesson 1 for the full grammar and editor bounds. The paired optimization view accepts at most four leaves so both trees stay readable.
Type: int. Direct evaluation: 20. VM result: 20. Original instructions: 5; after folding: 3.
Original bytecode
LOAD x
PUSH2PUSH3ADDMUL
After folding
LOAD x
PUSH5MUL
Folded VM result: 20. Counts measure these abstract instructions, not elapsed time or native CPU instructions.
The output is recalculated immediately. This bounded compiler implements expressions only, not the later pseudocode examples. The entered source is never passed to JavaScript eval or Function.
Evidence
Question it answers
Source and language contract
What behavior is required?
Tokens, AST, typed representation
Was the program understood as intended?
IR and generated code
What transformations occurred?
Runtime trace and measurement
What actually happened under these conditions?
A teaching model tests mechanisms, not production compiler correctness or speed.
Use the folding comparison to test a narrow claim
Select x * (2 + 3) to audit a specific claim. The original tree reads x on the left and computes a literal addition on the right. Folding replaces only that addition with five. With the fixed editor binding x equal to four, both return twenty. The surviving name node matters. This pass folds numeric literal subtrees; it does not propagate the current environment into the source. Seeing twenty in the result does not justify replacing every occurrence of the whole expression with twenty.
Follow the intermediate storage as well as the tree. Original code loads x, pushes two, pushes three, adds, and multiplies. The stack after the two pushes contains four, two, and three, bottom first. Addition replaces the top pair with five; multiplication then produces twenty. Folded code loads x, pushes five, and multiplies. Both have one final value and the same operand meaning. This identifies which work disappeared and which runtime read remains.
Folding changes intermediate stack storage
Original schematic stack snapshots at explicitly different instruction boundaries for x * (2 + 3), with x = 4. Each cell is one value, not a physical byte or time unit. The original ADD consumes 3 and 2; the following MUL consumes 5 and 4. Folded code begins directly with the latter pair.
Predict a contrasting case before selecting the full-fold preset. In (2 + 3) * (4 - 1), both children contain only numeric literals. Their results are five and three, so the multiplication can also fold, leaving the literal fifteen. The original stream has four pushes and three binary instructions, seven instructions in total. The folded stream has one push. This is a valid stronger transformation for this source because no name read remains. It is not evidence that the folder can specialize arbitrary expressions to the currently displayed environment.
Now select x * 0 + y. The result is seven with the fixed bindings, but the implemented folder leaves the tree unchanged. Neither multiplication nor the final addition has two literal children. A richer optimizer for this pure, checked integer language could use arithmetic identities, but this pass does not. Distinguish a missed opportunity from an invalid result. If an optimization claim says this particular pass deletes multiplication by zero, the unchanged tree refutes the claim about its implementation, even though deleting that multiplication could be semantically justified after validation.
To test that contrast yourself, edit the source to (2 + 3) * 0. It still fits the four-leaf paired limit. The literal addition folds to five, then multiplication folds to zero. Expect five original instructions and one folded instruction. Compare this with the unchanged five-instruction stream for x * 0 + y. The zero literal is not a universal trigger for deleting a subtree. Whether the children become literals during recursive folding determines the behavior of this narrow pass. Record the transformation, the count, and the value separately.
The reduction from five instructions to three removes two of five abstract instructions, or forty percent. It supplies no measured elapsed-time percentage. A native backend might choose a multiply with an immediate operand, allocate different registers, or optimize the surrounding program further. The teaching VM still incurs whatever overhead its implementation needs for each instruction. The page does not time that overhead. Use the simultaneous trees and code as evidence for translation behavior, and the matching values as evidence for these executions, without converting either observation into an application speedup.
Count only the work this pass removes
Source
Folded form
Instructions before → after
Value
x * (2 + 3)
x * 5
5 → 3
20
(2 + 3) * (4 - 1)
15
7 → 1
15
x * 0 + y
Unchanged
5 → 5
7
(2 + 3) * 0
0
5 → 1
0
Exact counts of PUSH, LOAD, and arithmetic instructions in the teaching VM with x = 4 and y = 7. The five-to-three change removes 40% of these instructions. No elapsed-time measurement or native instruction count is implied.
Measure the execution you actually intend to compare
Fewer abstract instructions need not mean less elapsed time. Keep workload and output equivalence fixed, separate compilation and startup from steady-state execution when relevant, and account for warmup, allocation, I/O, and measurement noise. Do not benchmark a computation the optimizer can remove and then infer the cost of useful work.
A suspected miscompilation deserves a reproducible report with the required semantics and observed difference. First check assumptions such as signed-overflow rules, pointer lifetimes, data races, and ABI compatibility in the language being used. This does not rule out compiler bugs; it makes a report capable of distinguishing one from a program whose required behavior was never established.
Separate a correctness report from a timing report
A correctness report needs a required behavior and a reproducible disagreement with it. Preserve the exact input, the implementation version, and the relevant options before reducing the program. For a native program, also preserve its target and linked dependencies. A changed optimization option is an experimental condition, not a diagnosis. First determine whether the execution has defined meaning under that language's rules. SmallExpr's wrapping contract does not transfer to another language whose signed overflow, pointer lifetime, or concurrent memory rules permit different conclusions.
Suppose a reduced native case calls a function from another compiled module and receives an unexpected value. A source-level type checker may have checked only the declaration visible to the caller. Successful linking may establish only that the symbol resolved. If the modules disagree about argument placement or structure layout, the runtime symptom does not establish that an arithmetic optimization was wrong. Inspect the boundary agreement. This is a separate ABI investigation, not an extension of the expression lab, which has neither calls nor object files.
A successful boundary check has a limited meaning
Evidence
What it supports
What still needs checking
Arithmetic expression is typed
Operand types satisfy these rules
Transformation preserves its value
Caller declaration is checked
Caller is consistent with that declaration
Callee matches the declaration and ABI
Symbol is resolved
Lookup found a definition
Argument layout and ownership agree
One trace reaches a block
That run used that path
Safety on other possible paths
The first row concerns SmallExpr. Calls, ABI boundaries, object files, and control-flow traces in the other rows are separate investigation models. A later failure can coexist with an earlier check that worked exactly as specified.
Use intermediate evidence to narrow a native report without pretending that every tool exposes the same stages. An IR listing can show that a computation disappeared before instruction selection. Disassembly can show a remaining machine operation, while a runtime trace can show whether control reached it on the supplied input. Static listings describe possible execution; a trace describes one run. A graph with a loop is not an iteration count. Likewise, one successful path does not prove that a moved operation is safe on paths that the run did not take.
A timing report asks a different question. Define whether the measured interval includes compilation, loading, startup, or only repeated execution after preparation. Check equivalent outputs and comparable work before comparing durations. Retain a consumed result so an optimizer cannot discard the benchmark's computation without consequence. In systems with JIT compilation, inspect warmup separately from later samples; in ahead-of-time systems, startup and cache state can still matter. Repeated samples help describe variability, but repetition alone cannot repair two benchmarks that do different work.
Treat diagnostic instrumentation as another changed condition. Printing every VM stack state adds formatting and output work that ordinary execution may omit. Measuring such a traced run answers a question about tracing enabled, not the uninstrumented arithmetic engine. Similarly, an allocation counter can help explain a timing difference without making allocation the only possible cause. Change one condition when practical, retain the output checks, and report the distribution of observations. Explain which alternatives the evidence rules out and which remain possible rather than naming the first plausible mechanism as proven.
Match the report to its evidence. For the folding exercise, report the source, fixed bindings, expected wrapped value, actual original and folded values, and exact instruction streams. If these disagree, name the earliest inspected boundary that violates the contract and include the reduced case. If they agree but one stream is shorter, report the count change without claiming timing. A native investigation requires its own build and execution evidence. Another reader can then reproduce the claim without guessing whether 'faster' meant fewer teaching instructions, shorter compilation, or measured application latency.
Defined startup interval including the intended preparation
Warm execution alone
Untraced execution is faster
Measurements without diagnostic trace overhead
Timing formatted stack output
This is an evidence checklist, not benchmark data. Compilation, startup, steady-state execution, and tracing are different measurement conditions. No timing, speedup, or production-compiler verdict is supplied by these static comparisons.
Exercises
An optimization changes a result
A native program changes output when optimization is enabled. What evidence is needed before concluding that the optimizer is wrong?
Hint
Establish both the program’s required behavior and the exact build conditions.
Worked answer
Step
Purpose
Minimal reproducible case
Make the behavior inspectable
Defined-semantics check
Establish the required result
Representation comparison
Locate a possible incorrect transformation
Investigate alternatives without assuming that compilers are infallible.
Preserve a minimal reproducer, inputs, target, compiler version, flags, libraries, and outputs. Check that the relevant execution is defined under the language and ABI rules. Then compare intermediate or generated code and, where useful, other implementations. Optimization-dependent behavior can reveal a compiler defect, but the symptom alone is not proof.
A smaller instruction list
The folding lab reduces five bytecode instructions to one. Can you claim a fivefold speedup for a native application?
Hint
The counter measures this abstract instruction stream, not elapsed CPU time.
Worked answer
Claim
Supported?
Five to one lab instructions
Yes, if those are the displayed counts
Fivefold native speedup
No timing evidence
Same returned value in this model
Check evaluator and VM equivalence
An operation count is evidence about the model that counted it.
No. It shows a reduction from five to one instructions in the defined teaching VM. Native instruction selection, dispatch cost, code size, register pressure, caches, and other work can change timing. A performance claim needs measurements of equivalent workloads under stated conditions.
Sources and further reading
The explanations, examples, and worked answers are complete without these optional links. The sources describe specific implementations and language contracts, not one universal compiler pipeline. SmallExpr and the later pseudocode models are constructed teaching examples. Source notes identify the selected material reviewed; this is not an exhaustive compiler survey, production conformance test, or independent specialist review.
Selected material on IR structure, SSA, phi semantics, integer operations, and semantic constraints. A production IR reference, not a specification of SmallExpr or a claim that every compiler uses LLVM.
Selected overview and register-allocation material describing instruction selection, target constraints, liveness, and native-code generation. The storage lab is not an implementation of LLVM’s allocator. The reviewed register-allocation example has inconsistent def/use prose; the lesson follows explicit assignment direction instead.