A lexer recognizes spelling boundaries and records positions without deciding the full syntax.
By the end: Read token spans, explain longest-match rules, and distinguish lexical errors from later failures.
Keep text and positions together
A token records a category, spelling, and location. In x + 12, the lexer can identify a name, an operator, and a number before it knows whether x is declared. Our locations are zero-based character offsets with an exclusive end: a token at [4,6) covers characters four and five.
Whitespace separates tokens here and is otherwise ignored. Comments and string escapes need their own lexical rules in languages that support them; SmallExpr does not. Keeping locations lets later stages report a useful error even after raw punctuation and spacing have been discarded.
Recover exact spellings from half-open spans
A lexer keeps a cursor into the source and marks the beginning of the next token. It advances through characters that belong to that spelling, then records the slice between the start and the new cursor. This interval is half open because it includes its start but excludes its end. The difference between the two offsets gives the spelling's length. Later stages can retain these offsets even when they no longer need to inspect individual characters, so an error can still refer to the original source.
For x + 12, number the character positions zero through five. The name occupies position zero, whitespace occupies one, and the plus occupies two. Another space occupies three, followed by digits at four and five. The recorded spans are [0,1), [2,3), and [4,6). In particular, six is the boundary after the last digit, not a seventh character included in the number. Taking the source slice at the number's two boundaries recovers exactly 12, with neither neighboring whitespace nor a missing digit.
A span ends at a boundary, not a final character
Source x + 12. Each equal-width cell is one UTF-16 code unit, here also one ASCII character. Dots represent spaces, not literal dots in the source. The empty end token is at boundary 6.
The scanner also appends an end token with an empty spelling. For this six-character input, its start and end are both six. It marks a boundary at which a parser can ask whether the complete expression ended. It is not an extra source character and does not occupy a cell after the final digit. The token view omits this internal marker from the chips but reports its offset separately. That distinction helps explain why the displayed token list and the internal list differ by one entry.
Now insert a second space before the number, producing x + 12. The number moves to [5,7), but its category and spelling remain unchanged. The end marker moves to seven. A parser can construct the same expression tree from these tokens because whitespace has not changed their categories or order. A diagnostic using the old number span would now point partly at whitespace and omit the last digit. Preserving meaning across an edit does not mean that old location metadata remains valid.
Whitespace skipping still has to advance the cursor. Otherwise the scanner would repeatedly inspect the same space without producing a token or reaching the end. A successful scan therefore accounts for every source position either as part of a token or as ignored whitespace. Unsupported characters are neither category and must cause a lexical failure here. This accounting is an implementation invariant that you can check independently of whether the later parser accepts the resulting stream as a complete expression.
Whitespace moves locations without adding operands
Source, with · standing for space
Number span
End boundary
x·+·12
[4,6)
6
x·+··12
[5,7)
7
x·+··12·
[5,7)
8
Exact half-open spans in string units. All three sources have the same name, symbol, and number token spellings; the end marker follows all trailing whitespace.
Try the two spacing variants in the token-only editor. Before reading each chip, mark the start and end boundaries of the number yourself. Then place a space after the final digit. Its span should stay unchanged, while the reported end offset increases by one. This separates the end of the last token from the end of the entire source. Use those two boundaries to explain why trailing whitespace changes a location report without introducing an additional arithmetic operand or altering the token spelling.
Choose a complete spelling, not the first prefix
A lexer usually follows a language-specific longest-match rule. If a language has both > and >=, it must decide whether the next character extends the operator. In our lexer, truex is one name, not boolean true followed by name x. The full identifier is recognized before keywords are classified.
The same spelling can be accepted lexically but rejected later. absent + 1 has valid tokens but an unbound name. flag + 1 has valid tokens and a declared name but violates the arithmetic type rule. 2 + * 3 has valid individual symbols but no expression in this grammar.
Finish a token before interpreting its role
At the beginning of a name, this scanner accepts an ASCII letter or underscore. Once scanning that name, it also accepts ASCII digits. The rule is different at the first position because a digit starts a number instead. For x12, all three characters therefore belong to one name token. For 12x, the scanner first finishes a number at the letter and then starts a name. It does not infer multiplication between them. Later, the parser rejects the adjacent operands as an incomplete expression structure.
The spelling true demonstrates a second decision after the name scan. The scanner first consumes the complete run of allowed name characters. Only then does it compare that spelling with true and false, the two boolean literals. Exactly true becomes a boolean token; truex becomes a name token. If it emitted true as soon as it recognized that prefix, it would split a longer valid name incorrectly. Consuming the complete spelling is necessary even when the shorter prefix has its own reserved meaning.
Consume the whole name-shaped spelling first
Source-span comparisons, not execution traces. Each letter cell represents one ASCII source unit. Keyword classification follows the completed spelling; a prefix alone does not end the token.
Case matters in this comparison. True starts with an allowed ASCII letter, but it does not match the lowercase boolean spelling. It becomes a name. Similarly, flag is a name token even though the fixed environment later gives that name a boolean value. The lexical category describes the source form, not the type of every value that form might denote. Name lookup supplies flag's meaning in a later responsibility, so a lexer need not consult the environment to recognize that spelling.
Longest-match rules operate within the available token vocabulary. SmallExpr recognizes each ASCII minus separately because its vocabulary has no decrement operator. Thus x--1 tokenizes as x, -, -, 1, and the parser can read it as x minus negative one. With x equal to four, that expression returns five. A different language with a -- token could divide the same characters differently. Its rules would need their own analysis; the SmallExpr result does not establish a universal treatment of repeated punctuation.
Unary minus also explains why -12 is not one negative number token here. The scanner records a minus symbol followed by the number 12. The parser later decides whether minus has one operand as negation or two as subtraction. For 8-3 it is a binary operation; at the beginning of -3 it is unary. Sharing a lexical symbol does not require the parser to build the same kind of node at every occurrence, because grammatical context determines the operator's role.
Token boundaries do not decide unary or binary minus
Source
Token spellings
Later interpretation
-12
-, 12
Unary negation: −12
8-3
8, -, 3
Binary subtraction: 5
x--1
x, -, -, 1
Subtract negative one: 5
12x
12, x
Parsing rejects leftover name
SmallExpr has no decrement token and no implicit multiplication. Each minus occupies one ASCII source unit, whether parsing later makes it unary or binary.
Compare true, truex, True, x12, and 12x in the token-only view. Predict categories before entering each source and check the exact spelling boundaries afterward. The view performs lexing only, so its acceptance of truex or 12x says nothing about whether either can execute. On paper, continue the analysis by asking whether the stream forms one expression and whether each name has a binding. This makes the division between recognizing characters and interpreting a complete expression visible in the examples themselves.
Work through the mechanism
Inspect token boundaries
Presets replace the whole source. Editing replaces the current calculation and resets any execution trace. Names: x = 4, y = 7, flag = true. Use ASCII - for subtraction and negation. Arithmetic is signed 32-bit wrapping; booleans are not numbers. See lesson 1 for the full grammar and editor bounds.
Token stream
2number[0,1)
+symbol[2,3)
3number[4,5)
*symbol[6,7)
4number[8,9)
The end marker is at offset 9. Spans exclude whitespace; token categories do not prove the expression is valid.
This view performs lexing only. Switch lessons to inspect later responsibilities. The entered source is never passed to JavaScript eval or Function.
Token in x + 12
Category
Span
x
name
[0,1)
+
symbol
[2,3)
12
number
[4,6)
Spans include the token spelling and exclude surrounding whitespace.
Use token evidence to classify a failure
The token lab intentionally stops before parsing. That makes it possible to inspect streams that a complete compilation would reject. Enter 2 + * 3 and predict four ordinary tokens. Each spelling is supported, so lexing succeeds. The problem appears when a parser expects an operand after plus and encounters multiplication instead. The token stream has no malformed character. Treating every rejection as a lexical error would misidentify both the evidence already obtained and the rule that actually needs correction.
Contrast 2 @ 3. After scanning the first number and skipping a space, the lexer reaches @ at offset two. No rule accepts that character, so it throws a lexical error rather than silently discarding it. The token-only interface does not display the partial token array from a failed call. Absence of chips in this case therefore does not mean that the scanner inspected no earlier characters. It means the operation failed before returning a completed stream to the view.
Use absent + 1 to examine a later failure. The name's six letters fit the editor bound, so the lexer can return a name, a plus, and a number. Those tokens also form an addition expression. A complete compilation then fails because absent has no declaration in the fixed environment. Replace it with flag and tokenization still succeeds, but the complete compilation fails for a different reason. Lookup finds a boolean binding, and the arithmetic rule refuses it rather than coercing it into an integer.
Valid tokens are evidence, not complete acceptance
Source
Token-only view
Complete compilation
2 + * 3
Four tokens
Parsing failure at *
2 @ 3
Lexical failure at offset 2
No completed token stream
absent + 1
Three tokens
Binding failure at absent
flag + 1
Three tokens
Typing failure at flag
First failures under this implementation. Failed lexing throws before returning its partial array, so the interface shows no partial chips.
An error offset identifies where a check reports trouble, not necessarily where the author should edit. For a missing right operand, x + reaches the end marker after the plus. The parser needs another expression component at that boundary. For an unsupported character, the lexer points directly at that character. For an unbound name, binding reports the name's start. These locations are useful because the original token spans remain available, but diagnosing the intended repair still requires understanding the violated rule.
A further contrast is 2 3. Both number tokens are valid, and neither is a misspelled operator. The parser can finish the first literal as an expression, but it must reject the leftover second literal. A parser that returned the first tree without checking for remaining tokens would make the second number disappear from the program. The end marker supports this completeness check. Tokenization's job is to expose the whole stream; parsing's job includes accounting for the complete stream it receives.
After predicting these outcomes, inspect each input in the token view and compare with the complete expression view in the preceding lesson. Keep a short record of the last successful responsibility and the first failing one. For x +, do not expect the lexical view to invent a missing operand token. For 2 @ 3, do not expect a parser tree. This exploration tests the limits of what each display establishes, rather than rewarding a guess based only on whether a red error message appears.
An empty end marker makes completeness explicit
Source
Internal stream, including end
Parser finding
x +
x [0,1); + [2,3); end [3,3)
An operand is missing at boundary 3
2 3
2 [0,1); 3 [2,3); end [3,3)
The second number remains after the first expression
x + 1
x [0,1); + [2,3); 1 [4,5); end [5,5)
The whole stream forms one expression
Offsets count UTF-16 string units, identical to ASCII character counts for these inputs. The end token consumes no source text.
Do not confuse a teaching bound with a language error
A lexical error means the character sequence cannot be tokenized under the specified language. An @ character is unsupported here. A six-character name limit and a 120-character input limit instead protect this teaching implementation and its diagram. They are reported as limit errors rather than evidence that real compilers cannot handle longer programs.
Real languages must also define encodings, identifier characters, source normalization, and line conventions. Our case-sensitive identifiers start with an ASCII letter or underscore, followed by ASCII letters, digits, or underscores. Offsets use JavaScript string indices. The tokenizer is not a Unicode identifier specification or a universal programming-language lexer.
Separate supported text from display limits
The scanner's character rules and its resource limits are distinct. A name with seven ASCII letters follows the same lexical pattern as a shorter name, but this implementation reports a limit error because names may contain at most six characters. It does not split the long name into a supported prefix and another token. Such splitting would alter the source's token boundaries. A limit failure tells you that this teaching implementation declines the input, not that seven-character identifiers are inherently impossible in programming languages.
The number rule distinguishes value from spelling length. A decimal run representing more than 999 triggers a literal limit error. Thus 1000 fails even though its four characters fit the source limit, while 00012 is accepted as a single number token with spelling length five. Parsing converts that spelling to the integer twelve. The token span still covers all five digits. A diagnostic or source tool needs the original spelling and extent even when evaluation uses a shorter numerical representation of its value.
Keep source spelling even when its value is simpler
Source
Lexical record
Later numeric meaning
12
number, [0,2), spelling 12
Integer 12
00012
number, [0,5), spelling 00012
Integer 12
1000
Literal limit failure at offset 0
No accepted numeric token stream
abcdefg
Name limit failure at offset 0
Not two shorter names
Exact SmallExpr behavior. The 0–999 rule bounds literal values; the six-character rule bounds name spellings. Neither rule defines a general compiler limitation.
The 120-character input bound and 31-token bound control different quantities. Whitespace contributes to source length without adding ordinary tokens. Parentheses contribute characters and tokens without adding value leaves to the eventual syntax tree. A short series of nested parentheses can therefore reach a parser nesting bound without approaching the source limit. Reading the reported failure category matters. One bound cannot serve as a reliable proxy for all the others, because each measures a different property of the representation being processed.
For a concrete boundary, fifteen opening parentheses, the literal 1, and fifteen closing parentheses form 31 source tokens. Lexing can return that stream, but parsing exceeds its separate nesting limit. Adding one more pair makes 33 source tokens, so the token bound fails first. These are constructed inputs for reasoning about the limits, not suggested expressions to fit into the normal tree display. They show why a failure can move between responsibilities even when the edit only adds punctuation around an unchanged literal.
This lexer uses ASCII rules for identifiers and decimal digits, but its whitespace test is the host regular expression's whitespace class. Do not infer that every non-ASCII character is necessarily rejected at the same point. A letter such as é cannot begin a supported name, whereas some non-ASCII spacing characters are skipped. Offsets are JavaScript string indices, measured in UTF-16 code units rather than screen columns or encoded file bytes. For the ASCII examples above, one character occupies one unit, which makes the ruler straightforward.
Different bounds count different representations
Constructed source
Lexical outcome
Parser outcome if reached
15 opening parentheses, 1, 15 closing
31 tokens, accepted
Nesting limit failure
16 opening parentheses, 1, 16 closing
Token limit failure on token 32
Not reached
x followed by 119 spaces
One token, source length 120
Valid name expression
Seven-letter name abcdefg
Name limit failure
Not reached
Ordinary token counts exclude the appended end marker. Source length counts string units. These are display and teaching bounds, not language-independent limits.
Production source tools must define their own encoding and position conventions. A tab can advance a displayed column by several spaces while occupying one string unit. Some visible characters require more than one UTF-16 unit, and combining marks can complicate the relationship further. The present diagrams deliberately avoid those cases. When transferring the span technique to another implementation, first identify its unit, its allowed character classes, and whether it preserves whitespace. Copy the boundary reasoning, not an unsupported assumption that all source coordinates are interchangeable.
Exercises
A keyword prefix
How does the SmallExpr lexer classify truex? Does tokenizing it prove that the program can run?
Hint
Keywords are checked after consuming the full name.
Worked answer
Stage
Outcome
Lexing
One name token: truex
Binding
No declaration
Execution
Not reached
Do not split a longer identifier at a keyword prefix.
It produces one name token, truex. This spelling is within the six-character bound. Binding then rejects it because the environment has no truex declaration. Tokenization does not establish that a name exists or that later operations are well typed.
Classify the failure
Compare 2 @ 3, 2 + * 3, and flag + 1. At which responsibility does each first fail?
Hint
Consider characters, expression structure, and operand types separately.
Worked answer
Input
First failure
2 @ 3
Lexing
2 + * 3
Parsing
flag + 1
Typing
Stage labels describe this implementation’s checks, not every compiler’s diagnostic order.
The @ character fails lexing. The second input tokenizes but fails parsing because * cannot start an operand. The third parses and resolves flag but fails typing because a boolean cannot be added to an integer in SmallExpr.
Sources and further reading
The explanations, examples, and worked answers are complete without these optional links. The sources describe specific implementations and language contracts, not one universal compiler pipeline. SmallExpr and the later pseudocode models are constructed teaching examples. Source notes identify the selected material reviewed; this is not an exhaustive compiler survey, production conformance test, or independent specialist review.
Selected passages on tokenization, identifier recognition, and maximal munch. The book implements Lox; SmallExpr is a separate, smaller teaching language.