What Is a Tokenizer?
A tokenizer scans raw source code character by character and groups the characters into meaningful units called tokens, such as keywords, operators, and literals. It discards whitespace and comments while attaching each token a type and position. The token stream it produces is the input the parser consumes to build a syntax tree.
Why it matters
Tokenizing first turns an unstructured stream of characters into discrete units the parser can reason about. Clear token boundaries make grammar rules and error messages far simpler to express.
Related guides
What Is Register Allocation?Register allocation is the compiler step that decides which program values are kept in fast CPU registers and…
What Is Template Instantiation?Template instantiation is the compiler generating concrete code from a generic template for each specific set…
What Is a Basic Block?A basic block is a straight-line sequence of instructions with one entry and one exit, used as the unit of an…