Back to series
· 9 min read

Grammar (Syntax)


I. What Is Syntax?

Syntax is the set of code organization rules defined by a programming language. It defines how various language constructs in a program should be written and how these constructs combine.

Simply put, syntax answers this question:

By what rules must a piece of code be organized to be a valid program expression?

For example, in Python:

x = 10
print(x)

This code conforms to Python's syntax rules.

The following code, however:

if x > 10
    print(x)

is missing a colon and does not meet Python's syntax requirements.

Syntax mainly specifies:

  • How keywords are used.
  • How identifiers are named.
  • How operators are combined.
  • How expressions are constructed.
  • How statements are organized.
  • How code blocks are divided.
  • How functions, classes, and modules are defined.

Different programming languages have different syntax rules.

For example:

# Python
if x > 10:
    print("大于 10")
// JavaScript
if (x > 10) {
    console.log("大于 10");
}

The two code snippets express the same logic, but their syntactic structures are different.

II. The Difference Between Syntax and Lexical Analysis

Before understanding syntax, we need to distinguish lexical analysis (Lexical) from syntax (Syntax).

They are two different stages in how the compiler front end processes source code.

1. Lexical Analysis: Identifying Basic Units in Code

Lexical analysis is responsible for splitting source code into meaningful basic units; these units are called tokens (Token).

For example:

total = price + 100

After lexical analysis, we get:

Source Code Fragment Token Type
total Identifier
= Assignment Operator
price Identifier
+ Addition Operator
100 Integer Literal

Lexical analysis mainly focuses on:

  • Whether characters are valid.
  • How identifiers are recognized.
  • How keywords are recognized.
  • How numbers and strings are recognized.
  • How operators are recognized.
  • How whitespace and comments are handled.

2. Syntax: Identifying Structural Relationships Between Tokens

Syntax analysis receives the token sequence produced by lexical analysis and, according to the language's syntax rules, determines whether these tokens can form a valid program structure.

For example:

total = price + 100

The syntax analyzer identifies the following:

  • price + 100 is an addition expression.
  • total = ... is an assignment statement.
  • The entire statement conforms to the syntactic structure of an assignment statement.

3. The Relationship Between Lexical Analysis and Syntax Analysis

flowchart TB
    A["Source Code"] --> B["Lexical Analysis"]
    B --> C["Token Sequence"]
    C --> D["Syntax Analysis"]
    D --> E["Syntax Tree"]
    E --> F["Subsequent Semantic Analysis"]

    style A fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style B fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style D fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style E fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style F fill:#FFF5DE,stroke:#E8D29A,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

The core differences between the two are:

Comparison Dimension Lexical Analysis Syntax Analysis
English Lexical Analysis Syntax Analysis
Input Character Sequence Token Sequence
Core Task Identify Basic Units Identify Structural Relationships
Output Token Syntax Tree or Abstract Syntax Tree
Typical Errors Illegal Characters, Illegal Numbers Missing Parentheses, Malformed Expression Structure

For example:

x = 10 + * 5

Here, 10, +, *, and 5 are all valid lexical tokens, but their combination does not conform to the syntax rules of Python expressions.

This is a syntax error, not a lexical error.

III. Basic Components of Syntax

The syntax of a programming language consists of a set of rules that define the legal forms of different code structures.

Common components include the following.

1. Keyword

Keywords are words predefined by the language that have special syntactic meaning.

For example:

Python:

if
else
for
while
def
class
return
import

Java:

public
private
class
static
void
if
else
return

Keywords generally cannot be used directly as ordinary identifiers.

For example:

class = 10

In Python, class is a keyword and cannot be used as an ordinary variable name.

2. Identifier

Identifiers are used to name various elements in a program, such as:

  • Variables.
  • Functions.
  • Classes.
  • Modules.
  • Parameters.
  • Attributes.

For example:

user_name = "Alice"

def calculate_total():
    return 100

Here, user_name and calculate_total are both identifiers.

Different languages have different naming rules for identifiers.

Python identifiers generally follow these rules:

  • They may contain letters, digits, and underscores.
  • They cannot begin with a digit.
  • They are case-sensitive.
  • They cannot use keywords.
user_name = "Alice"  # valid
_user = "Bob"        # valid
user2 = "Tom"        # valid

2user = "Jack"       # invalid

3. Literal

A literal is a fixed value written directly in the source code.

Common types include:

Type Example
Integer 100
Floating-point number 3.14
String "Hello"
Boolean True, False
Null value None
Collection literal [1, 2, 3]

For example:

age = 18
price = 99.5
name = "Alice"
enabled = True

Literals are different from variables.

A literal directly represents a value, while a variable references or binds data through an identifier.

4. Operator

Operators are used to form expressions and describe calculations, comparisons, and logical operations.

Common operators:

Category Operator Purpose
Arithmetic operators + - * / % Numeric calculation
Comparison operators == != > < >= <= Compare magnitude or equality
Logical operators and or not Logical evaluation
Assignment operators = += -= *= Assignment and update
Bitwise operators & \| ^ ~ << >> Bit-level calculation

Different languages may have different sets of operators, meanings, and precedence.

For example, Python uses and and or for logical operations, while Java and JavaScript use && and ||.

5. Delimiter

Delimiters are used to separate code structures and expressions.

Common delimiters include:

  • (): parentheses.
  • []: square brackets.
  • {}: curly braces.
  • ,: comma.
  • :: colon.
  • ;: semicolon.
  • .: member access operator.

Different languages use delimiters in different ways.

For example:

# Python
numbers = [1, 2, 3]

if len(numbers) > 0:
    print(numbers[0])
// JavaScript
const numbers = [1, 2, 3];

if (numbers.length > 0) {
    console.log(numbers[0]);
}

Python uses indentation to organize code blocks, while JavaScript uses curly braces.

IV. Syntax Rules and Formal Description

The syntax of a programming language cannot rely solely on natural language description; it also requires a formal method that can express rules precisely.

Among these, Context-Free Grammar (CFG) is an important theoretical foundation for describing the syntax of programming languages.

1. What Is a Grammar?

A grammar is a set of rules used to define the structure of a language.

It can usually be represented as:

G = (V, \Sigma, P, S)

Where:

  • V: set of non-terminals.
  • \Sigma: set of terminals.
  • P: set of production rules.
  • S: start symbol.

Let's understand each one:

Terminal

Basic symbols that cannot be further expanded according to grammar rules, such as keywords, operators, identifiers, and literals.

Non-terminal

Syntactic structures that can be further expanded through productions, such as expressions, statements, and programs.

Production Rule

Defines how a non-terminal can be expanded.

Start Symbol

The starting point of grammar derivation, usually representing a program or some complete syntactic structure.

2. A Simple Expression Grammar

For example, define a simple grammar for arithmetic expressions:

Expr   → Expr + Term
Expr   → Expr - Term
Expr   → Term

Term   → Term * Factor
Term   → Term / Factor
Term   → Factor

Factor → ( Expr )
Factor → NUMBER

Here:

  • Expr represents an expression.
  • Term represents a term.
  • Factor represents a factor.
  • NUMBER represents a number.
  • +, -, *, /, and parentheses are terminals.

This grammar splits expressions into different levels, reflecting the structure that multiplication and division take precedence over addition and subtraction.

For example:

1 + 2 * 3

It can be expanded according to the grammar as follows:

Expr
  → Expr + Term
  → Term + Term
  → Factor + Term
  → NUMBER + Term
  → NUMBER + Term * Factor
  → NUMBER + NUMBER * NUMBER

Finally, we get:

1 + 2 * 3

3. Structure of Syntax Derivation

flowchart TB
    A["Start Symbol Expr"] --> B["Expression production"]
    B --> C["Term + Term"]
    C --> D["Factor + Factor * Factor"]
    D --> E["NUMBER + NUMBER * NUMBER"]
    E --> F["1 + 2 * 3"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style C fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style D fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style E fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style F fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

Note that this demonstrates one approach to grammar derivation; it is not the only way it is implemented inside a compiler.

4. BNF and EBNF

When describing the syntax of programming languages, common formal notations include BNF and EBNF.

BNF (Backus-Naur Form)

Backus-Naur Form, used to describe context-free grammars.

For example:

<assignment> ::= <identifier> "=" <expression>

This means an assignment statement consists of an identifier, an equals sign, and an expression.

EBNF (Extended Backus-Naur Form)

Extended Backus-Naur Form, which adds more convenient notation on top of BNF.

Common symbols:

Symbol Meaning
= Definition
\| Multiple alternatives
[ ] Optional part
{ } Repeated part
( ) Grouping

For example:

assignment = identifier, "=", expression ;

Different language specifications may adopt different grammar notations. BNF and EBNF are tools for describing syntax, not programming languages themselves.

5. Parse Trees and Abstract Syntax Trees

A parser must not only determine whether the code is valid, but also recognize the hierarchical structure of the code.

This structure is usually represented by a parse tree.

1. Parse Tree

A parse tree, also called a derivation tree, records the complete structure derived from the source code according to the grammar rules.

For example:

1 + 2 * 3

According to the expression grammar given earlier, the following structure can be formed:

flowchart TB
    A["Expr"] --> B["Expr"]
    A --> C["+"]
    A --> D["Term"]
    B --> E["Term"]
    E --> F["Factor"]
    F --> G["1"]
    D --> H["Term"]
    D --> I["*"]
    D --> J["Factor"]
    H --> K["Factor"]
    K --> L["2"]
    J --> M["3"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style C fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style D fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style E fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style F fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style G fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style H fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style I fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style J fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style K fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style L fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    style M fill:#E8EDF2,stroke:#B8C3CE,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

The parse tree retains the intermediate structures in the grammar, such as Expr, Term, and Factor.

2. Abstract Syntax Tree (AST)

An abstract syntax tree is a structure obtained by abstracting the parse tree.

It omits some syntactic details and keeps only the information that is valuable for program structure and semantic analysis.

For:

1 + 2 * 3

The AST can be represented as:

flowchart TB
    A["Addition node +"] --> B["Number 1"]
    A --> C["Multiplication node *"]
    C --> D["Number 2"]
    C --> E["Number 3"]

    style A fill:#E0F0FE,stroke:#93BCE8,color:#1E4976
    style B fill:#E5F4EC,stroke:#A3CBB5,color:#334155
    style C fill:#F3EDFC,stroke:#C4B2E3,color:#334155
    style D fill:#FFF5DE,stroke:#E8D29A,color:#334155
    style E fill:#FFF5DE,stroke:#E8D29A,color:#334155
    linkStyle default stroke:#A3B8AC,stroke-width:1.5px

The AST clearly shows:

  • The outermost operation is addition.
  • The left operand is 1.
  • The right operand is a multiplication expression.
  • The two operands of the multiplication expression are 2 and 3.

3. Difference Between Parse Tree and AST

Aspect Parse Tree Abstract Syntax Tree
English Parse Tree Abstract Syntax Tree
Structure Fully reflects grammar derivation Retains the core program structure
Intermediate syntax nodes Usually more numerous Usually omitted
Syntax details More details retained Some details omitted
Primary use Syntax analysis and grammar derivation Semantic analysis, optimization, and code generation

The AST is an important data structure in modern compilers and many code analysis tools.

Note that the specific structure of an AST is determined by the language specification, parser implementation, and tool design; there is no single AST structure that is universal across all languages.

6. Operator Precedence and Associativity

When an expression contains multiple operators, their evaluation order must be clearly defined.

Two very important concepts are:

  • Operator precedence.
  • Operator associativity.

1. Operator Precedence

Precedence determines the evaluation order among different operators.

For example:

result = 1 + 2 * 3

Multiplication has higher precedence than addition, so:

1 + 2 \times 3 = 7

rather than:

(1 + 2) \times 3 = 9

Common precedence levels:

Operator Typical Meaning Common Precedence
() Parentheses High
* / % Multiplication, division, and modulo Higher
+ - Addition and subtraction Higher
> < >= <= Comparison Lower
== != Equality comparison Lower
and / && Logical AND Even lower
or / \|\| Logical OR Even lower
= Assignment Usually lower

This is only a common case; precedence tables differ across languages.

For example, Python and JavaScript differ in the precedence and associativity rules for some operators, so a single universal table cannot be applied directly.

2. Operator Associativity

When several operators of the same precedence appear consecutively, associativity determines how the expression is grouped.

For example:

result = 10 - 5 - 2

Subtraction is usually left-associative:

(10 - 5) - 2 = 3

rather than:

10 - (5 - 2) = 7

The assignment operator is right-associative in many languages.

For example:

a = b = 10

This can be understood as first assigning 10 to b, and then assigning the value of b to a.

3. Difference Between Precedence and Associativity

Concept Problem Solved Example
Precedence Which operator is evaluated first when different operators are used 1 + 2 * 3
Associativity How operators with the same precedence are grouped 10 - 5 - 2

Precedence and associativity together determine the syntactic structure of an expression.

During actual development, if an expression has a risk of ambiguity, explicitly grouping it with parentheses is usually the better choice.

VII. Syntax Errors and Semantic Errors

The fact that a program can pass syntax analysis does not mean the program is necessarily correct.

A programming language also needs to check whether the code is semantically sound.

1. Syntax Error

A syntax error means the code does not conform to the grammar rules specified by the language.

For example:

if x > 10
    print(x)

Because the colon is missing, Python cannot parse this code according to the grammar rules.

Another example:

result = 10 + * 5

The operator combination is invalid.

2. Semantic Error

A semantic error means the code structure conforms to the grammar rules, but its meaning does not conform to the language rules or the program's expectations.

For example:

age = "18"
result = age + 10

This code itself conforms to Python syntax, but a string and an integer cannot be directly added in this way, and a type error will occur at runtime.

Another example:

def calculate():
    return unknown_variable

The code is syntactically valid, but if unknown_variable is not defined, a name resolution error will occur at runtime.

3. Differences Between Syntax Errors and Semantic Errors

Comparison Dimension Syntax Error Semantic Error
Core issue The code structure is invalid The code meaning or usage is incorrect
Typical cause Missing parentheses or colons, invalid structure Type mismatch, undefined name
Discovery stage Usually during syntax analysis Semantic analysis or runtime
Whether a syntax tree can be constructed Usually the corresponding structure cannot be fully constructed Usually yes
Example if x > 10 "18" + 10

It is important to distinguish this: semantic errors are not necessarily all discovered during compilation.

For statically typed languages, many type issues can be checked at compile time; for dynamically typed languages, some issues may not be exposed until runtime.

In addition, business logic errors also fall under program errors in a broader sense. For example, if the formula for calculating a discount is written incorrectly, even if the syntax and types are correct, the result may still not meet business expectations.

VIII. How Does a Compiler Perform Syntax Analysis?

Syntax analysis is an important part of the compiler front end.

The compiler receives the tokens produced by lexical analysis, constructs the syntactic structure according to the language grammar, and provides the foundation for subsequent semantic analysis and code generation.

1. Basic Process of Syntax Analysis