Back to series
· 7 min read

Neurons, Weights, Biases, and Activation Functions: Understanding the Fundamental Computational Unit of Neural Networks


I. Introduction

A neural network is composed of a large number of interconnected computational units. Although modern deep learning models may contain billions or even more parameters, their fundamental computation is still built on a few core mechanisms:

  • Neuron: the basic unit that receives input and performs computation.
  • Weight: controls how different inputs affect the computed result.
  • Bias: adjusts the offset of the neuron's linear transformation.
  • Activation Function: introduces nonlinear expressive capability.

Together, these four concepts form the foundation of artificial neural networks.

Understanding them not only helps in understanding the Multilayer Perceptron (MLP), but is also a necessary foundation for further study of CNNs, RNNs, Transformers, and large language models.


II. Artificial Neuron: The Basic Computational Unit of a Neural Network

2.1 What Is an Artificial Neuron?

An artificial neuron is a mathematical computation model inspired by biological neurons.

It receives one or more inputs, performs a weighted sum of the inputs, and then passes the result through an activation function to generate an output.

An artificial neuron can be abstracted into two stages:

  1. Linear computation: perform a weighted sum of the inputs and add the bias.
  2. Nonlinear transformation: generate the final output through an activation function.

Mathematical expression:

z=\sum_{i=1}^{n}w_ix_i+b
a=f(z)

Where:

Symbol Meaning
x_i the i-th input
w_i the weight corresponding to the i-th input
b bias parameter
z linear computation result
f activation function
a neuron output
n number of input features

It can also be written as:

a=f(W^Tx+b)

Here it is assumed that the input vector x and the weight vector W are both represented as column vectors.

2.2 The Computation Process of a Neuron

Suppose a neuron has three inputs:

x_1=2,\quad x_2=3,\quad x_3=1

Corresponding weights:

w_1=0.5,\quad w_2=0.2,\quad w_3=0.8

Bias:

b=0.1

First perform the weighted sum:

z=0.5\times2+0.2\times3+0.8\times1+0.1
z=2.3

If the ReLU activation function is used:

ReLU(z)=\max(0,z)

Then:

a=ReLU(2.3)=2.3

This is the process by which an artificial neuron completes one computation.

2.3 Neuron Computation Structure

flowchart TB
    A["Input feature x1"] --> D["Weighted sum"]
    B["Input feature x2"] --> D
    C["Input feature xn"] --> D
    W["Weight parameters W"] --> D
    P["Bias parameter b"] --> D
    D --> E["Linear result z"]
    E --> F["Activation function f"]
    F --> G["Neuron output a"]

    style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style W fill:#EAF3EC,stroke:#83A995,color:#315D49
    style P fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49

It should be noted that an artificial neuron is not a complete simulation of a biological neuron, but rather a mathematical abstraction of part of the idea of information processing.


III. Weights: The Core Parameter That Determines How Much Each Input Matters

3.1 What Is a Weight?

Weight is one of the most important learnable parameters in a neural network.

Each input typically corresponds to one weight, which controls that input's contribution to the weighted sum.

The linear computation of a neuron:

z=w_1x_1+w_2x_2+\cdots+w_nx_n+b

Here, the weight determines how a change in the input affects the result of the linear computation.

For a given input x_i, we have:

\frac{\partial z}{\partial x_i}=w_i

This means that, with the other inputs held constant, the weight determines the magnitude of the local effect that a change in this input has on z.

3.2 The Sign and Magnitude of Weights

A weight can be positive, negative, or zero.

Positive weight:

When the input increases, the result of the linear computation usually increases.

w_i>0

Negative weight:

When the input increases, the result of the linear computation usually decreases.

w_i<0

Zero weight:

This input makes no direct contribution to the current neuron's linear computation.

w_i=0

For example:

z=2x_1-3x_2+0.5x_3+b

We can see that:

  • x_1 has a positive effect on the result.
  • x_2 has a negative effect on the result.
  • x_3 has a relatively small effect.

What is described here is a local effect within the linear computation; it does not mean that a causal relationship in the same direction necessarily exists between the input feature and the final prediction.

3.3 Why Do Weights Need to Be Trained?

When a neural network first begins training, its weights are usually set through random initialization or a specific initialization strategy.

At this point, the model has not yet learned the mapping required by the task.

During training, the model computes the weight gradients from the loss function, and the optimizer then adjusts the weights.

The basic update formula:

w_{t+1}=w_t-\eta\frac{\partial L}{\partial w_t}

where:

  • w_t: the current weight.
  • \eta: the learning rate.
  • L: the loss function.
  • \frac{\partial L}{\partial w_t}: the gradient of the loss with respect to the weight.

When there is a large discrepancy between the model's prediction and the target, the loss function produces a corresponding gradient, and the optimizer uses that gradient to adjust the weights.

After many training iterations, the weights gradually settle into a combination of values suited to the task at hand.

3.4 Weight Initialization

Weight initialization affects the distribution of values and the propagation of gradients in the early stages of neural network training.

Common initialization methods:

Initialization Method Core Idea Common Use Case
Zero Initialization Initialize parameters to zero Not suitable for ordinary hidden-layer weights
Random Initialization Generate initial parameters randomly Must be paired with a sensible distribution
Xavier Initialization Control variance based on input and output dimensions Sigmoid, Tanh, and other activation functions
He Initialization Control variance based on the input dimension ReLU and its variants

Pay special attention to this:

Neurons in a hidden layer must not simply be initialized with identical weights.

If all neurons in the same layer have exactly the same initial parameters and the same structure and computation, they will produce the same outputs and gradients and cannot develop distinct feature representations through training.

This is called the symmetry problem.

One of the key roles of random initialization is to break this symmetry.


IV. Bias: Adjusting the Neuron's Linear Transformation

4.1 What Is Bias?

Bias is a learnable parameter in a neural network that adjusts the offset of the linear computation's result.

The neuron's computation formula:

z=Wx+b

Here, b is the bias.

Without bias:

z=Wx

The linear transformation must pass through the origin.

With bias added:

z=Wx+b

The model can learn a more flexible linear transformation.

4.2 The Geometric Meaning of Bias

For a neuron with only one input:

z=wx+b

This is a linear function.

where:

  • w determines the slope of the line.
  • b determines the line's intercept on the vertical axis.

For example:

z=2x+1

and:

z=2x-3

The two lines have the same slope, but because their biases differ, their positions are shifted.

For a two-dimensional input:

z=w_1x_1+w_2x_2+b

When z=0, a linear decision boundary is obtained.

The bias can shift the position of that boundary.

4.3 What goes wrong without a bias?

Suppose the model needs to learn:

y=2x+5

If the model only has:

y=wx

then no matter how w is adjusted, this function cannot be represented accurately.

Because the model can only learn straight lines that pass through the origin.

If a bias is added:

y=wx+b

it can learn:

w=2,\quad b=5

Therefore, the bias increases the model's flexibility in expressing relationships in the data.

4.4 How bias differs from weight

Aspect Weight Bias
Main role Adjusts the contribution of the input Adjusts the offset of the linear transformation
Depends on input Multiplied with the input Added independently
Count Usually one per input connection Usually one per output neuron
Learnable Yes Yes
Effect Changes the slope, direction, and feature combinations Changes the intercept and the position of the decision boundary

Weights and biases together determine the linear transformation of a neuron.


V. Activation Functions: Giving Neural Networks Nonlinear Expressive Power

5.1 What is an activation function?

An activation function is a function applied to the result of a neuron's linear computation.

Its basic form is:

a=f(z)

where:

z=Wx+b

The activation function converts the result of the linear computation into a new output representation.

Its most important role is:

to introduce nonlinearity into the neural network, enabling the model to learn complex relationships between inputs and outputs.

5.2 Why can't we use only linear computations?

Suppose a neural network contains three layers of linear computation:

h_1=W_1x+b_1
h_2=W_2h_1+b_2
y=W_3h_2+b_3

Substituting the earlier expressions:

y=W_3[W_2(W_1x+b_1)+b_2]+b_3

After rearranging, it can still be written as:

y=Wx+b

In other words, no matter how many layers of linear transformations are stacked, the result is still a single linear transformation.

Although the network has more layers, it has not gained stronger nonlinear expressive power.

Therefore, activation functions need to be introduced between the computation layers:

h_1=f(W_1x+b_1)
h_2=f(W_2h_1+b_2)
y=f(W_3h_2+b_3)

Only in this way can the network build complex nonlinear mappings.

5.3 Main types of activation functions

1. Sigmoid

Formula:

\sigma(x)=\frac{1}{1+e^{-x}}

Output range:

(0,1)

Characteristics:

  • The output range is between 0 and 1.
  • The function is continuous and differentiable.
  • The output can be used to represent binary classification probabilities.
  • Gradient saturation tends to occur when the absolute value of the input is large.
  • It is not zero-centered.

Derivative:

\sigma'(x)=\sigma(x)(1-\sigma(x))

When the input is very large or very small, the derivative approaches zero, which easily causes vanishing gradients.

Common uses:

  • Output layer for binary classification.
  • Independent output for each label in multi-label classification.

2. Tanh

Formula:

\tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}

Output range:

(-1,1)

Characteristics:

  • The output is zero-centered.
  • It can express both positive and negative values.
  • Compared with Sigmoid, the zero-centered property can sometimes benefit optimization.
  • Gradient saturation still tends to occur at both ends.

Common uses:

  • Traditional recurrent neural networks.
  • Certain computation modules that require bounded positive and negative outputs.

3. ReLU

Formula:

ReLU(x)=\max(0,x)

Piecewise form:

ReLU(x)= \begin{cases} 0,&x\leq0\\ x,&x>0 \end{cases}

Characteristics:

  • Simple to compute.
  • The gradient is 1 in the positive region.
  • The output is 0 in the negative region.
  • It performs well in many deep neural networks.
  • The dying ReLU problem may occur.

Dying ReLU refers to a neuron remaining in the negative region for a long time, with its gradient staying at zero, so that its parameters are difficult to recover through gradient descent.

4. Leaky ReLU

Formula:

f(x)= \begin{cases} x,&x>0\\ \alpha x,&x\leq0 \end{cases}

where \alpha is a small positive number.

Characteristics:

  • A small gradient is retained in the negative region.
  • It can alleviate the dead ReLU problem to some extent.
  • It still maintains relatively low computational complexity.

5. GELU

GELU (Gaussian Error Linear Unit):

GELU(x)=x\Phi(x)

where \Phi(x) is the cumulative distribution function of the standard normal distribution.

Characteristics:

  • Smooth nonlinear transformation.
  • Does not use the hard-threshold cutoff of ReLU.
  • Widely used in modern Transformer architectures.

6. SiLU

SiLU (Sigmoid Linear Unit):

SiLU(x)=x\sigma(x)

Characteristics:

  • Smooth and non-monotonic.
  • Combines the input value with Sigmoid gating.
  • Commonly used in modern deep learning networks.

For example, some modern convolutional networks and Transformer variants use SiLU or related forms.

5.4 Comparison of Activation Functions

Activation Function Output Range Main Advantages Main Problems
Sigmoid (0, 1) Suitable for probability-form output Gradient saturation
Tanh (-1, 1) Zero-centered Gradient saturation
ReLU [0, +∞) Simple computation, high training efficiency Dead ReLU
Leaky ReLU (-∞, +∞) Retains gradient in the negative region The slope in the negative region must be set
GELU (-∞, +∞) Smooth nonlinearity Relatively complex computation
SiLU (-∞, +∞) Smooth, non-monotonic Relatively complex computation

5.5 Principles for Choosing Activation Functions

Different network architectures and tasks may use different activation functions.

Common choices:

  • Hidden layers of standard MLPs: ReLU, GELU, SiLU.
  • Hidden layers of CNNs: ReLU, SiLU, etc.
  • Traditional RNNs: Tanh, Sigmoid gating.
  • Transformers: GELU, ReLU, SwiGLU, etc.
  • Binary classification output layer: Sigmoid.
  • Mutually exclusive multi-class output layer: Softmax.
  • Regression output layer: usually uses a linear output.

Note that Softmax is usually not an ordinary hidden-layer activation function; rather, it is an output operation that converts a set of logits into a normalized probability distribution.


VI. How Do the Four Core Concepts Work Together?

Neurons, weights, biases, and activation functions are not independent techniques; together they form a complete computational unit.

The computation process is as follows:

  1. Receive input features.
  2. Multiply the inputs by the weights.
  3. Sum all weighted results.
  4. Add the bias.
  5. Pass the computed result into the activation function.
  6. Output a new feature representation.

Overall expression:

a=f(W^Tx+b)

6.1 Complete Computation Process

flowchart TB
    A["Input features x"] --> B["Multiply by weights W"]
    B --> C["Weighted sum"]
    D["Bias b"] --> E["Add bias"]
    C --> E
    E --> F["Obtain linear result z"]
    F --> G["Apply activation function f"]
    G --> H["Output feature a"]

    style A fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#E5F0E9,stroke:#83A995,color:#315D49