Back to series
· 12 min read

Features, Samples, Labels, and Datasets: The Data Foundations of Machine Learning


The core of machine learning is learning patterns from data, and how that data is organized, how it is represented, and how it is fed into a model directly affects both training results and generalization ability.

In machine learning, feature, sample, label, and dataset are the four most fundamental and most important concepts.

Together they form the data organization system of machine learning:

  • Sample: describes one specific observation object.
  • Feature: information describing the attributes of a sample.
  • Label: describes the target result corresponding to a sample.
  • Dataset: a collection of data formed by organizing multiple samples according to a certain structure.

These concepts look simple, but in real engineering they involve a large number of critical issues such as feature engineering, data annotation, data preprocessing, data splitting, data leakage, and model generalization.

Understanding these foundational concepts is an important prerequisite for learning machine learning algorithms and deep learning models.


1. Why Does Machine Learning Need Data?

1.1 Data Is the Foundation of Model Learning

Traditional programs are usually written by developers as explicit rules, and the program executes predefined logic based on its input.

Machine learning is different.

Developers provide the data, the learning objective, and the algorithm, and the model automatically learns parameters and patterns from the data.

For example, developing a house price prediction system.

Traditional programming might require developers to manually define rules:

  • The larger the area, the higher the price.
  • The better the location, the higher the price.
  • The more rooms, the higher the price is likely to be.

But real house prices are affected by many factors together, and it is very difficult to describe them completely with hand-written rules.

Machine learning can use historical housing data to automatically learn the relationship between these factors and house prices.

For example:

Area Rooms Distance to Subway House Age Price
80㎡ 2 500m 5 years 2 million
100㎡ 3 300m 3 years 2.8 million
120㎡ 3 800m 10 years 3.1 million
150㎡ 4 200m 2 years 4.5 million

The model learns the relationship between house prices and each attribute from this historical data.

Once training is complete, it can predict the price of a new house based on its set of attributes.

1.2 The Data Processing Pipeline in Machine Learning

Machine learning typically goes through stages such as data acquisition, data processing, feature construction, model training, and model evaluation.

flowchart TB
    A["Raw Data"] --> B["Data Cleaning"]
    B --> C["Data Preprocessing"]
    C --> D["Feature Construction and Selection"]
    D --> E["Build Dataset"]
    E --> F["Model Training"]
    F --> G["Model Evaluation"]
    G --> H["Model Deployment and Inference"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

These stages are not completely independent. For example, if model evaluation reveals that certain features perform poorly, feature engineering may need to be redone; if abnormal data distributions are found during training, the data may also need to be reprocessed.


2. Sample

2.1 What Is a Sample?

A sample is an independent data instance in a dataset, used to describe a specific observation object or event.

In English it is usually called a Sample, and is also often called an Instance or an Observation.

For example, in a house price prediction task:

One specific house can serve as a sample.

In spam detection tasks:

A single specific email can serve as one sample.

In customer churn prediction tasks:

One user's behavior records within a given observation period can serve as one sample.

Samples are the basic unit of data organization in machine learning.

2.2 What does a sample consist of?

A sample typically contains:

  • Sample identifier: used to distinguish different samples.
  • Features: attributes that describe the sample.
  • Label: the corresponding target outcome in supervised learning.

For example:

Sample ID Area Number of rooms Distance to subway House price
H001 80㎡ 2 500m 2 million
H002 100㎡ 3 300m 2.8 million
H003 120㎡ 3 800m 3.1 million

Here, each row represents one sample.

H001, H002, and H003 are three different housing samples.

Sample IDs are mainly used for data management and tracking, and generally should not be used directly as model features—unless the ID itself carries clear predictive meaning.

2.3 Mathematical representation of a sample

Suppose a sample contains n features; it can be represented as:

x_i=(x_{i1},x_{i2},\ldots,x_{in})

Where:

  • i: the sample index.
  • n: the number of features.
  • x_{ij}: the j-th feature value of the i-th sample.

For example:

x_1=(80,2,500,5)

can represent four features of a house:

  • Area: 80.
  • Number of rooms: 2.
  • Distance to subway: 500.
  • House age: 5.

In supervised learning, a sample usually also includes its corresponding label:

(x_i,y_i)

where y_i is the target value of the i-th sample.

2.4 The relationship between samples and features

A sample is jointly described by multiple features.

For example, an email can be described by the following features:

  • Email length.
  • Whether it contains links.
  • Sender reputation.
  • Number of occurrences of specific keywords.
  • Number of attachments.

A single feature is usually not enough to fully describe an object; multiple features must work together to form the sample representation.

flowchart TB
    A["An email sample"] --> B["Email length"]
    A --> C["Number of links"]
    A --> D["Number of keywords"]
    A --> E["Sender reputation"]
    A --> F["Number of attachments"]

    B --> G["Sample feature vector"]
    C --> G
    D --> G
    E --> G
    F --> G

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#E5F0E9,stroke:#83A995,color:#315D49

2.5 Sample quantity and sample quality

Sample quantity is often an important factor affecting model training results, but it does not mean that more data will necessarily lead to better model performance.

You need to pay attention to both:

Sample quantity

When the amount of data is insufficient, the model may not be able to fully learn the target patterns.

Sample quality

Duplicate data, erroneous data, missing data, and anomalous data can affect model learning.

Sample Representativeness

Training samples need to cover the data distribution in real application scenarios as much as possible.

Sample Independence

Although some data may be large in quantity, it actually comes from the same users, devices, or events, and the samples may be highly correlated.

For example, randomly splitting multiple records from the same user into the training set and test set may cause the model to indirectly memorize user characteristics, resulting in overly optimistic test results.

Therefore, sample quantity cannot be simply equated with the amount of effective information.


III. Feature (Feature)

3.1 What Is a Feature?

A feature is information used to describe sample attributes and is the main input for models to understand and process data.

In English, it is called Feature.

In a house price prediction task:

  • House area.
  • Number of rooms.
  • House age.
  • Region.
  • Distance to the subway.

All of these can be used as features.

In an image recognition task:

  • Pixel values.
  • Color distribution.
  • Edge information.
  • Texture information.
  • Semantic representations extracted by deep neural networks.

These can also constitute features used by the model.

The quality, representation, and information content of features directly affect the model's learning capability.

3.2 Mathematical Representation of Features

Suppose a dataset contains m samples, and each sample contains n features; a feature matrix can be constructed:

X= \begin{bmatrix} x_{11}&x_{12}&\cdots&x_{1n}\\ x_{21}&x_{22}&\cdots&x_{2n}\\ \vdots&\vdots&\ddots&\vdots\\ x_{m1}&x_{m2}&\cdots&x_{mn} \end{bmatrix}

Where:

  • m: number of samples.
  • n: number of features.
  • x_{ij}: the jth feature value of the ith sample.
  • X: the feature matrix of the entire dataset.

Each row of the matrix corresponds to one sample, and each column corresponds to one feature.

For example:

X= \begin{bmatrix} 80&2&500\\ 100&3&300\\ 120&3&800 \end{bmatrix}

There are 3 samples here, and each sample has 3 features.

In practical engineering, the feature matrix may also be a sparse matrix, a tensor, or another structure, and it is not necessarily stored as an ordinary two-dimensional dense matrix.

3.3 Main Types of Features

Different data types require different feature representation methods.

3.3.1 Numerical Feature

Numerical features represent numerical values that can undergo mathematical operations.

Common categories:

Type Description Example
Continuous Can take any real value within a certain range Height, weight, temperature
Discrete Values are usually finite or countable Number of rooms, number of purchases

Numerical features can usually be input directly into the model, but different features may differ greatly in scale and value range.

For example:

  • House area: 80.
  • House price: 3000000.
  • Number of rooms: 3.

Some algorithms are relatively sensitive to feature scale, so standardization or normalization may be needed.

3.3.2 Categorical Feature

Categorical features represent different categories or attributes.

For example:

  • Gender.
  • City.
  • Product type.
  • User level.
  • Payment method.

Categorical features usually cannot be directly treated as ordinary numerical values for mathematical calculations and need to be converted into a representation that the model can handle.

Common encoding methods:

Label Encoding

Assign integer codes to different categories.

For example:

City Encoding
Beijing 0
Shanghai 1
Shenzhen 2

This method is not necessarily suitable for all models, because integer encoding may introduce ordering relationships that do not exist.

One-Hot Encoding

Convert each category into an independent binary feature.

For example:

City Beijing Shanghai Shenzhen
Beijing 1 0 0
Shanghai 0 1 0
Shenzhen 0 0 1

One-Hot Encoding avoids directly assigning an ordering to categories, but when the number of categories is very large, it produces high-dimensional sparse features.

Embedding Encoding

Categories are represented by trainable vectors.

For example, user IDs and item IDs can be mapped to low-dimensional dense vectors.

Embeddings are extremely common in recommender systems, natural language processing, and deep learning.

3.3.3 Text Feature

Text data must be processed before it can be fed into a model.

Common methods in traditional machine learning:

  • Bag of Words.
  • TF-IDF.
  • N-gram.
  • Word frequency statistics.

Common methods in deep learning:

  • Tokenization.
  • Word Embedding.
  • Position Embedding.
  • Contextual Embedding.

For example, given a piece of text:

"Machine learning is powerful"

You can first perform word segmentation or tokenization, then map the token IDs to vector representations.

Modern language models typically use an embedding layer to convert token IDs into continuous vectors before feeding them into network structures such as Transformers.

3.3.4 Temporal Feature

Temporal features describe time and its related attributes.

For example:

  • Year.
  • Month.
  • Day of the week.
  • Hour.
  • Holidays.
  • Time interval.
  • Historical behavior sequence.

Temporal features are very important in time series forecasting, financial analysis, and user behavior prediction.

Note that temporal data usually has an inherent ordering and cannot be shuffled freely like ordinary independent samples.

3.3.5 Image and Audio Features

Images and audio are unstructured data.

An image can be expressed as a pixel matrix:

I\in \mathbb{R}^{H\times W\times C}

Where:

  • H: image height.
  • W: image width.
  • C: number of channels.

For example, an RGB image typically contains 3 color channels.

Audio can be represented as a continuous waveform, or converted into feature representations such as a spectrum or Mel spectrogram.

Deep learning models can typically learn features automatically from raw images, audio, or preprocessed data, and no longer rely entirely on hand-crafted features.

3.4 Feature Engineering

Feature engineering is the process of transforming, constructing, selecting, and representing raw data based on business knowledge and the characteristics of the data.

It is an important part of traditional machine learning engineering.

It mainly includes:

1. Feature Extraction

Extracting meaningful information from raw data.

For example, extracting from user order records:

  • Time of the most recent purchase.
  • Number of historical purchases.
  • Average order amount.
  • Spending in the last 30 days.

2. Feature Construction

Generating new features by combining existing features.

For example:

\text{Average order amount}= \frac{\text{Total spending}}{\text{Number of orders}}
\text{User activity}= \frac{\text{Active days}}{\text{Days in the statistical period}}

3. Feature Selection

Selecting the features that help the prediction target and reducing redundancy and noise.

Common methods:

  • Filter: filtering based on statistical metrics.
  • Wrapper: filtering based on model performance.
  • Embedded: feature selection completed during model training.

4. Feature Transformation

Changing the representation of features.

For example:

  • Standardization.
  • Normalization.
  • Log transformation.
  • Categorical encoding.
  • Missing value imputation.

3.5 Feature Standardization and Normalization

The numerical ranges of different features may differ greatly.

For example:

Feature Value range
Number of rooms 1~10
House area 30~300
House price 100000~10000000

If data is fed directly into certain models, differences in numerical scale may affect model training.

Standardization

Common Z-Score standardization:

x'=\frac{x-\mu}{\sigma}

Where:

  • x: Original feature value.
  • \mu: Feature mean.
  • \sigma: Feature standard deviation.
  • x': Standardized value.

After processing, features usually have a mean close to 0 and a standard deviation close to 1.

Normalization

Common Min-Max normalization:

x'=\frac{x-x_{\min}}{x_{\max}-x_{\min}}

Features are usually scaled to the [0,1] range.

Note the following:

  • Standardization does not guarantee that all data falls within a fixed interval.
  • Min-Max normalization is easily affected by extreme values.
  • Not all algorithms require feature scaling.
  • Decision trees and their common ensemble models usually do not rely on feature standardization.
  • Standardization parameters must be computed only from the training set and then applied to the validation and test sets to avoid data leakage.

3.6 Feature Selection and Model Performance

More features do not necessarily mean better model performance.

Irrelevant, duplicate, and noisy features may lead to:

  • Increased computational cost.
  • Increased model complexity.
  • More severe overfitting.
  • Reduced model stability.
  • Increased data collection and maintenance costs.

For example, when predicting whether a user will churn:

  • The number of logins in the last 30 days may be an important feature.
  • The time of the most recent purchase may be an important feature.
  • A randomly generated ID with no business meaning is usually not a valid feature.

However, whether a feature is effective cannot be judged solely by subjective judgment; it needs to be assessed through data analysis, model experiments, and validation set evaluation.


IV. Labels

4.1 What Is a Label?

A label is information used in supervised learning to describe the target result of a sample.

In English, it is called a Label and is also often called a Target.

For example, in a spam email detection task:

Email content Label
Congratulations on winning a reward; claim it now Spam
Project meeting tomorrow afternoon Legitimate email
Your account has an anomaly; please verify Spam

Here, the email content is the input, and spam or legitimate email is the label.

By learning from many samples and their corresponding labels, the model establishes a mapping relationship between the input and the target output.

4.2 Mathematical Representation of Labels

Supervised learning data can usually be represented as:

D=\{(x_i,y_i)\}_{i=1}^{N}

Where:

  • D: Dataset.
  • N: Number of samples.
  • x_i: Features of the i-th sample.
  • y_i: Label corresponding to the i-th sample.

The model's training objective is to learn the function:

f_\theta(x_i)\approx y_i

That is, making the model's predicted values as close as possible to the true labels.

4.3 Main Types of Labels

4.3.1 Binary Classification Labels

Binary classification labels have only two classes.

For example:

  • Yes / No.
  • Normal / Abnormal.
  • Churn / No churn.
  • Spam / Legitimate email.

They can usually be represented using 0 and 1:

y\in\{0,1\}

4.3.2 Multiclass Labels

Multiclass tasks contain three or more mutually exclusive classes.

For example:

  • Cat.
  • Dog.
  • Bird.
  • Car.

Labels can be represented using integer encoding or One-Hot encoding.

y\in\{1,2,\ldots,K\}

where K represents the number of classes.

4.3.3 Continuous Numerical Labels

Labels in regression tasks are usually continuous numerical values.

For example:

  • House price.
  • Sales.
  • Temperature.
  • Remaining useful life of equipment.
y\in\mathbb{R}

4.3.4 Multi-label Data

Multi-label tasks allow one sample to belong to multiple classes at the same time.

For example, an image may simultaneously contain:

  • People.
  • Cars.
  • Roads.
  • Buildings.

Unlike multiclass classification, the classes in a multilabel task are not necessarily mutually exclusive.

4.3.5 Structured Labels

For some tasks, the label is not a single value or class, but structured data.

For example:

  • Object detection: bounding box coordinates and class.
  • Semantic segmentation: the class corresponding to each pixel.
  • Sequence labeling: the label corresponding to each token.
  • Speech recognition: a sequence of text tokens.

4.4 Why Does Label Quality Matter?

Supervised learning relies on labels to provide training feedback, so label quality directly affects how well the model learns.

Common label problems include:

Label Noise

Manual annotation errors, data collection errors, or automatic labeling errors cause labels to be inconsistent with the actual situation.

Label Inconsistency

Different annotators apply different judgment criteria to the same sample.

Class Imbalance

The number of samples differs too greatly across classes.

For example:

  • Normal transactions: 99%.
  • Fraudulent transactions: 1%.

The model may achieve high accuracy by predicting normal transactions most of the time, yet fail to effectively identify fraudulent transactions.

Label Bias

Labels may inherit biases from historical decisions or biases in the data collection process.

Target Leakage

The features contain information that could only be obtained after the target outcome occurred, giving the model an unfair advantage during training and testing.

4.5 Labels Are Not Necessarily Objective Truth

Labels are usually targets constructed from business rules, human judgment, or observed outcomes.

For example:

A user churn label might be defined as:

A user who has not logged in for 30 consecutive days.

This is a business definition; it does not mean the user has permanently churned in any real sense.

Another example:

A credit risk label may be constructed from historical delinquency records, but the historical records themselves may be affected by credit granting policies, data coverage, and social factors.

Therefore, labels must not only be accurate, but also have clearly defined definitions, time ranges, and boundaries of applicability.


5. Dataset

5.1 What Is a Dataset?

A dataset is a set of data samples organized according to a certain structure.

In English it is called a Dataset.

A machine learning dataset usually contains multiple samples, each composed of features; in supervised learning tasks, it also contains the corresponding labels.

For example:

ID Area Number of Rooms House Age House Price
H001 80 2 5 200
H002 100 3 3 280
H003 120 3 10 310
H004 150 4 2 450

Here:

  • Each row is a sample.
  • Area, number of rooms, and house age are features.
  • House price is the label.
  • All rows together form a dataset.

5.2 Mathematical Representation of a Dataset

Suppose the dataset contains N samples, and each sample has d features.

Feature matrix:

X\in\mathbb{R}^{N\times d}

Label vector:

y\in\mathbb{R}^{N}

For classification tasks, labels may also be class indices or other discrete representations.

The entire supervised learning dataset can be represented as:

D=\{(x_i,y_i)\}_{i=1}^{N}

For unsupervised learning, there is usually only input data:

D=\{x_i\}_{i=1}^{N}

Note that real data is not necessarily all real-valued matrices; it may also contain strings, categories, time series, images, audio, or other structured objects.

5.3 Common Types of Datasets

5.3.1 Structured Data

Structured data usually has clear fields and a tabular structure.

For example:

  • User information tables.
  • Order data tables.
  • Transaction records.
  • Device sensor data.

Common storage formats:

  • CSV.
  • Excel.
  • SQL database tables.

Structured data is usually easy to analyze by field, compute statistics on, and construct features from.

5.3.2 Semi-Structured Data

Semi-structured data has a certain organizational structure, but its fields are not necessarily fully fixed.

For example:

  • JSON.
  • XML.
  • Log data.
  • Some event stream data.

It is commonly found in web systems, logging platforms, and data exchange scenarios.

5.3.3 Unstructured Data

Unstructured data does not have a uniform tabular field structure.

For example:

  • Images.
  • Audio.
  • Video.
  • Natural language text.
  • Document files.

This type of data usually needs specific data processing and feature representation methods before it can be input into a model.

5.4 Training, Validation, and Test Splits of a Dataset

In supervised learning, a dataset usually needs to be divided into three parts:

  • Training set.
  • Validation set.
  • Test set.

They serve different purposes.

Training set

Used for learning model parameters.

The model computes the loss on samples in the training set and updates the parameters.

Validation set

Used for model selection and hyperparameter tuning.

For example:

  • Choosing the learning rate.
  • Choosing the model architecture.
  • Adjusting the regularization strength.
  • Comparing different algorithms.

Test set

Used to evaluate final generalization performance after model selection is complete.

The test set should be kept as independent as possible and should not be repeatedly used for tuning.

5.5 Example of Dataset Splitting

Suppose there are 10000 samples:

Dataset Proportion Count Purpose
Training set 70% 7000 Train the model
Validation set 15% 1500 Tune the model
Test set 15% 1500 Final evaluation

This is only a common example, not a fixed standard.

The specific proportions need to be determined according to data scale, task type, and evaluation method.

flowchart TB
    A["Full dataset"] --> B["Dataset split"]
    B --> C["Training set"]
    B --> D["Validation set"]
    B --> E["Test set"]

    C --> F["Train model parameters"]
    D --> G["Model selection and hyperparameter tuning"]
    F --> H["Candidate model"]
    G --> H
    H --> I["Final model"]
    E --> J["Independent evaluation"]
    I --> J

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style J fill:#E5F0E9,stroke:#83A995,color:#315D49