The core of machine learning is learning patterns from data, and how that data is organized, how it is represented, and how it is fed into a model directly affects both training results and generalization ability.
In machine learning, feature, sample, label, and dataset are the four most fundamental and most important concepts.
Together they form the data organization system of machine learning:
- Sample: describes one specific observation object.
- Feature: information describing the attributes of a sample.
- Label: describes the target result corresponding to a sample.
- Dataset: a collection of data formed by organizing multiple samples according to a certain structure.
These concepts look simple, but in real engineering they involve a large number of critical issues such as feature engineering, data annotation, data preprocessing, data splitting, data leakage, and model generalization.
Understanding these foundational concepts is an important prerequisite for learning machine learning algorithms and deep learning models.
1. Why Does Machine Learning Need Data?
1.1 Data Is the Foundation of Model Learning
Traditional programs are usually written by developers as explicit rules, and the program executes predefined logic based on its input.
Machine learning is different.
Developers provide the data, the learning objective, and the algorithm, and the model automatically learns parameters and patterns from the data.
For example, developing a house price prediction system.
Traditional programming might require developers to manually define rules:
- The larger the area, the higher the price.
- The better the location, the higher the price.
- The more rooms, the higher the price is likely to be.
But real house prices are affected by many factors together, and it is very difficult to describe them completely with hand-written rules.
Machine learning can use historical housing data to automatically learn the relationship between these factors and house prices.
For example:
| Area | Rooms | Distance to Subway | House Age | Price |
|---|---|---|---|---|
| 80㎡ | 2 | 500m | 5 years | 2 million |
| 100㎡ | 3 | 300m | 3 years | 2.8 million |
| 120㎡ | 3 | 800m | 10 years | 3.1 million |
| 150㎡ | 4 | 200m | 2 years | 4.5 million |
The model learns the relationship between house prices and each attribute from this historical data.
Once training is complete, it can predict the price of a new house based on its set of attributes.
1.2 The Data Processing Pipeline in Machine Learning
Machine learning typically goes through stages such as data acquisition, data processing, feature construction, model training, and model evaluation.
flowchart TB
A["Raw Data"] --> B["Data Cleaning"]
B --> C["Data Preprocessing"]
C --> D["Feature Construction and Selection"]
D --> E["Build Dataset"]
E --> F["Model Training"]
F --> G["Model Evaluation"]
G --> H["Model Deployment and Inference"]
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#EAF3EC,stroke:#83A995,color:#315D49
style D fill:#EAF3EC,stroke:#83A995,color:#315D49
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F
These stages are not completely independent. For example, if model evaluation reveals that certain features perform poorly, feature engineering may need to be redone; if abnormal data distributions are found during training, the data may also need to be reprocessed.
2. Sample
2.1 What Is a Sample?
A sample is an independent data instance in a dataset, used to describe a specific observation object or event.
In English it is usually called a Sample, and is also often called an Instance or an Observation.
For example, in a house price prediction task:
One specific house can serve as a sample.
In spam detection tasks:
A single specific email can serve as one sample.
In customer churn prediction tasks:
One user's behavior records within a given observation period can serve as one sample.
Samples are the basic unit of data organization in machine learning.
2.2 What does a sample consist of?
A sample typically contains:
- Sample identifier: used to distinguish different samples.
- Features: attributes that describe the sample.
- Label: the corresponding target outcome in supervised learning.
For example:
| Sample ID | Area | Number of rooms | Distance to subway | House price |
|---|---|---|---|---|
| H001 | 80㎡ | 2 | 500m | 2 million |
| H002 | 100㎡ | 3 | 300m | 2.8 million |
| H003 | 120㎡ | 3 | 800m | 3.1 million |
Here, each row represents one sample.
H001, H002, and H003 are three different housing samples.
Sample IDs are mainly used for data management and tracking, and generally should not be used directly as model features—unless the ID itself carries clear predictive meaning.
2.3 Mathematical representation of a sample
Suppose a sample contains n features; it can be represented as:
Where:
- i: the sample index.
- n: the number of features.
- x_{ij}: the j-th feature value of the i-th sample.
For example:
can represent four features of a house:
- Area: 80.
- Number of rooms: 2.
- Distance to subway: 500.
- House age: 5.
In supervised learning, a sample usually also includes its corresponding label:
where y_i is the target value of the i-th sample.
2.4 The relationship between samples and features
A sample is jointly described by multiple features.
For example, an email can be described by the following features:
- Email length.
- Whether it contains links.
- Sender reputation.
- Number of occurrences of specific keywords.
- Number of attachments.
A single feature is usually not enough to fully describe an object; multiple features must work together to form the sample representation.
flowchart TB
A["An email sample"] --> B["Email length"]
A --> C["Number of links"]
A --> D["Number of keywords"]
A --> E["Sender reputation"]
A --> F["Number of attachments"]
B --> G["Sample feature vector"]
C --> G
D --> G
E --> G
F --> G
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style G fill:#E5F0E9,stroke:#83A995,color:#315D49
2.5 Sample quantity and sample quality
Sample quantity is often an important factor affecting model training results, but it does not mean that more data will necessarily lead to better model performance.
You need to pay attention to both:
Sample quantity
When the amount of data is insufficient, the model may not be able to fully learn the target patterns.
Sample quality
Duplicate data, erroneous data, missing data, and anomalous data can affect model learning.
Sample Representativeness
Training samples need to cover the data distribution in real application scenarios as much as possible.
Sample Independence
Although some data may be large in quantity, it actually comes from the same users, devices, or events, and the samples may be highly correlated.
For example, randomly splitting multiple records from the same user into the training set and test set may cause the model to indirectly memorize user characteristics, resulting in overly optimistic test results.
Therefore, sample quantity cannot be simply equated with the amount of effective information.
III. Feature (Feature)
3.1 What Is a Feature?
A feature is information used to describe sample attributes and is the main input for models to understand and process data.
In English, it is called Feature.
In a house price prediction task:
- House area.
- Number of rooms.
- House age.
- Region.
- Distance to the subway.
All of these can be used as features.
In an image recognition task:
- Pixel values.
- Color distribution.
- Edge information.
- Texture information.
- Semantic representations extracted by deep neural networks.
These can also constitute features used by the model.
The quality, representation, and information content of features directly affect the model's learning capability.
3.2 Mathematical Representation of Features
Suppose a dataset contains m samples, and each sample contains n features; a feature matrix can be constructed:
Where:
- m: number of samples.
- n: number of features.
- x_{ij}: the jth feature value of the ith sample.
- X: the feature matrix of the entire dataset.
Each row of the matrix corresponds to one sample, and each column corresponds to one feature.
For example:
There are 3 samples here, and each sample has 3 features.
In practical engineering, the feature matrix may also be a sparse matrix, a tensor, or another structure, and it is not necessarily stored as an ordinary two-dimensional dense matrix.
3.3 Main Types of Features
Different data types require different feature representation methods.
3.3.1 Numerical Feature
Numerical features represent numerical values that can undergo mathematical operations.
Common categories:
| Type | Description | Example |
|---|---|---|
| Continuous | Can take any real value within a certain range | Height, weight, temperature |
| Discrete | Values are usually finite or countable | Number of rooms, number of purchases |
Numerical features can usually be input directly into the model, but different features may differ greatly in scale and value range.
For example:
- House area: 80.
- House price: 3000000.
- Number of rooms: 3.
Some algorithms are relatively sensitive to feature scale, so standardization or normalization may be needed.
3.3.2 Categorical Feature
Categorical features represent different categories or attributes.
For example:
- Gender.
- City.
- Product type.
- User level.
- Payment method.
Categorical features usually cannot be directly treated as ordinary numerical values for mathematical calculations and need to be converted into a representation that the model can handle.
Common encoding methods:
Label Encoding
Assign integer codes to different categories.
For example:
| City | Encoding |
|---|---|
| Beijing | 0 |
| Shanghai | 1 |
| Shenzhen | 2 |
This method is not necessarily suitable for all models, because integer encoding may introduce ordering relationships that do not exist.
One-Hot Encoding
Convert each category into an independent binary feature.
For example:
| City | Beijing | Shanghai | Shenzhen |
|---|---|---|---|
| Beijing | 1 | 0 | 0 |
| Shanghai | 0 | 1 | 0 |
| Shenzhen | 0 | 0 | 1 |
One-Hot Encoding avoids directly assigning an ordering to categories, but when the number of categories is very large, it produces high-dimensional sparse features.
Embedding Encoding
Categories are represented by trainable vectors.
For example, user IDs and item IDs can be mapped to low-dimensional dense vectors.
Embeddings are extremely common in recommender systems, natural language processing, and deep learning.
3.3.3 Text Feature
Text data must be processed before it can be fed into a model.
Common methods in traditional machine learning:
- Bag of Words.
- TF-IDF.
- N-gram.
- Word frequency statistics.
Common methods in deep learning:
- Tokenization.
- Word Embedding.
- Position Embedding.
- Contextual Embedding.
For example, given a piece of text:
"Machine learning is powerful"
You can first perform word segmentation or tokenization, then map the token IDs to vector representations.
Modern language models typically use an embedding layer to convert token IDs into continuous vectors before feeding them into network structures such as Transformers.
3.3.4 Temporal Feature
Temporal features describe time and its related attributes.
For example:
- Year.
- Month.
- Day of the week.
- Hour.
- Holidays.
- Time interval.
- Historical behavior sequence.
Temporal features are very important in time series forecasting, financial analysis, and user behavior prediction.
Note that temporal data usually has an inherent ordering and cannot be shuffled freely like ordinary independent samples.
3.3.5 Image and Audio Features
Images and audio are unstructured data.
An image can be expressed as a pixel matrix:
Where:
- H: image height.
- W: image width.
- C: number of channels.
For example, an RGB image typically contains 3 color channels.
Audio can be represented as a continuous waveform, or converted into feature representations such as a spectrum or Mel spectrogram.
Deep learning models can typically learn features automatically from raw images, audio, or preprocessed data, and no longer rely entirely on hand-crafted features.
3.4 Feature Engineering
Feature engineering is the process of transforming, constructing, selecting, and representing raw data based on business knowledge and the characteristics of the data.
It is an important part of traditional machine learning engineering.
It mainly includes:
1. Feature Extraction
Extracting meaningful information from raw data.
For example, extracting from user order records:
- Time of the most recent purchase.
- Number of historical purchases.
- Average order amount.
- Spending in the last 30 days.
2. Feature Construction
Generating new features by combining existing features.
For example:
3. Feature Selection
Selecting the features that help the prediction target and reducing redundancy and noise.
Common methods:
- Filter: filtering based on statistical metrics.
- Wrapper: filtering based on model performance.
- Embedded: feature selection completed during model training.
4. Feature Transformation
Changing the representation of features.
For example:
- Standardization.
- Normalization.
- Log transformation.
- Categorical encoding.
- Missing value imputation.
3.5 Feature Standardization and Normalization
The numerical ranges of different features may differ greatly.
For example:
| Feature | Value range |
|---|---|
| Number of rooms | 1~10 |
| House area | 30~300 |
| House price | 100000~10000000 |
If data is fed directly into certain models, differences in numerical scale may affect model training.
Standardization
Common Z-Score standardization:
Where:
- x: Original feature value.
- \mu: Feature mean.
- \sigma: Feature standard deviation.
- x': Standardized value.
After processing, features usually have a mean close to 0 and a standard deviation close to 1.
Normalization
Common Min-Max normalization:
Features are usually scaled to the [0,1] range.
Note the following:
- Standardization does not guarantee that all data falls within a fixed interval.
- Min-Max normalization is easily affected by extreme values.
- Not all algorithms require feature scaling.
- Decision trees and their common ensemble models usually do not rely on feature standardization.
- Standardization parameters must be computed only from the training set and then applied to the validation and test sets to avoid data leakage.
3.6 Feature Selection and Model Performance
More features do not necessarily mean better model performance.
Irrelevant, duplicate, and noisy features may lead to:
- Increased computational cost.
- Increased model complexity.
- More severe overfitting.
- Reduced model stability.
- Increased data collection and maintenance costs.
For example, when predicting whether a user will churn:
- The number of logins in the last 30 days may be an important feature.
- The time of the most recent purchase may be an important feature.
- A randomly generated ID with no business meaning is usually not a valid feature.
However, whether a feature is effective cannot be judged solely by subjective judgment; it needs to be assessed through data analysis, model experiments, and validation set evaluation.
IV. Labels
4.1 What Is a Label?
A label is information used in supervised learning to describe the target result of a sample.
In English, it is called a Label and is also often called a Target.
For example, in a spam email detection task:
| Email content | Label |
|---|---|
| Congratulations on winning a reward; claim it now | Spam |
| Project meeting tomorrow afternoon | Legitimate email |
| Your account has an anomaly; please verify | Spam |
Here, the email content is the input, and spam or legitimate email is the label.
By learning from many samples and their corresponding labels, the model establishes a mapping relationship between the input and the target output.
4.2 Mathematical Representation of Labels
Supervised learning data can usually be represented as:
Where:
- D: Dataset.
- N: Number of samples.
- x_i: Features of the i-th sample.
- y_i: Label corresponding to the i-th sample.
The model's training objective is to learn the function:
That is, making the model's predicted values as close as possible to the true labels.
4.3 Main Types of Labels
4.3.1 Binary Classification Labels
Binary classification labels have only two classes.
For example:
- Yes / No.
- Normal / Abnormal.
- Churn / No churn.
- Spam / Legitimate email.
They can usually be represented using 0 and 1:
4.3.2 Multiclass Labels
Multiclass tasks contain three or more mutually exclusive classes.
For example:
- Cat.
- Dog.
- Bird.
- Car.
Labels can be represented using integer encoding or One-Hot encoding.
where K represents the number of classes.
4.3.3 Continuous Numerical Labels
Labels in regression tasks are usually continuous numerical values.
For example:
- House price.
- Sales.
- Temperature.
- Remaining useful life of equipment.
4.3.4 Multi-label Data
Multi-label tasks allow one sample to belong to multiple classes at the same time.
For example, an image may simultaneously contain:
- People.
- Cars.
- Roads.
- Buildings.
Unlike multiclass classification, the classes in a multilabel task are not necessarily mutually exclusive.
4.3.5 Structured Labels
For some tasks, the label is not a single value or class, but structured data.
For example:
- Object detection: bounding box coordinates and class.
- Semantic segmentation: the class corresponding to each pixel.
- Sequence labeling: the label corresponding to each token.
- Speech recognition: a sequence of text tokens.
4.4 Why Does Label Quality Matter?
Supervised learning relies on labels to provide training feedback, so label quality directly affects how well the model learns.
Common label problems include:
Label Noise
Manual annotation errors, data collection errors, or automatic labeling errors cause labels to be inconsistent with the actual situation.
Label Inconsistency
Different annotators apply different judgment criteria to the same sample.
Class Imbalance
The number of samples differs too greatly across classes.
For example:
- Normal transactions: 99%.
- Fraudulent transactions: 1%.
The model may achieve high accuracy by predicting normal transactions most of the time, yet fail to effectively identify fraudulent transactions.
Label Bias
Labels may inherit biases from historical decisions or biases in the data collection process.
Target Leakage
The features contain information that could only be obtained after the target outcome occurred, giving the model an unfair advantage during training and testing.
4.5 Labels Are Not Necessarily Objective Truth
Labels are usually targets constructed from business rules, human judgment, or observed outcomes.
For example:
A user churn label might be defined as:
A user who has not logged in for 30 consecutive days.
This is a business definition; it does not mean the user has permanently churned in any real sense.
Another example:
A credit risk label may be constructed from historical delinquency records, but the historical records themselves may be affected by credit granting policies, data coverage, and social factors.
Therefore, labels must not only be accurate, but also have clearly defined definitions, time ranges, and boundaries of applicability.
5. Dataset
5.1 What Is a Dataset?
A dataset is a set of data samples organized according to a certain structure.
In English it is called a Dataset.
A machine learning dataset usually contains multiple samples, each composed of features; in supervised learning tasks, it also contains the corresponding labels.
For example:
| ID | Area | Number of Rooms | House Age | House Price |
|---|---|---|---|---|
| H001 | 80 | 2 | 5 | 200 |
| H002 | 100 | 3 | 3 | 280 |
| H003 | 120 | 3 | 10 | 310 |
| H004 | 150 | 4 | 2 | 450 |
Here:
- Each row is a sample.
- Area, number of rooms, and house age are features.
- House price is the label.
- All rows together form a dataset.
5.2 Mathematical Representation of a Dataset
Suppose the dataset contains N samples, and each sample has d features.
Feature matrix:
Label vector:
For classification tasks, labels may also be class indices or other discrete representations.
The entire supervised learning dataset can be represented as:
For unsupervised learning, there is usually only input data:
Note that real data is not necessarily all real-valued matrices; it may also contain strings, categories, time series, images, audio, or other structured objects.
5.3 Common Types of Datasets
5.3.1 Structured Data
Structured data usually has clear fields and a tabular structure.
For example:
- User information tables.
- Order data tables.
- Transaction records.
- Device sensor data.
Common storage formats:
- CSV.
- Excel.
- SQL database tables.
Structured data is usually easy to analyze by field, compute statistics on, and construct features from.
5.3.2 Semi-Structured Data
Semi-structured data has a certain organizational structure, but its fields are not necessarily fully fixed.
For example:
- JSON.
- XML.
- Log data.
- Some event stream data.
It is commonly found in web systems, logging platforms, and data exchange scenarios.
5.3.3 Unstructured Data
Unstructured data does not have a uniform tabular field structure.
For example:
- Images.
- Audio.
- Video.
- Natural language text.
- Document files.
This type of data usually needs specific data processing and feature representation methods before it can be input into a model.
5.4 Training, Validation, and Test Splits of a Dataset
In supervised learning, a dataset usually needs to be divided into three parts:
- Training set.
- Validation set.
- Test set.
They serve different purposes.
Training set
Used for learning model parameters.
The model computes the loss on samples in the training set and updates the parameters.
Validation set
Used for model selection and hyperparameter tuning.
For example:
- Choosing the learning rate.
- Choosing the model architecture.
- Adjusting the regularization strength.
- Comparing different algorithms.
Test set
Used to evaluate final generalization performance after model selection is complete.
The test set should be kept as independent as possible and should not be repeatedly used for tuning.
5.5 Example of Dataset Splitting
Suppose there are 10000 samples:
| Dataset | Proportion | Count | Purpose |
|---|---|---|---|
| Training set | 70% | 7000 | Train the model |
| Validation set | 15% | 1500 | Tune the model |
| Test set | 15% | 1500 | Final evaluation |
This is only a common example, not a fixed standard.
The specific proportions need to be determined according to data scale, task type, and evaluation method.
flowchart TB
A["Full dataset"] --> B["Dataset split"]
B --> C["Training set"]
B --> D["Validation set"]
B --> E["Test set"]
C --> F["Train model parameters"]
D --> G["Model selection and hyperparameter tuning"]
F --> H["Candidate model"]
G --> H
H --> I["Final model"]
E --> J["Independent evaluation"]
I --> J
style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style B fill:#EAF3EC,stroke:#83A995,color:#315D49
style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style D fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style F fill:#EAF3EC,stroke:#83A995,color:#315D49
style G fill:#EAF3EC,stroke:#83A995,color:#315D49
style H fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F
style J fill:#E5F0E9,stroke:#83A995,color:#315D49