Back to series
· 16 min read

Video Understanding and Temporal Modeling: From Video Frame Analysis to Spatiotemporal Feature Modeling


I. What Is Video Understanding?

Video understanding refers to a model extracting visual information, motion information, and temporal relationships from continuous video data, and using them to perform tasks such as content recognition, event analysis, behavior understanding, and video reasoning.

Unlike image understanding, video contains not only spatial information but also dynamic information that changes over time.

A single image mainly answers:

  • What objects are in the frame?
  • Where are the objects located?
  • What spatial relationships exist between objects?
  • What is the current scene?

Video also needs to answer:

  • What is the object doing?
  • How does the object's state change?
  • When does the action occur?
  • What chronological relationships exist between multiple actions?
  • What event does a continuous sequence of behavior express?
  • How does an event begin and end?

For example, an image may show a person standing next to a bicycle, but video can further recognize:

  • The person walks toward the bicycle.
  • The person holds the handlebars.
  • The person gets on the bicycle.
  • The person starts riding.
  • The person leaves the frame.

This information depends on changes between consecutive frames, not on visual features from a single frame.

1. Differences Between Video Understanding and Image Understanding

Comparison Dimension Image Understanding Video Understanding
Input data A single image Continuous video frames
Core information Spatial information Spatial and temporal information
Main tasks Classification, detection, segmentation, OCR Action recognition, event understanding, temporal localization
Feature relationships Relationships within a single frame Intra-frame and cross-frame relationships
Computational complexity Related to image resolution and model architecture Also affected by the number of frames and video duration
Key challenges Object recognition, spatial relationships Motion changes, temporal dependencies, long-range relationships

2. Main Tasks of Video Understanding

Video Classification

Determine what category the entire video belongs to.

For example:

  • Sports competitions
  • Cooking process
  • Traffic accidents
  • People talking
  • Industrial production

Action Recognition

Recognize the action being performed by a person or object in the video.

For example:

  • Walking
  • Running
  • Jumping
  • Throwing
  • Clapping
  • Falling

Temporal Action Localization

Identify the specific time intervals in which actions occur in the video.

For example:

  • 00:12—00:18: The person starts running.
  • 00:19—00:23: The person jumps.
  • 00:24—00:30: The person falls.

Event Understanding

Understand the complete event formed by a combination of multiple actions.

For example, the model not only recognizes that a person picks up a cup, but also needs to understand the relationships among the sequence of actions: picking up the cup, pouring water, and drinking.

Video Question Answering

Answer natural-language questions based on the video content.

For example:

  • How many people are in the video?
  • When does the person leave the room?
  • Why does the car suddenly decelerate?
  • What did the person do before falling?

Temporal Grounding and Segment Retrieval (Temporal Grounding)

Locate the corresponding segment in a long video based on a textual description.

For example:

Input:

Find the segment in the video where someone opens the refrigerator and takes out a drink.

Output:

Start time: 00:35
End time: 00:48

II. Video Data Representation

A video is essentially a sequence of image frames arranged in chronological order.

Suppose a video contains T frames:

V = \{F_1,F_2,\ldots,F_T\}

Where:

  • V: Video sequence.
  • F_t: The image at frame t.
  • T: Number of video frames.

Each frame is itself an image and can be represented as:

F_t \in \mathbb{R}^{H \times W \times C}

Where:

  • H: Image height.
  • W: Image width.
  • C: Number of color channels; typically 3 for RGB images.

Therefore, the entire video can be represented as:

V \in \mathbb{R}^{T \times H \times W \times C}

This is a common video tensor representation.

Practical models usually convert it into different data layouts, for example:

[B,C,T,H,W]

Or:

[B,T,C,H,W]

Here, B denotes the batch size.

Different frameworks and models may use different tensor layouts, so the dimension order alone is not enough to determine how a model handles temporal information.

1. Frame Rate (FPS)

FPS (Frames Per Second) indicates how many frames are contained in each second.

For example:

  • 24 FPS: 24 frames per second.
  • 30 FPS: 30 frames per second.
  • 60 FPS: 60 frames per second.

If a video has a duration of D seconds and a frame rate of r, the approximate number of raw frames is:

T = D \times r

For example, for a 60-second video at 30 FPS:

T = 60 \times 30 = 1800

That is, 1800 frames.

However, models usually do not process all raw frames directly; instead, they sample frames according to the task.

2. Timestamp

There is a correspondence between frame indices and timestamps.

Under ideal conditions with a constant frame rate and timing starting from zero:

t_i = \frac{i}{r}

Where:

  • i: the frame index.
  • r: the frame rate.
  • t_i: the time position corresponding to that frame.

For example, frame 300 in a 30 FPS video corresponds to about 10 seconds.

Real videos may use a variable frame rate (VFR). In that case, the exact time cannot be obtained simply by dividing the frame index by the average FPS; the timestamp information provided by the video container or decoder must be used instead.

Timestamps are very important in video understanding, because a model needs not only to recognize what happened, but also to know when the event occurred.

3. Spatial and Temporal Dimensions of Video

A video has three main dimensions:

Dimension Meaning Primary Information
Spatial height H Vertical direction of the image Vertical position
Spatial width W Horizontal direction of the image Horizontal position
Time T Sequence of video frames Action changes and event order

A video can be understood as a three-dimensional spatiotemporal data volume:

V \in \mathbb{R}^{T \times H \times W \times C}

The spatial dimensions describe the structure within a frame, while the temporal dimension describes how that structure changes over time.

This is the core foundation that distinguishes video models from traditional image models.

III. Video Preprocessing and Frame Sampling

Raw video usually cannot be fed directly into a model as input; it must go through processing such as decoding, sampling, resizing, and format conversion.

1. Video Preprocessing Pipeline

flowchart TD
    A["Raw video file"] --> B["Video decoding"]
    B --> C["Extract video frames"]
    C --> D["Frame sampling"]
    D --> E["Image resizing"]
    E --> F["Pixel normalization"]
    F --> G["Timestamp alignment"]
    G --> H["Build video tensor"]
    H --> I["Input to video model"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#EAF3EC,stroke:#83A995,color:#315D49
    style H fill:#EAF3EC,stroke:#83A995,color:#315D49
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

2. Uniform Sampling

Uniform sampling selects frames at fixed intervals across the entire video time range.

Suppose the total video duration is D and N frames need to be sampled. The sampling time points can be approximated as:

t_i = \frac{iD}{N-1}

Where:

i \in \{0,1,\ldots,N-1\}

For example, sampling 8 frames from a 60-second video:

0s
8.57s
17.14s
25.71s
34.29s
42.86s
51.43s
60s

A practical implementation also needs to handle the end boundary of the video, rounding of frame indices, and timestamp mapping.

Advantages:

  • Simple to implement.
  • Covers the entire video time range.
  • Suitable for video summarization and overall scene understanding.
  • The number of samples is easy to control.

Disadvantages:

  • Easy to miss brief actions.
  • Cannot guarantee that key events are sampled.
  • Not sensitive enough to rapidly changing scenes.

3. Fixed-Interval Sampling

Extract frames at fixed time intervals.

For example, extract one frame every 1 second:

0s → 1s → 2s → 3s → 4s

Unlike uniform sampling, where the total number of sampled frames is specified, fixed-interval sampling usually determines the time interval first and then decides the final frame count based on the video length.

Suitable for:

  • Browsing long video content.
  • Frame extraction from surveillance video.
  • Building video datasets.
  • Tasks with explicit requirements for temporal coverage.

4. Random Sampling

Randomly select a number of frames from the video.

Mainly used for data augmentation during training, to keep the model from over-relying on fixed frame positions.

For example, the same video may yield different sampled frames across different training epochs.

Advantages:

  • Increases the diversity of training data.
  • Reduces dependence on fixed sampling positions.
  • Helps improve model generalization.

Disadvantages:

  • Sampling results are unstable.
  • May miss key actions.
  • Not suitable for inference tasks that require precise temporal localization.

5. Adaptive Sampling

Dynamically determine sampling positions and density based on video content.

For example:

  • Reduce sampling for static scenes.
  • Increase sampling at scene transitions.
  • Increase sampling when actions change noticeably.
  • Keep denser frames before and after events occur.

Adaptive sampling can be implemented by combining scene change detection, motion intensity, object changes and model feedback.

6. Comparison of Different Sampling Strategies

Sampling Strategy Core Idea Advantages Limitations
Uniform sampling Even distribution overall Comprehensive coverage Easy to miss short events
Fixed interval Extract frames at time intervals Clear time intervals Ignores content changes
Random sampling Randomly select frames Enhances data diversity Unstable results
Adaptive sampling Adjust density based on content Focuses on key changes Complex to implement

Note: the number of sampled frames is not equal to the number of visual Tokens the model actually processes. Each frame may undergo Patch splitting, multi-scale cropping or high-resolution encoding, and the final number of visual Tokens also depends on the vision encoder and model architecture.

IV. Video Feature Extraction: From Image Features to Spatiotemporal Features

Video models need to extract meaningful features from consecutive frames.

The core question is:

How can we simultaneously represent the spatial structure within a scene and how that structure changes over time?

1. Spatial Features

Spatial features describe the visual information in a single frame.

Mainly include:

  • Object category.
  • Object position.
  • Object shape.
  • Color and texture.
  • Person pose.
  • Scene layout.
  • Spatial relationships between objects.

For example, identifying in a single frame:

Person: one person
Object: bicycle
Position: the person is to the left of the bicycle
Relation: the person is holding the bicycle

Traditional vision encoders such as CNN and Vision Transformer can all be used to extract spatial features.

2. Temporal Features

Temporal features describe how video content changes between consecutive frames.

Mainly include:

  • Direction of object motion.
  • Motion speed.
  • Pose changes.
  • Action duration.
  • State changes.
  • Sequential ordering of events.
  • Long-term behavior patterns.

For example:

Frame 1: the person is standing
Frame 2: the person raises an arm
Frame 3: the person waves forward
Frame 4: the person lowers the arm

A single frame can only show one pose; only consecutive frames can express a waving action.

3. Spatiotemporal Features

Spatiotemporal features combine spatial information with temporal information.

This can be expressed abstractly as:

Z = f(V)

Where:

  • V: video input.
  • f: video feature extraction network.
  • Z: spatiotemporal feature representation.

Spatiotemporal features capture not only what is present in a given frame, but also what changes between frames.

For example:

  • The person's position in the scene changes.
  • The person's arm goes from hanging down to raised.
  • A car goes from stationary to moving.
  • Two people go from being separate to being in contact.

4. Fusion of Spatial and Temporal Features

Common video modeling approaches can be divided into:

Spatial-first

First extract spatial features for each frame independently, then perform cross-frame temporal modeling.

flowchart TD
    A["Input video frame sequence"] --> B["Frame-by-frame visual encoding"]
    B --> C["Spatial feature sequence"]
    C --> D["Temporal dimension modeling"]
    D --> E["Spatiotemporal feature representation"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Advantages:

  • Image pretrained models can be reused.
  • Spatial feature extraction and temporal modeling are relatively independent.
  • Computational cost is easier to control.

Disadvantages:

  • If each frame is processed completely independently during the spatial encoding stage, early cross-frame interaction is insufficient.
  • Subsequent temporal modules need to recover action relationships from the compressed frame features.

Joint Spatiotemporal Modeling

Perform joint feature extraction directly across the spatial and temporal dimensions.

For example, 3D CNNs perform convolution across the temporal, height, and width dimensions.

Advantages:

  • They can directly learn local spatiotemporal changes.
  • They represent short-term motion patterns more naturally.
  • Spatial and temporal information can interact early in feature extraction.

Disadvantages:

  • Computational cost and memory usage are relatively high.
  • They place higher demands on training data and model design.
  • Long-video modeling still requires additional temporal compression or long-range modeling mechanisms.

V. Core Technologies for Video Modeling

1. 2D CNN: Frame-by-Frame Spatial Feature Extraction

2D CNN (two-dimensional convolutional neural network) mainly processes a single image.

For video, each frame can be input into the same CNN separately:

Z_t = CNN(F_t)

This yields:

Z = \{Z_1,Z_2,\ldots,Z_T\}

The feature sequence is then passed to the temporal module for processing.

For example:

Video frame sequence
    ↓
2D CNN with shared parameters
    ↓
Frame-by-frame spatial features
    ↓
Temporal modeling module
    ↓
Video features

Here, the 2D CNN typically shares weights across all frames rather than training a separate network for each frame.

Limitations:

A standalone 2D CNN does not automatically understand cross-frame motion.

Without an additional temporal module, the model may know what appears in each frame but not how this content changes.

2. 3D CNN: Three-Dimensional Convolution

3D CNN (3D Convolutional Neural Network) performs convolution across the temporal, height, and width dimensions.

Standard 2D convolution kernel:

K \in \mathbb{R}^{k_h \times k_w}

3D convolution kernel:

K \in \mathbb{R}^{k_t \times k_h \times k_w}

where:

  • k_t: kernel size along the temporal dimension.
  • k_h: kernel size along the height dimension.
  • k_w: kernel size along the width dimension.

The core computation of 3D convolution can be expressed as:

Y(t,x,y)= \sum_{\tau,i,j} K(\tau,i,j) X(t+\tau,x+i,y+j)

The actual computation also includes input channels, output channels, and a bias term.

It can simultaneously learn:

  • Spatial textures.
  • Local motion.
  • Changes between consecutive frames.
  • Short-term action patterns.

For example, the model can learn local spatiotemporal features of a waving action from changes in arm position across multiple consecutive frames.

Representative architectures:

  • C3D
  • I3D
  • R(2+1)D
  • SlowFast

Among them, R(2+1)D decomposes 3D convolution into spatial convolution and temporal convolution, which helps separate the two feature-learning processes.

Limitations:

3D CNNs have high computational resource requirements; convolution computation and activation memory grow with temporal length, resolution, and number of channels.

3. Two-Stream Network: Dual-Stream Network

A two-stream network splits video information into two complementary branches:

  • RGB Stream: processes raw RGB images.
  • Optical Flow Stream: processes optical flow motion information.

The RGB branch mainly learns appearance and spatial semantics.

The optical flow branch mainly learns the motion changes of pixels or local regions between consecutive frames.

Optical flow can be expressed as:

F(x,y,t)=(u(x,y,t),v(x,y,t))

where:

  • u: horizontal motion component.
  • v: vertical motion component.

The two branches extract features separately and then fuse them.

flowchart TD
    A["Input video"] --> B["RGB frame sequence"]
    A --> C["Compute optical flow"]
    B --> D["RGB feature network"]
    C --> E["Optical flow feature network"]
    D --> F["Appearance features"]
    E --> G["Motion features"]
    F --> H["Feature fusion"]
    G --> H
    H --> I["Action classification or video understanding"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#EAF3EC,stroke:#83A995,color:#315D49
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

Advantages:

  • Uses both appearance and motion information.
  • Can explicitly model motion changes.
  • The two branches are highly complementary.

Limitations:

  • Optical flow computation increases preprocessing cost.
  • The two-branch design increases model complexity.
  • Optical flow quality affects final recognition performance.

4. RNN, LSTM, and GRU: Recurrent Temporal Modeling

After extracting spatial features frame by frame, a recurrent neural network can be used to process the feature sequence.

Suppose the visual feature of each frame is:

X = \{x_1,x_2,\ldots,x_T\}

RNN updates the hidden state in chronological order:

h_t = f(x_t,h_{t-1})

Where:

  • x_t: Current frame feature.
  • h_{t-1}: Hidden state at the previous time step.
  • h_t: Current hidden state.

The hidden state is used to retain historical information.

LSTM controls information retention and updating through the input gate, forget gate, and output gate, mitigating the vanishing gradient problem of ordinary RNNs during long-sequence training.

GRU uses an update gate and a reset gate to simplify the state update process, and its parameter count is usually smaller than that of a standard LSTM with the same hidden dimension.

Typical uses in video:

  • Human action recognition.
  • Video event classification.
  • Gesture recognition.
  • Continuous behavior analysis.
  • Temporal feature aggregation.

Limitations:

  • Traditional RNNs compute recursively in chronological order, limiting training parallelism.
  • Long-sequence modeling is susceptible to the compression of historical information.
  • The ability to model long-range event relationships is limited by network architecture and training method.

5. Transformer: Attention-Based Temporal Modeling

Transformer uses Self-Attention to model relationships between different positions in a sequence.

For video, the model can convert different frames or different visual Patches into a Token sequence.

Suppose the input features are:

X = [x_1,x_2,\ldots,x_T]

After linear mapping, we obtain:

Q=XW_Q,\quad K=XW_K,\quad V=XW_V

Attention computation:

Attention(Q,K,V) = softmax\left( \frac{QK^T}{\sqrt{d_k}} \right)V

The model can aggregate information from different temporal positions based on the correlation between the current Token and other Tokens.

For example:

  • The current frame focuses on the action preparation phase of the person in earlier frames.
  • The current frame focuses on the action outcome of the person in later frames.
  • The model establishes connections between the start, development, and end of an action.

The key advantage of Transformer is its ability to establish non-local temporal relationships without needing to process strictly recursively over time like an ordinary RNN.

However, the computational complexity of Self-Attention must also be considered.

For a Token sequence of length N, the size of the attention matrix in standard global attention is proportional to N^2.

When a video contains many frames and visual Tokens, computation and memory overhead grow rapidly.

6. Video Transformer: Video Transformer

Video Transformer extends the Transformer architecture to video data.

Unlike image Transformers, which process only the Patch sequence of a single image, Video Transformer also needs to represent the temporal dimension.

Common approaches include:

  • Divide each frame into image Patches.
  • Convert Patches into visual Tokens.
  • Inject spatial and temporal position information into Tokens.
  • Model cross-frame relationships through Attention.
  • Output video-level or clip-level features.

Assume a video has T frames, with each frame divided into P patches:

N=T \times P

If each frame has 196 patches and the video samples 16 frames:

N=16\times196=3136

In this case, if standard global Self-Attention is performed over all tokens, the size of the attention matrix is proportional to:

3136^2

This is also an important reason why video Transformers require spatiotemporal decomposition, token compression, and sparse attention design.

6. Spatiotemporal Attention: Spatial Attention and Temporal Attention

A core question for video Transformers is:

How can effective connections be established among a large number of spatial tokens and temporal tokens while controlling computational cost?

1. Spatial Attention

Spatial attention mainly establishes relationships between different patches within the same frame.

For example:

  • The relationship between a person's head and body.
  • The relationship between hands and tools.
  • The relationship between a car and the road.
  • The relationship between an object and the background.

It mainly answers:

In the current frame, what relationships exist between different regions?

2. Temporal Attention

Temporal attention mainly establishes relationships between different temporal positions.

For example:

  • Changes in a person's pose between two consecutive frames.
  • Changes in a car's position across consecutive frames.
  • The process of an object appearing and then disappearing.
  • The relationship between the preparation stage and the ending stage of an action.

It mainly answers:

What connections exist between visual information at different time points?

3. Divided Space-Time Attention

A common design is to perform spatial attention and temporal attention separately.

Suppose the video input is:

X \in \mathbb{R}^{T\times P\times D}

where:

  • T: number of frames.
  • P: number of patches per frame.
  • D: feature dimension.

Joint global attention needs to be performed over T\times P tokens:

O(T^2P^2D)

Here, some projection computations and constant terms are omitted.

If it is decomposed into:

  1. Spatial attention is performed within each frame.
  2. Temporal attention is performed across frames at the same spatial position.

Its main attention computation can be approximately expressed as:

O(TP^2D+PT^2D)

When both T and P are large, this decomposition can significantly reduce attention computation.

flowchart TD
    A["Video Token Sequence"] --> B["Spatial Attention"]
    B --> C["Intra-frame Regional Information Interaction"]
    C --> D["Temporal Attention"]
    D --> E["Cross-frame Information Interaction"]
    E --> F["Spatiotemporal Feature Fusion"]
    F --> G["Video Feature Output"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style G fill:#DCEFE5,stroke:#4F8B70,color:#24563F

This decomposition is a typical implementation approach for spatiotemporal attention; not all video Transformers use the same structure.

4. Temporal Positional Encoding

Self-Attention itself does not inherently know the order of tokens.

If temporal position information is not provided, the model may be unable to distinguish between:

  • Raising a hand first and then lowering it.
  • Lowering a hand first and then raising it.

Therefore, video models need to inject temporal positional encoding.

Common methods:

Absolute Temporal Positional Encoding

Assign a different position vector to each temporal position:

X_t' = X_t + E_t

where E_t denotes the temporal positional encoding.

Relative Temporal Positional Encoding

It describes the relative temporal distance between two tokens.

For example:

  • Two adjacent frames.
  • 5 frames apart.
  • 30 frames apart.

Timestamp Encoding

Directly using real-time information to build temporal position representations for different video clips.

Real timestamps are especially important for variable frame rate video, sparsely sampled video, and cross-clip retrieval.

VII. Motion Modeling in Video

Video understanding cannot rely on static appearance alone; it also needs to understand motion.

1. Optical Flow

Optical flow describes the motion of pixels or local regions between consecutive video frames.

Suppose a pixel's position in adjacent frames moves from:

(x,y)

to:

(x+\Delta x,y+\Delta y)

Then the motion can be expressed as:

(u,v)=(\Delta x,\Delta y)

where:

  • u: horizontal displacement.
  • v: vertical displacement.

By analyzing the motion vectors of a large number of pixels, one can estimate:

  • the motion of people.
  • the motion of objects.
  • camera motion.
  • the overall motion trend in the scene.

Note:

Optical flow is not the same as the true motion of objects.

Factors such as camera movement, occlusion, illumination changes, and motion blur all affect optical flow estimation.

2. Motion Vector

When compressing video, a video encoder may reduce redundant information through motion estimation and motion compensation.

In block matching or related coding mechanisms, motion vectors describe the displacement relationship between the current image block and an image block in a reference frame.

Compared with optical flow:

Comparison dimension Optical flow Video coding motion vector
Primary purpose Estimate visual motion Improve video compression efficiency
Representation Pixel-level or dense motion field Coding block-level motion information
Computation source Optical flow algorithms or models Video encoder
Main use Action understanding, motion analysis Video decoding, compression analysis

Coding motion vectors can serve as a source of motion features, but they are not equivalent to high-precision optical flow.

3. Human Pose Estimation

Human pose estimation detects human keypoints to build a human skeleton structure.

Common keypoints include:

  • Head.
  • Shoulder.
  • Elbow.
  • Wrist.
  • Hip.
  • Knee.
  • Ankle.

A single-frame pose can be represented as:

P_t = \{p_1,p_2,\ldots,p_K\}

where K is the number of keypoints.

Combining poses from consecutive frames:

P = \{P_1,P_2,\ldots,P_T\}

produces a skeleton temporal sequence.

Based on changes in the keypoints, a model can recognize:

  • Walking.
  • Running.
  • Jumping.
  • Falling.
  • Waving.
  • Bending over.

Skeleton-based action recognition usually focuses more on human body structure and joint motion than directly processing RGB video, but it may lose information about objects, background, and fine texture details.

4. Fusion of Motion Features

A practical system can use RGB, optical flow, and skeleton features simultaneously.

flowchart TD
    A["Input video"] --> B["RGB feature extraction"]
    A --> C["Optical flow feature extraction"]
    A --> D["Human pose estimation"]
    B --> E["Appearance features"]
    C --> F["Motion features"]
    D --> G["Skeleton temporal features"]
    E --> H["Multimodal feature fusion"]
    F --> H
    G --> H
    H --> I["Action recognition and event understanding"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#EAF3EC,stroke:#83A995,color:#315D49
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style G fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style H fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style I fill:#DCEFE5,stroke:#4F8B70,color:#24563F

VIII. Long Video Understanding: From Local Actions to Global Events

Short videos usually require understanding only a limited time span.

But long videos may contain several minutes or even hours of content, and the model needs to handle more complex temporal relationships.

For example, a 30-minute video may contain:

  • Multiple scenes.
  • Multiple characters.
  • Multiple scene transitions.
  • Multiple independent events.
  • Behaviors that are related over time.
  • Themes that run through the entire video.

Converting all video frames directly into tokens and feeding them into the model incurs enormous computational and context costs.

1. Main Challenges of Long Video Understanding

Too many tokens

If each frame produces P visual tokens and the video contains T frames:

N=T\times P

As the number of frames and the resolution increase, the number of visual tokens grows rapidly.

Long-range temporal dependencies

The outcome of an event may depend on information that occurred much earlier.

For example:

  • An object appears at the beginning of the video.
  • Partway through, a person puts the object into a drawer.
  • At the end of the video, the question is asked: where is the object?

The model needs to establish associations that span a long period of time.

Unclear event boundaries

Many events do not have an obvious starting or ending frame.

For example, a person gradually speeds up from walking normally to running, and the boundary of the action may be ambiguous.

Scene changes

A video may contain multiple cameras, multiple locations, or different viewpoints, and the associations between scenes need to be identified.

2. Hierarchical Temporal Modeling

A common approach is to divide a long video into multiple levels for processing.

flowchart TD
    A["Full long video"] --> B["Video segmentation"]
    B --> C["Clip-level feature extraction"]
    C --> D["Clip-level event understanding"]
    D --> E["Generate clip summaries"]
    E --> F["Global temporal modeling"]
    F --> G["Video-level semantic representation"]
    G --> H["Answer questions or perform retrieval"]

    style A fill:#DCEFE5,stroke:#4F8B70,color:#24563F
    style B fill:#EAF3EC,stroke:#83A995,color:#315D49
    style C fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style D fill:#EAF3EC,stroke:#83A995,color:#315D49
    style E fill:#F5F8F4,stroke:#B5CBBB,color:#40594A
    style F fill:#EAF3EC,stroke:#83A995,color:#315D49
    style G fill:#F7F4E8,stroke:#C7B77A,color:#675A2D
    style H fill:#DCEFE5,stroke:#4F8B70,color:#24563F

This structure can break a video down into different granularities:

Level Modeling target Main task
Frame level Single frame image Object and scene recognition
Local clip level Several consecutive frames Action recognition
Event level Combination of multiple actions Event understanding
Video level Complete video Global semantics and topic understanding

3. Video Segmentation

Segmentation can be performed based on the following information: