1. Bibliographic Information

1.1. Title

SABR: A Stable Adaptive Bitrate Framework Using Behavior Cloning Pretraining and Reinforcement Learning Fine-Tuning

1.2. Authors

The authors are Pengcheng Luo, Yunyang Zhao, Bowen Zhang, Genke Yang, Boon-Hee Soong, and Chau Yuen.

• Pengcheng Luo, Yunyang Zhao, Bowen Zhang, and Genke Yang are affiliated with the Ningbo Artificial Intelligence Institute, Shanghai Jiao Tong University, Ningbo, China, and the School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China.
• Boon-Hee Soong and Chau Yuen are affiliated with the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore. Boon-Hee Soong is a Senior Member of IEEE, and Chau Yuen is a Fellow of IEEE, indicating their established expertise and recognition in the field.

1.3. Journal/Conference

The paper is published as a preprint on arXiv. While the specific journal or conference is not stated, the reference list indicates a strong connection to prestigious computer networking and multimedia conferences such as ACM SIGCOMM, ACM Multimedia, and IEEE INFOCOM. These venues are highly reputable and influential in the fields of networking, multimedia systems, and artificial intelligence applications in these domains.

1.4. Publication Year

2025

1.5. Abstract

This paper introduces SABR, a novel training framework for Adaptive Bitrate (ABR) control that addresses the limitations of existing learning-based ABR methods, particularly their poor generalization to out-of-distribution (OOD) network conditions. The core problem is that most ABR models are trained on limited network traces, overlooking the diverse nature of real-world networks, which leads to unstable performance when faced with unseen scenarios. SABR tackles this by combining behavior cloning (BC) pretraining with reinforcement learning (RL) fine-tuning. The framework leverages Direct Preference Optimization (DPO) for stable BC pretraining on expert data, followed by Proximal Policy Optimization (PPO) for RL fine-tuning to enhance exploration and generalization. To facilitate evaluation, the authors also release two benchmarks, ABRBench-3G and ABRBench-4G+, which provide wide-coverage training traces and dedicated OOD test sets. Experimental results demonstrate that SABR achieves the best average rank compared to prominent baselines like Pensieve, Comyco, and NetLLM across these benchmarks, indicating its superior stability and generalization capabilities in diverse and unseen network conditions.

1.6. Original Source Link

https://arxiv.org/abs/2509.10486

PDF Link: https://arxiv.org/pdf/2509.10486v1.pdf Publication Status: Preprint on arXiv.

2. Executive Summary

2.1. Background & Motivation

With the advent of 5G networks, video content has become the dominant form of digital consumption, ranging from short-form platforms like TikTok to long-form services like Bilibili. This shift makes the Quality of Experience (QoE) for video streaming paramount. Adaptive Bitrate (ABR) algorithms are crucial for maintaining high QoE by dynamically adjusting video bitrate based on real-time network conditions to minimize issues like stalling and latency.

The core problem addressed by the paper is the limited generalization and stability of existing learning-based ABR methods. While Artificial Intelligence (AI) techniques, particularly deep learning and reinforcement learning (RL), have shown promise in optimizing ABR, they face two major challenges:

1. Limited generalization to unseen distributions: Most models are trained on narrow network trace sets, failing to perform well when encountering real-world network conditions they haven't seen before (out-of-distribution or OOD scenarios).

2. Degradation under wide-distribution training: When models are trained on a broad spectrum of network conditions, the training process often becomes inefficient and unstable.

   The paper draws inspiration from the success of large language models (LLMs), which effectively use a two-stage training framework of pretraining and fine-tuning (e.g., Supervised Fine-Tuning (SFT) + Reinforcement Learning from Human Feedback (RLHF)) to achieve robust generalization and alignment with human preferences across diverse tasks. This LLM paradigm motivates the authors to explore a similar approach for ABR.

2.2. Main Contributions / Findings

The paper's primary contributions are:

• A novel two-stage training framework, SABR: SABR combines Behavior Cloning (BC) pretraining with Reinforcement Learning (RL) fine-tuning to improve ABR generalization and stability. It uses Direct Preference Optimization (DPO) for fast and stable BC pretraining on expert data, followed by Proximal Policy Optimization (PPO) for deeper RL exploration and robust adaptation to challenging network dynamics.

• New benchmarks for ABR evaluation: The introduction of ABRBench-3G and ABRBench-4G+, which provide wide-coverage training traces and dedicated OOD test sets. These benchmarks allow for a more rigorous assessment of ABR models' generalization capabilities to unseen network conditions.

• Empirical validation of SABR's superiority: Experimental results show that SABR achieves the best average rank compared to state-of-the-art baselines, including Pensieve, Comyco, and NetLLM, across the proposed benchmarks. This demonstrates that SABR enables more stable learning across wide distributions and significantly improves generalization to unseen network conditions.

  The key finding is that the two-stage BC pretraining and RL fine-tuning approach, inspired by LLM training paradigms, effectively addresses the long-standing challenges of generalization and stability in learning-based ABR systems, leading to superior QoE performance in diverse and unseen network environments.

3. Prerequisite Knowledge & Related Work

3.1. Foundational Concepts

To understand the SABR framework, a few foundational concepts are essential:

• Adaptive Bitrate (ABR): A technology used in video streaming that dynamically adjusts the quality (bitrate) of a video stream in real-time based on current network conditions (e.g., available bandwidth, buffer occupancy). The goal is to provide a smooth playback experience, minimize buffering, and maximize video quality.
• Quality of Experience (QoE): A subjective measure of a user's overall satisfaction with a service, in this case, video streaming. For ABR, common QoE metrics consider factors like video bitrate (higher is better), rebuffering events (lower is better), and bitrate switching frequency (fewer switches are better).
• Reinforcement Learning (RL): A paradigm of machine learning where an agent learns to make decisions by interacting with an environment. The agent performs actions, receives rewards or penalties, and observes new states. The goal is to learn a policy (a mapping from states to actions) that maximizes the cumulative reward over time.
  • Agent: The ABR algorithm making bitrate decisions.
  • Environment: The video streaming system, including network conditions, video player buffer, and video content.
  • State: Information about the current environment (e.g., current buffer size, estimated bandwidth, recent bitrates).
  • Action: The choice of bitrate for the next video chunk.
  • Reward: A numerical value reflecting the immediate QoE resulting from an action.
  • Policy: The strategy the agent uses to choose actions given a state.
• Behavior Cloning (BC): A type of imitation learning where a machine learning model learns a policy by directly mimicking the behavior of an expert. The model is trained on a dataset of state-action pairs demonstrated by an expert, effectively learning a supervised mapping from observations to actions. It's often used to quickly bootstrap an RL agent with reasonable behavior.
• Direct Preference Optimization (DPO): An algorithm primarily used in LLM alignment that directly optimizes a policy to align with human preferences, without the need for a separate reward model. Instead of learning an explicit reward function, DPO directly maximizes the log-likelihood of preferred responses over dispreferred ones, making it simpler and more stable than traditional Reinforcement Learning from Human Feedback (RLHF) methods.
• Proximal Policy Optimization (PPO): A popular RL algorithm that belongs to the family of policy gradient methods. PPO aims to find a policy that maximizes expected reward. It's known for its balance of performance, stability, and sample efficiency. A key feature of PPO is its clipping mechanism, which limits the change in the policy at each update step, preventing large, destructive policy updates and promoting stable training.

3.2. Previous Works

The paper contextualizes SABR by reviewing several learning-based ABR methods:

• Pensieve [6]: One of the pioneering works in RL-based ABR. It applied the A3C (Asynchronous Advantage Actor-Critic) algorithm to train an RL policy using network states (e.g., throughput, buffer length) as inputs. Pensieve demonstrated the viability of RL for ABR control, particularly on 3G network traces.
• Comyco [8]: Further advanced RL-based ABR by incorporating quality-aware QoE metrics and utilizing imitation learning. It trained a policy by mimicking expert data generated by Model Predictive Control (MPC) algorithms. This approach significantly improved training efficiency and model performance compared to pure RL.
• Jade [9]: Focused on addressing user differences in video quality preferences. It integrated ranking-based QoE feedback into an RLHF framework, aiming to align the optimization objective with diverse human preferences and improve QoE across heterogeneous network conditions.
• Genet [10]: Introduced an automatic curriculum learning approach for ABR. It started training in network environments where rule-based baselines performed poorly and gradually expanded the training distribution. While this helped models improve progressively, it could suffer from distributional shift and forgetting issues as the training data became broader.
• NetLLM [12]: Explored adapting Large Language Models (LLMs) to various networking tasks, including ABR. It utilized multi-modal encoding and Low-Rank Adaptation (LoRA) to reduce training costs, showcasing the potential of LLMs in ABR, even though LLMs are not typically designed for control tasks.

3.3. Technological Evolution

The evolution of ABR algorithms has progressed from simple rule-based heuristics (like buffer-based methods) to more sophisticated optimization-based approaches (MPC, Lyapunov optimization), and more recently, to data-driven learning-based methods (Reinforcement Learning, Imitation Learning, Deep Learning).

• Rule-based: Early ABR algorithms relied on predefined rules (e.g., switch to higher bitrate if buffer is full, switch to lower if buffer is low). Simple but often sub-optimal and rigid.
• Optimization-based: Methods like MPC and BOLA use mathematical optimization to make decisions over a future horizon, considering network predictions and QoE models. These are more robust but can be computationally intensive and rely on accurate models.
• Learning-based (AI-driven):

  • Early RL (Pensieve): Applied RL to learn policies directly from network interactions, aiming to maximize QoE without explicit modeling.

  • Imitation Learning (Comyco): Combined RL with imitation learning from expert MPC policies to improve training efficiency and stability.

  • Human Feedback (Jade): Incorporated human feedback to align RL agents with user preferences, moving towards more personalized QoE.

  • Curriculum Learning (Genet): Focused on structured training over diverse environments to improve robustness.

  • LLM Adaptation (NetLLM): Explored leveraging powerful LLMs for ABR, hinting at a future where general-purpose AI models could adapt to specialized control tasks.

    SABR fits into this evolution by addressing a critical weakness of current learning-based ABR: their lack of generalization to diverse, unseen network conditions and the instability of training on wide-distribution data. It innovatively borrows the successful two-stage pretraining + fine-tuning paradigm from LLMs, applying it to the ABR domain.

3.4. Differentiation Analysis

Compared to the main methods in related work, SABR introduces several core differences and innovations:

• Two-stage BC pretraining + RL fine-tuning framework:

  • Unlike Pensieve (pure RL) or Comyco (imitation learning, but not explicitly two-stage $BC+RL$), SABR combines BC for initial stable learning from expert data (analogous to SFT in LLMs) and then RL for deeper exploration and optimization (analogous to RLHF in LLMs). This addresses the QoE performance degradation under wide-distribution training, a known issue with single-stage learning methods.
  • This approach aims for both stability and strong generalization, which is a key differentiator from methods that might focus on one aspect more than the other (e.g., Comyco focuses on efficiency through imitation, Genet on progressive improvement but faces stability issues).

• Leveraging DPO for BC pretraining: DPO is a relatively new and stable preference learning algorithm. By using DPO for BC, SABR aims to efficiently learn a strong initial policy from expert demonstrations, avoiding the complexities of traditional RLHF or simpler supervised learning that might not capture preference-like signals as effectively. This provides a robust base model before RL exploration.

• Comprehensive Benchmarks: The introduction of ABRBench-3G and ABRBench-4G+ with distinct training, test, and critically, Out-of-Distribution (OOD) sets, is a significant contribution. Previous works often rely on limited network trace sets. These new benchmarks enable a more rigorous and standardized evaluation of ABR models' generalization capabilities to genuinely unseen conditions, something not explicitly provided by earlier works.

• Improved Generalization and Stability: The experimental results demonstrate SABR's superior average rank across diverse network conditions and particularly in OOD scenarios. This directly tackles the limitations of limited generalization to unseen distributions that plague many existing learning-based ABR methods.

  In essence, SABR's innovation lies in its principled adoption of a proven AI training paradigm (pretraining + fine-tuning) to robustly address the unique challenges of ABR in highly dynamic and unpredictable real-world network environments.

4. Methodology

The SABR framework is designed to overcome the limitations of current learning-based ABR methods, specifically their poor generalization to unseen network conditions and instability during training on wide-distribution data. It achieves this by employing a two-stage training approach: Behavior Cloning (BC) pretraining followed by Reinforcement Learning (RL) fine-tuning.

4.1. Principles

The core idea behind SABR is to leverage the strengths of both behavior cloning and reinforcement learning.

1. BC Pretraining (Stability & Initial Understanding): The first stage, BC pretraining, focuses on providing the model with a strong initial policy by learning from expert demonstrations. This stage aims to quickly and stably establish a foundational understanding of effective ABR decision-making across a wide range of network conditions. By using Direct Preference Optimization (DPO), the model learns to prioritize expert actions, leading to a robust base model. This helps mitigate the instability often encountered when RL agents start learning from scratch in complex, wide-distribution environments.

2. RL Fine-tuning (Exploration & Generalization): The second stage, RL fine-tuning, takes the pre-trained base model and further refines it using Proximal Policy Optimization (PPO). This stage allows the model to explore the policy space beyond the expert demonstrations, adapt to subtle nuances, and optimize its behavior for maximum QoE in diverse and potentially unseen OOD scenarios. PPO's stability guarantees help ensure that this exploration and refinement process remains effective without destabilizing the policy learned during pretraining.

   The overall framework (Figure 2) illustrates this two-stage process: an initial model undergoes BC pretraining to become a base model, which is then further trained via RL fine-tuning to become the fine-tuned model used for inference. The ABR simulator acts as the environment for both stages of training.

The following figure (Figure 2 from the original paper) shows the proposed SABR framework:

2.jpg: VLM Description: The image is a schematic diagram illustrating the structure of the SABR framework, which includes the BC pretraining and RL fine-tuning processes. The framework involves the initial model, base model, and fine-tuned model, along with the ABR simulator. In the model inference section, factors such as bandwidth, past bit rate, and buffer are considered, outputting different bitrates like 360P, 480P, and 720P.

4.2. Core Methodology In-depth (Layer by Layer)

4.2.1. BC Pretraining with DPO

The BC pretraining stage utilizes Direct Preference Optimization (DPO) to learn from expert demonstrations. DPO was originally developed for aligning LLMs with human preferences by directly optimizing the likelihood ratio of preferred responses. In SABR, this principle is adapted to ABR by treating expert actions as "preferred" actions.

Original DPO Objective: The original DPO algorithm optimizes a policy $\pi_\theta$ by maximizing the log-ratio of probabilities for a "winner" trajectory $\tau_w$ over a "loser" trajectory $\tau_l$. This is typically done with respect to a reference policy $\pi_{ref}$, which is often the initial policy. The objective function is: $$\mathcal{L}_{\mathrm{DPO}}(\theta) = - \mathbb{E}_{(\tau_w, \tau_l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \cdot \Big[ \log \frac { \pi_{\theta} (\tau_w) } { \pi_{\mathrm{ref}} (\tau_w) } - \log \frac { \pi_{\theta} (\tau_l) } { \pi_{\mathrm{ref}} (\tau_l) } \Big ] \Big ) \Big ]$$ Here:

• $\theta$: Parameters of the current policy model $\pi_\theta$.
• $\mathbb{E}_{(\tau_w, \tau_l) \sim \mathcal{D}}$: Expectation over pairs of preferred (winner) and dispreferred (loser) trajectories sampled from the dataset $\mathcal{D}$.
• $\sigma(\cdot)$: The sigmoid function, which squashes its input to a range between 0 and 1.
• $\beta$: A scalar hyperparameter controlling the strength of the update. A higher $\beta$ means stronger preference for the winner trajectory.
• $\pi_\theta(\tau)$: The probability of trajectory $\tau$ under the current policy model.
• $\pi_{\mathrm{ref}}(\tau)$: The probability of trajectory $\tau$ under a fixed reference policy model (typically the model's initialization or a previous iteration), which helps stabilize training.
• $\log \frac { \pi_{\theta} (\tau_w) } { \pi_{\mathrm{ref}} (\tau_w) }$: The log-probability ratio for the winner trajectory, indicating how much the current policy prefers the winner compared to the reference.
• $\log \frac { \pi_{\theta} (\tau_l) } { \pi_{\mathrm{ref}} (\tau_l) }$: The log-probability ratio for the loser trajectory. The loss function encourages the model to increase the probability of preferred trajectories while decreasing that of dispreferred ones, relative to the reference policy.

Step-wise DPO for BC: In BC training for ABR, the focus is on learning from individual state-action pairs rather than full trajectories. Therefore, the original DPO loss is adapted into a step-wise formulation: $$\mathcal{L}_{\mathrm{DPO-step}}(\theta) = - \mathbb{E}_{(s, a^w, a^l) \sim \mathcal{D}} \Big[ \log \sigma \Big( \beta \cdot \Big[ \log \frac { \pi_{\theta} \left( a^w \mid s \right) } { \pi_{\mathrm{ref}} \left( a^w \mid s \right) } - \log \frac { \pi_{\theta} \left( a^l \mid s \right) } { \pi_{\mathrm{ref}} \left( a^l \mid s \right) } \Big ] \Big ) \Big ]$$ Here:

• $(s, a^w, a^l) \sim \mathcal{D}$: Sampled state-action pairs from the dataset, where $s$ is the current state, $a^w$ is an expert (preferred) action in that state, and $a^l$ is a less preferred (e.g., randomly sampled) alternative action in that state.
• $\pi_\theta(a^w \mid s)$: The probability of taking the expert action $a^w$ given state $s$ under the current policy.
• $\pi_{\mathrm{ref}}(a^w \mid s)$: The probability of taking the expert action $a^w$ given state $s$ under the reference policy.
• The terms are similar to the trajectory-based DPO, but applied to individual actions instead of entire trajectories. This loss encourages the policy to assign a higher probability to expert actions ($a^w$) compared to alternative actions ($a^l$) for any given state $s$.

BC Pretraining Procedure (Algorithm 1): The BC pretraining procedure follows the DAGGER (Dataset Aggregation) algorithm [14] framework. It iteratively collects data by having the current policy interact with the environment, and then uses that collected data to train.

$$Algorithm 1 BC pretraining with DPO 1: Input: Initial model π_θ, BEAM_SEARCH_POLICY, ABR simulator, iteration N_pretrain, rollout step T_pretrain, epoch E_pretrain, mini-batch size m_pretrain 2: Initialize π_ref, buffer B ← ∅, obtain initial state s_1 from ABR simulator 3: for 1, 2, ..., N_pretrain do 4: for 1, 2, ..., T_pretrain do 5: Select action a_t ~ π_θ(· | s_t) 6: Expert action a_t^w ← BEAM_SEARCH_POLICY(s_t) 7: Randomly select an alternative action a_t^l ≠ a_t^w 8: Append sample: B ← B ∪ {(s_t, a_t^w, a_t^l)} 9: Execute a_t in the ABR simulator to obtain next state s_t+1 10: end for 11: for 1, 2, ..., E_pretrain do 12: Sample mini-batch B̂ of size m_pretrain from B 13: Update π_θ using the DPO loss on B̂ (Eq. 2) 14: end for 15: end for 16: Output: Base model π_θ$$ Explanation of Algorithm 1:

• Inputs:
  • $Initial model π_θ$: The policy model to be trained.
  • BEAM_SEARCH_POLICY: The expert policy used to generate preferred actions. This policy is adopted from Comyco [8], [15].
  • ABR simulator: The environment for interacting and collecting data.
  • N_pretrain: Total number of outer iterations for pretraining.
  • T_pretrain: Number of steps (time steps) for policy rollout in each outer iteration to collect experience.
  • E_pretrain: Number of training epochs on the collected data in each outer iteration.
  • m_pretrain: Mini-batch size for DPO updates.
• Initialization (Line 2):
  • $\pi_{\mathrm{ref}}$: The reference policy, typically initialized as a copy of the $Initial model π_θ$.
  • $B$: An empty buffer to store collected (state, expert action, alternative action) samples.
  • $s_1$: The starting state obtained from the ABR simulator.
• Data Collection Loop (Lines 3-10):
  • The algorithm iterates N_pretrain times. In each iteration, it performs T_pretrain steps of interaction with the simulator.
  • Line 5: The current policy $\pi_\theta$ selects an action $a_t$ based on the current state $s_t$.
  • Line 6: An expert action $a_t^w$ is generated for the current state $s_t$ using the BEAM_SEARCH_POLICY. This serves as the "winner" action.
  • Line 7: A "loser" action $a_t^l$ is randomly selected from the available actions, ensuring it is different from the expert action $a_t^w$. This provides the contrasting action for the DPO loss.
  • Line 8: The tuple $(s_t, a_t^w, a_t^l)$ is added to the buffer $B$.
  • Line 9: The action $a_t$ (selected by the current policy) is executed in the simulator, leading to the next state $s_{t+1}$.
• Training Loop (Lines 11-14):
  • After collecting T_pretrain steps of data, the model undergoes E_pretrain epochs of training.
  • Line 12: In each epoch, a mini-batch $\hat{B}$ of size m_pretrain is sampled from the collected buffer $B$.
  • Line 13: The policy parameters $\theta$ are updated using the DPO-step loss (Eq. 2) on the sampled mini-batch.
• Output (Line 16): The final trained policy $\pi_\theta$ becomes the Base model.

4.2.2. RL Fine-tuning with PPO

After BC pretraining, the base model is fine-tuned using Proximal Policy Optimization (PPO). PPO is chosen for its stability and sample efficiency, allowing the model to further explore the environment and optimize for QoE without destabilizing the already learned policy. PPO uses an actor-critic architecture, consisting of an actor network ($\pi_\theta$) and a critic network ($V_\phi$).

PPO Actor Loss: The actor network $\pi_\theta$ (which is initialized from the base model after BC pretraining) is updated using the PPO actor loss. This loss encourages actions that lead to higher rewards while ensuring that policy updates are not too drastic. $$L^{\mathrm{Actor}}(\theta) = \mathbb{E}_t \Big[ \operatorname*{min} \big( r_t(\theta) A_t , \operatorname{clip}( r_t(\theta) , 1 - \epsilon , 1 + \epsilon ) A_t \big ) \Big ]$$ where $$r_t(\theta) = \frac { \pi_{\theta} ( a_t \mid s_t ) } { \pi_{\theta_{\mathrm{old}}} ( a_t \mid s_t ) }$$ Here:

• $\theta$: Parameters of the current actor network.
• $\mathbb{E}_t$: Expectation over time steps (or samples collected).
• $r_t(\theta)$: The probability ratio. It measures how much more or less likely the current policy $\pi_\theta$ is to take action $a_t$ in state $s_t$ compared to the previous policy $\pi_{\theta_{\mathrm{old}}}$.
  • $\pi_{\theta}(a_t \mid s_t)$: Probability of action $a_t$ in state $s_t$ under the current actor network.
  • $\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)$: Probability of action $a_t$ in state $s_t$ under the previous actor network (before the current update).
• $A_t$: The advantage estimate at time step $t$. This value indicates how much better an action $a_t$ is than the average action expected by the critic in state $s_t$. It is typically computed using Generalized Advantage Estimation (GAE) [18]. A positive $A_t$ means the action was better than expected, a negative $A_t$ means worse.
• $\operatorname{clip}(x, L, R)$: A function that clips the value of $x$ to be within the range [L, R]. In this case, $clip(r_t(θ), 1 - ε, 1 + ε)$ clips the probability ratio $r_t(\theta)$ to lie within $[1-\epsilon, 1+\epsilon]$.
• $\epsilon$: A small hyperparameter, the clipping threshold, which defines the range for policy updates. The PPO actor loss takes the minimum of two terms:
  1. $r_t(\theta) A_t$: The standard policy gradient term, scaled by the advantage.
  2. $\operatorname{clip}( r_t(\theta) , 1 - \epsilon , 1 + \epsilon ) A_t$: The clipped version of the policy gradient term. This clipping mechanism ensures that the policy does not change too drastically in a single update step, which helps prevent instability and performance collapse, especially when the advantage estimate $A_t$ is large.

Full PPO Objective: The full PPO objective combines the actor loss with a critic loss (for updating the value function network) and an entropy regularization term (to encourage exploration): $$L^{\mathrm{PPO}}(\theta, \phi) = \mathbb{E}_t \Big[ L^{\mathrm{Actor}}(\theta) - c_1 \big( V_{\phi}(s_t) - V_t^{\mathrm{target}} \big)^2 + c_2 S [ \pi_{\theta} ] ( s_t ) \Big ]$$ Here:

• $\phi$: Parameters of the critic network.
• $V_{\phi}(s_t)$: The state value predicted by the critic network for state $s_t$. The critic learns to estimate the expected cumulative reward from a given state.
• $V_t^{\mathrm{target}}$: The target value for the critic update. This is often calculated from the observed rewards and estimated future values, typically as $\hat{V}_t + \hat{A}_t$.
• $c_1$: Weighting coefficient for the critic loss. The term $c_1 \big( V_{\phi}(s_t) - V_t^{\mathrm{target}} \big)^2$ represents the critic loss, which is a mean squared error between the predicted value and the target value.
• $S [ \pi_{\theta} ] ( s_t )$: An entropy regularization term for the policy $\pi_\theta$ at state $s_t$. Entropy measures the randomness of the policy. Maximizing entropy encourages the policy to explore different actions, preventing it from collapsing into a deterministic or sub-optimal policy too early.
• $c_2$: Weighting coefficient for the entropy regularization term.

RL Fine-tuning Procedure (Algorithm 2): The RL fine-tuning procedure iteratively collects experience using the current policy and then updates both the actor and critic networks using the PPO objective.

Algorithm 2 RL fine-tuning with PPO
1: Input: Actor network π_θ (initialized from base model), critic network V_φ, ABR simulator, iteration N_finetune, rollout steps T_finetune, PPO epochs E_finetune, mini-batch size m_finetune, clipping parameter ε, discount factor γ, GAE parameter λ
2: Empty buffer B ← ∅, obtain initial state s_1 from ABR simulator
3: for 1, 2, ..., N_finetune do
4:   for 1, 2, ..., T_finetune do
5:     Select action a_t ~ π_θ(· | s_t)
6:     Execute a_t in the ABR simulator to obtain reward r_t and next state s_t+1
7:     Append transition: B ← B ∪ {(s_t, a_t, r_t, s_t+1)}
8:   end for
9:   For all transitions in B compute V̂_t = V_φ(s_t) and V̂_t+1 = V_φ(s_t+1)
10:  Compute TD errors δ_t = r_t + γ V̂_t+1 - V̂_t, then advantages Â_t via GAE with (γ, λ)
11:  Set target value V_t^target = V̂_t + Â_t for critic updates
12:  Augment each transition in B to {(s_t, a_t, r_t, s_t+1, Â_t, V_t^target)}
13:  for 1, 2, ..., E_finetune do
14:    Sample mini-batch B̂ of size m_finetune from B
15:    Update parameters θ and φ using the full PPO objective (Eq. 5) on B̂
16:  end for
17:  Clear B ← ∅
18:  π_θ_old ← π_θ
19: end for
20: Output: fine-tuned model π_θ

Explanation of Algorithm 2:

• Inputs:
  • $Actor network π_θ$: Initialized from the Base model obtained from BC pretraining.
  • $Critic network V_φ$: A separate network to estimate state values.
  • N_finetune: Total number of outer iterations for fine-tuning.
  • T_finetune: Number of steps for policy rollout to collect experience in each outer iteration.
  • E_finetune: Number of PPO epochs for training on the collected data.
  • m_finetune: Mini-batch size for PPO updates.
  • $ε$: Clipping parameter for PPO.
  • $γ$: Discount factor, used to weigh future rewards.
  • $λ$: GAE parameter, used in advantage estimation.
• Initialization (Line 2):
  • $B$: An empty buffer to store collected (state, action, reward, next state) transitions.
  • $s_1$: The initial state.
• Data Collection Loop (Lines 3-8):
  • The algorithm iterates N_finetune times. In each iteration, it performs T_finetune steps of interaction with the simulator.
  • Line 5: The current actor policy $\pi_\theta$ selects an action $a_t$.
  • Line 6: The action $a_t$ is executed in the ABR simulator, yielding a reward $r_t$ and the next state $s_{t+1}$.
  • Line 7: The transition $(s_t, a_t, r_t, s_{t+1})$ is added to the buffer $B$.
• Advantage and Target Value Computation (Lines 9-12):
  • After T_finetune steps, the critic network $V_\phi$ is used to estimate values $\hat{V}_t$ and $\hat{V}_{t+1}$ for all collected states.
  • Line 10: Temporal Difference (TD) errors $\delta_t$ are computed, and then advantages $\hat{A}_t$ are calculated using GAE with $γ$ and $λ$. GAE provides a balance between bias and variance in advantage estimation.
  • Line 11: Target values $V_t^{\mathrm{target}}$ are set for the critic updates.
  • Line 12: The buffer $B$ is augmented with the computed advantages and target values.
• Training Loop (Lines 13-16):
  • The model undergoes E_finetune epochs of training on the augmented buffer.
  • Line 14: A mini-batch $\hat{B}$ of size m_finetune is sampled from $B$.
  • Line 15: Both actor ($\theta$) and critic ($\phi$) parameters are updated using the full PPO objective (Eq. 5) on the mini-batch.
• Reset and Policy Update (Lines 17-18):
  • The buffer $B$ is cleared for the next iteration.
  • $\pi_{\theta_{\mathrm{old}}}$ is updated to the current $\pi_\theta$ to prepare for the next round of rollouts.
• Output (Line 20): The final trained policy $\pi_\theta$ is the fine-tuned model.

4.2.3. Model Architecture and ABR Simulator

• ABR Simulator: The ABR simulator follows the design of Pensieve's Python environment [6] but uses a C++ implementation from [8], [15] for improved efficiency. The state, action, reward function, and state transition are consistent with those defined in Pensieve.
• State Representation: The input features for the model, which are typically represented as a 6x8 matrix in Pensieve and Comyco, are flattened into a 48-dimensional vector for SABR.
• Network Architectures:
  • Actor Network ($\pi_\theta$): A fully connected neural network with the structure [48, tanh, 64, tanh, 64, 6].
    • 48: Input layer size (corresponding to the flattened state vector).
    • tanh: Hyperbolic tangent activation function.
    • 64: Two hidden layers with 64 neurons each.
    • 6: Output layer size, corresponding to the 6 possible bitrate choices.
  • Critic Network ($V_\phi$): A fully connected neural network with the structure [48, tanh, 64, tanh, 64, 1].
    • 48: Input layer size.
    • tanh: Activation function.
    • 64: Two hidden layers with 64 neurons each.
    • 1: Output layer size, predicting a single scalar value for the state value.
• Parameter Sharing: The actor and critic networks do not share parameters.
• Optimizer: The Adam optimizer [33] is used for both DPO and PPO training.
• Parallelization: The Vector Environment module from $StableBaselines3 (SB3)$ [32] is used to enable parallel sample collection across 4 environments, enhancing training efficiency.

5. Experimental Setup

5.1. Datasets

SABR introduces two new benchmarks, ABRBench-3G and ABRBench-4G+, each comprising video content and network traces. The traces are curated from publicly available datasets like Lumos 4G/5G [19], [20], and FCC [6], [21], [22]. Each benchmark is divided into training, testing, and Out-of-Distribution (OOD) sets.

• Training and Testing Sets: Created by proportionally splitting each trace set (e.g., 75% for training, 30% for testing in FCC18). Models are trained on the entire training set with shuffled traces.
• OOD Set: Designed specifically to evaluate generalization to unseen distributions. Trace sets included in the OOD set are not split or reused in other sets, ensuring they are truly novel.
• Evaluation Granularity: Evaluation is performed separately for each trace set within the test and OOD sets to avoid high-bandwidth traces skewing average QoE results and masking performance under other conditions.

5.1.1. ABRBench-3G Trace Statistics

The following are the results from Table I of the original paper:

• Video Content for ABRBench-3G: Uses the Envivio-Dash3 [29] video.
• Available Bitrates for ABRBench-3G ($R^{3G}$): $R^{3G} = \{300, 750, 1200, 1850, 2850, 4300\}$ kbps.

5.1.2. ABRBench-4G+ Trace Statistics

The following are the results from Table II of the original paper:

• Video Content for ABRBench-4G+: Uses the Big Buck Bunny [30] video.
• Available Bitrates for ABRBench-4G+ ($R^{4G+}$): $R^{4G+} = \{1000, 2500, 5000, 8000, 16000, 40000\}$ kbps.

5.2. Evaluation Metrics

The performance of the ABR algorithms is primarily evaluated using two metrics: Quality of Experience (QoE) and Average Rank.

5.2.1. Quality of Experience (QoE)

Conceptual Definition: QoE quantifies the overall satisfaction of a user watching a video stream. It balances high video quality with minimal interruptions (rebuffering) and smooth transitions between quality levels. The goal of an ABR algorithm is to maximize this QoE.

Mathematical Formula: $$QoE = \sum_{n=1}^{N} q(R_n) - \delta \sum_{n=1}^{N-1} \left| q(R_{n+1}) - q(R_n) \right| - \mu \sum_{n=1}^{N} T_n$$ Symbol Explanation:

• $N$: The total number of video chunks in the stream (set to 49 chunks for experiments).

• $R_n$: The bitrate of the $n$-th video chunk.

• $q(R_n)$: A function that maps the bitrate $R_n$ to a corresponding quality score. Consistent with prior work, $q(R_n) = R_n$ is used, meaning the quality score is directly proportional to the bitrate.

• \sum_{n=1}^{N} q(R_n): Represents the total quality obtained from all video chunks. Higher is better.

• $\left| q(R_{n+1}) - q(R_n) \right|$: The absolute difference in quality scores between consecutive video chunks, representing bitrate switches or fluctuations.

• $\delta$: The smoothness penalty coefficient. It penalizes frequent or large changes in video quality. A higher $\delta$ means more penalty for switches. (Set to 1 for experiments).

• \delta \sum_{n=1}^{N-1} \left| q(R_{n+1}) - q(R_n) \right|: Total penalty for bitrate fluctuations. Lower is better.

• $T_n$: The rebuffering time (stalling duration) that occurs at the $n$-th step (chunk download).

• $\mu$: The rebuffering penalty coefficient. It penalizes playback interruptions. A higher $\mu$ means more penalty for rebuffering. (Set to 4.3 for ABRBench-3G and 40 for ABRBench-4G+).

• $\mu \sum_{n=1}^{N} T_n$: Total penalty for rebuffering events. Lower is better.

  A higher overall QoE value indicates better video streaming quality and user experience.

5.2.2. Average Rank

Conceptual Definition: The Average Rank metric provides a consolidated way to compare algorithms across multiple trace sets within a benchmark. Instead of just comparing absolute QoE values, which might be dominated by a few high-performing trace sets, ranking assesses an algorithm's relative performance within each individual trace set. A lower average rank indicates more consistent and superior performance across diverse network conditions.

Mathematical Formula: $$\operatorname { AveRank } ( i ) = \frac { 1 } { M } \sum _ { j = 1 } ^ { M } r _ { i , j }$$ Symbol Explanation:

• $\operatorname{AveRank}(i)$: The average rank of algorithm $i$.
• $M$: The total number of trace sets in the benchmark (e.g., 5 for ABRBench-3G test set, 3 for ABRBench-4G+ test set).
• $r_{i,j}$: The rank of algorithm $i$ on trace set $j$. For each trace set, algorithms are ranked from 1 (best QoE) to $K$ (worst QoE, where $K$ is the number of algorithms).

5.3. Baselines

SABR's performance is compared against a range of established ABR algorithms, including heuristic, optimization-based, and learning-based methods:

• Buffer-Based (BB): A simple, traditional heuristic ABR algorithm that adjusts bitrates based solely on the current buffer occupancy. If the buffer is full, it tries a higher bitrate; if the buffer is low, it tries a lower bitrate to prevent stalling.

• BOLA [35]: Buffer Occupancy-based Lyapunov Adaptation. An optimization-based ABR algorithm that uses Lyapunov optimization theory. It aims to maximize video quality while keeping the buffer occupancy within a desired range to prevent underflow or overflow, making bitrate decisions primarily based on buffer observations.

• RobustMPC [34]: An extension of the Model Predictive Control (MPC) approach. MPC algorithms predict future network conditions and buffer states to make optimal bitrate decisions over a short future horizon. RobustMPC enhances this by considering uncertainties in network predictions, maximizing a given QoE metric (typically over 5 future chunks).

• QUETRA [36]: Queueing-theoretic Rate Adaptation. This algorithm models the ABR task as an M/D/1/K queueing system (representing a single server, deterministic service time, finite buffer capacity). It makes bitrate decisions by estimating future buffer occupancy based on queueing theory.

• Pensieve [6]: The pioneering Reinforcement Learning (RL)-based ABR method. It trains an A3C (Asynchronous Advantage Actor-Critic) policy network using network states to maximize a defined QoE reward.

• Comyco [8]: A learning-based ABR method that improves upon Pensieve by employing imitation learning. It trains its policy by mimicking expert trajectories generated by an MPC controller, thereby enhancing training efficiency and performance.

• NetLLM [12]: An approach that adapts Large Language Models (LLMs) for various networking tasks, including ABR. It uses multi-modal encoding and Low-Rank Adaptation (LoRA) to efficiently fine-tune LLMs for control tasks.

  Each algorithm is executed ten times, and the average performance is reported. For learning-based methods, ten separate models are trained, and their average performance is reported.

5.4. Hyperparameters

The following are the results from Table III of the original paper:

6. Results & Analysis

6.1. Core Results Analysis

The experimental evaluation focuses on demonstrating SABR's improved generalization capabilities and robustness across diverse network conditions, including out-of-distribution (OOD) scenarios. The performance is assessed on the ABRBench-3G and ABRBench-4G+ benchmarks, comparing SABR against a range of baselines. The key metric for overall comparison is Average Rank, where a lower value indicates better performance.

6.1.1. Performance on ABRBench-3G Test Set

The following are the results from Table IV of the original paper:

• Analysis: On ABRBench-3G, SABR demonstrates strong performance, achieving the highest QoE on FCC-16 (36.68), FCC-18 (145.18), Oboe (99.68), and Puffer-22 (40.05). While its QoE on Puffer-21 (36.05) is not the highest (QUETRA is 42.48), its consistent top performance across most trace sets results in the best overall Average Rank of 1.8. This significantly outperforms RobustMPC (3.4), Pensieve (3.8), and other baselines, indicating its superior ability to adapt to varying 3G network conditions.

6.1.2. Performance on ABRBench-4G+ Test Set

The following are the results from Table V of the original paper:

• Analysis: For ABRBench-4G+, SABR again achieves the lowest Average Rank of 1.7. It performs best on Lumos 4G (1309.65 QoE). While Comyco has a slightly higher QoE on Lumos 5G (1835.42 vs. SABR's 1832.14) and RobustMPC on Solis Wi-Fi (589.64 vs. SABR's 576.33), SABR's overall consistent performance across all three trace sets ensures its top average rank. This indicates its robust applicability to higher bandwidth 4G/5G and Wi-Fi networks.

6.1.3. Performance on OOD Datasets

The following are the results from Table VI of the original paper:

• Analysis: This is arguably the most critical evaluation for SABR, as it directly addresses the generalization problem. On the OOD datasets, SABR achieves the lowest Average Rank of 2.0. It yields the highest QoE on HSR (142.20). While Comyco (1595.09) performs slightly better on Lab than SABR (1561.18), and RobustMPC (1075.17) slightly better on Ghent than SABR (1023.56), SABR's overall performance across the OOD sets is superior to all other baselines. This result strongly validates SABR's ability to generalize effectively to unseen network conditions, which is a major advancement over previous methods.

6.2. Overall Summary of Results

Across all benchmarks (ABRBench-3G test, ABRBench-4G+ test, and OOD sets), SABR consistently achieves the best Average Rank. This is a strong indicator of its superior performance, stability, and, most importantly, its improved generalization to diverse and unseen network conditions compared to heuristic, optimization-based, and other learning-based ABR algorithms. The two-stage BC pretraining with DPO and RL fine-tuning with PPO framework successfully addresses the identified challenges of limited generalization and training instability in wide-distribution settings.

6.3. Ablation Studies / Parameter Analysis

The paper does not explicitly present ablation studies to dissect the individual contributions of BC pretraining versus RL fine-tuning, or the specific choice of DPO versus other BC methods. However, the comparison against baselines that represent pure RL (Pensieve) or imitation learning (Comyco) implicitly highlights the benefits of SABR's combined approach. The presented hyperparameters (Table III) show the specific configurations used for DPO and PPO, but a detailed analysis of how varying these parameters affects performance is not included in this paper.

7. Conclusion & Reflections

7.1. Conclusion Summary

This paper introduces SABR, a novel and robust framework for Adaptive Bitrate (ABR) control that effectively addresses the critical issues of generalization to unseen network conditions and training instability in learning-based ABR methods. Inspired by the success of pretraining + fine-tuning paradigms in Large Language Models (LLMs), SABR employs a two-stage approach: Behavior Cloning (BC) pretraining using Direct Preference Optimization (DPO) to establish a stable base model from expert demonstrations, followed by Reinforcement Learning (RL) fine-tuning with Proximal Policy Optimization (PPO) to enhance exploration and generalization.

To rigorously evaluate ABR models, the authors contribute two new benchmarks, ABRBench-3G and ABRBench-4G+, featuring wide-coverage training traces and dedicated Out-of-Distribution (OOD) test sets. Experimental results across these benchmarks consistently demonstrate that SABR achieves the best Average Rank compared to state-of-the-art baselines, including Pensieve, Comyco, and NetLLM. This empirical evidence validates SABR's ability to learn more stably across wide network distributions and significantly improve its generalization performance in unseen and challenging network environments.

7.2. Limitations & Future Work

The authors explicitly state a limitation and future work direction:

• Limitations: The current benchmarks, while comprehensive, are still limited.
• Future Work: The authors plan to extend their benchmarks with more traces and videos to provide an even more comprehensive evaluation platform for ABR research. This continuous expansion is crucial as network conditions and video content evolve rapidly.

7.3. Personal Insights & Critique

The SABR framework presents a compelling and well-motivated approach to learning-based ABR. The inspiration from LLM training paradigms is a clever transfer of a successful AI methodology to a different domain, highlighting the universality of certain learning challenges and solutions.

Strengths:

• Addressing a Core Problem: The paper directly tackles generalization, which is a major hurdle for deploying learning-based ABR in real-world, unpredictable networks.
• Principled Approach: The BC pretraining provides a strong, stable foundation, while RL fine-tuning allows for exploration and optimization. This combination often yields better results than either method alone, especially when expert data is available but full exploration is also desired.
• Novel Application of DPO: Using DPO for BC in ABR is innovative. DPO's stability benefits (compared to traditional RLHF) likely contribute to the robustness of the pretraining phase.
• Valuable Benchmarks: The introduction of ABRBench-3G and ABRBench-4G+ with dedicated OOD sets is a significant contribution to the ABR research community. Standardized, diverse benchmarks are crucial for fair and reproducible evaluation.

Potential Issues/Areas for Improvement:

• Expert Policy Dependence: The BC pretraining relies on a BEAM_SEARCH_POLICY for generating expert actions. The quality and diversity of this expert policy directly impact the base model. If the expert policy has biases or limitations, these could be transferred to SABR. The paper could benefit from a discussion on how robust this expert policy is or how sensitive SABR is to its quality.
• Computational Cost: While DPO is more stable than RLHF, a two-stage $BC + RL$ framework, especially with extensive rollouts, can still be computationally intensive. A discussion on the training time and resource requirements compared to single-stage methods would be valuable.
• Hyperparameter Sensitivity: As with most RL algorithms, PPO can be sensitive to hyperparameters. While a table is provided, an ablation study on key DPO and PPO parameters (e.g., $β$, $ε$, $c1$, $c2$) could provide deeper insights into their impact on stability and performance.
• Real-world Deployment Challenges: The simulator environment is a good approximation, but real-world networks introduce complexities like varying packet loss, jitter, and dynamic user behavior that are hard to capture. A discussion on the gap between simulation and real-world performance, and how SABR might handle these additional challenges, would be insightful.
• Dynamic QoE Metrics: The paper uses a fixed QoE metric. In reality, user preferences for QoE components (bitrate, smoothness, rebuffering) can vary. Future work could explore integrating dynamic or personalized QoE models.

Broader Impact: The success of applying an LLM-inspired pretraining + fine-tuning paradigm to a control problem like ABR suggests a powerful general recipe for AI agents. This could be transferable to other domains where stable initial behavior is crucial, followed by fine-tuned optimization in complex, dynamic environments, such as robotics, industrial control, or other networking tasks. The emphasis on OOD generalization is particularly relevant in an era of increasingly diverse and unpredictable digital infrastructures.