Enhancing Video Streaming Quality: A Comprehensive Analysis of Fugu's In Situ Learning Experiment

paper link: https://arxiv.org/pdf/1906.01113.pdf

Summary:

The paper proposes Fugu an ABR algorithm based on MPC (Model Predictive Control) model, that robustly outperformed other schemes by enhancing the QoE(Quality Of Experience), by leveraging data from its deployment and limiting the scope of ML only to making predictions trained in situ on real data. The system uses supervised learning in situ, with data from the real deployment environment, to train a probabilistic predictor of upcoming chunk transmission times which then provide inputs to a classical control policy. Substantial support from Puffer, which is a video streaming website open to the public, is taken to enhance and monitor the designed ABR because The newer algorithms were developed and evaluated using throughput traces that may not have captured enough of the Internet’s heavy tails and other dynamics when replayed in simulation or emulation. Fugu predictor has features like it predicts transmission time given a chunk’s file size, it outputs a probability distribution and it considers low-level congestion-control statistics among its input signals. Fugu outperformed existing techniques including the simple algorithm in stall ratio (with one exception), video quality, and the variability of video quality. It is observed that users who were randomly assigned to Fugu without pre-knowledge of it chose to continue streaming for 5–9% longer, on average, than users assigned to the other ABR algorithms. Fugu’s predictor outputs not only a single predicted transmission time but a probability distribution of possible outcomes. Fugu is the first to use the concept of TTP (Transmission Time Predictor) and Stochastic MPC and can be trained in supervised learning. To test the efficiency the authors built Puffer, a free, publicly accessible website that live-streams six over-the-air commercial television channels. The idea was to collect data from enough participants and network paths to draw robust conclusions about the performance of algorithms for ABR control and network prediction. Puffer encodes each video chunk in ten different H.264 versions, using libx264 in very fast mode. Puffer then uses ffmpeg to calculate each encoded chunk’s SSIM. Looking at the other ABR schemes, almost each of them lies somewhere along the SSIM/stall frontier in emulation (left side of figure). Authors tested Pensieve and found rebuffering was the least and MPC delivered the highest quality video. In the real-life experiment a more muddled picture, with a different qualitative arrangement of schemes was found. Finally, The Fugu algorithm robustly outperformed other schemes, both simple and sophisticated, on objective measures (SSIM, stall time, SSIM variability) and increased the duration that users chose to continue streaming.

Strengths:

Well, the paper achieved the suggested objective from the author where they want to create a learned algorithm that can face and tackle the wild internet and they achieved it by training the algorithm in situ on data from the real deployment environment, and using an algorithm whose structure is sophisticated enough and yet also simple to benefit from that kind of training.

Weaknesses:

The research paper in question exhibits a conspicuous weakness in its assessment of emulation-trained Fugu's real-world performance. It presents a perplexing scenario where, despite rigorous training in a controlled environment, emulation-trained Fugu's performance in actual usage falls drastically short when compared to both in situ Fugu and other Adaptive Bitrate (ABR) schemes. This disconcerting revelation casts doubts on the practicality and efficacy of the emulation training approach. Furthermore, the paper delves into the intricate challenges associated with Reinforcement Learning (RL), highlighting the formidable requirement for a training environment that can effectively respond to control decisions and discern consequential rewards. The inherent statistical noise and the immense variability of inputs pose formidable obstacles to the practicality of RL in this context, making it a method of exceedingly slow learning or necessitating an extensive deployment of algorithms for training. Consequently, it underscores the imperative need for future research to explore techniques aimed at mitigating this variability and developing more precise simulators and emulators that can capture the multifaceted dynamics of the real Internet, including aspects such as varying Round-Trip Time (RTT), cross-traffic patterns, and the intricate interplay between throughput and chunk size. Without addressing these deficiencies, the research remains tantalizingly incomplete in its quest to unravel the underlying drivers of the observed phenomenon, leaving the industry in a state of uncertainty regarding the optimization of video quality and user experience.

Applicability:

The discussed shortcomings in emulation-trained Fugu's performance raise suspicion about its immediate execution in real-world scenarios, suggesting a need for further refinement and optimization