Crowdsourcing Framework for QoE-Aware SD-WAN

doi:10.21203/rs.3.rs-31021/v2

Download PDF

Research

Crowdsourcing Framework for QoE-Aware SD-WAN

https://doi.org/10.21203/rs.3.rs-31021/v2

This work is licensed under a CC BY 4.0 License

Journal Publication

published 14 Aug, 2021

Read the published version in Future Internet →

Version 2

posted 13 Jan, 2021

You are reading this latest preprint version

The exponential increase in bandwidth-sensitive multimedia traffic on the Internet has created new challenges and opportunities. With the shifting focus from service availability to service quality, there is a need to have quality management measures to serve the high needs of efficient transmission and delivery in time-constrained environments over IP networks. Quality of Experience (QoE) is now considered the most important measure to achieve the twin goals of application efficiency and user satisfaction from a user perspective. In this paper, we propose a framework that can be used to collect real-time QoE feedback through crowdsourcing and forward it to SD-WAN controllers to enhance streaming routes based on real-time user quality perceptions. We analyze how QoE can be affected by different network conditions, and how different streaming protocols compare against each other when the network parameters change dynamically. We compare the real-time user feedback to predefined network changes to measure if participants will be able to identify all degradation events, and to examine which combination of degradation events are noticeable to the participants. These QoE timestamped feedback is sent back to the SD-WAN controller continuously in order to locate problems and bottlenecks in the current service paths and to enable network controllers to take corrective action by rerouting the streamed traffic. Our aim is to demonstrate that real-time QoE feedback can enhance cloud-based services and can adjust services quality based on real-time, active participants’ interaction[1].

[1] This paper extends the preliminary results presented by the authors at the PVE-SDN 2019 workshop in IEEE Conference on Network Softwarization (NetSoft 2019).

Technical Communication

Systems and Networking

QoE

SDN

QoS

crowdsourcing

A. Background and Motivation

Real-time multimedia content streaming over the Internet has gained prime applications in several industries such as communication, education, interactive gaming and entertainment. The main bulk of Internet traffic nowadays is multimedia content, in particular on-demand video and live video streaming.

The multimedia service can be characterized by: 1) the content it supplies, 2) the transmission methods it uses to supply this content, and 3) the services it employs to enable content exchange between different parties. Multimedia service design also takes into consideration the user requirements, which include: cost of the service, ease of content accessibility, content quality and multimedia desirability [2]. Furthermore, the Internet usage has shifted towards content-centric rather than host-centric. At the same time, user expectations are also continuously elevating and the multimedia content providers are becoming more aware of the importance of service quality.

With the emergence of high functioning mobile devices, network service providers are in continuous efforts to support a wider range of applications and quality of service (QoS) requirements with highly utilizing network capacities [1]. Traditionally, service providers evaluated the quality of multimedia streaming by focusing on network conditions and its corresponding QoS parameters such as delay, jitter, bandwidth and packet loss. However, more recently the attention has shifted toward Quality of Experience (QoE), which is a more subjective and user-centric assessment technique that is concerned with the user perception of the service. Once mobility is introduced, packet delays and losses in multimedia streams become more common and maintaining QoS becomes more challenging. As such, QoE-based assessments are quickly becoming the guidelines for managing user quality expectations.

Most past research contributions in this area have focused on the quality of video streaming, video analytics, network QoS, and metrics for conducting QoE. In those works, QoE is usually measured and assessed after completion of streaming session/or paired comparisons, and results are sent back to management systems to make changes for enhancing the user QoE in future streaming. However, current proposals often do not provide details on how such mechanisms are deployed and used in a network in real time; e.g. how QoE feedback can be gathered in real-time and how it should be communicated to the network controllers to allow dynamic network changes to enhance QoE and streaming quality while participants are observing the session. This is the research gap that we are attempting to address in this work, with a particular focus on deployment in a software-defined transport network

In terms of transport services, Real-time Transport Protocol (RTP), along with Real-Time Control Protocol (RTCP) or Real-Time Streaming Protocol (RTSP), provide a reliable foundation for real-time services [3]. The emergence of Software-Defined Networking (SDN) has promised better control and management of end-to-end service quality in the networks [4]. Leveraging SDN’s advantages such as dynamic programmability, central control, cost efficiency and adaptability to changes in networking environment, makes the Software-Defined Wide-Area Network (SD-WAN) a desirable architecture to control QoE for multimedia streaming applications and services. With real-time user feedback during live video streaming, it will provide a better QoE by trying to enhance streaming routes using SD-WAN controllers. Furthermore, SD-WAN controllers incorporate the emerging concept of intent-based networking (IBN), which would allow network administrators to define their QoE requirements in form of service intents; i.e. collection of policies, goals and network-level performance targets rather than specific device-level configuration parameters. Such implementation greatly facilitates the management of service-level QoE requirements for multimedia service delivery.

We propose a framework based on a combination of real-time QoE measurement application and QoS quality parameters which can accommodate a variety of streaming protocols while deployed in an SD-WAN. This framework emphasizes the correlation between QoE and QoS and how the overall user QoE perspective can be affected. We study how dynamic changes in the network could affect the performance of different streaming protocols, and how the streaming protocol adjusts to network changes, and consequently, the perceived QoE of the streaming content. The main protocols used in our model are RTP, RTP over TCP, SCTP, and UDP.

The proposed framework is based on real-time alerts of quality degradation from users’ side during live video streaming over a cloud-based SD-WAN environment. A QoE-rating application is deployed on the user end devices to collect user feedback during a real-time streaming session. Through this application (which can be deployed as a plugin on web browsers or multimedia players) users send individual or cumulative feedback to the SD-WAN controller (directly or indirectly through a data summarization gateway) to inform the controller about potential problems in the video streaming. This QoE feedback would enable the SD-WAN controller to investigate problems in the service path and to take corrective action by making changes to the topology, resources or routes of the content delivery network. The main feature of our proposed framework is the ability to identify changes in the QoE of video streams with dynamically changing network conditions in real-time.

B. Contributions

The list of contributions in this paper includes the following:

Design of a four-tier framework for QoE-aware content delivery over SD-WAN (Section 4)
Performance Analysis of SD-WAN transport protocol candidates for multimedia delivery, through a set of Human-Paired Comparison (HPC) experiments (Section 6.A)
Performance analysis of real-time SD-WAN re-routing under quality degradation in an intent-based network environment (Section 6.B).

The experiments presented in this research aim to answer the following questions:

1) Will the participants be able to identify quality degradations at the same time as the network changes?

2) Does the content of the video affect quality-rating decisions by the participants despite a noticeable quality degradation?

3) When comparing objective analysis against Human perceived quality, would the results be consistent?

4) Which protocols are most suited for unstable or rapidly changing networks?

5) Is there consistency in protocol performances when video content changes, or when different events and scenarios occur?

For each case, we present an analysis of the experimental results and reflect on how they address the relevant question from the above-mentioned list.

C. Paper Outline

The rest of this paper is organized as follows. In Section 2, a review of the relevant literature is presented. In Section 3 we present an overview of the SD-WAN network and QoE measurement models. In Section 4 we present our crowdsourcing QoE-Aware design. In Section 5, we will describe the implementation of our test bed. Section 6 discusses performance results and an analysis of the proposed model. Section 7 provides conclusions and directions for future work.

A comprehensive discussion of network and service performance indicators for multimedia applications have been presented in [2]. The most important performance indicators include the following:

• One-way end-to-end delay (including network, propagation and equipment) for video or audio should be within 100 to 150 ms.

• Mean-opinion-score (MOS) levels for audio should be within 4.0 and 5.0. MOS level for video should be between 3.5 and 5.0.

• End-to-end delay jitter must be short, normally less than 250 μs.

• Synchronization of intermedia and intramedia should be maintained using suitable algorithms. To maintain intermedia synchronization, differential delay between audio and video transmission should be within −20 ms to +40 ms.

• The following parameters should also be taken into consideration while designing a QoE framework for multimedia services are described in [5]:

Video quality at the source.
How the content is delivered over the network and QoS quality parameters.
User perception, expectations and ambiance.

More recently, crowdsourcing techniques have been considered for collecting user QoE feedback. In [6], authors designed a crowdsourcing framework that overcomes some of the disadvantages of MOS technique, namely: 1) difficulty and inconsistency for participants to map their ratings to 5-point MOS scaling, 2) rating scale heterogeneity, and 3) the lack of a cheat detection mechanism. By introducing the ability to have QoE measured in a real-life environment using crowdsourcing rather than a controlled environment in a laboratory, this new approach provides comparable consistency as of the MOS methodology. Another approach, OneClick framework [7], captures user perception in a simple one-click procedure where experiments are held to gather user feedback and then collected data are processed to calculate the accumulative QoE of all users. Programmable QoE-SDN APP was discussed in [8], which aims to improve QoE for video service customers by minimizing the occurrence of stalling events in HTTP Adaptive Streaming (HAS) applications, and by utilizing the forecast and rating estimates provided by mobile network operators.

In order to tackle the requirements of multimedia over IP, multimedia services should have the ability to classify traffic, prioritize different applications and make the necessary reservations accordingly. The Internet Engineering Task Force (IETF) developed an Integrated Service framework that consists of real-time and best effort services. RTP, along with RTCP, and RTSP, provide a reliable foundation for real-time services. However, this framework has had limited deployment due to complexity and backward compatibility.

Some past research efforts have focused on the specific use of SDN controllers and the importance of the selection of SDN controllers in designing network models. Recently, research done in [9] focuses on using intent-based programming in Open Network Operating System (ONOS) [10] to allow more dynamic monitoring and rerouting services by using intents. Intent Framework [11] enables applications to provide network requests in form of a policy, not as a mechanism. Intents provide a high-level abstraction where programmers only focus on the task that should be accomplished, rather than how these tasks will be translated into low-level rules and how these rules can be installed into network devices, by only expressing required intentions via high-level policies. Those researches aim to enhance Intent Framework to compile more than one intent at the same time and to re-optimize paths based on flow statistics.

Leveraging SDN in routing would allow service providers to customize routing services for applications [12]. This approach is based on a new open framework called Routing as a Service (RaaS) by reusing virtualized network functions. Upon selecting appropriate functions, the authors build customized routing services on the routing paths for different applications.

Several prior works have examined the possibility of managing QoS and QoE using the advantages of SDN architecture. In [13, 14] the authors focused on how QoE could be managed efficiently over cloud services, and investigated the challenges facing QoE management in cloud applications, especially the quality of multimedia streaming. The goal of QoE management in that environment was to provide high quality services to user on the cloud while taking into consideration relaying costs behind such quality.

Implementing QoS over SDN in [15], the authors designed an approach to introduce QoS into IP multicasting using SDN in order to have proper and flexible control management of the network environment. OpenFlow protocol was adopted to allow a controller to monitor IP multicasting statistics for each flow to provide end-to-end QoS. They implemented a learning algorithm to allocate required network resources without dropping those low priority packets which could impact the performance for those flows. It thus demonstrated that SDN could be used for network quality management. The next section will discuss past contributions in the area of QoE Models.

A. In-Network QoE Models

In [16], the authors proposed an In-network QoE Measurement Framework (IQMF). The user feedback is not considered as an input parameter in this scheme, however the streams are being monitored within the network. Two QoE metrics are adopted by IQMF for measuring experience: 1) quality of video and 2) switching impact over HTTP Adaptive Streams. IQMF offers those measurements for QoE as a service through an API (Figure 1). This service could be provided to a content distributor or a network provider. By leveraging SDN, it allows control plane to interact with IQMF framework allowing it more flexibility to analyze and measure participant’s QoE. It also enables IQMF to utilize traffic managements dynamically and provide scalability for deploying more measurement agents. IQMF interacts with OpenFlow controller that keeps the forwarding behavior of the network to ensure that all necessary information about flow duplications is provided to allow better monitoring of QoE.

The QoE measurement framework operates by filtering HTTP packets in the traversing traffic. It then identifies HTTP GET requests, and examines those requests for identification of the manifest files such as Media Presentation Description file (MPD). MPD parser extracts information from the MPD file - different representations that includes references to different resolutions, quality multitude and playback codecs. The measurement engine then merges the parsed information with supplementary details from the HTTP packet filter in order to monitor the behavior of the user whilst playback continues.

Another model, described in [17], aims to enhance the capabilities of Dynamic Adaptive Streaming over HTTP (DASH) – a standard for multimedia streaming that changes the quality of content presentation automatically in accordance with network conditions. In this research, authors take into consideration the QoE as perceived by the user, and then integrate user perception with dynamic changes in the content. Such enhancements will provide more efficient QoE measurements and increases positive feedback by the users. This model allows automated estimation of MOS measurements from QoS parameters such as video bitrate, video frame rate and video quantization parameter.

The following three metrics were used in this model:

1. Buffer underflow/overflow: to prevent freezing images and losing packets, buffer thresholds were specified. TCP has been used as well for reliable transmission.

2. Frequencies and amplitude of switching quality: the frequency of quality switches of the represented content was identified as one of factors affecting QoE.

3. QoS media parameters: the parameters associated with the content of media.

It was found through experiments that it takes seconds to measure presentation intervals that are affected by media parameters. However, it takes more time intervals in terms of switching quality and re-buffering impacting QoE. Where, “the representation quality switch rate, required a recursive approach where the MOS is calculated based on previous MOS variations in order to take into account the entity of the quality switch in addition to the rate” [17]. This model has shown potential in enhancing DASH logic of adaptation capabilities for selecting the best video quality levels, through integrating the QoE monitoring.

OpenE2EQoS model, discussed in [15], aims to introduce QoS into IP multicasting using SDN to have a flexible control management of the network environment. In this approach, OpenFlow protocol was adopted to allow controller to monitor IP multicasting statistics for each flow to provide end-to-end QoS. The system makes use of the Additive-Increase / Multiplicative Decrease (AIMD) algorithm to enhance adaptive learning of efficient bandwidth utilization over time to improve QoS. An N-dimension statistical algorithm is used in that approach to redirect low priority traffic packets from overly crowded links while maintaining priority for multimedia packets.

In [18] a method has been proposed for predicting QoE by using machine learning algorithms leveraging SDN. An architecture is designed that uses previously-measured MOS values from users, collected during different network conditions. This data along with objective measures are supplied to machine learning algorithms to predict MOS values for the current network conditions. The SDN QoE Monitoring Framework (SQMF) [19] is a monitoring application that aims to preserve QoE for both video and VoIP applications in real-time regardless of unexpected network issues, by continuously monitoring network parameters and using QoE estimation models. In [20] a new QoE-Aware management architecture over SDN was proposed, which was able to predict MOS by mapping different parameters of QoS into QoE. The proposed framework was designed to autonomously control and allocate underlying network resources infrastructure with the ability to avoid QoE degradation, optimize resource use and improve QoS performance.

B. Crowdsourcing QoE Models

A general crowdsourcing framework for QoE capture was discussed in [6]. The objective in that work is to overcome some disadvantages of MOS technique by utilizing paired comparison technique, as well as the ability to have QoE measured in a real-life environment using crowdsourcing rather than a controlled environment in a laboratory. Four case studies were conducted using audio and video content to evaluate the effectiveness of the proposed framework.

The key features of using this framework for QoE evaluation is:

It could be generalized for different types of multimedia content with no need for adjustments.
Use of pair-comparison rating technique that provides a simpler user feedback comparing to MOS technique
Results from compared judgements can be evaluated by probability models
Using reward and punishment where users are given appropriate incentives to give honest feedback in order to obtain trustable quality measures.

This framework is a promising evaluation technique to measure QoE, however as the authors of the study pointed out, this is not a QoE evaluation; rather, it is Quality of Perception (QoP). As they point out, “QoP reflects a user detectability of a change in quality or the acceptability of a quality level” [6].

The OneClick Framework [7] captures user perception in a simple one-click procedure. Whenever a user is not satisfied with the quality of the viewed content, they can click on a button that informs the system of their dissatisfaction. In contrast to MOS technique, a user does not have to decide between different grading scales and what best suits their perception. OneClick is a real-time framework which means the clickable button is available along the whole viewing experience. The user can record their dissatisfaction several times along the process where each click is time captured. This framework is based on PESQ (Perceptual Evaluation of Speech Quality) and VQM (Video Quality Metric), where both techniques are objective measurements.

The key advantages of OneClick Framework:

Initiative: Participants are not required to decide about the perceived quality, however, they only report their dissatisfaction through one click button
Lightweight: Framework doesn’t require any specific deployments nor expensive to roll out.
Efficient: The user can record their dissatisfaction several times along the test, accordingly, enough to know participant perception
Time-aware: Participant can record their dissatisfaction several times along the process where each click is time captured, which indicates how perception can be changing over time.
Independent: OneClick can be used in conjunction with several applications and not limited to a specific one.

OneClick includes two main steps: 1) experiments are held to gather users’ perception feedback during different network conditions. 2) Collected data are then processed to identify QoE measurements. Figure 2 shows the full OneClick process assessment technique with the following steps: 1) Preparing test materials (optional); 2) asking subjects to do experiments; 3) inferring average response delays; 4) modeling the relationship between network factors and click rates; 5) predicting the click rate given each network factor; 6) summarizing an application’s QoE over various network conditions by comfort region.

Given the simplicity of OneClick’s approach to user feedback, we have used a similar idea for the user side of our framework; namely to allow the users to express their displeasure with a click when the quality of the streaming deteriorates.

A. Network Model

A real-time cloud-based content-delivery network model is shown in Figure 3. The network model is based on four tiers: The content provider tier produces the multimedia content for distribution over the network. The content can be live stream or broadcast, music, video on demand etc., though live video content is our main focus in this paper. Content is made to be accessible and processed by the various servers and forwarding modules in SD-WAN in the next tier.

SD-WAN consists of a collection of interconnected network switches, controlled by one or more centralized controller(s) through a southbound protocol such as OpenFlow. The OpenFlow protocol is an open-source, standard protocol that governs communication between the SDN controllers and switches. OpenFlow allows full control over packet flows where the controller specifies the routing paths as well as how packets should be processed.

The end-user tier contains the wide range of end devices that are used for accessing the content and provide continuous, real-time feedback of QoE in form of perceived video quality through SD-WAN. The end-user tier connects to SD-WAN through a number of gateway servers which form a data summarization tier and are responsible for delivering the content, as well as collecting and summarizing the crowd feedback to the controllers. We will describe the role and functionality of each tier in more detail in the following sections.

The software-defined centralized control plane in SD-WAN allows efficient traffic engineering to meet dynamic service requirements. An SD-WAN controller can have a global view of the network, making it an ideal physical substrate for cloud-based content-delivery network (CCDN) environments. Implementing QoE applications over SD-WAN allows us to further enhance the quality of streaming by adjusting video quality based on user feedback, not just by relying on QoS SLA.

B. QoE Model

There are two main approaches for QoE assessments: subjective and objective methods. Subjective techniques are based on user interaction and feedback. As discussed earlier, a common subjective approach is Mean Opinion Score (MOS) based on a quality rating system on a scale from 1 to 5, where 1 stands for ‘Bad’ and 5 stands for ‘Excellent’. 3.5 is the minimum acceptable threshold for a video MOS [5]. MOS scaling still may give room for inaccurate representation of user perception [6], due to the non-similarity of the scale interpretation by different participants.

While subjective approaches are considered to reflect the user perception more accurately, they are expensive to roll out because such QoE assessments require a large scale of participants in order to obtain reliable results. They are also time-consuming, as traditional QoE experiments are conducted in controlled lab-environment, making it difficult to collect sufficient results from experiments in a limited time-frame [6]. QoE crowdsourcing techniques have been proposed to overcome these constraints; where taking advantage of employing a diverse group of online participants, getting subjective results becomes relatively cheaper and more efficient than traditional ways. Crowdsourcing allows subjective measures based on both video-pair comparisons as well as MOS-based rating comparisons, with the flexibility to choose participants’ demographics if certain demographics required for specific results.

Despite their scalability, crowdsourcing experiments lack supervision, which makes some results not fully trustable. Researchers should be able to identify trusted and untrusted participants. This can be achieved by designing the crowdsourcing campaigns based on certain best practices. These best practices are majorly concerned with technical implementation aspects of the experiment, campaign and test design and thorough statistical analysis of results [21]. Campaigns should be simple enough for participants to understand how the experiment is designed and what is required to complete it.

As opposed to subjective assessment techniques, objective QoE assessment techniques are mostly based on network analysis and technical comparisons that aim to produce a quantitative assessment. Quantitative QoE assessment is tightly related to the QoS of an application or service. Peak-Signal-to-Noise-Ratio (PSNR) is considered an objective approach for measuring quality, as it assesses how much similarity exists between two different video images. It is widely used in video streaming assessment, where the higher PSNR value the higher similarity between the original and received video images. One drawback of PSNR is that it does not take into consideration how human perception works.

Structural Similarity Index (SSIM) [22] is another measurement approach for estimation of perceived visual distortion based on the structural distortion of the video. SSIM addresses the shortcoming of PSNR by combining other factors such as contrast, luminance and structure similarity and compares the correlation between the perceived video images and the original video images, hence considered a Full-Reference model. The higher the ratio, the higher the structural similarity. Another objective approach is Video Quality Metric (VQM) [23], which is a metric to measure the perception of video quality as closely as to human perception. The VQM metric is designed to be a general- purpose quality model for a range of video systems with different resolutions, frame rates, coding techniques and bit rates. VQM measurement takes noise, blurring, and block and color distortions into consideration. VQM gives an output value of zero if no impairment perceived, and increases with a rising level of impairment.

A. QoE Crowdsourcing

The proposed framework uses real-time QoE crowdsourcing feedback of quality degradation during live video streaming over a cloud-based SD-WAN environment. Figure 4 highlights details of our framework, in which a streaming server transmits multimedia content over the SD-WAN environment. During the video streaming, dynamic network changes and events may affect the perceived quality at the user end. Several streaming protocols can be considered such as RTP, SCTP, TCP and UDP for data transmission. The SD-WAN is responsible for routing the video stream and delivering it to the participating users. A QoE-rating application is deployed on the user-end devices to return user feedback during a real-time streaming, through which the participants click on a dislike button when they feel the quality has been degraded.

The QoE-rating feedback application is designed to send REST API requests (if needed, could be summarized through intermediate servers for scalability) to the SDN controller to inform the controller about potential problems in the current video stream and possible request of corrective actions such as traffic rerouting. The QoE rating application can be deployed as a plugin on web browsers, multimedia players, or as a desktop application. In contrast to the MOS technique, participants do not have to decide between different grading scales; instead, they only alert the SD-WAN controller to quality degradation, therefore providing more decisive feedback. The timing between feedbacks could be an indication of quality; i.e. more frequent feedback (“dislike” clicks) indicate a lower quality than less frequent feedback. As such, all clicks are timestamped before transmission. The intermediate servers will collect this feedback and provide summaries (e.g. number of dislikes within a given time interval) to the QoE-control algorithm on the SD-WAN controller for potential actions.

Providing QoE feedback would enable the SD-WAN controller to detect problems in the service path and to take corrective action by making changes to the virtual topology of the content delivery network, reassigning users, or rerouting traffic. Ideally, a resource optimization algorithm such as [24] can be executed in real time to respond to QoE degradation, however the complexity and processing time of such algorithms must be considered in order to provide an effective remedy in real time. The use of crowdsourcing would provide a more scalable and efficient method for collecting feedback, whether to correct real-time problem
or for creating performance benchmarks, as we shall explain in the next sections.

The SD-WAN QoE-aware rerouting mechanism operates as following:

The transport route on SD-WAN experiences delay or loss on specific network links, causing quality degradation in delivery and resulting in delay, jitter or loss.
when the users notice the video quality degradation, they click ‘Dislike’ button. The feedback is transmitted to the Feedback server through HTTP.
Server communicates the feedback to the SDN controller
Controller resolves client and server IPs into MACs using REST.
Controller retrieve current intent information
Controller queries current traffic path.
Controller computes alternative paths between client and server
Controller installs the modified intent and query new path.
New streaming path established and QoE enhancement is verified through user feedback (i.e. lack of dislike feedback).

B. Data Summarization

The issue of scalability is an important challenge in the use of crowdsourcing for quality of service control. Certain streaming content services may have millions of users at any time, and receiving and analyzing feedback from them in real time could become a bottleneck. In order to address this issue, we add two elements in our design:

1. An intermediary data summarization layer is placed between the end-user tier and SD-WAN to implement a hybrid fog computing operation. The nodes in this layer are responsible for receiving feedback from users in their region, summarizing the feedback and sending it back to the network controller(s). For instance, this feedback could include the number of dislikes received over a reporting period. The number and location of these intermediary nodes can be optimized to accommodate any processing limits at the controllers.

2. Furthermore, a minimalist approach is used in the design of the user feedback data. As discussed in Section 3, our framework relies on negative feedback (dislike button) that is only given when there is a problem. Therefore, no feedback is expected in all regions and all times as long as the streaming quality is acceptable. When a congestion or failure scenario causes a spike of negative feedback in a region, the intermediary-layer server in that region will collect the feedback and send a summary report to the central controller. This approach will allow for service scalability to a wide area network.

It must be noted that QoE response time is inherently different from QoS responses. In case of user-initiated QoE, the feedback is controlled by user actions that typically span a few seconds. QoE-correcting actions from the controller can also be executed within a similar time scale, and as such, the limiting impact of propagation delays is less significant in QoE-aware services, as opposed to QoS-aware systems

C. Use of the proposed framework for performance baseline

Aside from real-time response to QoE feedback, this framework can also be used for planning and performance analysis of future streaming. In such scenario, the video stream is saved under different protocols and network conditions to measure QoE via user participation. These saved videos are used to construct a paired comparison stimulus on any chosen crowdsourcing platform in a campaign to be advertised to users asking for participation. Such campaigns include instruction of requirements for participation, consent forms, and terms of any designed stimuli. It includes processed videos and paired comparison stimuli are constructed accordingly. Figure 5 illustrates the crowdsourcing pair comparison model design.

Paired comparisons can be seen as an alternative for the MOS-based method, due to its simplicity where the need to decide between five different ratings is eliminated, making it more intuitive judgment. The paired comparison technique makes it easier for users to express their opinions, decision making and have an easier experience interaction when multiple factors are applied.

In order to build the performance benchmark, participants are hired to rate videos in a paired comparison stimulus with different streaming protocols to provide feedback on which video has the higher QoE. In any crowdsourcing model design, there must be a method to identify reliable trustworthy participants since experiments are not controlled and lack monitoring that traditional controlled lab environments provide. To verify the reliability of the participant, trap questions must be introduced. Participants are presented with a golden question, where a stimulus constructed between the original video with no applied processing as a reference and an event degradation processed video. If participant didn’t rate original video with higher QoE, then the participant is considered unreliable and their results are excluded. A campaign is run over a period of time depending on the number of participants required to conduct the study. Rating scores can be computed at any point during the campaign to show trends and results. Different methods can be used to compute these scores such as Crowd Bradley-Terry Model [25].

A. Emulation Environment

We created an SD-WAN environment using Mininet and ONOS remote controller, which provides an emulation of a software-defined virtual network similar to a real networking environment, running kernel, switching and application code in one single virtual machine using a simple in-line code. ONOS also supports the concept of network intents, which allows the service requirements to be defined in form of policies. VLC server was used as the video streaming application. User feedback was collected through a custom-designed plug-in for VLC client. For the purpose of these tests, we integrated the role of the intermediary layer into the hosts; i.e. assumed that the feedback response from the hosts represent a summarized feedback from the region they represent. We created the network intents using ONOS IFWD application reactively.

B. Use case scenarios

Two experiments were conducted using human participants, one based on paired video comparison using QoE crowdsourcing technique and the second based on real-time feedback during video streaming in a controlled lab environment. The purpose of these experiments was to attempt to provide answers to the research questions that were posed in Section 1; i.e. to have a better understanding of how quickly and reliably the participants can identify quality degradation; how do their results compare to objective analysis of videos, and how different transport protocols perform with regard to QoE of multimedia content.

The first experiment, Human Paired Comparison (HPC), was tested using a QoE crowdsourcing campaign of a paired video comparison on a set of processed videos using subjectfy.us web service [26]. We created comparable video sets based on four selected HD videos. Each video is 40 sec in time, which allows sufficient time for introducing quality degradation and user reactions without being too long. The videos also vary in bitrate and frame rate as well as in content type, so that the analysis of results for research questions 1-3 can be reasonably separated from video quality and content. Details of these videos are presented in Table 1.

We also created 5 random scenario files with changing delay and loss events every 10 or 5 sec shown in Table 2, and applied RTP, Legacy UDP and RTP over TCP as streaming protocols in different scenarios. These processed videos were used as the golden question for verifying the reliability of participants in comparing original HD video against processed (deteriorated) video. If participants chose the processed copy, then their feedback results were considered as untrustworthy. The choice of different transport protocols and network conditions allow us to evaluate results for research questions 4-5.

For the second experiment, we created random scenario files of 40 sec in length with timed changing events every 0.5 sec during the video streaming session. We applied these scenarios to 3 HD-Quality videos, and applied RTP, Legacy UDP, SCTP, RTP over UDP and RTP over TCP as streaming protocols with a result of 150 processed videos. Using MSU-VQM objective analysis tool [27], we computed comparative results against the original videos for PSNR, VQM and SSIM.

For these experiments, we created a network topology consisting of one controller, one switch, and two hosts. By using minievent module script [28] we applied changes to adjust delay and loss over a period of time during video real-time streaming, in order to measure the impact of changes in network conditions on QoE. The parameters of our event scenarios were selected to check how we considered three network change scenarios:

1. Fixed link delay and changing packet loss over time

2. Switching from high delay to a substantial lower delay with a fixed packet loss

3. Gradually decreasing delay over time and then gradually increasing it while applying packet loss.

The selection of these particular scenarios was to allow us to evaluate how different transport protocols and user perception respond to abrupt or gradual packet loss or delay. In our experiments, link delay was set in the range of [0-80ms] and packet loss was set in the range of [0 or 1%]. The flow bandwidth was fixed at 50 Mbps. The values of link delay, loss and bandwidth were chosen through trial and error to generate sufficiently large quality degradation at the user end, and had no other significance in our study. We created scenario files of 40 sec in length with timestamped events where delay and loss were changed during the video streaming session. For each scenario, we streamed the videos with different streaming protocols, recorded all output videos and ran a comparative analysis.

In order to examine the performance of the real-time QoE feedback on SD-WAN operation, we also set up an SDN-based re-routing experiment to demonstrate that QoE feedback can be captured and processed in real-time during live video streaming session. Feedback can be sent on-spot to the SDN controller to alert the controller of issues in the steaming service. Our intention was to demonstrate how traffic reroute could be done instantly based on QoE feedback and how the results would compare from the perception of a participant.

For this scenario we created a network of three hosts: one acts as a streaming server, the second acts as a client and the third an un-namespaced server which can communicate with ONOS SDN controller. The hosts were connected through a network of 10 OVS switch devices with 22 links and 50 flows, where three edge-disjoint paths exist between the client and the server. The choice of this network topology and the number of flows were to allow the network controller to discover alternate paths without having to solve a full optimization problem (which was not part of this study and is considered for future work). RTP used as default streaming protocol. The network topology is shown in Figure 6.

During video streaming experiences, we applied network degradation parameters which affect the quality of the network to indicate if and by how much the participating user will be able to detect such changes during video streaming, and how these changes affect their QoE. Furthermore, we intended to see the timing of their QoE feedback within time window when QoS parameters change events occurs. And how streaming protocols will be able to adapt to these changing events, and if all or some of these changing events will be noticeable to participants or not.

A. HPC Experiment Analysis

For the HPC experiment, there were 2430 paired comparison questions with a total of 259 participants. 243 participants were successful and 16 failed the reliability test. Ranks were computed based on Crowd Bradley-Terry model [19]. Figure 7 shows participants ratings for Video 1 for each protocol and scenario of events. We collected similar results for other videos in Table 1.

Using MSU-VQM, we computed VQM for processed videos from experiment 1 against the original video. Table 3 shows VQM values for each resulted video. By comparing subjective and objective results, we found that HPC and VQM results were consistent, where mostly the highly ranked protocols in HPC had the best (lowest) VQM value across the three protocols per video. This result demonstrates that, in response to the research question 3, objective analysis and human perceived quality do provide consistent results.

Table 4 shows the highest ranked protocol in HPC against best VQM value for each video across different scenarios:

Among all the events and scenarios that we studied in the HPC experiment, RTP on average had the highest rating. Same results were concluded from VQM values; i.e. RTP had the lowest VQM values, followed by TCP. UDP only stood out with better values and ranking in Video 3. For Video 1, it was noticed that TCP has highest VQM values across all scenarios except in scenario 5, with a very low difference against RTP, however in HPC, it was ranked either 2nd or first with very close rating difference to RTP. UDP on the other hand had the lowest rating among all scenarios in Video 1. It was noted during Video 2 analysis that, there was a high-ranking gap between RTP and the other two rated protocols in HPC experiment, however this gap was not noticeable in VQM values. UDP was ranked the lowest among 3 scenarios out of 5 in Video 3 at HPC ranks, and in VQM it had highest values in only 2 scenarios. For Video 4, TCP ratings in HPC are consistent with VQM calculated values.

We note that the sequence of events in the scenarios in Table 2 was designed to allow us to monitor how protocols differ in recovering from packet loss and delay (research question 3); how the QoE will be affected; and if participating users will detect changes with one parameter fixed and the other changes (research questions 1 and 2). It was found in our experiments that for the packet loss events the quality degradation is most noticeable, however changing delay parameters in some cases was not noticeable by participants, likely due to the buffering ability of the video player on the client side.

B. SD-WAN Re-routing experiment Analysis

The SD-WAN rerouting experiment was conducted in a controlled lab environment in which each participant watched a 3-minute live streaming video and was asked to provide QoE feedback in form of a ‘dislike’ click when quality degrades based on timely changed scenarios of event. Participants could provide feedback clicks any time during the viewing session. In our experiment, due to limited number of participants, we assume that the response from a participant represents a summary of feedback from a group of users; i.e. each user essentially represents a summarization server. As such, a 10-second waiting time was enforced between acceptance of two consecutive clicks to better represent the summarization and also to ensure that the controller’s rerouting is complete, thus avoiding a ping-pong routing problem.

Figure 8 shows the timing of the vents for QoE-aware rerouting experiment for a sample participant, where red dots represent ping time (right axis), blue line is iperf loss percentage (left axis), green bars are VLC player errors and the gray background indicate active minievents scenarios. We observe that the streaming quality improved after every participant feedback click, where it was noted that the blue line indicated no further loss on the streaming link. It was noticed that when applying a typical delay without loss, users did not detect quality degradation and they typically do not click the feedback button, primarily due to the playback buffering capabilities of the VLC player. With gradually increasing packet loss% with minimum delay, the QoE was affected immediately and accordingly the feedback for reroute was received. By applying degrading changes on two paths out of three, it was found that ONOS kept searching for the alternative routes with every click until the quality was acceptable for the user.

With all participants, it was found that ONOS managed to reroute traffic based on users’ feedback, and the quality of the stream was enhanced after controller’s enhancement action. Among all participants, ONOS reaction and response for rerouting decision was consistent and timely. It took ONOS between 10-15 ms to construct a new route (intent) as a new streaming path and between 15-20 ms to reroute the traffic. These results demonstrate that the users’ interactive feedback can be taken into consideration during streaming sessions and this feedback can be communicated in a timely manner to the SDN controller to alert of an existing issue and that a corrective action is required. It was shown that rerouting decision can be made on spot and quality can be enhanced based on external user feedback.

In this paper, we proposed a real-time QoE crowdsourcing framework for SD-WAN. The proposed framework is based on a combination of QoE measurement application and QoS quality parameters that accommodate a variety of different streaming protocols. It emphasizes the dependability between QoE and QoS and how the overall user QoE perspective can be affected. We have analyzed how dynamic changes of events can affect the performance of different streaming protocols, and accordingly perceived quality. We have compared objective and subjective results and found that both results are mostly consistent. We also demonstrated that an SD-WAN controller can receive feedback and detect problems in the service path and take corrective action by making changes to the routing paths of the content delivery network in a timely manner. Our goal was to prove that real-time QoE feedback could enhance cloud-based services and could adjust services quality based on real-time active participants’ interaction.

We are considering several paths for future work in this area. Firstly, the scalability of the framework can be further confirmed by setting up environments with larger number of participants in real world. Also, the use of Artificial Intelligence (AI) can be considered in this operation for both data summarization as well as network-based rerouting decisions. With AI, it is possible to learn feedback patterns received by external user participation and an AI algorithm can decide if rerouting is required or not, where we could have a threshold for number of clicks. This AI algorithm can determine this threshold dynamically and adjust it by learning from user feedback. AI could also be used to determine the suitable level of data abstraction/summarization for the trade-off between prompt QoE feedback reaction versus the volume of feedback traffic. Another path shall focus on algorithms for optimizing SDN environments, cloud resources and paths, taking into consideration the QoS SLA and minimum QoE requirements. Intent-based programming is evolving currently and new research could leverage the Intent Framework of SDN controller to optimize re-routing based on the QoE.

AI Artificial Intelligence

AIMD Additive-Increase/Multiplicative-Decrease

API Application Programming Interface

CCDN Cloud-based Content Delivery Network

DASH Dynamic Adaptive Streaming over HTTP

HPC Human-Paired Comparison

HTTP HyperText Transfer Protocol

IETF Internet Engineering Task Force

IQMF In-network Quality Measurement Framework

MOS Mean Opinion Score

MPD Media Presentation Description

ONOS Open Networking Operating System

PESQ Perceptual Evaluation of Speech Quality

PSNR Peak-Signal-to-Noise Ratio

QoE Quality of Experience

QoP Quality of Perception

QoS Quality of Service

REST Representational State Transfer

RTP Real-time Transport Protocol

RTCP Real-time Control Protocol

RTSP Real-time Streaming Protocol

SCTP Stream Control Transmission Protocol

SDN Software-Defined Network

SD-WAN Software-Defined Wide-Area Network

SLA Service-Level Agreement

SQMF SDN Quality Measurement Framework

SSIM Structural Similarity Index Metric

TCP Transmission Control Protocol

UDP User Datagram Protocol

VoIP Voice-over- Internet Protocol

VQM Video Quality Metrics

WAN Wide-Area Network

A. Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to the requirements of the project funding sources, but may be available from the corresponding author on reasonable request and subject to the approval of the funding sources.

B. Competing Interests

The authors declare that they have no competing interests in relation to this work.

C. Funding

This research was supported through funding from Natural Sciences and Engineering Research Council of Canada (NSERC) and Ericsson Canada.

D. Authors’ Contributions

This research work was completed by IE as part of her thesis (supervised by SSH) for the degree of Master of Computer Science at University of Ontario institute of Technology. IE and SSH both contributed to the ideas and methodologies behind the work. Implementation of the test bed and data collection was performed by IE. Manuscript was written by IE and SSH. All authors have read and approved the final manuscript.

E. Acknowledgements

We thank Subjectify.us and MSU Graphics & Media Lab for helping us to conduct subjective and objective measurements with MSU Video Quality Measurement Tool.

F. Ethics approval and consent to participate

Human-based Experiments in this work were approved by the Research Ethics Board of the University of Ontario Institute of Technology under application file number 14780.

[1] Dutta A, Chennikara JM, Chen W, Altintas O, Schulzrinne H. Multicasting streaming media to mobile users. IEEE Communications magazine. 2003 Oct 14;41(10):81-9.

[2] Roy RR. Handbook on Session Initiation Protocol: Networked Multimedia Communications for IP Telephony. CRC Press; 2016.

[3] Liu C. Multimedia over IP: RSVP, RTP, RTCP, RTSP. Handbook of emerging communications technologies: the next decade. CRC Press; 1997. p. 29-46.

[4] Kim H, Feamster N. Improving network management with software defined networking, IEEE Communications Magazine. 2013;51:114-119.

[5] Kuipers F, Kooij R, De Vleeschauwer D, Brunnström K. Techniques for measuring quality of experience. Proceedings of International Conference on Wired/Wireless Internet Communications. 2010. p. 216-227.

[6] Wu CC, Chen KT, Chang YC, Lei CL. Crowdsourcing multimedia QoE evaluation: A trusted framework. IEEE transactions on multimedia. 2013 Jan 18;15(5):1121-37.

[7] Chen KT, Tu CC, Xiao WC. Oneclick: A framework for measuring network quality of experience. In IEEE INFOCOM 2009 2009 Apr 19 (pp. 702-710). IEEE.

[8] Liotou E, Samdanis K, Pateromichelakis E, Passas N, Merakos L. QoE-SDN APP: A rate-guided QoE-aware SDN-APP for HTTP adaptive video streaming. IEEE Journal on Selected Areas in Communications. 2018 Mar 15;36(3):598-615.

[9] Sanvito D, Moro D, Gulli M, Filippini I, Capone A, Campanella A. ONOS Intent Monitor and Reroute service: enabling plug&play routing logic. In 2018 4th IEEE Conference on Network Softwarization and Workshops (NetSoft) 2018 Jun 25 (pp. 272-276). IEEE.

[10] Berde P, Gerola M, Hart J, Higuchi Y, Kobayashi M, Koide T, Lantz B, O'Connor B, Radoslavov P, Snow W, Parulkar G. ONOS: towards an open, distributed SDN OS. In Proceedings of the third workshop on Hot topics in software defined networking 2014 Aug 22 (pp. 1-6).

[11] ONOS Intent Framework. The ONOS project. https://wiki.onosproject.org/display/ONOS/Intent+Framework. Accessed May 22, 2020.

[12] Bu C, Wang X, Cheng H, Huang M, Li K. Routing as a service (RaaS): An open framework for customizing routing services. Journal of Network and Computer Applications. 2019 Jan 1;125:130-145.

[13] Hobfeld T, Schatz R, Varela M, Timmerer C. Challenges of QoE management for cloud applications. IEEE Communications Magazine. 2012 Apr 5;50(4):28-36.

[14] Sezer S, Scott-Hayward S, Chouhan PK, Fraser B, Lake D, Finnegan J, Viljoen N, Miller M, Rao N. Are we ready for SDN? Implementation challenges for software-defined networks. IEEE Communications Magazine. 2013 Jul 12;51(7):36-43.

[15] Lin TN, Hsu YM, Kao SY, Chi PW. OpenE2EQoS: Meter-based method for end-to-end QoS of multimedia services over SDN. In 2016 IEEE 27th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC) 2016 Sep 4 (pp. 1-6). IEEE.

[16] Farshad A, Georgopoulos P, Broadbent M, Mu M, Race N. Leveraging SDN to provide an in-network QoE measurement framework. In 2015 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS) 2015 Apr 26 (pp. 239-244). IEEE.

[17] Alberti C, Renzi D, Timmerer C, Mueller C, Lederer S, Battista S, Mattavelli M. Automated QoE evaluation of dynamic adaptive streaming over HTTP. In2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX) 2013 Jul 3 (pp. 58-63). IEEE.

[18] Letaifa AB. Real time ml-based qoe adaptive approach in SDN context for HTTP video services. Wireless Personal Communications. 2018 Dec 1;103(3):2633-56.

[19] Xezonaki ME, Liotou E, Passas N, Merakos L. An SDN QoE Monitoring Framework for VoIP and Video Applications. In 2018 IEEE 19th International Symposium on" A World of Wireless, Mobile and Multimedia Networks"(WoWMoM) 2018 Jun 12 (pp. 1-6). IEEE.

[20] Volpato F, Da Silva MP, Gonçalves AL, Dantas MA. An autonomic QoE-aware management architecture for software-defined networking. In 2017 IEEE 26th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE) 2017 Jun 21 (pp. 220-225). IEEE.

[21] Hossfeld T, Keimel C, Hirth M, Gardlo B, Habigt J, Diepold K, Tran-Gia P. Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. IEEE Transactions on Multimedia. 2013 Nov 20;16(2):541-58.

[22] Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. 2004 Apr 13;13(4):600-612.

[23] Watson AB. Toward a perceptual video-quality metric. InHuman Vision and Electronic Imaging III 1998 Jul 17 (Vol. 3299, pp. 139-147). International Society for Optics and Photonics.

[24] Haghighi AA, Shahbazpanahi S, Heydari SS. Stochastic QoE-aware optimization in cloud-based content delivery networks. IEEE Access. 2018 Jun 8;6:32662-72.

[25] Chen X, Bennett PN, Collins-Thompson K, Horvitz E. Pairwise ranking aggregation in a crowdsourced setting. InProceedings of the sixth ACM international conference on Web search and data mining 2013 Feb 4 (pp. 193-202).

[26] Crowd-sourced subjective quality evaluation platform. Moscow State University. http://www.subjectify.us. Accessed May 22, 2020.

[27] Vatolin D, Moskvin A, Petrov O, Trunichkin N. Msu video quality measurement tool. 2009]. http://www.download3k. com/Install-MSU-Video-Quality-Measurement-Tool.htrnl. 2009 Aug. Accessed May 22, 2020.

[28] Giraldo C. Minievents: A mininet Framework to define events in mininet networks. 2015. https://github.com/mininet/mininet/wiki/Minievents:-A-mininet-Framework-to-define-events-in-mininet-networks. Accessed May 22, 2020.

Download PDF

Journal Publication

published 14 Aug, 2021

Read the published version in Future Internet →

Version 2

posted 13 Jan, 2021

You are reading this latest preprint version

Crowdsourcing Framework for QoE-Aware SD-WAN

Status:

Journal Publication

Version 2

Abstract

Figures

1. Introduction

A. Background and Motivation

B. Contributions

C. Paper Outline

2. Related Works

3. Network Model And Assumptions

4. Proposed Framework

5. Test Bed Implementation

6. Performance Evaluation

7. Conclusion And Future Works

List Of Abbreviations

Declarations

A. Availability of data and materials

B. Competing Interests

C. Funding

D. Authors’ Contributions

E. Acknowledgements

F. Ethics approval and consent to participate

References

Status:

Journal Publication

Version 2