Quantitative Evaluation of QoS Prediction in IoTGary White, Andrei Palade, Christian Cabrera, Siobhan Clarke

School of Computer Science and Statistics,Trinity College Dublin, Dublin, Ireland

{whiteg5, paladea, cabrerac, siobhan.clarke}@scss.tcd.ie

Abstract—Internet of Things (IoT) applications, are typicallybuilt from services provided by heterogeneous devices, which arepotentially resource constrained and/or mobile. These servicesand applications are widespread and a key research questionis how to predict user side quality of service (QoS), to ensureoptimal selection and composition of services. Invoking allavailable IoT services for evaluation is impractical due to theexponential growth in these services. To assess the current state ofthe art related to this research question we conduct a quantitativeevaluation of QoS prediction approaches, particularly those thatuse matrix factorisation (MF) for collaborative QoS prediction.These approaches derive from the collaborative filtering modelused in recommender systems, which avoids the problems ofmany other QoS prediction approaches of requiring additionalinvoking of available IoT components on behalf of the user. Weconduct comprehensive experiments based on a real-world large-scale QoS dataset as well as a transformation of this datasetto more closely estimate IoT services, to show the predictionaccuracy of these approaches. We also give a demonstration ofhow they can be used in a small scale example.

I. INTRODUCTION

The emergence of a new generation of cheaper and smallerwireless devices with a number of communication protocolshas led to the formation of the IoT. The number of thesedevices is predicted to grow at an exponential rate, with thelatest forecasts predicting that there will be between 26 and 50billion connected devices by 2020 [1], [2]. The huge number ofdevices in the IoT will lead to a wide variety of services, whichcan provide information from a number of sources such ashome applications, surveillance cameras, monitoring sensors,and actuators [3]. This leads to applications in many differentdomains, such as home automation, industrial automation,medical aids, traffic management and many others [4].

These applications have a range of QoS requirements, whichcan be typically categorised as best effort (no QoS), differ-entiated services (soft QoS) and guaranteed services (hardQoS). As these devices are usually resource constrained [5]there is a high probability that the QoS provided will fail ordegrade during service execution. These devices are capableof providing multiple services, which means that there willbe a large number of functionally similar services available tousers. A key requirement is to be able to select the best servicebased on the non-functional requirements such as QoS, eitherat design time, or runtime using runtime service adaptation,if the currently executing service begins to degrade. This isespecially important for safety critical services such as those inhealthcare, where a service failing can have a serious impact.

Switching to a service with better QoS is not a simpletask as the QoS of a service takes into account a number ofdifferent factors such as response time and throughput, whichmay be influenced by user preference. There are also a numberof QoS factors, which are difficult to quantify such as thesecurity and functional stability of a service. Quantifiable QoSfactors, such as response time and throughput, are not stableand vary by time and location of invocation [6]. A number ofapproaches have been proposed to deal with these challengesand to make predictions of QoS values based on time-seriesanalysis [7], [8], [9]. These approaches make predictions aboutthe future QoS they will receive based on the current valuesbeing monitored, but they cannot make predictions on theQoS of services that they have not invoked, unless they startexecuting them. This is a problem in the IoT as there will bea huge number of candidate services, which could be invokedand when a currently executing service begins to degrade, weneed to be able to find an alternative functionally equivalentservice with the requested QoS, without having to invoke allpossible services.

An alternative approach to predicting QoS is inspired fromrecommender systems, where the QoS information of similarusers is used to make predictions about the QoS from possibleservices, by collaborative filtering. These approaches use MFto allow the user to receive QoS predictions of services thatthey have not invoked, based on the QoS values from similarusers. Using the QoS from other similar users gives moreinformation about candidate services, which can be choseneither at design time or during runtime service execution. Thisalso has the additional benefit that we can gather the additionalQoS information without harming the performance of theinfrastructure with needless invocations of services to retrievethe QoS. This is a key challenge in SOAs [10], especially inIoT where devices can be resource constrained and/or mobile.

MF algorithms have not been used in the IoT and in thispaper we conduct a comprehensive evaluation of existing MFmethods on a dataset of QoS information from real worldservices. We also conduct experiments with a transformationof this dataset to show how the algorithms perform withdata, which more closely resembles what would be found inthe IoT, with more service failures and longer delay times.The remainder of the paper is organised as follows: SectionII describes the collaborative framework in IoT, Section IIIdescribes the experiments that were conducted, Section IVdescribes the results of the experiments and Section V outlinesthe conclusions and future work.

User Request Request Handler Service Registration Engine

Service DiscoveryEngine Web Service

Provider

QoS MonitorMonitoring

EnginePrediction

Engine

Service Composition& Execution Engine

Service Registry

AutonomousService Provider

AsynchronousRequest

Publish/SubscriptMQTT

Service Interface

WSN

Fig. 1: Middleware Architecture

II. COLLABORATIVE FRAMEWORK IN IOT

QoS information from a number of different users is neededto use collaborative filtering. Figure 1 shows our evaluationmiddleware architecture, including the QoS monitor used tocollect QoS information from a number of different userrequests. The QoS prediction component is implemented usinga number of different MF algorithms. Figure 2 illustrateswhere the middleware is implemented in gateways that executeservices from heterogeneous service providers. This figureshows a typical IoT scenario, with services provided fromdifferent service types including web services, wireless sensornetworks (WSN), and autonomous service providers (ASP).The services have a range of different QoS properties includingresponse-time, throughput and availability. The communica-tion links for invoking these services are diverse, which willaffect the personal QoS experience of the users, so are mon-itored using the monitoring engine. These monitored resultsare used by the prediction engine to provide personalised QoSprediction to service consumers, which is the focus of thispaper.

Within the middleware there are several components whichare used to handle requests from service consumers. Therequest handler exposes a request interface to the serviceconsumers, which then subscribe to response messages thatare published when the execution has finished. The serviceregistration engine collects the service description informationby subscribing to WSN providers, which publish their servicedescription and expose a service to register web services. Thisis stored in a service registry, which the service discoveryengine searches and publishes messages to the service com-position and execution engine if the services are available.The QoS monitor provides personalised QoS prediction of theselected services for the user and recommends the best serviceto the service execution engine. The service execution enginethen invokes the components and publishes the results to therequest handler when the execution finishes.

User Request

WSP1: Flood PredictionService

ASP1: Noise Service

WSP2: Air PolutionService

Service Providers

Gateways

User Request

ASP3: Water LevelService

WSN1: Water LevelService

WSDN: Flood PredictionService

ASP2: Air PollutionService

WSP3: Best Route Service

GW2:192.168.2.2

GW1:192.168.2.1

GW3:192.168.2.3 GW5:192.168.2.5

GW4:192.168.2.4

Fig. 2: Small Scale Demonstration

The QoS monitor contains two sub-components: A Moni-toring Engine, which is responsible for collecting a numberof QoS factors from users such as the response time, costand throughput and the Prediction Engine which provides thepersonalised QoS prediction values for different componentusers. In this paper, we focus on the design of the predictionengine and quantitatively evaluate a number of algorithmsbased on their prediction accuracy.

Even in a small scale scenario such as the demonstrationshown in Figure 2, traditional time-series approaches cancause a number of additional service requests for each userwhich can flood the network. The collaborative frameworkallows for users to share local QoS usage experiences andcombine this information to get a global view of the QoSof the services. We can then use algorithms such as MF tomake predictions of QoS values for new users based on localinformation from similar users and global information fromthe service providers. Each user keeps a record of their QoSexperiences in the service registry. If a service consumer wouldlike to get personalised QoS information, they can allow theirQoS records to be used for QoS prediction. Based on thecollected QoS information, the Prediction Engine can thenmake personalised QoS prediction for each individual user andforward the recommended services to the service executionengine.

III. EXPERIMENTAL SETUP

A. Small Scale Demonstration

To provide QoS values on m IoT services for n users, youneed to invoke at least n×m services, this is almost impossible

in an IoT environment where we expect a large number ofservices and users. Without this QoS information, the serviceexecution engine cannot select the optimal components basedon their QoS and must make a choice based on whateverservices are available. This leads to choosing potentiallynon-optimal services which can cause service degradation atruntime and service execution errors.

To address these challenges, we have conducted a quantita-tive evaluation of matrix factorisation approaches, for person-alised QoS prediction in IoT on an open source dataset [6] aswell as a small scale demonstration based on Figure 2. Theapproaches use ideas from recommender systems where usersthat share similar characteristics (e.g., location, response time,etc.) will receive similar QoS when executing the same service.The QoS value of IoT service s observed by user u can bepredicted by exploring the QoS experiences from a user similarto u. A user is similar to u if they share similar characteristics,which can be extracted from their QoS experiences on differentcomponents by performing non-negative matrix factorisation(NMF). By sharing local QoS experience among users theseapproaches can predict the QoS value of a range of IoTservices which include ASP, web services and WSN even ifthe user u has never invoked the service s before.

We demonstrate an example based on the implementationin Figure 2, where we have a number of different serviceproviders, who are able to provide functionally equivalentservices from heterogeneous devices. We model this as abipartite graph G = (U ∪ S,E), such that each edge in Econnects a vertex in U to S. Let U = {u1, u2, ..., u4} be theset of component users and S = {ASP1, ASP2, ...,WSN2}denote the set of IoT services and E (solid lines) represent theset of invocations between U and S. Given a pair (i, j), ui ∈ Uand cj ∈ S, edge eij corresponds to the QoS value of thatinvocation. Given the set E the task is to predict the weightof potential invocations (broken lines).

We visualise the process of matrix factorisation for thedemonstration in Figure 3b, in which each table entry showsan observed weight in Figure 3a. The problem that matrixfactorisation solves is how to predict the missing values inthe user service matrix based on the existing entries. Thesepredicted entries can then provide users with personalised QoSinformation which can be used for service selection in servicecomposition at both design and runtime.

One of the problems that emerge due to the large amountservices in the IoT is that there are a number of func-tionally similar services for users to choose, which reducesthe possibility of having commonly invoked services. Sincethe similarity of users is calculated by comparing commonservices, the problem of having few common services makesit difficult to calculate the similarity between users. By usingthe latent factor model [11] a number of algorithms havebeen able to first factorise the sparse user-component matrixand then use V TH to approximate the original matrix, wherethe low-dimensional matrix V denotes the user latent featurespace and the low-dimensional matrix H represents the low-dimensional item latent feature space. The latent feature space

represents the underlying structure in the data, computed fromthe observed features using matrix factorisation. The rows inthe two matrices represent different features with each columnin V representing a user and each column in H denoting aservice. As the matrices V and H are dense it is then possibleto use a neighbourhood-based collaborative method, as shownin Figure 3c.

B. Dataset Description

Invoking thousands of IoT services in the wild is difficultbecause some of the services may have limited range and maynot be available on the Internet. To evaluate the predictionquality of these approaches, we use the dataset released byZheng et al. [6], which consists of a matrix of the responsetime and throughput of 339 users by 5,825 web services. Wealso conduct experiments with a transformation of this dataset,which more closely resembles services in IoT, where there is agreater distribution in response time and throughput and thereare more device failures. We transform the response time andthroughput matrix by adding random values from a poissondistribution, which is standard practice for modelling networktraffic in cellular networks and the IoT [12].

C. Metrics

To evaluate the prediction accuracy of the proposed algo-rithms, we use two standard error metrics, the Mean AbsoluteError (MAE) and the Root Mean Square Error (RMSE). Themetric MAE is defined as:

MAE =1

N

∑i,j

|wi,j − w∗i,j | (1)

and RMSE is defined as:

RMSE =

√1

N

∑i,j

(wi,j − w∗i,j)

2 (2)

where wi,j is the QoS value of service cj observed by userui, w∗

i,j denotes the predicted QoS value of the service cj byuser ui. N is the number of predicted QoS values. MAE givesequal weight to individual differences, RMSE on the otherhand gives high weights to large errors due to the squaringterm.

D. Evaluated Algorithms

The MF algorithms used to make the QoS predictions ofthe web services and the dataset which we have transformed,are as follows:

1) CloudPred: A neighbourhood-based approach, enhancedby feature modelling on both users and components.[13].

2) EMF (Extended Matrix Factorisation): A frameworkwith relational regularization to make missing QoS valuepredictions [14].

3) LN LFM (Latent Factor Models): A Latent FactorsModel, which can be utilised to predict the unknownQoS values [15].

U1 U2 U3 U4

WSN1 WSN2WS2WS1ASP2ASP1

(a) Invocation Graph for IoT services

0.9U1

ASP1

0.43U2 0.23

WS1

0.33

WSN1

0.340.23U4

0.23ASP2

0.30U3 0.37

0.12WS2

0.34 0.24

WSN2

(b) User-service Matrix (c) Predicted User-service Matrix

Fig. 3: Demonstration of QoS Prediction in IoT

0 100 200 300User ID

0

5

10

15

20

Res

pon

seT

ime

(s)

Service 1

Service 2

Service 3

(a) Response Time for Users

0 100 200 300User ID

0

20

40

60

80

Th

rou

ghp

ut

(kb

ps)

Service 1

Service 2

Service 3

(b) Throughput for Users

Fig. 4: Sample QoS Data

4) NMF (Non-negative Matrix Factorisation): A non-negative matrix factorisation approach on user-item ma-trix for missing value prediction [16].

5) PMF: A personalised neighbourhood matrix approach[17].

These algorithms represent a number of alternative ap-proaches to MF, which we consider to be the current stateof the art, based on results in their respective papers.

IV. RESULTS

To evaluate the performance of the different approaches,we remove random entries from the matrices and comparethe values which were predicted to the original ones. Theprediction accuracy is calculated using Equation (1) and (2).To evaluate the impact of matrix density we use matrices withmissing values in different densities. For example, 10% meansthat we randomly remove 90% entries from the original matrixand use the remaining 10% to predict the original values. Totest the impact of the dataset we conduct experiments on atransformation of the original dataset as described in SectionIII-B, which more closely models what would be expected inthe IoT with longer tails in the distribution. Finally, we conductexperiments on the dimensionality of the algorithms, whichdetermines the number of latent features used to characterisethe user and the IoT services. This parameter allows for thetuning of the individual algorithms to reduce error in certainconditions such as a low matrix density.

A. Impact of Matrix Density

Table I and Figure 5 show the impact of matrix density onthe prediction accuracy. To estimate the impact of the matrix

density, we randomly remove some entries from the matricesand compare the predicted values to the original ones. Wetest a number of different matrix densities, we change thematrix density from 10% to 90% with a step value of 10%and repeat each step sizes 20 times. In Figure 5 we draw thestandard deviation in the region around the average error forthe 20 trials. The parameter settings for each of the algorithmsare set to the default parameters mentioned in their respectivepapers.

Figure 5a and 5b show the experimental results of responsetime, while Figure 5c and Figure 5d show the experimentalresults conducted on the throughput dataset. We can see fromthe figures that matrix density has a large impact on predictionaccuracy, with an increase in density improving the predictionaccuracy of all the algorithms. In an IoT environment, wewould expect the matrix density to be quite low due to thelarge number of potential services, so it is important to identifyhow these algorithms perform with low matrix density.

Table I shows the detailed results of the experiments, wherethe best preforming result for each density is in bold. For theresponse-time dataset the CloudPred algorithm has performedthe best for all matrix densities in both the MAE and RMSE.Figure 5a and 5b visualise the response time results from thetable and show how each of the algorithms follow a similarslope as the matrix density increased.

The throughput dataset shows more varied results with thebest algorithm switching from NMF, EMF and LN LFM fordifferent martix densities as can be seen in Table I. We canalso see that the errors are much larger for the throughputdataset, due to the scale of data, which varies more than theresponse time as can be seen in Figure 4. One solution maybe to use an ensemble of the existing algorithms to reduceerror. The overall results from this experiment show how muchof an influence matrix density has on these algorithms, andthe amount of error to be expected when there is low matrixdensity, which may be the case in IoT.

B. Impact of Dataset

As mentioned in Section III-B we transform the originaldataset using random values drawn from a poisson distributionto more closely resemble QoS values which would be expectedin the IoT. We use the CloudPred algorithm to demonstratethe effect of the change in dataset and the matrix density from

TABLE I: Performance comparison of algorithms

Matrix Density Metrics Respose-Time (seconds) Throughput (kbps)CloudPred NMF EMF LN LFM PMF CloudPred NMF EMF LN LFM PMF

10% MAE 0.4773 0.4775 0.4836 0.4998 0.4867 20.1655 15.5929 15.8031 17.6602 15.9616RMSE 1.2230 1.2824 1.2893 1.2272 1.3164 50.6640 47.8526 48.3729 46.8979 48.0285

20% MAE 0.4180 0.4285 0.4269 0.4525 0.4308 17.7078 13.5480 13.5164 16.3540 13.9386RMSE 1.1363 1.1609 1.1575 1.1552 1.1692 45.9991 41.6983 41.6403 44.0707 41.7737

80% MAE 0.3529 0.3723 0.3741 0.4131 0.3749 14.3136 11.9228 12.0601 15.7978 12.1213RMSE 1.0186 1.0513 1.0478 1.0922 1.0504 39.2664 36.8135 36.3161 42.8756 36.4876

90% MAE 0.3485 0.3690 0.3715 0.4131 0.3725 14.0215 11.7736 11.9262 15.7664 12.0380RMSE 1.0108 1.0483 1.0400 1.0900 1.0430 38.4532 36.2391 35.8774 42.5764 36.0491

10 20 30 40 50 60 70 80 90Matrix Density

0.34

0.36

0.38

0.40

0.42

0.44

0.46

0.48

0.50

0.52

MA

E

Response TimeCloud PredictNMFEMFLN LFMPMF

(a)

10 20 30 40 50 60 70 80 90Matrix Density

0.95

1.00

1.05

1.10

1.15

1.20

1.25

1.30

1.35R

MSE

Response TimeCloud PredictNMFEMFLN LFMPMF

(b)

10 20 30 40 50 60 70 80 90Matrix Density

12

14

16

18

20

MA

E

ThroughputCloud PredictNMFEMFLN LFMPMF

(c)

10 20 30 40 50 60 70 80 90Matrix Density

34

36

38

40

42

44

46

48

50

52

RM

SE

ThroughputCloud PredictNMFEMFLN LFMPMF

(d)

Fig. 5: Impact of Matrix Density

10% to 90% with a step value of 10%. The parameters for thisexperiment are Top-K=10, dimensionality=20 and λ=0.5, werepeat the experiment 20 times and draw the standard deviationin the region around the average error.

Figure 6 shows the impact that the change in dataset has hadon the results, we can see that for the response time dataset thechange has increased the error considerably with over twicethe amount of error for the transformation by lambda=1, 2in the MAE as can be seen in Figure 6a and the RMSE inFigure 6b. For the throughput dataset, the change has hadmuch less of an impact and the error values remain consistentin the MAE in Figure 6c, and in the RMSE in Figure 6d. Theimpact of the change in dataset is influenced by the originaldata, as the scale of the response times is much smaller, 0-20seconds compared to throughput, which is 1-1000kbps as canbe seen in Figure 4. This suggests that the QoS factors whichthe algorithms are used on can have an impact on accuracy ofthe results. This is something to take into consideration whenusing these algorithms to model a number of factors, whichhave a large range of values such as throughput as they maynot be able to accurately predict within a close bounds.

C. Impact of DimensionalityThe dimensionality parameter is used to determine the

number of latent features, which characterise the users andservices. We study the impact of this parameter by varyingthe value of dimensionality from 10 to 50 with a step sizeof 10 on the CloudPred algorithm. Other parameter settingsremain constant and are set to Top-K=10 and λ=0.5 and thematrix density is set to 10%. We repeat the experiment 20times and draw the standard deviation in the region aroundthe average error.

Figure 7 shows the results of this experiment and theeffect that the dimensionality parameter has on the results.

We can see that a low value for matrix dimensionality canlead to underfitting and an increase in prediction error forthe algorithm. As the dimensionality increases the algorithmadds more latent features to the model which can reduce theaccuracy by overfitting to the data. In Figure 7, we see howthe error seems to stay constant even as more latent featuresare added to the model.

V. CONCLUSION AND FUTURE WORK

We have evaluated a number of matrix factorisation ap-proaches and have shown how these approaches can be usedto make QoS predictions by exploring the past usage fromboth the user and other similar users both for web servicesand a transformed dataset to more closely resemble IoT data.The results illustrated in Figure 5 and 6 show the impactthat the density and dataset have on the results. The matrixdensity has an obvious effect on the prediction accuracy as theerror reduces for all of the algorithms as the matrix becomesmore dense. The transformation of the dataset to more closelyresemble IoT QoS has led to an increase in the predictionerror most notably in response time, where the error rate morethan doubled. We also show the impact of internal modelparameters such as the dimensionality of the algorithm inFigure 7, which shows that the model parameters may haveto be tuned as they can have a large impact on the predictionaccuracy of the model.

The results have shown that these approaches have increasederror for prediction IoT services. However, they have a numberof additional benefits which would make them extremelyuseful in the IoT. They require no additional invocation, whichreduces the load on the network. This is a key requirementfor resources constrained devices, which are used in the IoT.They also allow the user to utilise both local usage information

10 20 30 40 50 60 70 80 90Matrix Density

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6M

AE

Response TimeOriginal

λ = 1

λ = 2

(a)

10 20 30 40 50 60 70 80 90Matrix Density

0.8

1.0

1.2

1.4

1.6

1.8

2.0

RM

SE

Response TimeOriginal

λ = 1

λ = 2

(b)

10 20 30 40 50 60 70 80 90Matrix Density

12

13

14

15

16

17

18

19

20

21

MA

E

ThroughputOriginal

λ = 1

λ = 2

(c)

10 20 30 40 50 60 70 80 90Matrix Density

36

38

40

42

44

46

48

50

52

RM

SE

ThroughputOriginal

λ = 1

λ = 2

(d)

Fig. 6: Impact of Dataset

10 15 20 25 30 35 40 45 50Dimensionality

0.472

0.474

0.476

0.478

0.480

0.482

0.484

0.486

0.488

MA

E

Response Time10% Density

(a)

10 15 20 25 30 35 40 45 50Dimensionality

1.210

1.215

1.220

1.225

1.230

1.235

1.240

RM

SE

Response Time10% Density

(b)

Fig. 7: Impact of Algorithm Dimensionality

from similar users and global information from the serviceproviders. The IoT is highly dynamic in nature and theQoS performance of services can vary dramatically with timeand service providers moving in and out of the environmentcausing a service to suddenly not be available either due tomobility or running out of battery. This is in combination withthe usual QoS performance constraints of the IoT such as thenetwork traffic, device power, device location etc. The datasetthat we use is based on QoS observed over a long time, whichrepresents the average QoS.

For future work, we aim to build a QoS dataset of IoTservices, which will include services from WSN, web servicesand autonomous service providers. This will give a morerealistic evaluation compared to our current dataset which isa transformation. We also aim to add time information tothis dataset to allow for experimentation with online models.We will conduct more experiments to show how the hetero-geneity of services affects the prediction accuracy of thesemodels, which we have replicated for these experiments witha transformation of the dataset. We also aim to investigate anumber of alternative techniques for preprocessing the dataand conducting similarity comparison between the users (e.g.data smoothing, clustering, PCA, etc.).

ACKNOWLEDGMENT

This work is supported by Science Foundation Ireland (SFI)under the project named SURF: Service-centric networking forurban-scale feedback systems.

REFERENCES

[1] H. Bauer, M. Patel, and J. Veira, “The internet of things: Sizing up theopportunity,” McKinsey, 2014.

[2] D. Evans, “The internet of things: How the next evolution of the internetis changing everything,” CISCO white paper, vol. 1, p. 14, 2011.

[3] A. Al-Fuqaha, M. Guizani, M. Mohammadi, M. Aledhari, andM. Ayyash, “Internet of things: A survey on enabling technologies,protocols, and applications,” IEEE Communications Surveys Tutorials,vol. 17, no. 4, pp. 2347–2376, Fourthquarter 2015.

[4] P. Bellavista, G. Cardone, A. Corradi, and L. Foschini, “Convergence ofmanet and wsn in iot urban scenarios,” Sensors Journal, IEEE, vol. 13,no. 10, pp. 3558–3567, 2013.

[5] D. Miorandi, S. Sicari, F. D. Pellegrini, and I. Chlamtac, “Internetof things: Vision, applications and research challenges,” Ad HocNetworks, vol. 10, no. 7, pp. 1497 – 1516, 2012. [Online]. Available:http://www.sciencedirect.com/science/article/pii/S1570870512000674

[6] Z. Zheng, Y. Zhang, and M. R. Lyu, “Investigating qos of real-worldweb services,” IEEE Transactions on Services Computing, vol. 7, no. 1,pp. 32–39, Jan 2014.

[7] Z. Ye, S. Mistry, A. Bouguettaya, and H. Dong, “Long-term qos-awarecloud service composition using multivariate time series analysis,” IEEETransactions on Services Computing, vol. 9, no. 3, pp. 382–393, May2016.

[8] A. Amin, L. Grunske, and A. Colman, “An automated approach toforecasting qos attributes based on linear and non-linear time seriesmodeling,” in 2012 Proceedings of the 27th IEEE/ACM InternationalConference on Automated Software Engineering, Sept 2012, pp. 130–139.

[9] A. Amin, A. Colman, and L. Grunske, “An approach to forecasting qosattributes of web services based on arima and garch models,” in 2012IEEE 19th International Conference on Web Services, June 2012, pp.74–81.

[10] C. Areias, J. C. Cunha, and M. Vieira, “Studying the propagation offailures in soas,” in Dependable Systems and Networks Workshops (DSN-W), 2015 IEEE International Conference on. IEEE, 2015, pp. 81–86.

[11] R. Salakhutdinov and A. Mnih, “Probabilistic matrix factorization.” inNips, vol. 1, no. 1, 2007, pp. 2–1.

[12] M. Mirahsan, R. Schoenen, and H. Yanikomeroglu, “Hethetnets: Hetero-geneous traffic distribution in heterogeneous wireless cellular networks,”IEEE Journal on Selected Areas in Communications, vol. 33, no. 10, pp.2252–2265, Oct 2015.

[13] Y. Zhang, Z. Zheng, and M. R. Lyu, “Exploring latent features formemory-based qos prediction in cloud computing,” in 2011 IEEE 30thInternational Symposium on Reliable Distributed Systems, Oct 2011, pp.1–10.

[14] W. Lo, J. Yin, S. Deng, Y. Li, and Z. Wu, “An extended matrixfactorization approach for qos prediction in service selection,” in 2012IEEE Ninth International Conference on Services Computing, June 2012,pp. 162–169.

[15] D. Yu, Y. Liu, Y. Xu, and Y. Yin, “Personalized qos prediction forweb services using latent factor models,” in 2014 IEEE InternationalConference on Services Computing, June 2014, pp. 107–114.

[16] D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” Nature, vol. 401, no. 6755, pp. 788–791,1999.

[17] Z. Zheng and M. R. Lyu, “Personalized reliability predictionof web services,” ACM Trans. Softw. Eng. Methodol., vol. 22,no. 2, pp. 12:1–12:25, Mar. 2013. [Online]. Available:http://doi.acm.org/10.1145/2430545.2430548