Energy Systems https://doi.org/10.1007/s12667-020-00376-x ORIGINAL PAPER A comparison of machine learning algorithms for forecasting indoor temperature in smart buildings Sadi Alawadi"@ - David Mera? - Manuel Fernandez-Delgado? - Fahed Alkhabbas' - Carl Magnus Olsson’ - Paul Davidsson! Received: 28 August 2019 / Accepted: 10 January 2020 (© The Author(s) 2020 Abstract The international community has largely recognized that the Earth’s climate is chang- ing. Mitigating its global effects requires international actions. The European Union (EU) is leading several initiatives focused on reducing the problems. Specifically, the Climate Action tries to both decrease EU greenhouse gas emissions and improve energy efficiency by reducing the amount of primary energy consumed, and it has pointed to the development of efficient building energy management systems as key. In traditional buildings, households are responsible for continuously monitoring and controlling the installed Heating, Ventilation, and Air Conditioning (HVAC) system Unnecessary energy consumption might occur due to, for example. forgetting devices turned on, which overwhelms users due to the need to tune the devices manually. Nowadays, smart buildings are automating this process by automatically tuning HVAC systems according to user preferences in order to improve user satisfaction and opti- mize energy consumption. Towards achieving this goal, in this paper, we compare 36 Machine Learning algorithms that could be used to forecast indoor temperature in a smart building. More specifically, we run experiments using real data to compare their accuracy in terms of R-coefficient and Root Mean Squared Error and their per- formance in terms of Friedman rank. The results reveal that the ExtraTrees regressor has obtained the highest average accuracy (0.97%) and performance (0.058%) over all horizons. Keywords Smart buildings - Time series prediction - Energy efficiency - Machine Learning - Internet of Things a Sadi Alawadi sadi.alawadi @mau.se Extended author information available on the last page of the article Published online: 24 January 2020 QD Springer5S. Alawadi et al. 1 Introduction Climate change is one of the bigger challenges we face today, needing immediate and long-term action. In general, climate change affects all regions of the world-posing a threat to global economy, holding negative environmental effects, and bringing worry- ing health implications. These growing threats require international action to mitigate and minimize their negative effects. Initiatives such as the European Climate Action outline the recorded negative effects of climate change and list urban infrastructure as the key to effectively working towards the goals set forth by the European Union (EU), who is currently responsible for 71% of global gas emissions and thus has a vital role to play [60]. To achieve this, the European Climate Action initiative aims to both decrease EU greenhouse gas emissions and improve energy efficiency by reducing the amount of primary energy consumed. It also aims to find sustainable solutions from an environmental as well as an economic standpoint. Within smart buildings, the automation of existing residential as well as commercial buildings (built prior to modern low- or zero-energy buildings) plays a significant role, as such buildings make up the majority of energy consumption. The EU has pointed to the development of efficient building energy management systems as key to achieving the identified objectives due to the fact that buildings account for 40% of energy consumption and 36% of total CO emissions within the EU [3.72]. The majority of energy in those buildings is consumed by Heating, Ventilation, and Air Conditioning (HVAC) systems, which have strong impact on households comfort as well as on the environment [29] Increasing affordability as well as rising temperatures have meant that HVAC sys- tems are increasingly being used to improve comfort and thus quality of everyday life. At the same time, such systems can quickly consume a considerable amount of energy. Particularly for systems with limited intelligent behavior, energy efficiency is not emphasized, and simple matters quickly add up to energy waste—such as a household forgetting to turn off an air conditioner before going to work or systems not adapting when the weather changes by, for instance, turning off when not needed. In traditional (non-smart) buildings, users (residents) are responsible for monitoring. and controlling available devices. However, contemporary smart buildings are increas- ingly equipped with Internet of Things (ToT) devices and objects such as sensors. actuators, connected air conditioners, and heaters. In such buildings, unlike traditional buildings, IoT devices collaborate to automatically adjust temperature and optimize the use of HVAC systems, for instance. by forecasting the indoor temperature and generating plans for tuning HVAC devices to optimize energy consumption Previous studies have shown that Machine Learning (ML) algorithms can be exploited to model most of the systems in the smart buildings. In particular, ML can be used to model the current HVAC systems [23] to improve energy efficiency and reduce consumption in such buildings. ML is a sub-field of Artificial Intelligence (AD) that combines a set of mathematical algorithms to give systems the ability to learn automatically and improve the experience without being explicitly programmed [4]. Nowadays, ML is widely used in many fields, including health care, public trans- portation, and smart cities systems [2.1645]. ML is divided into several categories based on the learning method, such as supervised, semi-supervised, unsupervised, and ®D SpringerA comparison of machine learning algorithms.. enforced learning. In this paper we will be using supervised learning, which is divided into two main branches; classification [7,28,62] and regression [1,27] depending on the problem that needs to be solved. In our case we will be using regression to forecast the indoor temperature. In this paper, we describe an experiment that compares 36 offline ML algorithms used for forecasting the indoor temperature for three consecutive hours in a smart building. A real dataset was collected from the CiTIUS research center and the closest weather station sensor measurements that belongs to different winter periods with different weather conditions as reported in Table 1. All algorithms were evaluated based on their accuracy, performance, and robustness to weather changes. The main aim of this study is to find the most suitable ML algorithm in terms of the performance and robustness that can be integrated into building management systems (BMS) to improve building energy efficiency. Specifically to tune HVAC system parameters taking into consideration user comfort levels and reduction of energy consumption. We concluded that inereasing the forecasting time does not decrease the accuracy of the best model. Moreover, we found that the difference between the obtained results for three consecutive forecasting hours is insignificant (around 0.01) for both R-coefficient and RMSE: This means that the inerease of the horizon does not rapidly affect the accuracy of extraTrees, The remainder of this paper is organized as follows. In the next Sect. 2, we review existing studies to forecast the building’s indoor temperature using different ML algo- rithms. In Sect. 3, we describe the dataset we used to develop the experiments and explain the ML algorithms used to develop the experiments. Section 4 shows the results and discussion. Finally, Sect. 5 draws the conclusions and outlines of the future work. 2 Related research Previous studies have determined that the HVAC systems have the highest energy demand ina building. Therefore, managing HVAC systems in current buildings should be addressed to improve energy efficiency by improving energy plans. In particular, developing a ML model that considers the surrounding factors is necessary to configure the best HVAC system parameters. Those parameters have a relevant impact on both energy consumption and user comfort [23]. The ML model Artificial Neural Network (ANN) is widely used for indoor temperature forecasting. Nivine et al. [8] proposed a new approach to forecast the indoor temperature up to4 h based on ANN by considering the outdoor parameters. Further, Kwok et al, [44] modulated the cooling load ina smart building by incorporating a Neural Network (NN) into an intelligent model that allows forecasting and examining the energy demand of the building as well as determining the critical factors that impact on energy consumption. The study reveals that the building occupaney is a significant factor in forecasting the cooling load of the HVAC system In [54], the authors studied the impact of both users’ activities and their behaviors on potential energy saving in smart buildings. The authors classified the user as the most important factor and divided the user impact on energy demands into three main subsystems: HVAC, light, and plug load systems QD Springer5S. Alawadi et al. Moreover, Varick et al. [25] used real-time data to study building occupancy and its influence on energy saving. They proposed an occupancy model that could be successfully integrated into the HVAC system in the building through Markov Chains. The study revealed that this mode! could annually save 42% of consumed energy. Zhao [62] argued that external factors also have a significant influence on a building's energy performance through reviewing various energy forecast methods implemented into ML algorithms and studying the engineering and statistical techniques utilized to predict a building's energy consumption [72]. Tn [48]. a new model was developed based on Support Vector Regression (SVR) to predict the hourly cooling load inside office buildings. The model's hyper-parameters were tuned to get the best temperature forecast. The study compared the developed model with the classical multi-layer perceptron neural network (MLP) and showed that the SVR outperforms the MLP in both accuracy and mean squared error (MSE). In addition, Dong [21] examined the feasibility of forecasting building energy con- sumption by applying SVR for regression and determined the impact of different SVR parameters on the prediction accuracy. The study exposes that SVR obtained the highest accuracy compared with other relevant research approaches using genetic programming and neural networks. Previous studies addressed external weather con- ditions and their influence on indoor temperature through autoregressive model (ARX) and autoregressive moving average model (ARMAX). The selection of the suitable structure of both models has been determined to obtain the best prediction accuracy. These models can become a flexible controller because of their dynamic structure, which permits to increase the user's comfort level inside the building and to improve the energy efficiency of HVAC systems [58]. The outcomes exhibited that the ARX model achieved the best forecasting accuracy. Siilo et al. [67] developed a deep learning model to predict the energy consumption value of each building resides in the City University of New York (CUNY) campuses. Each one of those buildings has different energy expenditures. Where. the optimal conditions and forecasting the future energy usage of those buildings have been inves- tigated to determine the loss of energy, using long short-term memory (LSTM) Neural Networks models. The experiments were conducted using time series data that were collected from several campuses of CUNY. Furthermore, Xu et al. [18] used an LSTM deep learning model to forecast the indoor temperature for 5 and 30 min in advance. The LSTM model was compared to three standard ML models Back Propagation Neu- ral Network (BPNN), Support Vector Machine (SVM) and Decision Tree (DT). which it outperformed. In [42] Jin et al. used deep learning to forecast the optimal indoor temperature with the aim to adjust the air conditioner automatically without any user interference. Abdullatif et al. [10] proposed a cooling load forecasting model for buildings, uti- lizing the generalized regression neural network (GRNN) taking into consideration the building orientational characteristics and occupancy in order to optimize the thermal energy storage of the HVAC. Catalina [14] developed polynomial regression models based on neural networks to predict the monthly heating demand for residential buildings. considering the res- idential constructional structure. Catalina used 270 different scenarios to validate the developed models to find the best approach. Several other recent investigations ®D SpringerA comparison of machine learning algorithms.. proposed models using different ML algorithms for forecasting a building's energy consumption [24.53.6172]. In these studies, various external factors were consid- ered, such as building structure, orientation, isolation, and environmental variables. The statistical results showed that these factors have a significant influence on indoor temperature prediction and energy consumption in a building. Kangji et al. [47] devel- oped a GA-ANFIS model to predict the indoor building temperature. This approach obtained the optimal configuration of subtractive clusters, using a genetic algorithm (GA) to optimize the fuzzy if-then rule base. The adaptive network-based fuzzy infer- ence system (ANFIS) adjusted the premise and subsequent parameters to match the training data, The results showed that GA-ANFIS obtained higher performance levels compared to neural networks in terms of prediction accuracy. Recently, Rodriguez-Mier et al. [59] used FRULER-GFS (fuzzy rule learning through evolution for regression-genetic fuzzy system) to develop a rule-based model for forecasting indoor temperature. The knowledge bases learned by FRULER include Takagi-Sugeno-Kang fuzzy rules that correctly predict the temperature dynamics mea- sured by several different predictors obtained from both inside and outside the building. The experiment results demonstrated that FRULER-GFS had the best accuracy rate compared with ElasticNet and random forest regressors [59] Further, Doukas etal. [22] developed an integrated decision support system based on rule sets. Their study aimed at improving the energy management system of a building. Their system allowed central control over energy consumption in the building, which made it exceptionally flexible. Furthermore, they created a reliable energy profile using expert knowledge in the system. The HVAC control optimization (On/Off) provided the system with the capability to recognize and discard any wrong decision. The study confirmed that expert experience has a notable impact on improving the building energy management system. When reviewing previous studies on improving energy efficiency of HVAC systems in smart buildings, none has compared a large set of ML algorithms to predict the indoor temperature of buildings. This study provides a baseline for future studies on forecasting the indoor temperatures in smart buildings using ML algorithms. All models developed have been trained using the same settings for different weather conditions to check the robustness and the performance of these algorithms. 3 Experiments ‘1 Experiments setups As part of the European OPERE project [26]. which aims at improving the energy man- agement system of the Universidade de Santiago de Compostela (USC), the USC has deployed sensors in 45 university buildings. In this paper, we conducted experiments considering one of those smart buildings, called Centro Singular de Investigacién en Tecnoloxias Intelixentes (CiTIUS), using a medium-sized sensor network. The net- work collects and reports sensor readings as illustrated in Table 1. It produces 667 signals every 10s. QD Springer5S. Alawadi et al. Table 1 Pattern features, where (*) represents features from CiTIUS, and (+) symbolizes features from Meteogalicia Features ‘Abbr. Type Description Underfloor Heating Status * UHS Binary Status of the underfloor heating system (on/off) in the office Underfloor Heating Temperature UHT Continuous ‘Temperature of the water linked to * the underfloor heating system Air Condition Status * ACS Binary Status of the air conditioning system (on/oft) in the office Air Conditioning Temperature ACT Continuous The desired temperature of the central air conditioning system. Air Conditioning Humidity * ACH — Continuous The percentage of the humidity, attached to central air conditioning flow Humidity + OutH Continuous Degree of the outdoor relative humidity Temperature + OuT Continuous Outdoor temperature Solar radiation + SR Continuous Level of solar radiation Indoor temperature * T Continuous Indoor temperature in one particular Previous indoor temperature * Tl Continuous The actual office temperature in a specific time period (1, 2 and 3 h) The dataset we used to develop the experiments composed both the sensor measure- ments linked to the CiTIUS HVAC system and weather data collected from the closest Meteogalicia weather station. The CiTIUS building has two functionality modes: win- ter and summer modes. The dataset patterns were retrieved every 10 min during two different time periods: from October 1, 2015, to March 31, 2016 (26.321 patterns). and from November |, 2016, to January 31, 2017 (13,083 patterns). Both periods cor- respond to the HVAC winter working mode, which has the highest energy demand. It must be noted that the second period corresponds to an unusually dry winter season in Galicia. Thus, the weather conditions in both periods are different enough, Each dataset pattern comprises 10 features, seven of them are provided by the CiTIUS and the rest by Metogalicia weather station, Each variable indicates a mea- surable phenomenon that can reduce the energy demand for heating and cooling the building: these features are described in Table 1. 3.2 Machine learning algorithms In this paper, we compared 36 batch learning algorithms belonging to 20 different families (as listed in Table 2) [27]. All algorithms were selected based on the ree- ommendation of the study conducted by Sirsat et al. [63]. The main purpose of the experiment was to identify which of those algorithms is the most accurate to forecast the indoor temperature of the studied building. The majority of the algorithms were ®D SpringerA comparison of machine learning algorithms.. selected from the Classification and Regression Training package! in the statistical computing language R.? The experiments for each algorithm were repeated 10 times using different seeds generated randomly. The data partitions were generated randomly in such a way that 70%, 15%, and 15% of the patterns were used for training, validating, and testing the models, respectively. For each algorithm, the hyper-parameters were tuned using the values reported in Table 3, The selected final values for the hyper-parameter are those that maximize the average performance over the validation sets. Furthermore, we implemented three more popular methods using other platforms: support vector regression (SVR) using the LibSVM library was implemented in C++, and Generalized Regression Neural Network (GRNN) and Extreme Learning Machine (ELM) with Gaussian kernels were both implemented in MATLAB.‘ Moreover, we trained the regressors by exploiting the values reported in Table 3, and stated in the R package documentation to tune the algorithm hyper-parameters, We then evaluated the tested algorithms’ performance using Pearson correlation (R- coefficient) that falls between (+1, —1), shown in Eq. 1, and the Root Mean Squared Error (RMSE), shown in Eq. 2. se \ /y~ PP (2) a oy where jp and oy are the mean and standard deviation of the predicted temperature P, while jry and oy are the mean and the standard deviation of the real temperature and N is the number of test patterns. loose RMSE = |—Y (ij -Yi)? 2 yee! y 2) The final regressor performance matrices were computed in the developed experiments by taking the average of both RMSE and R-coefficient over the 10 repetiti 4 Results and discussion Satisfying users by achieving and maintaining their comfort levels and optimizing energy consumption inside buildings should be core aspects when realizing smart buildings. This requires developing accurate and reliable HVAC systems that are auto- matically adaptable to different weather conditions. Towards achieving this goal, we compared 36 ML algorithms, over a real data set, to predict the indoor temperature in the CiTIUS office. The results can be utilized to generate energy plans that tune hup://topepo.github.io/caret/train-models-by-tay hl. hup:/fe-project.org, hupsv/www.esic.ntu.edu.tw/~jlin/libsvm. hup://mathworks.com. QD Springer5S. Alawadi et al. Table2 Regressors considered in this work, grouped by families No. Family Regressors 1 Bayesian models Bayesian GLM (Bayesglm) [36] Bayesian regularized neural network (Brnn) [30.49] 2 Bagging ensembles Bagging ensemble of eonditional inference regression trees (Bag) [1] Bagged multivariate adaptive regression (BagEarth) [6.43] Bagging ensemble of regression trees (Treebag) [55] 3 Boosting ensembles RandomGLM [5,65] 4 Gaussian processes Gaussian processes regression with Tinear kernel (GaussprLinear) [69] 5 Generalized additive models Generalized additive model (Gam) [70] 6 Generalized linear Generalized Linear Model (Gim) [20] regression Penalized Linear Model (Penalized) [38] 7 Gradient boosting Gradient boosting machine with linear regressors (BstLm) [9,32,50] machines Gradient boosting with regression base trees (BstTree) [9.50] Generalized boosting model (Gbm) [32] Independent Component Analysis Independent component regression (ler) [41] Least absolute shrinkage Least absolute shrinkage and selection operator (Lasso) [73] 10 Least squares Non-negative least squares regression (NLS) [46] Wl Linear regression Linear Model (Lm) [15] 12 ‘Neural networks Multi-layer perceptron (MLP) [68] Averaged neural network (AvNNet) [43] Generalized regression neural network (Gmnn) [66] Extreme learning machine (Elm) [40] Deep belief neural network (Dnn) [39] Elm-kernel [40] B Other methods Multivariate adaptive regression (Earth) [31] Projection pursuit regression (PPR) [33] 4 Partial least squares ‘Sparse partial least squares regression (Spls) [17] Statistically Inspired Modification of PLS (Simpls) U9] 1s Prototype models Cubist [56] 16 Quantile regression Ralasso regressor (Rqlasso) [52] 7 Random forests Random forest ensemble (Rf) [12] Quantile regression forest (Qef) [51] Ensemble of extremely randomized regression trees (ExtraTrees) [37] ®D SpringerA comparison of machine learning algorithms.. Table2 continued No. Family Regressors. 18 Regression trees Recursive partitioning and regression tre (Rpt) U3) Multivariate linear tree-based model (M5) [57] 19 Ridge (or Tikhonov) regression Forward—Backward Greedy algorithm (Foba) [71] 20 Support vector regression Support vector machine for regression (Svr) [64] Table3 List of the regressors, with their tunable hyper-parameters (tried values and packages) Regressor _ Hyperp. (values) Packages Regressor — Hyperp. (values) Packages AVNNet size (7) nnet Gran spread(14) Mattaby decay (3) Bag - caret er neomp(10) fastICA BagEarth —_nprune(10) caret. Lasso - elasticnet Bayesgim —— am Lm - MASS BsLm mstop (10) bot, ply MS pruned (2) RWeka smoothed (2) rules (2) BstTree mstop (4), maxdepth (5) bst, plyr- MLP n.hidden (20) inet Brn neurons (15) bmn NNLS - nls Cubist committees (3) Cubist Penalized 44(5), 4214) penalized neighbors (3) Dan layer! (10) deepnet PPR nterms(10) stats layer? (10) layer3 (10) Earth prune (15) earth Qrf try (2) quantregForest Elm hid (20) elmNN RE miry(10) randomForest actfun (4) Elm-kernel o (25),C Matlab ExtraTrees —-mtry (10) extraTrees Rpart complexity (10) numRandomCuts 2) Foba KA 10) foba —Ralasso 2. (10) Gam. select (2) gam Simpl comp (10) GaussprLinear — kernlab —Spls K B)an« Gbm natrees (5) gbm, plyr Svr oC) interaetion.depth (3) Gim - gbm, plyr Treebag - RandomGLM maxinterationOrder (3) randomGLM_ rpart rqPen pls spls kernlab ipred, plyr e107 the HVAC system parameters and consequently both increase user satisfaction and optimize energy consumption. We plan to address those aspeets in our future work. QD Springer5S. Alawadi et al. Table4 Friedman rank of the RMSE (left) and R-coefficient (right) Order RMSE rank R-coefficient rank Regressor Rank — MSE Avg. Regressor Rank coefficient Avg 1 extraTrees: 1 0.05807 extraTrees, 1 0.97052 2 ca 2 0.06046 wf 2 0.96916 3 cubist 4 0.06255 cubist 3.7 0.96801 4 avNNet 4 0.06382 avNNet 4 0.96727 5 bstTree 53 0.06362 bstTree 0.96738 6 elm-kernel 37 0.06484 elm-kemel 0.96673 7 bmn 1 0.06811 bm 0.96595 8 gbm 1 0.06635 brnn 0.96506 9 sv 10.3 0.06832 sor 10 0.96505 10 at 10.7 0,06831 ant 10.7 0.96503 ut ppr 1.7 0.07480 ppr 2 0.96154 12 bag 133 0.07721 bag 13.3 0.96023 3 gemn 14 0.07571 arnn 137 0.9614 4 penalized 143 0.08885 penalized 143 0.95398 15 simpls 17.7 0.09578 simpls 17.7 0.9503 16 mip 183 0.09182 mip 18.3 0.95268 7 earth 18.7 0.12496 earth 187 0.93754 18 lasso 18.7 0.09719 rqlasso 18.7 0,95003 19 bagEarth 19.3 0.16418, bagEarth 19.3 0.92212 20 ns 20.7 0.10783 nis 20.3 0.94505 au BstLm 20.7 0.10757 BstLm 213 0.94431 2 lasso 21.7 0.10847 lasso 213 0.94468 23 bayeselm 253° 0.12207 bayeseln 25 0.93695 4 elm 26 0.11665 alm 26 0.93687 25 aim 263 0.12224 gam a 0.93687 26 spls 273 0.22177 spls 273 0.89188 7 gaussprLinear 27.3 0.12219 gaussprLinear 27.3 (0,93688 28 gam 273° 0.12224 elm 27.3 0.93927 29 M5 277 0.17291 M5 277 0.91652 30 Im 283 0.12224 Im 28 0.93687 31 treebag 29 0.12731 29 0.93359 32 xpart 29.3 0.13357 part 29.3 0,92993, 33 ier 293° 0.12732 ier 293 0.93426 34 randomGLM 29.7 0.35537 randomGLM — 29.70.8464 35 foba 30 0.12355 foba 30 0.93617 36 dan 35.7 053150 dan 35.7 0.70223 Best algorithm amongst the whole group and has obtained the best results are shown in bold In the performed experiments, we calculated the Friedman ranks [35] for both RMSE and R-coefficient for all regressors (see Table 4). The Friedman test is a non- ®D SpringerA comparison of machine learning algorithms.. Table5 The best R-coeflicient 2 and RMSE are achieved by th 2h Sh extTives forthe foreeasing SE 0.04041 0.06011 0.07370 Recoelficient 0.97958 0.96951 0.96245 parametric statistical test. Similar to the parametric repeated measures ANOVA, it compares three or more matched or paired groups. It scores the values in each matched row in ascending order, where each row is ranked individually. It then sums the ranks in each column [34]. This test determined the actual position of each algorithm on average over all the horizons. The regressors must be sorted in a descending order based on their performance on each data set (e.g.. by increasing RMSE or by decreasing R- coefficient), and the Friedman rank of each regressor is its average position over the horizons, Figure | illustrates the Friedman rank for both MSE and R-coefficient in ascending order (i.e., by decreasing performance). The best results were achieved by two regressors that belong to the random forest family (ExtraTrees and RF) in both performance measurements. Generally, both figures are quite similar, with small changes in some regressor positions. Table 4 summarizes the Friedman ranks of both the MSE and the R-coefficient average for each regressor, and it clearly shows the small change in the position over all three horizons. Namely. the algorithms fall between the 24th and 28th positions and also between the Bayesian regularized neural network (Brnn) and the generalized boosting model (Gbm). Figure 2 shows the average R-coefficient of the most reliable 20 regressors over the three prediction horizons, sorted decreasingly. The highest R-coefficients are achieved by extremely randomized regression trees (ExtraTrees)—with the accuracy R-coefficient (0.97) and the lowest RMSE average (0.058) as reported in Table 4— followed by Rf, Cubist, BstTree. and AvNNet. The Figure also shows that all the algorithms that appear in the top 10 list belong to random forest family, and the accu- racy obtained by Qrf is quite similar to the Bayesian model (Brn) and Support Vector Regression (Svr). On the other hand, NNLS, Lasso, and Bstlm are at the bottom of the top 20 list, with good performance in terms of R-coefficient (around 0.94 over all horizons). These results (ploted in Fig. 2) are quite similar to the Friedman rank of R-coetficient shown in Fig. |. The BstTree is substituted with AvNNet, so they come in 4th and 5th position, respectively. Moreover, Bag and Grnn algorithms swap positions, becoming 12th and 13th, respectively. Regarding the last three positions, NNLS has improved its position. Unfortunately, Earth and Bagearth regressors disappeared from the top 20, while lasso and BstLm replaced them in the 19th and 20th position The outcomes of this comparative experiment are as follows: the extraTrees algo rithm achieved the highest accuracy for the three prediction horizons in terms of Friedman rank, average values of RMSE, and R-coefficient (Table 5), ExtraTrees is less sensitive to noise and outlier values while ANN models are more sensitive, which means that extraTree is more robust. Moreover, the difference between the obtained results for three consecutive forecasting hours is quite small (around 0.01) for both R-coefficient and RMSE: this means the increase of the horizon does not rapidly QD SpringerS.Alawadiet al. 2 16 “ 7 Friedman rank of R-Coefficient 18 1“ 2 10 Friedman rank of MSE. SASS PEEP EPEPEIEL SF? Fig. 1 Friedman rank of R-coefficient (upper panel) and RMSE (lower panel) for the 20 best regressors 0 £ @ affect the extraTrees accuracy. Other regressors with good performance are random forest, cubist, gradient boosting of regression trees (bstTree), average neural network committee (avNNet), and kernel ELM (elm-kernel). There is a high agreement between average values and Friedman ranks in the results. This comparison might be useful for indoor temperature prediction for any smart build- ®D SpringerA comparison of machine learning algorithms.. 0978 0.950 — 094s sanded 0940 EO PEBP EEF P REAP EPL EIS Fig.2. Average values of R-coeflicient over the datasets of the 20 best regressors to forecast three consecutive hours ing, which facilitates building a ML forecast model to improve the energy efficiency, reduce energy consumption, and manage a building’s assets. Threats to validity A potential threat is that our results may not be valid in all HVAC systems. As we have not made any particular assumptions, and as the HVAC does not have any unique features, we believe that our results can be generalized to most other HVAC Systems. However, further research is needed to confirm this. Our study may have been internally biased from the settings of the experiments because the data was collected during winter periods in two different years with different weather conditions. Testing all algorithms using data collected during summer periods may produce different results, however, based on previous studies, the ExtraTree will obtain the best results in all scenarios [63]. Moreover, the algorithm hyper-parameters values were tuned according to the default settings shown in the Table 3 used in our study and the results are quite good. However, if we search for the optimal values of those parameters which will affect the learning process, we may get a slight improvement in the accuracy of the algorithms. The experiments were repeated 10 times to make it statistically significant, and the mean was calculated to ensure the result was correct and avoid any execution errors. 5 Conclusions and future work In this paper, we compared a set of 36 ML algorithms that belong to 20 different families to forecast the indoor temperature for three consecutive hours using real data collected from both a smart building and a weather station every 10 min. This comparison showed that the ExtraTrees algorithm performs best in terms of both the R-coefficient (0.97%) and RMSE (0,058% ): it also ranks the highest according to the Friedman test. Other algorithms performed well are the random forest, averaged neural QD Springer5S. Alawadi et al. network (AVNNet), cubist, gradient boosted machines with regression trees, extreme learning machine with Gaussian kernels, and support vector machine for regression The outcomes of this study show that the extraTrees is more robust to outliers and data noise, while most of the algorithms such as ANN are highly sensitive to data noise. Furthermore, increasing the forecasting time does not decrease the accuracy of the best model. We found that the difference between the obtained results for three consecutive forecasting hours is insignificant (around 0.01) for both R-coefficient and RMSE: this means that the increase of the horizon does not rapidly affect the accuracy of extraTrees. Finally, it is possible to use a standard ML algorithm to forecast the indoor temperature with reasonable accuracy based on weather and sensors data linked to the smart building However, more research efforts should be made in the future to optimize the HVAC parameters based on the prediction of the indoor temperature. Researchers need to con- sider the following: integrating an incremental training and online learning approach to improve the accuracy and the robustness of the identified model. Real time user feedback during the deployment phase (Interactive learning) for new data behavior that will help in improving model efficiency. Raising the forecast horizon for longer time periods (days ahead), considering user satisfaction (comfort level), and energy consumption. Integrating the winner model (ExtraTree) with building management systems and predicting in real-time. Validating the results in other buildings using other sensor data. Finally, addressing possible noise or missing data linked to sensor failure scenarios during the run time Acknowledgements Open css funding provided by Malmé University. Open Access Thisarticlcis licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If ‘material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit hitp://creativecommons.org/licenses/by/4.0 References 1. Alawadi, S., Delgado, MF, Pérez, D.M.: Machine learning algorithms for pattern visualization in classification tasks and for automatic indoor temperature prediction. Ph.D. thesis, Universidade de Santiago de Compostela (2018) 2. Alawadi, S., Mera, D., Fernéndez-Delgado, M., Taboada, J.A.: Comparative study of artificial neural network models for forecasting the indoor temperature in smart buildings. In: 2017 2nd International Conference on Smart Cities, pp. 29-38. Springer (2017) Aliberti, A., Bottaccioli, L., Macii, E., Di Cataldo, S., Acquaviva, A., Patti, E.: A non-linear autore- _gtessive model for indoor air-temperature predictions in smart buildings, Electronics 8(9), 979 (2019) 4. Alaubi. J., Nayar, A., Kumar, A.: Machine learning from theory to algorithms: an overview. In: 2018, 2nd National Conference on Computational Intelligence (NCCI): Journal of Physics, vol. 1142, p. 012012. 1OP Publishing (2018) Almubi, J.A.: Diversity based improved bagging algorithm, In: 2015 Ist Proceedings of The Interna- tional Conference on Engineering & MIS (ICEMIS) 2015, pp. 35-40. ACM (2015) 6. Alaubi, J.A.: Diversity-based boosting algorithm, Int, J. Adv. Comput. Sei. Appl.7(5} 524-529 (2016) ®D SpringerA comparison of machine learning algorithms.. 7. Alaubi, J.A.: Research article optimal classifier ensemble design based on cooperative game theory. Res. J. Appl. Sci. Eng. Technol. 11(12), 1336-1343 (2015) 8. Attoue, N., Shahrour, L, Younes, R.: Smart building: use of the arti indoor temperature forecasting. Energies 1(2), 395 (2018) 9. ATLAS Collaboration et al: The evolution of boosting algorithms-from machine learning to statistical modelling. Methods Inf. Med. 53(6), 419-427 (2014) 10. Ben-Nakhi, A.E., Mahmoud, M.A.: Cooling load prediction for buildings using general regression neural networks. Energy Convers. Manag. 45(13), 2127-2141 (2004) LL. Breiman, L.: Bagging predictors, Mach, Learn, 24, 123-140 (1996) 12. Breiman, L.: Random forests. Mach, Learn, 45, 5-32 (2001) 13 man, J., Olshen, R.A., Stone, C.J.: Classification and Regression Trees, Wadsworth c Grove (1984) 14, Catalina, T., Virgone, J., Blanco, E.: Development and validation of regression models to predict monthly heating demand for residential buildings. Energy Build. 40(10), 18251832 (2008) 15. Chambers, 1M: Linear models, chapter 4, In: Chambers, J.M., Hastie, TJ. (eds.) Wadsworth & Brooks/Cole (1992) 16. Chen, M., Hao, Y., Hwang, K., Wang, L., Wang, L.: Disease prediction by machine learning over big data from healtheare communities. IEEE Access 8, 8869-8879 (2017) 17. Chun, H., Keles, S.: Sparse partial least squares for simultaneous dimension reduction and variable selection. J. R. Stat. Soe. 72, 3-25 (2010) 18, Chengliang, X., Chen, H., Wang, J., Guo, ¥., Yuan, Y.: Improving prediction performance for indoor temperature in public buildings based on a novel deep learning method. Build. Environ. 148, 128-135 2019) 19, De Jong, S.: SIMPLS: an alternative approach to partial least squares regression, Chemom. Intell. Lab, Syst. 18, 251-263 (1993) 20. Dobson, A.J.: An Introduction to Generalized Linear Models, Chapman and Hall, London (1990) 21. Dong, B..Cao,C., Lee, S.E.: Applying support vector machines to predict building energy consumption in tropical region, Energy Build, 37(5), 545-553 (2005) 22. Doukas, H., Patlitzianas, K.D., latropoulos, K., John, P: Intelligent building energy management system using rule sets. Build, Environ, 42(10), 3562-3569 (2007) 23. Dounis, A.L., Caraiscos,C.: Advanced control systems engineering for energy and comfort management in a building environment a review, Renew. Sustain, Energy Rev. 13(6), 1246-1261 (2009) 24. Ekici B.B., Aksoy, U.T.: Prediction of building energy consumption by using artificial ncural networks. Adv. Eng. Softw. 40(5), 356-362 (2009) 25. Erickson, V.L., Carrcira-Perpiiiin, M.A., Cerpa, A.E.: OBSERVE: Occupancy-based system for cfli- cient reduction of HVAC energy. In: Proceedings of the 2011 10th ACM/IEEE International Conference ‘on Information Processing in Sensor Networks, pp. 258-269. IEEE (2011) 26. EU. Life-OPERE. hitp://www.life-opere.org/en. Accessed June 2019 27. Fernander-Delgado, M., Sirsat, M.S., Cemadas, E., Alawadi, S., Barro, S., Febrero-Bande, M.: An extensive experimental survey of regression methods. Neural Netw. 111, 11-34 (2018) 28. Femdnder-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? J. Mach, Learn. Res. 15(1), 3133-3181 (2014) 29. Fong, K.., Hanby, V.I., Chow, T.-T.: Hvac system optimization for energy management by evolutionary programming. Energy Build. 38(3), 220-231 (2006) 30. Foresee, F.D., Hagan, M.T.; Gauss-Newton approximation to Bayesian regularization. In: International Joint Conference on Neural Networks, pp. 1930-1935 (1997) Friedman, J.H.: Multivariate adaptive regression splines. Ann. Stat, 19(1), I-141 (1991) . Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29, 1189- 1232 (2001) 33. Fricdman, J.H., Stuctale, W. Projection pursuit regression. J. Am. Stat. Assoc. 76(376), 817-823 981) 34. Friedman, M.: Phe use of ranks to avoid the assumption of normality implicit in the analysis of variance. J. Am. Stat, Assoc. 32(200), 675-701 (1937) 35. Garefa, S., Fernandez, A., Benitez, A.D., Herrera, F: Statistical comparisons by means of non- e study on genetic based machine learning. In: Proceedings of the II Congreso (CED12007). V Taller Nacional de Mineriade Datos y Aprendizaje (TAMIDA), al neural network approach for parametric tests: a Espatiol de Informatic pp. 95-104 (2007) QD Springer5S. Alawadi et al. 36. Gelman, A., Jakulin, A., Pittau, M.G., Su, YS.: A weakly informative default prior distribution for logistic and other regression models. Ann. Appl. Stat. 2(4), 1360-1383 (2009) 37. Geurts, P., Emst, D., Wehenkel, L.: Extremely randomized trees. Mach, Learn, 63(|), 3-42 (2006) 38. Goeman, J.J.: L-1 penalized estimation in the cox proportional hazards model. Biom. J. 52, 70-84 010) 39. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527-1554 (2006) 40. Huang, G.-B.,. Zhou, H., Ding, X.. Zhang, R.: Extreme learning machine for regression and multiclass classification, IEEE Trans, Syst. Man Cybern, Part B Cybern, 42(2), 513-529 (2012) 41. Hyvarinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural Netw. 13, 411-430 (2000) 42. Jin,J., Shu, S., Lin, Pz Prediction of 2029-2042 (2019) 43. Kuhn, M., et al.: Building predictive models in r using the caret package. J. Stat. Softw. 28(5), 1-26 (2008) 44, Kwok, S.S.K., Yuen, R.K.K., Lee, E.W.M. An intelligent approach to assessing the effect of building \door air temperature based on deep learning. Sens. Mater. 31(6), ‘occupancy on building cooling load prediction, Buil. Environ, 46(8), 1681-1690 (2011) 45. Langley, P., Simon, H.A.: Applications of machine learning and rule induction. Commun, ACM 38(11), 54-64 (1995) 46. Lawson, C.L., Hanson, R.J.: Solving least squares problems, vol. 15 of Classies in Applied Mathemat- is, Society for Industrial and Applied Mathematics (SLAM) (1995) 47. Li, K., Hongye, S.. Chu, J: Forecasting building energy consumption using neural networks and hybrid neuro-fuzzy system: a comparative study. Energy Build. 43(10), 2893-2899 (2011) 48, Li, Q., Meng, Q., Cai, J., Yoshino, H., Mochida, A.: Applying support vector machine to predict hourly cooling load in the building. Appl. Energy 86(10), 2249-2256 (2009) DJ.C.: Bayesian interpolation, Neural Comput, 4, 415-447 (1992) 50. Maloney, K.O., Schmid, M., Weller, D-E.: Applying additive modelling and gradient boosting to assess, the effects of watershed and reach characteristics on riverine assemblages. Methods Ecol. Evol. 3(1), 116-128 (2012) 51. Meinshausen, N.: Quantregforest: quantile regression forests. R package version 0.2-2 (2007) 52. Mizera, I, Koenker, R.: Convex optimization in r. J. Stat. Softw. 60(5), 1-23 (2014) 53. Morosan, P-D., Bourdais, R., Dumur, D., Buisson, J.: Building temperature regulation using a dis tributed model predictive control. Energy Build. 42(9), 1445-1452 (2010) 54, Nguyen, T.A., Aiello, M.: Energy intelligent buildings based on user activity: a survey. Energy Build. 56, 244-257 (2013) 55. Peters, A., Hothorn, T., Lausen, B.: ipred: Improved predictors. R News 2(2), 33-36 (2002) 56. Quinlan, R.: Combining instance-based and model-based learning. In Proc. Intl, Conf. on Machine Learning, pp. 236-243 (1993) 57. Quinlan, J.R. et al: Learning with continuous classes. In: 1992 Sth Australian Joint conference on Artificial Intelligence, vol. 92, pp. 343-348. World Scientific (1992) 58. Rios-Moreno, G.J.. Trejo-Perea, M., Castaneda-Miranda, R., Hernindez-Guzmdn, V.M.. Herrera-Ruiz, G.: Modelling temperature in intelligent buildings by means of autoregressive models. Autom. Constr. 16(5), 713-722 (2007) 59. Rodniguez-Mier, P,, Fresquet, M., Mucientes, M., Bugarin, A.: Prediction of indoor temperatures for ‘energy optimization in buildings. In: Conference of the Spanish Association for Artificial Intelligence, pp. 675-684 (2016) 60. Rosenzweig, C., Solecki, W., Hammer, action, Nature 467(7318), 909 (2010) 61, Santin, 0.G., lard, L., Visscher, H.: The effect of occupancy and building characteristics on energy use for space and water heating in dutch residential stock. Energy Build. 41(11), | (2009) 62. Shanthamallu, U.S., Spanias, A., Tepedelentioglu, C., Stanley, M.: A brief survey of machine learning methods and their sensor and iot applications. In: 2017 8th International Conference on Information, Intelligence, Systems and Applications (IISA), pp. I-8. IEBE (2017) 63. Sirsat, M.S., Garefa, E.C., Delgado, M.E: Application of machine learning to agricultural soil data. Ph.D. thesis, Universidade de Santiago de Compostela (2017) 64. Smola, A.J., Schélkopf, B.B.: A tutorial on support vector regression. Stat. Comput. 14(3), 199-222 (2004) .A.. Mehrotra, S.: Cities lead the way in climate-change ®D SpringerA comparison of machine learning algorithms.. 65. Song, L., Langfelder, P., Horvath, S.: Random generalized linear model: a highly accurate and inter pretable ensemble predictor. BMC Bioinform. 14(1), 1-22 (2013) 66. Specht, D.F: A general regression neural network, IEEE Trans. Neural Netw. 2, 568-576 (1991) 67. Siilo, Idil, K., Seref, R., Dogan, G.,. Brown, T.: Energy efficient smart buildings: LSTM neural networks for time series prediction, In: 2019 International Conference on Deep Learning and Machine Learning in Emerging Applications (Deep-ML), pp. 18-22. IEEE (2019) 68. Tang, J.. Deng, C., Huang, G.-B.: Extreme learning machine for multilayer perceptron, IEEE Trans. Neural Netw. Learn, Syst. 27(4), 809-821 (2015) 69, Williams, C.K.L, Barber, D.: Bayesian classification with Gaussian processes. IEEE Trans. Pattern Anal, Mach, Intell. 20(12), 1342-1351 (1998) 70. Wood, S.N.: Fast stable restricted maximum likelihood and marginal likelihood estimation of semi- parametric generalized linear models, J. R. Stat, Soe. 1(73), 3-36 (2011) 71. Zhang, T.: Adaptive forward-backward greedy algorithm for learning sparse representations. IEEE ‘Trans. Inf. Theor. $7(7), 4689-4708 (2011) 72. Zhao, H., Magoules, F.: A review on the prediction of building energy consumption, Renew. Sustain. Enerey Rev. 16(6), 3586-3592 (2012) 73. Zou, H., Hastie, T.: Regularization and variable selection via the elastic net. J.R. Stat, Soe. Ser B (Stat. Method.) 67(2), 301-320 (2005) Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations, Affiliations Sadi Alawadi'@ - David Mera? - Manuel Fernandez-Delgado? - Fahed Alkhabbas! - Carl Magnus Olsson! - Paul Davidsson! David Mera david.mera@ use.es Manuel Ferndndez-Delgado manuel [email protected] Fahed Alkhabhas [email protected] Carl Magnus Olsson [email protected] Paul Davidsson [email protected] 1 Internet of Things and People Research Center Department of Computer Science and Media ‘Technology, Malm University, 20506 Malm, Sweden 2 Centro Singular de Investigacién en Tecnoloxias da Informacién (CiTIUS), Universidade de Santiago de Compostela, Santiago de Compostela, Spain QD Springer