Abstract:

With new devices and new network technologies coming up, it has become an
inevitable task to provide services of a minimum quality. Setting feasible Service
Level Agreements (SLAs) is the need of the hour. This, being a part of network
provisioning and providing the best possible Quality of Service (QoS) is very vital
and helps improve user perceived quality or the Quality of Experience (QoE). QoE
evaluation helps Internet Service Providers (ISPs) understand their user satisfaction
better and this goes hand in hand with providing adequate network QoS. Moreover, in
this era of competition, the ISPs themselves will have to be evaluated based on their
QoE and QoS metrics to know their true position in the market in terms of
performance against their peers/competitors. This evaluation is usually done on a per-
metric basis. However, we see from current performance data that all the ISPs fare
well on some metrics and need improvement in the others. It is a fact that no ISP fares
bad on all given metrics and leads to an understanding that per-metric based
evaluation may be a biased form of evaluating performance. Hence, this research has
attempted to use an intelligent, robust mathematical technique called the Data
Envelopment Analysis (DEA) with its Slack Based Measure (SBM) approach. DEA is
a proven, tested and tried technique that is in use in major industries even today.
Being a multiple criterion evaluation methodology based on linear programming, it
works well on multiple inputs and multiple outputs. DEA gives the overall, relative
efficiency of the ISPs which gives us the true position of the provider against its
peers.
The Slack Based Measure provides the Output Slacks that show the potential
improvement that the lagging ISPs can make to be in par with their peers/competitors.
The Output targets that are provided by the technique can be used as benchmarks for
SLAs. These could also be used as achievable goals to be met by the ISPs needing
improvement in any of the QoE/QoS metrics, so they can be in par with their peers.
The technique also recommends Reference/Benchmark providers that can be used to
learn/adopt best practices from. In all this, the DEA technique provides feasible goals
that can well be achieved by the ISPs that are less performing/ lagging compared to
the others ahead. This technique could thus be used to evaluate an ISP end-to-end
based on user QoE and network QoS management and understand the providers’ true
position, also paving the way to their improvement by the feasible goals that are
provided. Moreover, as a second step, the research also uses the Radial Super
ii
Efficiency approach that helps in ranking the best performing countries/ISPs and
helps set stretch goals/ stretch targets, so they can challenge themselves and achieve
these goals in the long run, thereby providing the best possible network services to
their citizens/subscribers. The research holds good for 4G, 5G and future generations
of networks as well. The study has been conducted both at an operator level and at a
regional level and has proved to be a promising effort.
This kind of benchmarking is needed in this era of changing and turbulent business
scenarios. Benchmarking is a process that is necessary for continued improvement in
processes and results. Benchmarking can be done within the organization or between
firms. There is also a need for optimal benchmarking processes that really help the
managers achieve their organizational goals in an easy and the best achievable way.
DEA is a proven technique that caters to this need of the organizations. It has been
used in many major industries and the telecommunications industry is no exception.
This technique has evolved over the years to accommodate the changing needs of the
organizations. What started as conventional DEA has evolved into many different
models of DEA that overcome the limitations of the traditional model.
For instance, the conventional DEA did not have answers as to how the potential
improvement can be measured for an inefficient DMU (Decision Making Unit-ISPs or
countries) and the result was the development of the Slack Based Measure in DEA
that provided the output slacks that help in quantifying the improvement needed by
inefficient DMUs and this helps in understanding by how much the DMU must
improve in a particular metric, as the analyses in this research show. Moreover, the
traditional DEA models did not have a way to differentiate and rank the efficient
DMUs. This has been overcome with the development of the Radial Super efficiency
model that gives the efficient DMUs different scores of over unity and thus helps with
effective ranking of the efficient DMUs. This technique also helps provide these best
performing DMUs with stretch goals or targets that can be met in the long run.
QoE and QoS of organizations, especially in the telecommunications service arena are
very important indicators of performance. Hence their evaluation is an important
aspect of organizational management. This research takes us through how this can be
done using the DEA technique and its advancements in an optimal way. DEA utilizes
linear programming that makes it the right technique to use when there are multiple
inputs/outputs involved. The research shows us how this technique can be utilized
effectively for end-to-end performance evaluation, effective QoE/QoS management
iii
and Internet Service Provider enhancements. The advantage of this technique is that it
helps set achievable goals for QoE/QoS metrics for the DMUs needing improvement
and has enhancement plans for both the DMUs that do not perform in par with their
peers and for those that excel in their performance.
Keywords: Data Envelopment Analysis, Telecom service providers, Benchmarking,
Slack Based Measure, Super-efficiency

iv
 TABLE OF CONTENTS

1. Introduction 1
2. Motivation of research 2
3. Design phases 2
4. Scope of research 3
5. Business relevance / Social impact 3
6. Base references 4
7. Review of literature 5
7.1 Video experience 5
7.2 Games experience 5
7.3 Voice app experience 5
7.4 Download speed Experience 6
7.5 Peak download speed Experience 6
7.6 Upload speed Experience 6
7.7 Fault Repair > 90% in 1 Working Day and >= 99% in 3 Working
Days 6
7.8 Broadband connection speed from ISP to user (Download speed) 7
7.9 Service availability/uptime 7
7.10 4G Coverage 7
7.11 Reach/Location Experience 7
7.12 Data Envelopment Analysis (DEA) 8
7.13 Slack Based Measure (SBM) 8
7.14 Radial Super efficiency DEA model 9
8. Findings and summary from literature 9
9. Methodology 10
9.1 Conceptual Framework 10
9.1.1 Functional blocks of framework 11
10. Analysis and results 12
10.1 Analysis 1 results & discussion 12
10.2 Analysis 2 results & discussion 15
10.3 Analysis 3 results & discussion 20

v
10.4 Analysis 4 results & discussion 25
10.5 Analysis 5 results & discussion 31
11. Managerial Contributions 35
12. Limitations and Directions for Future Research 36
13. Goals achieved by research/Conclusion 39
14. Publications 40
References 40

vi
 List of Tables

1 Data Envelopment Analysis – Overall Efficiency 13

2 SBM recommendations for all providers to achieve
100% efficiency 13
3 Super efficiency recommendations of stretch targets
for efficient DMUs 14
4 Current performance of small and medium wireline
broadband service providers 15
5 Reference/Benchmark providers for the inefficient
service providers 16
6 Optimal Targets using SBM in DEA 17
7 Stretch targets using super efficiency Output targets for
goal setting using DEA 19
8 Current User experience on 4G mobile networks 20
9 Relative overall efficiency of top 4 Indian telecom service
Providers 21
10 Improvement needed for BSNL 22
11 SBM recommendations for all providers to attain/be at 100%
Efficiency 22
12 Super-efficient service providers with their stretch targets 24
13 Current QoE performance of key countries in the Asia Pacific
Region 26
14 Relative benchmark countries and their weightages 27
15 Output targets/recommendations for APAC countries/QoE
Management 28
16 Stretch targets for efficient countries using Radial Super
Efficiency 30
17 QoE performance of EMEA countries as of June 2021 31
18 Benchmark countries for inefficient EMEA countries 33
19 Output targets for goal setting using DEA 33
20 Super-efficient DMUs and stretch targets 35

vii
 List of Figures

1 Process flow/methodology 10
2 End to end evaluation of CSPs using DEA

11
3 QoE/QoS management using Slack Based Measure in
DEA 11
4 CSP enhancement using DEA 11
5 Efficiency scores after applying Radial Super efficiency
Model 14
6 Efficiency vs Super efficiency 18
7 BSNL – Improvement needed 21
8 Super-efficient DMUs with their new efficiency scores 23
9 References/Benchmark providers for all service providers 24
10 Efficient and Inefficient DMUs (Countries) 27
11 Efficient DMU ranking using super efficiency 29
12 Efficient and inefficient DMUs 32
13 Super-efficient countries with their new efficiencies 34

viii
ix
1. Introduction
Quality of Experience (QoE) and Quality of Service (QoS) are two important parts of
network services. QoE is an indicator of customer/user satisfaction of network services
quality, while QoS is part of network services provisioning. These go hand in hand in
that QoE provides feedback over the network, and this helps with better network
provisioning. It is important that these are benchmarked optimally for high quality of
QoE/QoS management. There are many processes in place for this purpose. However,
this benchmarking is done at a per-metric level today and contributes to bias. This
research has attempted to eliminate this bias by using a mathematical technique based
on Linear programming, by name, Data Envelopment Analysis (DEA) with Super
efficiency. This technique has provided an ideal framework for optimal benchmarking
by considering a whole set of QoE/QoS metrics as opposed to the current per-metric
benchmarking standards. The use of Linear Programming enables the analysis to be
carried out in the multidimensional space, with multiple inputs and multiple outputs.
The technique helps with the three main functionalities of Telecom management –
Evaluation of service providers end-to-end, QoE/QoS management and Service
provider enhancement.
The end-to-end service provider performance evaluation is made possible by the
overall efficiency scores that are given out by the technique. The super efficiency
approach further ranks the relevant peer providers and gives the true position of the
service providers. Thus the true standing of the service provider with respect to its
relevant peers is identified using the DEA approach. Moreover, the Slack Based
Measure (SBM) gives out output slacks for the inefficient/less performing providers,
that help estimate output targets that can be set as benchmarks. These are feasible
targets that are set by the DEA technique, making it easier to achieve. These targets
are short term targets for the lagging providers. Since there is always scope for
improvement, the super efficiency approach provides stretch/long term targets for the
efficient/best performing providers. This helps further enhance the service providers.
The DEA technique also provides references/benchmark providers. The inefficient
providers could learn from these relevant reference peers that are efficient in
themselves and adopt best practices. This way the telecom management cycle is fully
covered by the application of this sophisticated, intelligent technique that is so
mathematically precise.

1
2. Motivation of research
• Current benchmarking of QoE/QoS metrics for purposes like SLA targets are
done on a per- metric basis.
• Different service providers have their own strengths and weaknesses.
• This leads to bias in the way realistic targets are set.
• Unrealistic targets result in penalising service providers in an unfair manner.
• DEA, a Linear programming application helps set optimal targets in QoE/QoS
management by evaluating the service providers across a whole set of
performance metrics, eliminating bias.
• An ideal framework is identified for Network planning and management that
helps prepare infrastructural and other functional aspects towards roll out of
5G and future network technologies.
3. Design phases
I. End to end evaluation of Communication Service providers; QoE/QoS
management; CSP Enhancement.
Phase 1
 To evaluate and benchmark the CSPs over the 4G mobile broadband
network based on multiple user QoE metrics using the DEA-SBM approach.
Phase 2
 To optimally benchmark and provide recommendations for the ISPs over the
wireline broadband network based on multiple service provider’s QoS
metrics using the DEA-SBM approach
Phase 3
 To deepen network penetration and restrain service Pricing by improving
Quality of Experience in the Indian 4G public telecommunication Sector
II. Network Planning and Management
Phase 4
 To rank and benchmark the countries in the Europe and Middle Eastern
(EMEA) region that have adopted the 5G technology and provide
recommendations, increasing scope for improvement for the ones lagging and
for the ones that are performing well

2
 Phase 5
 To evaluate Asia Pacific region countries (APAC) that have adopted 5G
and provide insights on the countries that are lagging and for those doing well
in the current scenario.
4. Scope of research
 Works across all network types irrespective of technology used. Tested on 4G
and 5G networks and can positively be used in future network technologies.
 Multidimensional analysis made possible.
 Any set of KPIs/KQIs could be used for analysis for efficient target setting
 Multiple applications can be supported, for example,
 evaluating service provider performance
 Improving performance of public telecom sector
 Network planning and management
 Granular research possible at different DMU levels – Mobile Network
operator level/ Country level/ regional level

5. Business relevance / Social impact

 The proposed method can be used to save costs and maximize profit
 Saves significant managerial effort and time in setting and achieving targets
 Helps managers in realistic goal setting
 Impacts Business productivity in a significant way
 Helps understand competition within industry
 Smaller organizations with less analytical presence are benefitted.

3
6. Base references

S.No. Reference Technical Discussion

1 Halvaiee et al., (2022) 1.Service providers strive towards high conversion

rate and customer loyalty
2.Prediction and optimization of QoE before and after
providing service to user respectively highly essential
3.Robust and comprehensive method or set of
parameters for QoE evaluation, need of the hour
2 Chatterjee (2022) 1.4G integrates with 3G with fixed internet
bandwidth to support wireless mobile internet
2.5G brings about a real Wireless World Wide Web
(WWWW)
3.6G integrates 5G with satellite networks for global
coverage
3 Koltai et al., (2022) 1.Accurate and scientific efficiency information
needed for managers and healthcare policy makers
2.Several measures and indicators used in the field
lead to complication in evaluation
3.Output oriented, slack based DEA models are used
for effective efficiency evaluation
4 Wanke et al., (2022) 1.Dynamic slack-based measure model used to
measure bank efficiency
2.Regression of efficiency scores on bank ownership
types and other contextual factors
3.Foreign banks outperform their domestic
competitors
5 Sun et al., (2021) 1.Low latency critical for QoE for live video
streaming
2.Live streaming algorithms developed to achieve
almost optimal performance
6 Lafta et al., (2021) 1.Video and audio transmission in real time apps
leads to high demand on cellular networks
2.QoS therefore is a major problem in these cases
3.Analysis of existing QoS parameters helps improve
QoS in UMTS networks
7 Morita et al., (2005) 1.DEA is a data-oriented performance evaluation
method where multiple inputs and outputs are used
2.Non-zero slack values indicate inefficiency
8 Tone (2001) 1.DEA deals with input excesses and output shortfalls
of Decision- Making Units involved
2.DEA is units-invariant
3.Valid efficiency measurement tool
9 Avkiran et al., (2008) 1.DEA has two measures of efficiency – radial and
non-radial
2.Slacks Based Measure was an advancement over the
conventional DEA model and is non-radial in nature

4
 10 Ranjha et al., (2021) 1.6G marks the beginning of URLLC (Ultra Reliable
Low Latency Communication)
2.URLLC empowers real-time wireless systems,
smart grids, and industrial applications

7. Review of Literature
Benchmarking is known as “an essential tool for continuous improvement of quality”.
Benchmarking is an important priority in business practice, Per [1], 50% of the 1000
French companies used benchmarking on a regular basis and 80% of them looked at
them as a tool that brought about effective change.
The most widely accepted form of benchmarking is comparing organizational
performance within or between companies [2].
7.1 Video experience:
The user perceived Quality of Service or the QoE is largely influenced by the network
QoS of the service provider. This also applies to Video streaming and video
conferencing applications. User experience or QoE for video streaming expects low
jitter and very low packet loss. Video conferencing too needs higher QoS while audio
streaming can afford to have median QoS [3].
7.2 Games experience:
With more people embracing gaming through PC, mobile applications, online services
and cloud services, it has become popular today. Research made on gaming through
mobile applications and online platforms show that user satisfaction inclined towards
application based gaming and big screen displays where quality of visuals was better
than on online gaming. Multiplayer gaming needs very low latencies and for this, new
standards in data transmission and routing need to be introduced (Panwar, 2020).
Experience in gaming depends on game development, gaming server deployment and
game play [4,5].
7.3 Voice app Experience
With the recent trend in mobile users being flexible with voice and video usage, there
arises the need for good speech quality at the receiver. Voice and video transmission
over the same network reduces speech quality, being affected by factors like coverage
area, data rate and movement. Using heterogenous networks like 4G is a good
solution [6].

5
For voice to be transmitted through the internet, low bandwidth channels are needed.
Packets sent must be of low latency, so there is less jitter. Voice over the internet
requires low latency and jitter [7]. QoE of voice user also depends on the rates of
packet loss, delay and encoding [8].

7.4 Download speed Experience

The rate at which data reception from the internet occurs is download speed and this
is a good indicator of browsing experience. Test cases were made on single and
multiple video browsing and interactive behaviour-based browsing. Each case used a
unique combination of upload and download speeds. Different KPIs (Key
Performance Indicators) like Page Load time, Initial loading delay, Total stalling time
were studied. These parameters affected the QoE of the user [9].
7.5 Peak download speed Experience
A Cloud based Applications Speed Platform (CLASP) has been proposed by studies
for measuring the performance between virtual machines and different networks in
cloud regions using speed test servers from the internet. The measurements were done
for 5 months and showed a reduction in the Download throughput by 50% compared
to the peak throughput 1.3-3% of the time. For more than 10% days of research
congestion was observed in 30-70% ISPs causing 50% drop in the download
throughput than the peak throughput of the day. The download/upload throughput
variability was analyzed to detect congestion. Congestion was observed in the peak
hours and there was high intra-day variability. Some regions of study showed a packet
loss of 3-50% during peak hours [10].
7.6 Upload speed Experience
The rate at which data is sent to the internet is upload speed and is helpful to study
responsiveness of applications in real-time [11]. Uploading large files over the
internet via mobile communications has increased a lot, resulting in bottlenecks in
server processing ability and insufficient bandwidth availability. Many measures are
being taken to improve the QoE of users. One such study used network resources at
the edges and uploaded parallely divided files. This improved the upload speed ten
times over [12].
7.7 Fault Repair (>90% In 1 Working Day And >=99% In 3 Working Days)
Fault repair is an essential KPI in evaluating service providers’ QoS. In India the
service providers are expected to complete 90% of the repairs in one working day and
6
99% repairs in 3 working days, according to the Telecom Regulatory Authority of
India (TRAI). For networks to perform well, the Mean Time to Repair (MTTR) must
be less than the Mean Time Between Failures (MTBF). This is closely related to
reliability and availability of the network [13]. Any delays in correcting faults will
result in service providers paying the price, since customers will move away from
them, as QoE deteriorates.
7.8 Broadband connection speed from ISP to user (Download speed):
Network utilization by consumers includes video, voice and other applications. Since
consumers use varied applications, their speed needs vary and pose distinct demands
on their broadband connections. For instance, basic browsing and email handling
require low download speeds of 100-300 kbps. Standard Definition (SD) video
streaming needs 1Mbps-5Mbps, while High Definition (HD) needs 5Mbps-10Mbps or
more. Download speed is also required for upstream traffic, for example, HD video
conferencing requires fast upload speeds to transmit voice, image and also requires
good download speeds for retrieving the voice and image on the other side. QoS
parameters like availability, latency and jitter are significant like download and
upload speeds of a network connection for many applications. Based on these
demands distinct customer profiles are possible on the network [14].
7.9 Service availability /uptime (for all users)
To keep the survivability of the network despite attacks, availability plays a vital role,
ensuring mission-critical data when needed. Availability thus caters to customers with
a high standard of services and plays a key role on applications. System-level failures
are not appreciated, with only a small service loss acceptable. System failures from a
service provider’s point will lead to large losses.
Long Term Evolution (LTE) technology is expected to provide strong privacy and
guaranteed availability to users. Factors like issues in hand-off, synchronising
location, coherence in addition of new users, multicasting and QoS support, wireless
security and authentication, failure and backup of the network have an adverse impact
on availability of networks. QoS support issues could occur at multiple levels of
network flow.
7.10 4G Coverage
To initiate communication with mobile terminals, it is essential to have suitable
wireless coverage. Good radio coverage must therefore be guaranteed by mobile

7
network operators [15]. Cellular networks like 4G are expected to provide seamless
coverage with guaranteed QoS to the user over the complete service area [16].
7.11 Reach/Location Experience
mmWave communication has caused a paradigm shift in localization capabilities
where both joint communication and position location utilize the same infrastructure.
Accurate localization in futuristic smartphones is made possible by massive
bandwidths along with high gain directional MIMO (Multiple Input, Multiple Output)
antenna at mmWave frequencies [17]. 5G has brought about accurate localization
experiences with wide bandwidth and narrow beamwidth through highly directional
antenna arrays [18]. The IEEE 802.11 ad standard supports an indoor usage of 60
GHz mmWave band from 57 GHz-71 GHz [19].
7.12 Data Envelopment Analysis (DEA)
DEA is a multiple criterion evaluation methodology that maximises output and
minimises input. The overall performance of an organization is gauged using DEA.
This overall performance is obtained by the composite measure that is a result of
aggregating a set of individual indicators [20]. DEA is an empirical based
methodology and eliminates some assumptions and limitations of conventional
efficiency measurement techniques.
Many studies have used DEA to determine the efficiency of services provided by
telecommunications industry and others. A study evaluated 42 countries in the
wireless communications sector [21]. Yet another study proposed benchmarks to the
wireless communications sector of the USA [22]. DEA was also used to evaluate the
performance of Korean telecommunication companies [23]. An International
Telecommunications Union (ITU) based ranking of the telecommunication industry
and performance assessment was done [24].
7.13 Slack Based Measure (SBM)
SBM is an optimum solution that is better than the efficiency scores of DEA, in that,
it deals with input excesses and output shortfalls of the DMU-Decision-Making Unit.
The conventional Charnes, Cooper and Rhodes (CCR) model provided ratio
maximization while the SBM model provides output maximization. Being a tool of
efficiency measurement, SBM is compatible with other efficiency measures. This
technique considers both ratio efficiencies and the slacks. The SBM technique could
be understood as the product of output and input efficiencies [25].

8
Some research that utilized the SBM approach are as follows: The regional green
efficiency was estimated by analysing regional differences and suggested
improvements towards achieving sustainable social development in China [26]. Mix
efficiencies in a fuzzy environment were measured for the State Bank of Patiala, India
[27]. Industrial green efficiency of provinces in China were measured [28].

7.14 Radial Super efficiency DEA model

The Radial Super efficiency model is used to rank the efficient DMUs which are
otherwise capped at the same value of unity. The DMU under evaluation is taken out
of the reference set, hence an efficient DMU is removed from the efficiency frontier
of the production set. This lets them become super-efficient with different scores that
are above 100% .
Recent studies have used a Super efficiency SBM DEA approach to provide
suggestions to enhance flood disaster ability and considered vulnerability and
resilience changes in 28 provinces of China [29]. A super efficiency SBM model was
used to construct a relative efficiency frontier. A regression model was built using
Machine learning algorithms to construct an absolute efficiency frontier. A
conventional DEA model was then used to compare these [30]. An investigation of
economic role of multinational construction companies was done with 41 countries
and regions by measure of overall economic and linkage performances. Efficiency
indicators were measured, and ranking was done using a super efficiency DEA model
[31].

8. Findings and summary from Literature

 QoE/QoS management need of the hour
 Current industry standards are inefficient for benchmarking/setting QoE/QoS
targets
 Efficient target setting standards not in place today
 All QoE/QoS management is done at a per metric basis
 Need for a model to evaluate network performance across multiple QoE/QoS
metrics
 Data Envelopment Analysis with its Slack Based Measure, a mathematically
precise technique for benchmarking widely used in major industrial areas today
 Radial DEA approach overcomes limitations of conventional DEA

9
 Together, conventional and advanced models of DEA promising in development
of the proposed conceptual framework in telecommunications industry

9. Methodology
9.1 Conceptual framework

Fig 1 Process flow/methodology

10
9.1.1 Functional blocks of the framework:
 Evaluation of CSPs across a set of QoE/QoS Metrics-Facilitates end-to-
end evaluation of CSP:
The Envelopment model of the DEA technique gives the relative efficiencies
of CSPs that can be used to group the CSPs as efficient (best performing) and
inefficient (lagging their peers/competitors).

Fig 2 End to end evaluation of CSPs using DEA

 QoE/QoS Management:
The Slack Based Measure (SBM) is used to set the output targets for QoE/QoS
metrics for the lagging CSPs for efficient QoE/QoS management.

Fig 3 QoE/QoS management using Slack Based Measure in DEA

 CSP Enhancements:
Both inefficient and efficient CSPs can improve themselves, with the
inefficient ones learning from their efficient peers and the efficient ones by
achieving the stretch targets set by the Radial Super-efficiency approach.

11
 Fig 4 CSP enhancement using DEA

10. Analysis and results

There were five analysis that were attempted using the DEA technique with Super-
efficiency.
1) Top 4 providers over 4G cellular network
2) Top 20 small and medium providers over wireline Broadband network
3) Compare public and private telecom sectors for fair pricing and better network
penetration
4) Evaluate 5G adopted Asia Pacific regional countries
5) Evaluate 5G adopted Europe and Middle Eastern countries
Data sources:
Opensignal.com
Telecom Regulatory Authority of India
Time period:
Analysis 1: As of September 2020, including data over 90 days starting from May 1,
2020
Analysis 2: July 2020- September 2020
Analysis 3: July 2020-September 2020
Analysis 4: As of September-2021 over a period of 90 days
Analysis 5: June 2021
Software used: DEA Frontier Solver, dear

10.1 Analysis 1 Results & Discussion

12
On Improving Quality of Experience of 4G Mobile Networks – A Slack Based
Approach
From the data, we see that Airtel performs the best for Video experience and
Vodafone-Idea (Vi) follows closely while Vi performs the best for all the other
metrics in the study. However, BSNL seems to lag all the other service providers.
The question that needs to be answered here is this – Is Vi really the undisputed leader
as the data shows and is BSNL really lagging by a huge margin thus performing way
below the others?
We answer these questions by using the Data Envelopment Analysis. This technique
provides the relative efficiencies of all the service providers.
Table 1 shows the resultant relative efficiencies of the providers.

Table 1: Data Envelopment Analysis – Overall Efficiency

Service Efficiency
Provider score (0-1) Efficiency %
Airtel 1.00000 100%
BSNL 0.89108 89.1%
Jio 0.96325 96.3%
Vi 1.00000 100%

Table 2 shows the final outcome from the DEA-SBM technique that summarizes the
entire analysis.
Table 2 SBM recommendations for all providers to achieve 100% efficiency

Video experience Games experience Voice app experience Download speed Upload speed experience
Service
(0-100 pts) (0-100 pts) (0-100 pts) experience (Mbps) (Mbps)
Provider

Actual Target Actual Target Actual Target Actual Target Actual Target
Airtel 57.9 57.9 56.3 56.3 75.6 75.6 10.5 10.5 2.9 2.9
BSNL 43.1 51.5↑ 39.1 56.0↑ 67.9 67.9 4.3 10.1↑ 1.7 3.6↑
Jio 52.3 55.7 ↑ 50.1 60.5 ↑ 73.4 73.4 6.9 10.9↑ 2.3 3.9↑
Vi 57.8 57.8 62.8 62.8 76.2 76.2 11.3 11.3 4.0 4.0

If all the four service providers were to have the metrics as shown in Table 5, they
would perform equally.

13
The DEA technique shows how these two providers Jio and BSNL need not really
meet each metric value that their peers are currently at, but only need to follow the
shortest or optimal path that the technique recommends, in order to reach the
efficiency frontier or the envelop or in simple terms the highest efficiency and be in
par with their peers Airtel and Vi. The Conventional DEA, however, does not
differentiate the efficient DMUs. Hence, an effective ranking of efficient DMUs is not
possible. For this purpose, the Radial Super efficiency model was developed.
Figure 2 shows the new efficiency trend after the Radial Super efficiency approach
has been applied.

Fig 5 Efficiency scores after applying Radial Super efficiency model

Besides ranking the efficient DMUs, the Radial Super efficiency model also provides
some targets that can be set as stretch targets or long-term goals for the efficient
DMUs. This gives the efficient DMUs an opportunity to enhance themselves,
although they are already performing well.
The DMUs that appeared as inefficient in the conventional DEA approach are not
given any more target to achieve by the Super efficiency approach. They continue to
have only the feasible targets that were assigned to them by the traditional DEA
model. This way, they do not have the pressure to achieve unrealistic targets.

14
 Table 3 Super efficiency recommendations of stretch targets for efficient DMUs

Thus, both efficient and in efficient DMUs have the opportunity to upscale
themselves so they could provide the best services to their subscribers/customers.

10.2 Analysis 2 Results & Discussion

Towards a Smarter Connected Society by Enhancing Internet Service Providers' QoS
metrics using Data Envelopment Analysis
Twenty small and medium wireline broadband service providers who had an All-India
presence at the time of study and did not undergo a name change during the year of
study were used for the analysis.
The conventional DEA model is used as a first step in this analysis to identify the
inefficient providers. These inefficient providers are given the recommendations of
the right group of peers to learn best practices from, to enhance themselves.
Table 4 shows the current performance data of the 20 small and medium wireline
providers chosen for study.
Table 4 Current performance of small and medium wireline broadband service
providers

15
Table 5 shows this data where both reference/benchmark providers and their
weightages are provided by the DEA technique.

Table 5 Reference/Benchmark providers for the inefficient service providers

16
Service
Optimal Lambdas with Benchmarks
Provider
Airlink Meghbela Cable ONEOTT
Communicatio 0.160 & Broadband 0.840 Intertainment
ns Pvt. Ltd Services Limited
Asianet
RailTel World Phone
Satellite
0.578 Corporation of 0.418 Internet Services
Communicatio
India Ltd Pvt Ltd
ns Ltd
Atria
DEN Limras Eronet
Convergence RailTel Corporation
Broadband Pvt. 0.170 0.724 Broadband Service 0.103
Technologies of India Ltd
Ltd Pvt.
Pvt
ESTO Limras Eronet
Broadband 1.000 Broadband
Pvt.Ltd Service Pvt.
GTPL D-Vois RailTel
Broadband 0.513 Communications 0.483 Corporation of
Pvt.Ltd. Pvt.Ltd. India Ltd
D-Vois
Hathway Cable Meghbela Cable & RailTel Corporation
0.166 Communications 0.230 0.601
& Datacom Ltd Broadband Services of India Ltd
Pvt.Ltd.
RailTel World Phone
Indinet Service
0.333 Corporation of 0.667 Internet Services
Pvt Ltd
India Ltd Pvt Ltd
D-Vois ONEOTT
RI Networks Fusionnet Web
0.000 Communications 0.935 0.065 Intertainment
Pvt Ltd Services Pvt. Ltd
Pvt.Ltd. Limited
Siti Broadband D-Vois ONEOTT
Fusionnet Web Meghbela Cable & RailTel Corporation
Services 0.355 Communications 0.007 0.192 0.304 Intertainment 0.137
Services Pvt. Ltd Broadband Services of India Ltd
Pvt.Ltd. Pvt.Ltd. Limited

12 providers, i.e., 3 in 5 providers needed no improvement in any of the 6 QoS

metrics. However, the rest of the 8 providers, i.e., 2 in 5 providers needed
improvements to be made in atleast one metric. These providers are therefore
recommended benchmark/reference providers so they can improve themselves. Table
2 gives these details. The number of benchmark/reference providers needed for these
8 providers range from 1 for Esto Broadband network to as high as 5 as for Siti
Broadband Services Pv. Ltd. These reference/benchmark providers are arrived at
based on the position of each provider on the envelop that is generated by the DEA
technique.
As for the 12 providers that do not need improvements, 8 of them emerge as
benchmark providers. We see that the frequency of appearance of these benchmark
providers also varies, as seen from Table 5.
Additional vital information that these results give us is the weightage of each of these
benchmark/reference providers. This is made possible with the help of distance
mapping using the multidimensional envelope generated by the DEA technique.
When a service provider needs improvement in performance, it is essential to
understand which QoS metrics need to be worked on and by how much they must
improve. These insights and results are displayed in Table 6.

17
 Table 6 Optimal Targets using SBM in DEA

↑Indicates improvement needed to achieve 100% efficiency

From the results, we understand that no service provider needs improvement in all of
the 6 QoS metrics. The improvements that are recommended are all optimal paths
provided by the DEA model to achieve an overall efficiency score of 100%.
Table 6 shows the shortest/best/optimal path to the envelope that is formed by the
DEA technique. The advantage of using this mathematical technique is that it also
recommends marginal improvement.
These references are recommendations to the inefficient providers to learn best
practices from to enhance themselves.
We see from Figure 5, that 12 out of 20 providers are efficient and the other 8
providers are inefficient or ‘weakly efficient’ (all 8 are tagged as inefficient in this
research). All efficient providers were assigned a value of 1. Some of these providers
are weakly efficient and are also assigned a value of ‘1’. In this research we have
considered these weakly efficient providers as inefficient as they also have
improvement needs with non-zero output slacks. So, the ones that stand out efficient
outrightly, i.e., they are assigned a value of ‘1’ and they do not have any output

18
slacks, need to be differentiated to know the true position of each of these efficient
providers.
To overcome the ranking problem in traditional DEA, the radial super efficiency
approach is used as a second step. Here all the efficient providers emerge super-
efficient and are assigned different efficiency scores of above ‘1’. This enables us to
realize the true position of all the providers, including the efficient ones.
Figure 5 shows the efficiency scores of all the providers, both inefficient and super-
efficient.

Fig 6 Efficiency vs Super efficiency

Besides suitably ranking the efficient providers, the super efficiency approach also
sets stretch targets that enable the efficient, now super-efficient providers, to reach
greater heights in their network services quality. Table 6 shows the super-efficient
providers with their stretch targets.

19
 Table 7 Stretch targets using super efficiency
Limras World
Atria Fusionnet Meghbela You
Actual vs D-Vois Eronet Ortel ONEOTT RailTel RI Tata Sky Phone
Convergence Web Cable & Broadban
QoS metrics stretch Communications Broadband Communication Intertainmen Corporation Networks Broadband Internet
Technologies Services Broadband d India
targets Pvt.Ltd. Service s Ltd. t Limited of India Ltd Pvt Ltd Pvt. Ltd. Services
Pvt. Ltd. Pvt. Ltd Services Pvt. Ltd.
Pvt. Pvt Ltd
Actual 0.9153 0.9431 0.9240 0.9900 0.9500 0.9500 0.9700 0.9830 0.9130 0.9822 0.9133 0.9967
Stretch
Faults Repair(>90%) target using
0.9533 0.9601 0.9422 0.9900 0.9729 0.9598 1.0000 0.9830 0.9270 0.9822 0.9566 0.9967
super
efficiency
Actual 1.0000 0.9968 0.9940 1.0000 1.0000 0.9943 0.9900 1.0000 0.9912 0.9920 1.0000 0.9986
Stretch
Faults Repair (=>99%) target using
1.0000 0.9968 0.9940 1.0101 1.0000 0.9943 1.0526 1.0000 0.9938 0.9998 1.0000 1.0068
super
efficiency
Actual 0.8533 0.8550 0.7120 0.9000 0.9500 1.0000 1.0000 0.6933 0.6792 0.8107 0.9900 0.5276
Response Time to the Stretch
Customer for target using
assistance(>60%) 0.9551 0.8550 0.7239 0.9908 0.9500 1.0000 1.0000 0.8165 0.7306 0.8107 0.9900 0.9061
super
efficiency
Actual 1.0000 0.9110 0.8440 1.0000 1.0000 1.0000 1.0000 0.7633 0.8541 0.8439 1.0000 0.6338

Response Time to the Stretch

Customer for target using
assistance(>80%) 1.0000 0.9110 0.8440 1.0000 1.0031 1.0022 1.0526 0.8706 0.8541 0.8595 1.0046 1.0068
super
efficiency

Actual 0.9565 0.9000 1.0000 0.9500 0.9500 0.9000 1.0000 1.0000 1.0000 1.0000 0.8500 0.8695

Bandwidth
utilisation/throughput Stretch
Download target using
0.9565 0.9856 1.0000 1.0000 0.9500 0.9697 1.0000 1.0000 1.0000 1.0000 0.9300 0.9564
speed(>80%) super
efficiency

Actual 0.9941 1.0000 1.0000 0.9940 0.9990 0.9901 0.9998 0.9995 1.0000 0.9988 0.9983 0.9894
Service availability
/uptime (for all users)
in %age(>98%) Stretch
target using
1.0001 1.0000 1.0000 1.0163 0.9996 1.0016 1.0516 1.0050 1.0000 1.0024 0.9983 1.0007
super
efficiency

Thus, both efficient and inefficient providers are given an opportunity to enhance
themselves so they can render better network services to their subscribers/customers.
The DEA technique is a mathematically precise technique, and all targets are accurate
and feasible to achieve. Moreover, the technique strives to give out the maximum
efficiency of the DMUs, so any inefficiency identified is true. Thus, this is a reliable
technique for service providers of all sizes as shown in these two chapters. Also, it is
very useful to service providers with less analytical presence. It is a simple technique
to use, flexible and can be used on large data volumes as well. It helps managers save
time and effort when it comes to goal setting.

20
10.3 Analysis 3 Results & Discussion
An Optimal Benchmarking Technique to Improve Quality of Experience in 4G Indian
Public Telecom Sector for Better Network Penetration and Fair Service Pricing
Besides being useful for end-to-end evaluation of service providers, the DEA
technique can be applied to other useful measures like improving network penetration
and restraining service pricing in the telecom industry by helping in improvement of
the Quality of Experience in the Indian 4G Public Telecommunications Sector. This
study shows how in the current Indian Telecom scenario, the private sector companies
are performing better then the public sector ones and how the upscaling/enhancement
of the public sector will help in not only improving network penetration within the
country but also in maintaining fairness in the pricing system for network services.
The role of DEA in this kind of QoE enhancements, that could scale up the
performance of the public telecom sector company in the current Indian scenario
could prove vital in the growth of the telecommunications industry in India, with not
much trade-off in quality vs cost that could benefit millions of customers all over the
country.
BSNL is currently the only major public telecom company. Its private contenders are
Airtel, Vi and Jio. The technique used is the DEA-SBM with Super efficiency. The
data in Table 8 shows the current performance of the top Indian 4G providing public
and private telecom companies at the time of study.
Table 8: Current User experience on 4G mobile networks

21
From Table 8 data, we see that across each of the seven QoE metrics, all the 3 private
sector companies – Airtel, Vi and Jio perform better than the only public sector
company, BSNL. We would like to understand if this is also mathematically true and
if so, is BSNL lagging by a very large margin. Also, our idea is to eliminate any
improper ways of setting targets for each metric, like setting the highest value in each
column. Rather, we use the DEA technique to determine the shortest/best/optimal
path that would provide BSNL with a feasible target to achieve. Airtel, Vi and Jio
each seem to perform well in some of the QoE metrics.
Table 9 shows the overall, relative efficiencies of the 4 service providers that are
taken for the study. The Envelopment model of DEA is used for this purpose. The
maximum efficiency that is assigned to the DMUs (service providers in this case) is
‘1’. The service providers that hold a value of ‘1’ are said to be efficient and these fall
on the efficiency frontier or envelop that is generated by the DEA technique. All
providers with an efficiency value less than ‘1’ are said to be inefficient and fall
within the envelop.
Table 9 Relative overall efficiency of top 4 Indian telecom service providers
Efficiency
Service Provider score (0-1) Efficiency %
Airtel 1.00000 100%
BSNL 0.93368 93.40%
Jio 1.00000 100%
Vi 1.00000 100%

Figure 6 shows the details of the improvement needs of BSNL that must be achieved

22
in order to become equal in performance with the private providers that are currently
doing better.

Fig 7 BSNL – Improvement needed

Table 10 below shows some quick wins for BSNL for the immediate, short and long
terms.
Table 10 Improvement needed for BSNL

Level of
Measure of
Metrics improvement
improvement
needed

Download speed 197.9% High

Upload speed 133.4%
Medium
Coverage 122.6%
Video experience 31.7%
Games experience 30.7% Low
Availability 35.4%
Voice app experience 0.0% None
Table 11 below, shows the ideal picture, and if all the 4 providers were to hold these
values, they would all be in par with each other. The Slack based measure helps arrive
at these targets by the output slacks that are provided by the DEA-SBM approach for
the inefficient DMUs.
Table 11: SBM recommendations for all providers to attain/be at 100% efficiency

23
 Service
Airtel BSNL Jio Vi
provider
Actual Target
Video
experience
(0-100 pts) 59.2 42.4 55.8↑ 60.1 59.8
Games
experience
(0-100 pts) 57.2 42.3 55.3↑ 54.7 59.2
Voice app
experience
(0-100 pts) 76.8 71.8 71.8 75.8 76.9
Download
speed
experience
(Mbps) 12.4 4.2 12.5↑ 12.1 13.4
Upload
speed
experience
(Mbps) 3.8 1.8 4.2↑ 3.3 4.5
Availability
%
97.8 61.8 83.7↑ 99.3 89.6
Coverage
(%)
7.8 2.6 5.8↑ 9.4 6.2

The importance of this technique is that the targets set for BSNL to achieve, are not
higher or equal to the top 3 private providers at all. It is rather the shortest path to
attain the highest efficiency.
As much as the efficient providers are in par with each other, they cannot be suitably
ranked using the conventional DEA technique, since all of them have an efficiency of
‘1’. Hence, we use the Radial Super efficiency approach that provides these efficient
DMUs with different efficiency scores of above ‘1’. This is because, the DMU under
evaluation is taken off from the efficiency frontier of the production set and this goes
on to shrink the production set, resulting in efficiency scores more than ‘1’ for only
the efficient DMUs (service providers). Now the efficient service providers can also
be ranked aptly. Figure 7 shows the all the providers with their efficiency scores, the
efficient ones with their new super efficiency scores.

24
nce weightages Fig 8 Super-efficient DMUs with their new efficiency scores
It is to be noted that BSNL that emerged inefficient still holds the same efficiency
score while all others – Airtel, Vi and Jio that emerged as efficient before, have new
efficiency scores. We see that Jio is performing the best, followed closely by Vi and
then Airtel.
Although the top 3 providers are already performing well in the Indian Telecom
sector, there is always scope for improvement. So, the radial super efficiency model
recommends some references/benchmark providers for each of them, so they could
learn from each other, in the case of having missed out on some best practice. Figure
6 shows the super-efficient and inefficient DMUs with their references/benchmarks
that have been recommended by the technique.

Fig 8 References/Benchmark providers for all service providers

Besides being able to rank the efficient DMUs that are now super-efficient, so their
true position is realized, the super-efficiency approach also provides them with stretch

25
targets that can be achieved in the long term. These targets are shown in Table 5
below. All this drives healthy competition among the service providers enabling them
to enhance themselves, so they are able to provide better services every time.
Thus, this technique provides an all-round development of the service providers, both
the ones that need improvements in their QoE metrics and those that are already
performing well. This kind of analysis helps improve QoE that results in better
customer satisfaction. This could serve as feedback for the service providers who can
check their network QoS provisioning and improve it.
Table 12 Super-efficient service providers with their stretch targets
Service
Airtel BSNL Jio Vi
provider
Actual Target
Video
experience
(0-100 pts) 59.2 42.4 55.8↑ 60.1 59.8
Games
experience
(0-100 pts) 57.2 42.3 55.3↑ 54.7 59.2
Voice app
experience
(0-100 pts) 76.8 71.8 71.8 75.8 76.9
Download
speed
experience
(Mbps) 12.4 4.2 12.5↑ 12.1 13.4
Upload
speed
experience
(Mbps) 3.8 1.8 4.2↑ 3.3 4.5

Availability
%
97.8 61.8 83.7↑ 99.3 89.6

Coverage
(%)
7.8 2.6 5.8↑ 9.4 6.2

The advantage of this technique is that the inefficient DMUs are not given any stretch
targets but only the feasible targets set by the DEA technique. It is the super-efficient
DMUs that are given stretch targets that they can achieve in the long run. Thus, BSNL
could try to achieve these targets in the short term, while the other 3 providers could
challenge themselves and try to achieve their stretch goals over a longer period.
This kind of target setting is essential for public sector companies like BSNL, as it
helps upscale them to the level of their private peers, leading to fierce competition.
This results in deepening network penetration in the country. Improving the QoE/QoS
of the public sector companies also helps in competitive pricing among the service
providers, as public sector companies tend to have lower prices for their service as

26
opposed to their private peers. It helps place a check on the pricing system of the
telecom sector of the country eventually resulting in fair pricing. This helps the
customers enjoy high quality services at competitive costs.

10.4 Analysis 4 Results & Discussion

5G Planning and management of QoE Metrics in the APAC Region Using
Mathematical Benchmarking Techniques
The study shows the true position of the APAC region countries with 5G, at the time
of study, with the available data. Since, 5G has not been completely rolled out in the
country by all providers at the time of study, the study has been done at a country and
provincial level.
A two-step approach is followed in the study, where the countries are first evaluated
based on relative efficiency scores generated by the DEA-SBM (Data Envelopment
Analysis - Slack Based Measure) technique. The APAC countries are identified as
either efficient or inefficient. The inefficient countries are then given
recommendations of reference/benchmark providers to improve themselves. In
addition to this, these inefficient countries are also provided with feasible output
targets to meet, in order to improve the QoE metrics that need improvement. Thus, an
optimum solution is found for the inefficient countries to upscale themselves. As a
second step, the efficient countries are further ranked using the super efficiency
approach in DEA where the relative efficiency scores are altered only for the efficient
countries, making it possible to rank them suitably. Moreover, these efficient
countries are also recommended with references and set stretch targets to be met by
them in the long run. Thus, both inefficient and efficient countries are provided with
optimal opportunities/recommendations to enhance themselves.
Table 13 below shows the current performance of the 8 key APAC countries at the
time of study.
Table 13 Current QoE performance of key countries in the Asia Pacific region

27
 Reach -
Peak Video Games Voice app
Download Upload Location
download experience experience experience Availability
QoE metrics speed speed experience
speed (0-100 (0-100 (0-100 (% time)
(Mbps) (Mbps) (0-10
(Mbps) points) points) points)
points)
South
380.5 771.4 30.6 82.3 90.4 84.9 26.7 5.8
Korea
Taiwan 353.3 847.3 51.8 84.1 84.2 83.9 23.8 5

Australia 242.1 595.6 16.1 80.4 82 83.3 10.1 2.9

Japan 180.9 664.2 12.5 76.7 64.9 78.9 2 1

APAC countries

Singapore 143.5 579.1 21.2 80.5 87.2 83.1 8.2 2.9

Phillipines 141.7 426.2 12.1 79.3 68.1 79.7 10 3

Hong
Kong (SAR 134.8 347.5 21.5 81.6 82.3 81.7 27 6.2
China)

Thailand 122.5 414.9 26.4 80.7 82.2 82.7 18.8 4

From the data, we see that the countries have all performed at a varied range across all
the QoE metrics.
Overall, South Korea and Taiwan have lead the list of countries in most of the metrics
and are above average in each of the QoE metrics chosen for the study. The countries
have all performed well in some metrics and less in others. No country has performed
bad in all of the metrics considered. Instead of just taking the average of the values in
each metric and concluding the performance/position of the countries, we use an
intelligent optimization technique, Data Envelopment Analysis that considers all the
given metrics and gives the overall relative efficiency of the countries and
recommends the true position of these countries.
Figure 9 shows the overall relative efficiency scores by the DEA technique.

28
 Fig 9 Efficient and Inefficient DMUs (Countries)
If we were to just look at Table 13 or use a simple technique like taking the average of
each of the metric values to study the performance of the countries, we would have
interpreted the results in a different way. DEA, however, is a multiple criterion
evaluation methodology and provides the optimal/best path to the efficiency frontier
giving an accurate picture of where each DMU stands.
The DEA technique also provides the references/benchmark countries for the
inefficient DMUs and their weightages as shown as in Table 14.
Table 14 Relative benchmark countries and their weightages

DMU Name Efficiency Optimal Lambdas with benchmarks

South Korea 1.00000 1.000 South Korea

Taiwan 1.00000 1.000 Taiwan
Australia 0.98115 0.981 South Korea
Japan 0.93025 0.852 South Korea 0.078 Taiwan
Singapore 0.97880 0.979 South Korea
Phillipines 0.94743 0.211 South Korea 0.737 Taiwan
Hong Kong (SAR Hong Kong
1.00000 1.000
China) (SAR China)
Thailand 0.97635 0.784 South Korea 0.192 Taiwan

The top two countries using 5G – South Korea and Taiwan appear as benchmarks for
the countries that need to improve themselves in some of the QoE metrics. For
instance, the DMU4 – Japan falls between South Korea (0.852) and Taiwan (0.078).
This shows that Japan may learn best practices from these two countries to improve
its QoE. Similarly, the DMU3 Australia has been benchmarked with one country
South Korea.

29
Table 15 shows the output targets/recommendations, i.e., the optimal benchmarks for
the APAC countries on each of the QoE metrics considered for the study. If all the
countries were to achieve this level of QoE, then they would all perform equally.
Table 15 Output targets/recommendations for APAC countries/QoE management
APAC countries
Actual vs Hong
QoE metrics
Target South Taiwan Australia Japan Singapore Phillipines
Kong
Thailand
Korea (SAR
China)
Download Actual 380.5 353.3 242.1 180.9 143.5 141.7 134.8 122.5
speed
(Mbps) Target 380.5 353.3 373.3* 351.8* 372.4* 340.5* 134.8 366.3*
Peak
Actual 771.4 847.3 595.6 664.2 579.1 426.2 347.5 414.9
download
speed
Target 771.4 847.3 756.9* 723.5* 755.0* 786.8* 347.5 767.8*
(Mbps)

Upload Actual 30.6 51.8 16.1 12.5 21.2 12.1 21.5 26.4
speed
(Mbps) Target 30.6 51.8 30.0* 30.1* 30.0* 44.6* 21.5 34.0*

Video Actual 82.3 84.1 80.4 76.7 80.5 79.3 81.6 80.7
experience
(0-100
points) Target 82.3 84.1 80.7* 76.7 80.6* 79.3 81.6 80.7

Games Actual 90.4 84.2 82.0 64.9 87.2 68.1 82.3 82.2
experience
(0-100
points) Target 90.4 84.2 88.7* 83.6* 88.5* 81.1* 82.3 87.1*

Voice app Actual 84.9 83.9 83.3 78.9 83.1 79.7 81.7 82.7
experience
(0-100
points) Target 84.9 83.9 83.3 78.9 83.1 79.7 81.7 82.7

Actual 26.7 23.8 10.1 2.0 8.2 10.0 27.0 18.8

Availability
(% time)
Target 26.7 23.8 26.2* 24.6* 26.1* 23.2* 27.0 25.5*

Reach -
Location Actual 5.8 5.0 2.9 1.0 2.9 3.0 6.2 4.0
experience
(0-10 Target 5.8 5.0 5.7* 5.3* 5.7* 4.9* 6.2 5.5*
points)

*Indicates that the DMU needs an improvement in that QoE metric

These are feasible targets that are recommended by the slack based measure in DEA.
The output slacks are a measure of potential improvement that the DMU/country will
have to make in each of the QoE metrics. Achieving these targets helps the
DMU/country to reach the maximum efficiency and be in par with their efficient
peers.

30
Of the eight QoE metrics considered for the study the key APAC countries are all at
their best performance in two of the metrics. There is a tie between the top three
countries – South Korea, Taiwan and Hongkong who stand first, per the analysis
across all QoE metrics considered. Are these three countries really at par with each
other? Is there a way to rank these efficient countries using the traditional DEA? To
answer these questions, we use the Radial Super efficiency approach which is an
evolution over the conventional DEA approach. This technique helps in
differentiating the efficient countries/DMUs obtained by the basic Envelopment
model of conventional DEA.
From Figure 10, we see the top three countries that were all at an efficiency score of 1
using the conventional DEA approach now have different scores of efficiency of over
1.

Fig 11 Efficient DMU ranking using super efficiency

Only the efficient DMUs have now emerged super-efficient. This kind of ranking
helps in realizing the true position of efficient DMUs. The super efficiency approach
also helps in setting stetch targets for the efficient DMUs/countries.
Table 16 shows the stretch goals for the efficient APAC countries to be achieved in
the long term.

31
 Table 16 Stretch targets for efficient countries using Radial Super efficiency
Actual vs Hong Kong (SAR
QoE metric South Korea Taiwan
Stretch targets China)
Download speed Actual 380.5 353.3 134.8
experience
Target using
(Mbps) 380.5 644.1 406.7
Super efficiency
Peak download Actual 771.4 847.3 347.5
speed experience Target using
(Mbps) 914.9 1305.8 824.6
Super efficiency
Upload speed Actual 30.6 51.8 21.5
experience Target using
(Mbps) 56.0 51.8 32.7
Super efficiency
Actual 82.3 84.1 81.6
Video experience
(0-100 pts) Target using
95.4 139.3 88.0
Super efficiency
Actual 90.4 84.2 82.3
Games experience
(0-100 pts) Target using
95.5 153.0 96.6
Super efficiency
Voice app Actual 84.9 83.9 81.7
experience Target using
(0-100 pts) 95.2 143.7 90.8
Super efficiency
Actual 26.7 23.8 27.0
Availability (%) Target using
27.4 45.2 28.5
Super efficiency
Actual 5.8 5.0 6.2
Reach (0-10 pts)
Target using
5.8 9.8 6.2
Super efficiency

The efficient countries also have an output target to achieve, although they could
choose to do so in the long run. The efficient countries are the only DMUs to have a
stretch target. The inefficient ones still only must achieve the feasible short-term
targets that were already set by the conventional DEA. Thus, this technique
intelligently sets suitable targets for both inefficient and efficient countries. The
stretch targets help the efficient/best performing countries to challenge themselves
and be able to provide the best possible services by improving their QoS provisioning.
This could possibly lead to better customer satisfaction, leading to increased 5G usage
by the citizens they cater to.
The latest 5G technology plays a vital role in all major industrial areas. Per the HIS
2017 report, the record of the impact of 5G enabled technology in sales activity of 16
industries shows that investment in wireless internet domain has led to high global
sales turnover and has impacted economy the world over. By 2035, this sales activity
is expected to reach $12.3 Trillion and this accounts to 4.6% of all global real output.
Therefore, 5G planning is the need of the hour and it is essential for countries to adopt
an intelligent benchmarking tool for proper goal setting purposes, so that the best QoE

32
is achieved. This would in turn enable the network operators to provide their citizens
with the best possible QoS.

10.5 Analysis 5 Results & Discussion

5G Planning and QoE Management Using Mathematical Benchmarking Techniques
for Europe and Middle Eastern Countries
The research could be extended to the Europe and Middle Eastern countries as
well. This gives us a clear picture of how the other continents are faring after their 5G
roll outs. Future research could compare the performances of both regions for better
insights on 5G performance, that could possibly help with better 5G planning.
The 14 key EMEA countries are evaluated across 8 QoE metrics. The entire study is
framed out in the coming sections.
The whole study is done in two stages – the conventional DEA approach with the
slack based measure as the first step, where the countries are evaluated across 8 QoE
metrics and feasible output targets set for QoE. The second step is the radial super
efficiency model where the efficient/best performing countries are given stretch/long
term targets/goals to achieve so they could enhance themselves further to cater better
network services to their citizens.
Table 17 shows the current QoE scores metric-wise exhibited by key countries in the
EMEA region after their 5G roll outs.
Table 17 QoE performance of EMEA countries as of June 2021

33
 EMEA countries
QoE
metrics Saudi
Austria Finland Germany Greece Ireland Italy Kuwait Netherlands Poland Spain Switzerland UAE UK
Arabia
Voice app
experience
83.8 83.7 84.2 82.8 84.4 80 78.1 84.8 82.6 79.3 82.3 82.7 80.7 82.7
(0-100
points)
Games
experience
82.8 81.2 86.6 75.2 87.5 70 61.1 90.3 74.4 59.2 75.9 83.5 72.5 80.8
(0-100
points)

Video
experience
79.7 81.3 82.2 80.9 81.5 76 77.3 84.5 75.1 79 78 81.5 80.8 79.8
(0-100
points)

Download
speed 182.5 221.4 108.4 188.7 165 93 194.9 126.2 81.8 237.4 155.2 188 274.9 121.4
(Mbps)
Peak
download
567.2 654.9 419.2 777.4 539.8 485 517.4 289.2 283.8 793.1 600.1 608.2 655.1 326.4
speed
(Mbps)
Upload
speed 23.2 25.1 25.6 23.2 17.7 16 18.9 36.2 14.4 22.8 20.6 37.3 23.5 13.8
(Mbps)

Availability
8.8 12.1 7.3 7.9 5.8 10 27.6 15.3 11.9 24.8 5.3 11.2 12.2 5.6
(% time)

Reach -
Location
experience 2.4 2.7 2.2 2.5 1.7 1.7 5.4 4.9 2.5 4.5 1.2 3.7 4.1 2
(0-10
points)

From the data in Table 17, some countries fare well in some metrics, but do not fare
equally well in others. Therefore, it would be an unfair comparison if the metrics are
evaluated at an individual level. Moreover, we try to avoid using simple measures like
estimating the mean of the values or setting the biggest value in the column as the
benchmark/target. Rather, we use DEA which is a multiple criterion evaluation
methodology that helps evaluate the key EMEA countries across a whole set of 8 QoE
metrics chosen for the study.
Figure 11 shows the overall, relative efficiencies of the EMEA countries in terms of
the 8 key QoE metrics chosen for the study.

34
 Fig 12 Efficient and inefficient DMUs
Eight out of the chosen fourteen countries have achieved 100% efficiency (efficiency
DMUs) and fall on the envelope/efficiency frontier generated by the DEA technique.
The other six countries that have an efficiency score below the 100% mark fall within
the envelop (inefficient DMUs). A line starting at the origin through the current
position of the inefficient DMUs to the efficiency frontier gives the optimal/shortest
path for countries that need to improve themselves, to reach the highest efficiency.
The DEA technique also provides references/benchmark countries with weightages.
The lagging countries must analyse their reference countries and adopt best practices
from these efficient peers.
Table 18 shows the EMEA countries with improvement needs along with their
references/benchmark countries.

Table 18 Benchmark countries for inefficient EMEA countries

EMEA
countries
Benchmark countries
needing
improvement
UK 0.176 Ireland 0.800 Netherlands
Spain 0.617 Finland 0.364 Ireland
Poland 0.008 Ireland 0.966 Netherlands
Germany 0.524 Ireland 0.471 Netherlands
Austria 0.486 Finland 0.404 Ireland 0.106 Netherlands
Italy 0.316 Finland 0.373 Ireland 0.262 Netherlands

The non-zero output slacks are a measure of the potential improvement for

35
the lagging countries to reach the efficiency frontier (100% efficiency). The output
targets are shown in Table 19.
Table 19 Output targets for goal setting using DEA.
EMEA countries Austria Germany Italy Poland Spain UK
Actual 182.5 108.4 92.7 81.8 155.2 121.4
Download
speed
(Mbps) Target 187.71 145.95 164.71 123.25 196.51 130.01

Peak Actual 567.2 419.2 484.5 283.8 600.1 326.4

download
speed Target 567.20 419.20 484.50 283.80 600.10 326.40
(Mbps)

Upload Actual 23.2 25.6 15.7 14.4 20.6 13.8

speed
(Mbps) Target 23.20 26.34 24.03 35.11 21.91 32.08

Video Actual 79.7 82.2 75.7 75.1 78 79.8

experience
(0-100 pts) Target 81.43 82.54 78.28 82.29 79.77 81.95
Games Actual 82.8 86.6 69.7 74.4 75.9 80.8
experience
(0-100 pts) Target 84.42 88.42 82.01 87.94 81.89 87.65

Voice app Actual 83.8 84.2 80.2 82.6 82.3 82.7

experience
(0-100 pts) Target 83.80 84.20 80.20 82.60 82.30 82.70

Actual 8.8 7.3 10 11.9 5.3 5.6

Availability
(% time) Target 9.85 10.25 10.00 14.82 9.57 13.26

Reach - Actual 2.4 2.2 1.7 2.5 1.2 2

Location
experience
(0-10 Target 2.52 3.20 2.77 4.75 2.28 4.22
points)

Table 19 shows the target outputs that must be achieved by all six countries that have
improvement needs in some of their QoE metrics to be in par with the ones not
needing improvement. All the six countries perform well in Voice app and peak
download speed experiences and require no improvement. However, these countries
need to improve themselves in some of the other metrics. These target outputs that are
provided by the DEA technique are therefore feasible targets to be achieved.
With the traditional/conventional DEA, the real/true position of the efficient countries
(DMUs) was not realized since there is no suitable ranking method for them. For this
reason, we use the super efficiency approach, where the DMU under evaluation is
removed from the production set and this shrinks the production set. As a result, the
efficient DMUs are all given different efficiency scores of over ‘1’. Now, the efficient
DMUs can be ranked suitably.

36
Figure 12 shows the new scores of efficiency for the efficient/best performing
countries that were initially provided with the same efficiency scores using the
traditional DEA model.

Fig 13 Super-efficient countries with their new efficiencies

The output slacks provided by the super efficiency model helps set the target outputs.
These could be set as stretch/long term goals.
Table 20 shows the super-efficient DMUs with their stretch targets.

Table 20 Super-efficient DMUs and stretch targets

37
 EMEA countries
QoE metrics Actual vs
stretch Saudi
targets Netherlands Greece Switzerland Ireland Finland Kuwait UAE
Arabia
Actual 84.80 82.80 82.70 84.40 83.70 78.10 79.30 80.70
Voice app Target
experience using
95.30 87.41 100.52 85.44 84.28 94.87 101.70 93.31
Super
efficiency
Actual 84.50 80.90 81.50 81.50 81.30 77.30 79.00 80.80
Video
experience Target
(0-100 using
94.03 85.97 100.16 84.44 83.44 94.52 100.11 92.64
points) Super
efficiency
Actual 90.30 75.20 83.50 87.50 81.20 61.10 59.20 72.50
Games
experience Target
(0-100 using
90.30 75.20 94.22 87.50 81.20 75.81 85.00 72.50
points) Super
efficiency
Actual 126.20 188.70 188.00 165.00 221.40 194.90 237.40 274.90
Download
Target
speed
using
(Mbps) 222.45 246.18 210.22 176.60 221.40 260.46 244.44 274.90
Super
efficiency

Actual 289.20 777.40 608.20 539.80 654.90 517.40 793.10 655.10

Peak
download Target
speed using
(Mbps) 681.88 777.40 608.20 539.80 654.90 845.72 793.10 905.50
Super
efficiency

Actual 36.20 23.20 37.30 17.70 25.10 18.90 22.80 23.50

Upload
speed Target
(Mbps) using
37.54 25.68 37.30 37.99 32.88 29.46 26.26 26.98
Super
efficiency

The table above shows how the efficient DMUs which did not have an output target to
meet in the conventional DEA approach. However, when these countries/DMUs
become super-efficient, they are given stretch/output targets to meet with the super
efficiency approach that can be achieved in the long run.
The main advantage of using such an intelligent technique is that although the
inefficient countries/DMUs have a benchmark to target/meet, it is achievable. It is
only the efficient countries/DMUs that are given the stretch targets, so they could
further enhance themselves, thereby providing the best services for the citizens.

11. Managerial Contributions

Many significant contributions have been made by the research as can be seen from
the previous chapters. These are listed here.
As a first step, realistic and scientific targets are provided for service providers who
fall short against their competitors/peers. Secondly, all targets recommended are at a
granular level pertaining to each metric. Thirdly, as much as organizations have their
stretch targets, this research has attempted to provide the service providers with

38
optimal benchmarks with the shortest path to achieve 100% efficiency, so they could
all perform equally. Fourthly, this analysis is quick to replicate for even large volumes
of data, like state level drill downs.
Besides helping the top service providers with their pulse check and competitive
analysis, this research has also attempted to evaluate small and medium wireline
service providers that have very little analytical presence. This goes to prove that this
research is useful for service providers of all kinds irrespective of their size and type
of network service they provide. This provides managers of organizations of all sizes
and capacities with same quality of analysis and insights. A significant contribution of
the research is its ability to consider a whole set of QoS metrics to provide an overall
relative efficiency score. This eliminates the use of simple ways like taking the mean,
median and mode, also crude methods like taking the sum of the scores. Yet another
significant contribution is the provision of references/benchmark providers
recommended by the analysis along with their weightages from whom the lagging
service providers could learn best practices from, so they could enhance themselves.
This also helps managers to save time and effort towards benchmarking exercises
helping them plan in a more focussed manner on how to lead the organization
forward.
While there are many ways to reach the highest efficiency theoretically, the DEA
technique employed in the research provides the service providers with the optimal
path to attain full efficiency. It would otherwise be a challenge for managers to arrive
at accurate, realistic and achievable targets in terms of time and effort. Yet another
advantage of this technique is that it is easy to run multiple simulations and choose an
alternate path to attain highest efficiency. It is also mathematically precise and is a
flexible technique and of invaluable use to managers in goal setting.
This research is versatile, in that it can be used for varied use cases and is not
restricted to the above mentioned. The research has also attempted to study the 4G
public telecom sector of India and has provided insights/recommendations to the
public sector companies for better performance. As shown above, achievable targets
have been provided to the public sector company that is currently lagging its private
peers across the QoE metrics it needs to improve on. Some quick wins that can be
achieved with immediate effect and in the short and long term have been identified
and recommended. This could prove vital in the upliftment of the Indian 4G public
telecommunications sector. This in turn would lead to deepening of network
39
penetration and fairness in service pricing. As always, it is a proven technique that is
very flexible and can be run multiple times. This research can also be carried out on
large data volumes. Targets/benchmarks set are optimal helping the service provider
that is lagging in the current scenario to find the best/optimal/shortest path to attaining
the highest efficiency, so it can stand in par with its private competitors/peers.
This research could be extended to future networks like 5G as well for network
planning and QoE/QoS management. For effective QoE/QoS management, it is vital
to set the right benchmarks so unrealistic goals can be avoided. This research caters to
this need by enabling countries to set these targets right. Moreover, in planning a new
network setup, it is very important to learn best practices from the relevant peer group
and the research facilitates the same. Maintaining quality is yet another goal to reach
while planning a network. The research helps achieve the same with the DEA
technique and its super efficiency approach. Further enhancements are also possible
using this method, since this approach also helps the countries that are already
performing well in their network services quality to reach greater heights through
achieving/trying to achieve the stretch targets/goals that are set. This method is
flexible and mathematically precise since it is based on Linear programming. The
technique allows for simulations to be run multiple times and to be run on voluminous
data.
With DEA and super efficiency, an ideal framework has been identified for 5G
deployment planning in countries yet to roll out. Post adopting 5G network
technology too, this research could prove to facilitate effective QoE/QoS
management. Analysis at granular level is made possible using this approach, by
means of which both immediate and long-term goals can be achieved at a network
operator level or at a country/regional level thereby enhancing user experiences on the
network services provided.

12. Limitations and Directions for Future Research

Throughout the research, for all use cases, inputs have been assumed equal. This is
because there is a need for a separate, thorough study of all significant input variables
that could impact the output metrics of the research. Secondly, significant input
variables are not available publicly. For this reason, it was decided that usage of few
and arbitrary outputs be avoided in the research, while making recommendations for
QoE enhancements. Future studies could make use of these input variables should

40
they be available then, for analysis. The study presents results for the whole of India.
It would be more useful for service providers to have state/region level breakdowns of
the same analysis. This study has considered five key QoE metrics. Future studies
could involve usage of QoS metrics like packet loss, bandwidth, latency and other
KPIs (Key Performance Indicators). Also, this study considered the top four 4G
service providers of the Indian telecommunications sector. Future studies could
include other network service providers like wireline providers.
Data availability is sometimes limited in some of these use cases in the research. This
is mainly due to unavailability of data for providers like small and medium ones that
have been chosen for this part of the study. Input variables are not completely
available, hence the assumption here is that all providers have same inputs of unity.
Future studies could involve the use of relevant input variables that could impact the
results significantly, making it more holistic. Moreover, there are more than the 20
small and medium providers considered for the study. However, as not all these have
a PAN country presence, region specific studies could prove to be more useful.
Fourth, this study has considered data over one quarter of a year. Although it is an
acceptable time frame in the industrial standards, covering a longer duration of time
could provide better results.
The study could possibly result in an end-to-end evaluation of the service providers
when it is done from both a QoE and a QoS perspective. While public and private
players could have a marked difference in terms of their infrastructure and all other
functionalities, this study has assumed all inputs to be equal due to input variables not
being available on the public domain. Addition of the right input variables could lead
to a significant impact on results, leading to different targets for the public sector
companies to achieve, if they still lag their private peers.
As the research extends to the key countries in the APAC region that have rolled out
5G technology, at the time of study, only few countries had adopted 5G and so this
may not be a holistic view of the performance of the region overall. Moreover, the
fact that these countries have embraced this technology at different times, with few of
them being well established, while others just starting to use it could impact the
results significantly if the analysis were done over a future time-period. Future
research could include more countries as they begin to adopt the 5G network services,
that would provide a broader perspective of the performance of this region. In the
future, a comparison of the performances of the APAC region with other regions with
41
a similar pattern of 5G usage could lead to impactful research and eventually aid
planning, deployment, and maintenance of this network in a more inclusive manner.
More granular level analysis could be done as more network operators start providing
5G services at each country level.

13. Goals achieved by research/Conclusion

• Conceptual framework for Telecom performance management realized in
terms of all functional capabilities
1) End to end evaluation of CSPs at both network and user levels
2) Effective QoE/QoS management
3) CSP enhancement for both CSPs that perform well and those lagging their
relevant peers
 An ideal method for better Network Planning and Resource Management has
been identified.
• A versatile, optimal solution towards realistic scientific target setting with
high degree of mathematical precision has been identified.
• This research fills in the gap that has been in place in current industry
standards towards effective benchmarking of telecommunications performance
indicators
• Helps in provision of better network services and improving customer
satisfaction levels that would enable subscribers around the world enjoy better
quality of services.

Therefore, this research has been instrumental in bringing about an intelligent,

sophisticated, robust method for Telecommunication planning and management. This
is applicable for all future networks as well, as it is very versatile. This research is
also very flexible and can be used at different levels, for instance, at an overall
country performance level or at an individual network operator level. Similarly, it can
be done at a very granular level like region/state wise drilldowns as well.
This research study therefore recommends this method utilizing the DEA technique
and its super efficiency and slack based measures as an ideal choice for performance
management in the telecom industry.

42
14. Publications
Patent submitted
International Conferences / Book chapters
International Journals
References
[1] J. L. Maire, “A model of characterization of the performance for a process of
benchmarking,”, Benchmarking, vol. 9, no. 5, pp. 506-520, 2002.
doi:10.1108/14635770210451491.
[2] G. H. Watson, Strategic Benchmarking: How to Rate Your Company’s
Performance Against the World’s Best. Chichester, UK: John Wiley & Sons Inc.,
1993.
[3] Available at: https://trai.gov.in/.
[4] M. S. El-Nasr et al., Game Analytics. Berlin: Springer, 2016.
[5] I. Slivar et al., “Empirical QoE study of in-home streaming of online games” in
Network and Systems Support for Games (Netgames) 13th Annual Workshop on,
2014, pp. 1-6. doi:10.1109/NetGames.2014.7010133.
[6] Lafta, Sameer AS, Mohaned Mahdi Abdulkareem, Raed Khalid Ibrahim, Marwah
M. Kareem, and Adnan Hussein Ali. "Quality of service performances of video and
voice transmission in universal mobile telecommunications system network based on
OPNET." Bulletin of Electrical Engineering and Informatics 10, no. 6 (2021): 3202-
3210.
[7] S. Malisuwan et al., “Quality of service (QoS) and quality of experience (QoE) of
the 4G LTE perspective,” Int. J. Future Comput. Commun., vol. 5, no. 3, pp. 158-162,
2016. doi:10.18178/ijfcc.2016.5.3.463.
[8] J. Janssen et al., “Assessing voice quality in packet-based telephony,” IEEE
Internet Comput., vol. 6, no. 3, pp. 48-56, 2002. doi:10.1109/MIC.2002.1003131.
[9] W. Robitza et al., “Measuring YouTube QoE with ITU-T P. 1203 under
constrained bandwidth conditions” in Tenth International Conference on Quality of
Multimedia Experience (QoMEX). IEEE Publications, 2018, May, pp. 1-6.
doi:10.1109/QoMEX.2018.8463363.
[10] Z. Zhang, “Impact of environmental regulation on industrial green efficiency-
based on non-radial SBM model” in J. Phys. Conf. S. (Vol. 1744, No. 4, p. 042179).
IOP Publishing, vol. 4, 1744, 2021, Febr.. doi:10.1088/1742-6596/1744/4/042179.

43
[11] A. Kumar et al., “4G wireless technology: A brief review,” Int. J. Eng. Manag.
Res. (IJEMR), vol. 3, no. 2, pp. 35-43, 2013.
[12] R. K. P. Mok et al., “Measuring the quality of experience of HTTP video
streaming” in 12th IFIP/IEEE International Symposium on Integrated Network
Management (IM 2011) and Workshops. IEEE Publications, 2011, pp. 485-492.
doi:10.1109/INM.2011.5990550.
[13] İ. Duman and U. Eliiyi, “Performance metrics and monitoring tools for
sustainable network management,” Bilişim Teknol. Derg., vol. 14, no. 1, pp. 37-51,
2021.
[14] Available at: https:// transition.fcc.gov
[15] A. D. Kora et al., “Accurate radio coverage assessment methods investigation for
3-G/4G networks,” Comput. Netw., vol. 107, pp. 246-257, 2016.
doi:10.1016/j.comnet.2016.03.028.
[16] S. Wang and C. Ran, “Rethinking cellular network planning and optimization” in
IEEE Wirel. Commun., vol. 23, no. 2, pp. 118-125, Apr. 2016.
doi:10.1109/MWC.2016.7462493.
[17] R. K. P. Mok et al., “Measuring the network performance of Google cloud
platform” in Proc. 21st ACM Internet Measurement Conference, 2021, pp. 54-61.
doi:10.1145/3487552.3487862.
[18] O. Kanhere and T. S. Rappaport, “Position location for futuristic cellular
communications: 5G and beyond,” IEEE Commun. Mag., vol. 59, no. 1, pp. 70-75,
2021. doi:10.1109/MCOM.001.2000150.
[19] T. S. Rappaport et al., “Wireless communications and applications above 100
GHz: Opportunities and challenges for 6G and beyond,” IEEE Access, vol. 7, no. Jun.,
pp. 78729-78757, 2019. doi:10.1109/ACCESS.2019.2921522.
[20] D. Sotiros et al., “Analysing the export potentials of the Portuguese footwear
industry by data envelopment analysis,” Omega, vol. 108, p. 102560, 2022.
doi:10.1016/j.omega.2021.102560.
[21] A. Azadeh et al., “An integrated fuzzy DEA algorithm for efficiency assessment
and optimization of wireless communication sectors with ambiguous data,” Int. J.
Adv. Manuf. Technol., vol. 52, no. 5-8, pp. 805-819, 2011. doi:10.1007/s00170-010-
2741-0.

44
[22] H. B. Kwon et al., “Measuring comparative efficiencies and merger impacts of
wireless communication companies,” Benchmarking, vol. 15, no. 3, pp. 212-224,
2008. doi:10.1108/14635770810876566.
[23] J. Zhu, “Efficiency evaluation with strong ordinal input and output measures,”
Eur. J. Oper. Res., vol. 146, no. 3, pp. 477-485, 2003. doi:10.1016/S0377-
2217(02)00360-0.
[24] M. A. Azadeh et al., “Total assessment and optimization of telecommunication
sectors by multivariate approach” in 4th IEEE International Conference on Industrial
Informatics. IEEE Publications, 2006, Aug., pp. 659-663.
doi:10.1109/INDIN.2006.275639.
[25] Y. Tan et al., “Hotel performance in the UK: The role of information entropy in a
novel slack-based data envelopment analysis,” Entropy (Basel), vol. 23, no. 2, p. 184,
2021. doi:10.3390/e23020184.
[26] L. Yang et al., “A research on the efficiency of regional green innovation in
China—Based on DEA-SBM model and Malmquist index method” in, E3S Web
Conf. EDP Sciences, vol. 236, 2021. doi:10.1051/e3sconf/202123604012.
[27] J. Puri and S. P. Yadav, “A concept of fuzzy input mix-efficiency in fuzzy DEA
and its application in banking sector,” Expert Syst. Appl., vol. 40, no. 5, pp. 1437-
1450, 2013. doi:10.1016/j.eswa.2012.08.047.
[28] Z. Zhang, “Impact of environmental regulation on industrial green efficiency-
based on non-radial SBM model” in J. Phys. Conf. S. (Vol. 1744, No. 4, p. 042179).
IOP Publishing, vol. 4, 1744, 2021, Febr.. doi:10.1088/1742-6596/1744/4/042179.
[29] L. Yang et al., “A research on the efficiency of regional green innovation in
China—Based on DEA-SBM model and Malmquist index method” in, E3S Web
Conf. EDP Sciences, vol. 236, 2021. doi:10.1051/e3sconf/202123604012.
[30] K. Zhong et al., “Super efficiency SBM-DEA and neural network for
performance evaluation,” Inf. Process. Manag., vol. 58, no. 6, p. 102728, 2021.
doi:10.1016/j.ipm.2021.102728.
[31] R. Zhu et al., “Investigating economic roles of multinational construction
industries: A superefficiency DEA approach,” Appl. Econ., vol. 53, no. 41, pp. 4810-
4822. Available at: https://www.opensignal.com/2021/06/14/benchmarking-the-
5gexperience-asia-pacific-june-2021, 2021. doi:10.1080/00036846.2021.1910133.
[32] Available at: https://deafrontier.net
[33] Available at: https://rbensua.shinyapps.io/deaR/

45
46