Transition from KPI to KQI
-- Huawei Service Quality Improvement Solution
HUAWEI TECHNOLOGIES CO., LTD.
Content
What is Service Quality Improvement Case Study
Technology Trend
User Email Video/Voice Messaging MMS Device Video telephony
Cafe
Working
Office
Watching TV
Subway
Learning
Home
Driving
Multimedia Apps
Social Networking
Network IP Transformation
SMS
PTT Location Based Services IMS
Mobile TV
User culture drives Technology Trend
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 3
End-user Experience
CamGSMs Challenges in 3G Era
To understand end user Demand
-Understand end user interest & service distributions
Maintaining network
Behavior Performance
Total Value of Ownership
Performance during rapid growth
-Maintain VIP urban 3G network During 300 NodeB launch
Complexity
Increasing network Complexity after new services introduction
-Maintain 2G&3G networks simultaneously
Visibility Services
Defend and grow top line Services
-Support new service to launch & grow
Visibility of user experience
on offered services
-KPI User Experience (QoE), visibility to manage user experience, E2E, All-IP etc
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 4
What is Service Quality Improvement about?
Service Quality Improvement
Is about End-To-End Visualization of real end-user experience Improve application service quality Root cause analysis of service quality issues
Network KPI
RRC Setup Success Rate Call Drop Rate Call Setup Success rate
User Experience
No service Poor voice quality Slow to web
Customer Hotline
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 5
From KPI to Service Quality Improvement
As is To be
Good KPI, but Poor User Experience
CTOs Experience: Full signal bar, but difficult to call out Poor voice quality, One-way audio Connecting to internet, but cant browse internet
Build SQI System, Improve End-User Experience
Level 1 QoE: Quality of Experience, ranking from 1-5; Level 2 KQI: Key Quality Index, measure end user experiences; Level 3 Measurement KQI: Measurable, Manageable, Assessable, Improvable KQIs;
Cell KPI Value RAN Access Success >99% Rate Call Drop Rate <0.3% Cell Average KPI is Good
QoE WEB page access satisfaction
KQI WEB page access success ratio WEB page access delay
Measurement KQI E2E access success ratio DT access success rate E2E page access delay DT page Access delay
Sample DT KPI Value Coverage >99% Voice Quality (MOS) 3.6 Sample DT KPI is Good
Good KPI Good Service Quality
Huawei Confidential Page 6
HUAWEI TECHNOLOGIES CO., LTD.
From Network layer to User layer
As is To be
Indicator Care ---Monitoring Network KPI
Call Setup Success Ratio-NCR[%]
0 00 .001
99.20 99.25 99.30 99.30
VIP Care ---Quick Response to User Complaint
At same time:
Cause types: Normal Access failure Handover failure Call drop Web refresh delay Other exceptions Call record Call information Subscriber Access/release CS service PS service Handover
..
10:00 am 18:00 pm
Several Key users worse experience will not indicate from KPI, then complaint exists
HUAWEI TECHNOLOGIES CO., LTD.
00 .99 00 .89 00 .79 00 .69
Week 39 Week 40 Week 41 Week 42
Call Set-up Success Rate
KPI Normal
10:00 am
10:30 am
11:00 am
11:10 am
Signaling
Huawei Confidential
Page 7
Huawei SQI Capability - Tools & Platform
Accessibility Service Quality KQI System
E2E access completion rate E2E access duration
Retainability
Web offline ratio Speech Quality
Integrity
-One-way Audio -Echo -Time delay -Chopping -Noise -Crosstalk
User perception of the entire network
VIP&VAP user perception
Monitoring (tool) Tool Service
M2000 (traffic/log) PRS U2520 Nastar MR
Assessment & Delimiting & Identification (Service)
Nastar
E2E Active/passive joint testing
Assessment tool set
Equipment capability
End-to-End Network
BTS Abis
Resource-based refined tracking
User-based refined tracking
IP
BSC A UMG MSC MSC UMG A BSC Abis BTS
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 8
Huawei SQI Capability User Experience Research Team
Special Experts Dr. Dong Biography
The User Experience Principle at Huawei. Ph.D from Purdue University in the USA. More than 15 years of research and Design experience in top companies, such as IBM and eBay. He is now in charge of the user research and consumer programs in the Department of User Centered Design. Ph.D. degree in psychology from the Zhejiang University, Zhejiang, China. More than 6-year research experiences on human perception and mental processing. Expert of QoE modeling- establishing quantitative relation between KQIs and QoE.
Dr. Wan
Dr. Cui
Ph.D in Cognitive Psychology from the Zhejiang University, China. Senior user researcher at UCD center, Huawei R&D. Expert of QoE research User Experience Design and Human-Computer Interaction.
From Technologies to Research in End User Experience & Human Perception
HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential Page 9
Tina
Zhang Tingting received her master of Arts (Sociology) in Nanjing University, accumulated wealth of experience in user research. She organized and participated in lots of large sample social research and focused on quantitative analysis.
Q1: How to Establish SQI System?
Example
Project Kick-off Tool & Service Setup KQI Data Collection KQI Analysis & Optimization KQI Monitoring & QoE Survey QoE KQI Modeling
Project End
Step 5 KQI Monitoring
1.Precision KQIs for use of each specific service 2.Time Span: 2~4 weeks, 7*24 hours 3.Sample Size About 20,000 4.Sample Structure User groups of different user behavior, and from different regions.
5
Step5 QoE Survey
1.Method Phone survey 2.Questionnaire Satisfaction of the quality of Service, including QoE and sub-QoE. 3.Performer Call center (Trained by QoE Experts) 4.Sample Size: 900~1,000 users from those monitored 5.Potential Risk: Might annoy the end users, how to control?
5
Objective
Data from KQIs monitoring
KQI monitored
Mapping KQI to QoE Step6 QoE-KQI Modeling
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Method
Regression analysis+ QoE = f(KQI1, KQI2, KQI3 ) Page 10
Subjective
QoE score from phone survey
QoE surveyed
Q2: How to Measure QoE?
Mapping Category KQI name KQI value QoE Weight score QOE score 4.5 4~4.5 3.5~4 3~3.5 <3 QoE grade Excellent Very good Good Fair Poor
E2E access completion rate (Passive Test) Accessibility Front page access delay
Example
User perception Very satisfied Satisfied Neutral Very unsatisfied Extremely unsatisfied
Integrity
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
%51
1 2 3
spbk46 < spbk821~46 spbk652~821
Web page refresh delay
%52
Web page refresh success ratio
%01
Retainability
Web browsing data transfer cut-off ratio
%04 %01
5 1 2 3 4 5 1 2 3 4 5 5 1 2 3 3 3 3 4 5 1 2 3 5
4 4 4 4 4 4 4 4
spbk215~652 spbk215~652 spbk215~652 spbk215~652
spbk215 %07 < %08 ~ %07 %59 ~ %08 %99~%59 %99 %2 > %2 ~ %1 %1 ~ %5.0 %5.0 ~ %1.0 %5.0 ~ %1.0 %1.0 %1.0 spbk46 < spbk821~46 spbk652~821 spbk652~821 spbk652~821 spbk652~821 spbk215~652 spbk215 %07 < %08 ~ %07 %59 ~ %08 %99
%99~%59 %99~%59 %99~%59 %99~%59
QoE
3* 4 * 40% + 3*10% + 4*10% + 3*25% + 4*15%
3.65 (Good)
Initialized with default weight value & benchmark for KQIs , yet need to get agreement with CamGSM first.
Page 11
Content
What is Service Quality Assurance Case Study
Data Service Quality Improvement China Mobile
Customer Needs
Measurable Establish WAP/MMS/FTP/Stream data service measurement system, which is tailored for China Mobile Hangzhou; Visible Quality monitoring of WAP/MMS/Stream data services; Manageable Establish methodology to identify key factors and improve KQI; Improvement 3G Quality Improve 3G network quality. (E.g. WEB front page access delay/Unstable MMS downloading speed/Frequent 3G complaints etc)
Hangzhou, China Mobile
Solutions
Establishment - China Mobile Hangzhou SQI system through partnership
Analysis & Optimization
- Data Service Quality Analysis - User complaints Analysis - 2G/3G Hotspot Analysis - User Behavior Analysis - P2P Service Analysis & Restriction - PS Core Network Parameter Audit & Resource Evaluation
Delivery Period: Aug.2010 Dec.2010 Organization: An End-to-End project team including: Wireless / PS / SQI PDT team / Nastar Tool Team Tools: - Nastar
- Wireless tool set - Probe /CDS - U2520
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 13
Data Service Quality Improvement China Mobile
TOP regions WEB Delay comparison
Front Page Access Delay Before & After (s) Refresh delay Page Refresh Delay Before & After (s)
Connection access delay
BF BF AF BF AF BF AF BF AF
AF
BF
AF
BF
AF
BF
AF
TOP regions WEB access failure comparison
TOP Cell Access Failures Before & After
Access failure rate
Values: - Established customized SQI solution for Hangzhou, China Mobile. - New O&M methodology for long-term benefits; - WEB Data Service Quality was remarkably Improved; - Joint Benefits Analysis is undertaken.
BF
AF
BF
AF
BF
AF
BF
AF
HUAWEI TECHNOLOGIES CO., LTD.
Huawei Confidential
Page 14
Thank you
www.huawei.com
