Purpose
• The purpose of this process is to standardize the communication process and
improve the communication efficiency between the R&D team, IT team, and
the Support, Customer Success, Sales, Marketing team regarding
Emergencies about the product.
• The internal teams mentioned in this document include the NA Enterprise
Sales Team, the Customer Success Team, SMB Technical Sales Team, APAC
Enterprise Sales Team, Marketing Team, and IT Team, R&D Team.
Hereinafter referred to as internal teams.
Emergency Issue Definition
The following situations are regarded as emergency issues. We may have system-
wide latency or outage, or the whole server/system might be down, rather than
single site/client issue.
• All the agents cannot log into Control Panel or Agent Console
• Unusual error message on login page
• Login page keeps loading forever
• Chat messages cannot be sent out during chat
• Cannot send attachments/files
• The chat button disappears from the website
• All Site Account Cannot send Ticket Replies or receive Email Tickets
• All Site Account Cannot send or receive Social Messages from any channel
• Cannot initiate chats, send messages with bot
Emergency Process
1. Call Emergency Contacts
Usually emergency issues are related to the system, servers, data centers,
infrastructure issues. If we receive reports from customers regarding these issues,
please contact the emergency contacts IMMEDIATELY.
After you make the call, please create a customer ticket in DevOps with the issue
details. For how to create ticket in DevOps, please refer to this wiki page.
Emergency Contacts List:
1. During Vancouver Working Hours (9AM to 6PM, Pacific Time),
1. For emergencies you cannot decide the Product/Category, please call
Summer (backup: Elei) and Vancouver R&D team Grant (backup:
Saikiran).
2. For emergencies you know about the Product/Category,
• For infrastructure issues, please call Summer (backup: Elei)
• For product issues, please call Vancouver R&D team - Grant
(backup: Saikiran).
2. During the rest of the time,
1. For emergencies you cannot decide the Product/Category, please call
Summer (backup: Elei) and Live Chat team Emergency Contact
number (backup).
2. For emergencies you know about the Product/Category, please call
the contacts of each team below accordingly.
Contact Departmen Contact Backup
Product/Category Name t Number Number
Infrastructure Summer IT Team +86
1815813715
7
Infrastructure Elei IT Team +86
(backup) 1350581696
7
Live Chat & Booking Surey or R&D Live +86 +86
& Queue Jason Chat Team 1570017018 13867448920,
(See 6 +86
rotation 13588732041
note
below)
Vancouver Team Grant Li R&D +1 236-865-
Vancouver 3702
Team
Vancouver Team Saikiran R&D +1 647-313-
(backup) Dundigal Vancouver 7548
Team
Global & Internal, Hardy R&D Global +86
Login Issues Team 1867483542
6
Bot Davy R&D Bot +86 +86
Team 1867483542 16680802108,
6 +86
17352835210,
Contact Departmen Contact Backup
Product/Category Name t Number Number
+86
17352830210
Ticketing & Frank R&D +86
Messaging Ticketing 1869224977
Team 3
Knowledge Base, File Allon R&D +86
Upload/Downloading Architect 1865886658
Team 0
Rotation Note: (new process)
Emergency Contact for Live Chat team will be rotated quarterly. The business
team should call the emergency contact number +86 15700170186, and the person
on duty for the quarter will answer.
• Q1 & Q3: Surey
• Q2 & Q4: Jason
2. Escalate - Call Product Line Manager
Once the emergency issue is determined to be product line specific but no
solution is provided by emergency contact within 30 mins, please follow the
below table to call the emergency escalation point of the corresponding
product line to escalate.
The product line manager should be engaged with no more than 30 mins if the issue
is product line specific and not resolved. In the meantime, inform CTO William, head
of IT Elei & head of Product Line if they’re not yet aware.
Emergency escalation point for each team:
Contact Contact Backup
Name Department Number Number
Elei IT Team +86
13505816967
Henry Live Chat/Queue/Booking +86 +86
15067144897 13958006596
William CTO/Global & +86
Internal/Vancouver R&D Team 13575751035
Davy Bot +86 +86
Contact Contact Backup
Name Department Number Number
17673634755 16680802108
Yogesh Ticketing & Messaging +91
7838351911
Allon Knowledge Base +86
18658866580
3. Communicate - Set up Emergency Call Teams Meeting
For issues that cannot be resolved within 30mins, Emergency Call teams meeting is
required. And the emergency contacts will post updates to the business teams
regularly.
You can use this pre-created emergency call link and then add the emergency
contacts to the call -
• https://teams.microsoft.com/l/meetup-join/
19%3ameeting_NmQyYjg1NzAtMjAwYi00MjgzLWJkMGQtOTEzYTRhMWJjN
mVj%40thread.v2/0?context=%7b%22Tid%22%3a%2267648efa-5ddc-
4372-b505-62b86a1f61c6%22%2c%22Oid%22%3a%2214836f60-8393-
4155-b250-794a6fda2a7b%22%7d
4. Emergency Updates and Post-Morten
Emergency issues need to be investigated immediately - response in 15 minutes and
resolve in 4 hours. IT team and R&D team would be online when we try to solve
these kinds of issues and we will update you frequently in the Emergency Call
Teams Meeting or in the Teams >Service Outages.
Regarding Teams >Service Outages, it’s used emergency issue commutation rather
for support/CS team to report issues, so “New conversation” should be initiated only
by IT or R&D Team. Once “New conversation” is created, all teams can follow up and
update under the thread/discussion.
After the emergency issue is resolved, an incident/issue report will be updated of
this issue in DevOps, either by IT team or R&D team. The report will add in the
Discussion and include the following info:
1. Which platforms or what clients were influenced?
2. What was the issue and what’s the status?
3. When did the issue happen?
4. Why did the issue happen?
5. What we have done to solve it?
6. How long did we take to solve it?
7. What we will do to avoid the same issue from happening again?
If the emergency Issue/outage caused by internal change (e.g. not external reason
like AWS), the responsible person needs to fill in the “RCA Report” and upload to
Teams -> Outage channel.
