Customer Visual
How to Break your App Best Practices in Mobile App Testing
Daniel Knott | @dnlkntt
14/13/16
4/13/16
Who am I ?
Daniel Knott
Software Test Manager/ Mobile Tester @AOE
Mobile Testing since the end of 2010
Before AOE, Team Lead QA @XING Mobile Team
Blogger and Speaker
http://www.adventuresinqa.com
http://www.linkedin.com/pub/daniel-knott/1a/925/993
https://www.xing.com/profile/Daniel_Knott
@dnlkntt
24/13/16
34/13/16
4/13/16
Company Profile
Intro
Current
Status
Mobile
Challenges
The Customer
Mobile Pyramid
Tools
44/13/16
Mobile Specific
Testing
Current Status
54/13/16
4/13/16
Author: Barry Caruth, https://www.flickr.com/photos/caruth/8401943928/
Current Status | Some Numbers
2+ millions apps available
100+ billion downloads
80% 90% of apps are deleted after the first
usage
56% of apps that requires a registration gets
deleted user checks device 150 times per
Average
day
45% are not well tested
64/13/16
KIFSU To Remember the important Things
K
Keep
74/13/16
I
It
F
Fast
Useabl
Simple
e
84/13/16
4/13/16
Mobile Challenges
Mobile Challenges | All known!?
94/13/16
http://opensignal.com/reports/2014/android-fragmentation/
Mobile Challenges | All known!?
104/13/16
http://opensignal.com/reports/2014/android-fragmentation/
Mobile Challenges | How to handle fragmentation
Know the target group
Group your devices, A B C
Test only on those devices
Test on real device
Dont use simulators or emulators
114/13/16
Mobile Challenges | How to handle fragmentation
Group 1, Prio A
High End
Devices
Dual/ Quad
Core CPU
RAM >1024MB
Retina, Full HD
Display >=
4,5
Based on your target group
124/13/16
Group 2, Prio B
Group 3, Prio C
Mid-range
devices
Dual Core CPU
RAM <1024MB
Screen size <=
4
Software not
older than 1
year
Small devices
Single Core
RAM < 512MB
Low screen size
and resolution
Older OS, older
browser
Mobile Challenges | More
Author: Davide Vizzini Limbte, http://www.flickr.com/photos/limbte/4007115435/
Author: Juan de Dios Santander Vela, http://www.flickr.com/photos/juandesant/209098053
Author: Rain Rabbit, http://www.flickr.com/photos/37996583811@N01/8033259053/
134/13/16
144/13/16
4/13/16
The Customer
Author: Michael Newman, https://www.flickr.com/photos/mzn37/122601628
The Customer | Better know them
Know the target group
Will downsize testing effort
Talk to your customer
Do usability testing with them
Author: Grzegorz Lobinski, http://www.flickr.com/photos/gregloby/3515990945/
Invite them to a beta testing program (Sonos,
FB, Twitter)
154/13/16
Mobile Specific Testing
164/13/16
4/13/16
Author: Theus Falcao, http://www.flickr.com/photos/theusfalcao/956376714
Mobile Specific Testing
Known functional testing is still required
Boundary values
Equivalence classes
Testing against acceptance criteria
Test the app as black box
Try to break it
Be creative and think in edge cases
174/13/16
Mobile Specific Testing
Author: Juan de Dios Santander Vela, http://www.flickr.com/photos/juandesant/209098053
Author: Ian Britton, http://www.flickr.com/photos/freefoto/728651045/
Author: @Doug88888, http://www.flickr.com/photos/doug88888/7149187135/
Author: Dan Levy, http://www.flickr.com/photos/danlevy/5607259711/
184/13/16
Author: Moyan Brenn, https://www.flickr.com/photos/aigle_dore/5626341059/
Mobile Specific Testing
Author: amateur_photo_bore, http://www.flickr.com/photos/andresv/233667770/
Author: Rain Rabbit, http://www.flickr.com/photos/37996583811@N01/8033259053/
Author, Daniel Sancho, http://www.flickr.com/photos/teclasorg/2852716491/
194/13/16
Mobile Specific Testing
Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4311105154/
Author: Surat Lozowick, http://www.flickr.com/photos/suratlozowick/4464587302/
Author: Theus Falcao, http://www.flickr.com/photos/theusfalcao/9563767145/
204/13/16
Mobile Specific Testing
Android
JouleUnit, to measure battery usage
Monkey, to stress test the app
iOS
Energy Usage in Instruments
UI AutoMonkey, to stress test the app
214/13/16
Mobile Specific Testing
Author: Molly Holzschlag, http://www.flickr.com/photos/mollyeh11/272434559/
224/13/16
Author: Quinn Dombrowski, http://www.flickr.com/photos/quinnanya/4718635484/
Mobile Specific Testing
Author: minxlj, http://www.flickr.com/photos/minxlj/3270996158/
Author: Ryan B. BulaPhotography, http://www.flickr.com/photos/nayralub/3763581778/
Author: Sterling Coffey, http://www.flickr.com/photos/n0ssc/7046915929/
234/13/16
Tools
Author: OZinOH, http://www.flickr.com/photos/75905404@N00/712614712
244/13/16
4/13/16
Test Automation Pyramid
Manual
Testing
UI Tests
Integration Tests
Unit Tests
254/13/16
Flipped Test Pyramid
Manual Tests
UI Tests
E2E
Unit Tests
264/13/16
Mobile Test Pyramid
Unit Tests
Beta Tests
End2End Tests
(UI Tests)
Manual Tests
274/13/16
Tools
Do test automation!
Dont buy huge test automation tools
Start simple and discover different tools
No tool is perfect
See which tool fits in your development
environment
284/13/16
Tools | Android
Robotium (ExtSolo)
Robolectric
Spoon
Espresso
Selendroid
Monkey
Calabash for Android
MonkeyTalk
Appium
Great mobile tools provided by the company square: http://square.github.io/
294/13/16
Tools | iOS
UIAutomation
ios-driver
Keep It Functional
Kiwi
MonkeyTalk
XCTest
Calabash for iOS
Appium
304/13/16
Security
314/13/16
4/13/16
Author: Alexandre Dulaunoy, http://www.flickr.com/photos/adulau/771254542
Security
Author: Anthony Catalano, http://www.flickr.com/photos/badwsky/34164244/
Author: Dennis Wong, http://www.flickr.com/photos/denniswong/2406135310/
uthor: orkomedix, http://www.flickr.com/photos/orkomedix/5211437958/
324/13/16
Security
Security testing is not easy
Get help from experts
Think twice about app permissions
Check the OWASP mobile security page
https://www.owasp.org/index.php/OWASP_Mo
bile_Security_Project
334/13/16
344/13/16
4/13/16
Summary
Mobile Specific Testing | Summary
Be creative and on the move while testing mobile apps
Combine your in house testing with crowd
testing
Be up to date! Know your target group
Keep the mobile features in mind
Keep the mobile test pyramid in mind
Do test automation and know the right tools
354/13/16
Av
ai
la
bl
e
so
on
Thanks for your attention!
More information about mobile testing
can be found in my book
Hands-On Mobile App Testing
which will be available very soon.
www.handsonmobileapptesting.com
364/13/16
in Germany
AOE GmbH
LuisenForum, Kirchgasse 6
65185 Wiesbaden
Germany
Phone: +49 6122 70 70 7 - 0
Fax:
199
+49 6122 70 70 7 -
E-Mail: [email protected]
4/13/16
Web:374/13/16
www.aoe.com
Links & References
World Quality Report [http
://www.sogeti.com/en-us/looking-for-solutions/Services/Software-Control-Testing/TheWorld-Quality-Report-2013--2014
/]
Robotium [https://code.google.com/p/robotium/]
Spoon [https://github.com/square/spoon]
Robolectric [http://robolectric.org/]
Calabash for Android [https://github.com/calabash/calabash-android]
Selendroid [http://selendroid.io/]
ExtSolo [
http://docs.testdroid.com/_static/extSolodocs/com/bitbar/recorder/extensions/ExtSolo.
html
]
Espresso [https://code.google.com/p/android-test-kit/wiki/Espresso]
Monkey [http://developer.android.com/tools/help/monkey.html]
UI AutoMonkey [https://github.com/jonathanpenn/ui-auto-monkey]
384/13/16
Links & References
MonkeyTalk [https://www.cloudmonkeymobile.com/monkeytalk]
UIAutomation[https
://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/Instr
umentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
]
Keep It Functional [https://github.com/kif-framework/KIF]
XCTest [
https://developer.apple.com/library/ios/documentation/ToolsLanguages/Conceptual/X
code_Overview/UnitTestYourApp/UnitTestYourApp.html
]
Calabash for iOS [https://github.com/calabash/calabash-ios]
ios-driver [http://ios-driver.github.io/ios-driver/?page=home]
Kiwi [https://github.com/allending/Kiwi]
Appium [http://appium.io/]
JouleJunit [https://code.google.com/p/jouleunit/]
Energy Usage Instruments [
https://developer.apple.com/library/ios/documentation/AnalysisTools/Reference/Ins
394/13/16
truments_User_Reference/EnergyUsageInstrument/EnergyUsageInstrument.html
]
404/13/16
