Wednesday, April 26, 2017

Xamarin Test Cloud Best Practices



Do’s:
  • Get test cases from QA. One great reason for Test Cloud is to be able to complete a base set of smoke tests or comprehensive testing so that QA can do exploratory testing for new issues / rare cases.  
  • Take a lot of screenshots using ‘app.Screenshot()’ and use a very descriptive name / sentence to describe the step occurring. The only views you will see on Test Cloud will be these screenshots.
  • Name your elements the same between iOS and Android, otherwise you will have a lot of nasty if/else statements in your code.
  • Repeat code as much as possible. If you have 10 tests for buying an item.. take a separate reusable method that logs you in, and call this method many times. The make another method that will add something to your cart, and call this method many times. In case the ‘login’ button name changes, you do not want to be changing your hardcoded app.Tap(“element_name”) in 50 locations, you want to have to only change this is one place if possible.
  • Naming conventions: Try to keep really good naming conventions like large_button_text_on_button or icon_close so that it is easy to identify or even identifiable without going through REPL.  The less time you have to search for elements in REPL, the quicker it will be to complete development.
    • *Make sure you do have a label of some kind to hook onto. If there is no class/id/other name, add one!!! Otherwise you end up doing a hackish thing where you hook something onto the 5th element on the page that is a button or something, and it’s not pretty.
  • Tests can run on average 30 s – 6 minutes or so depending on the app. Try to break down your test scenarios into smaller pieces—as everything is hardcoded, the longer the test the higher probability it can break with any new changes / updates to the app.
  • If you are debugging your app, add app.Repl(); right before it breaks so you can debug and figure out what is going on.
  • Get really good at app.Query();, you will probably use these a lot to find elements.
  • Try to automate when a build goes out, so does a script to test cloud to run smoke/full sets of tests.

Do Not’s:
  • Map anything to x/y coordinates. This changes a lot based on the device you use.
  • For whatever element you choose to wire your test cloud to.. make sure you do not name multiple things on the page the same thing whether it’s a class/id, etc..

Notes:
  • There is an option for a live recording of your phones on Test Cloud. It is a setting when you submit your tests to the cloud to run. You can choose to send a subset of tests or all of the tests.  Subsets are defined using an attribute above the test name. You can add more than one category to a test, for ex. If it is part of a smoke test, part of a login screen test, etc.
  • You probably are not going to get 100% of tests working all the time, phones on Test Cloud will
  • It seems safe to run about 6-10ish types of varied devices per run to conserve hours to get a good idea of if your app is working across most devices.  You can run more if you are not concerned about saving on hours.  
  • You can plug in a physical device also to your computer and run test cloud on that. Make sure you enable debugging on your device. This is a good way to test on iOS if you are only using a Windows computer.
  • You are probably going to have to use some platform specific if statements, for ex. Typing text into an input may not work with .Tap() and you need .Enter(). Make sure you always inject the platform into your classes using DI everywhere.  It happens, sometimes things just don’t work in both platforms the same.
    • if (platform == Platform.Android) {}
    • if (platform == Platform.iOS) {} 

Pain points:
  • If your naming convention is bad or non-existent, consider going through the app to fix this.  It will save you time to re-name elements in a clean way as opposed to trying to hookup Test Cloud to poorly named pieces.  Especially if iOS/Android are different, try to keep these aligned. You can eat a lot of development time if the names are bad, different between iOS/Android, and if elements are hard to find.
  • Remember everything is completely specific to the flow. If you do a UI test to login, add an item, and hit checkout.. and someone adds a new page in for a promotion before you checkout.. it will break your tests. If an element is removed or changed, the tests will break.  There is some upkeep/maintenance to consider, though it should be quick changes.