User Experience Testing for Connected Devices 2

USER EXPERIENCE TESTING
FOR CONNECTED DEVICES

Performance, Functionality, Localization, Compatibility and Usability Testing

Your Challenge

If you’re developing Wireless IoT devices, you absolutely need to consider the end users in all your target markets. And you have to consider their varying needs and desires every single time, which means you have to incorporate User Experience testing into your connected device roadmap.

There are a number of challenges which need to be addressed when considering launching a smart device to market, two of the key ones being:

1) Determine how your target consumers use the connected device and associated APP?
Before you know what to focus on for usability, you have to understand how your target market will use your IoT device:

  • To show off.
  • To provide entertainment.
  • To automate simple tasks to make their lives easier.
  • To be able to control as much of their world as possible from their cell phones (see who’s ringing the doorbell, turning lights off and on, keeping track of their kids, etc.).
  • To monitor and control health issues, communicate with doctors and other health care providers, etc.
  • To increase efficiency, reduce costs, improve planning and forecasting, etc.
  • To support the educational process.
  • To simplify critical tasks.

2) Establish a Process for User Experience (UX) Testing

Clearly there’s a LOT to think about when it comes to testing the usability of your IoT devices and associated APP. The solution relies on a structured testing process that every new and modified connected IoT device must go through. This then must become part of your ongoing product development policy. 

Bureau Veritas Solution

To supplement its existing Product Usability Testing service, Bureau Veritas has launched a straightforward 3-Star UX Testing service for Connected Devices. Some of the tests we perform include:

Functionality & Compatibility (1-Star)

  • Labeling. Does the product have the correct labeling for the target markets?
  • Pairing. How easy is it for customers to pair / connect their devices? This is tested with common operating systems including Android and IoS as well as multiple controlling devices including smart phone and tablet
  • Actual Use - Functionality. Do all the features proposed in the APP actually work on the connected device? Using a controller device (i.e. phone), we verify that each of the functions listed or claimed operate as indicated. For all ‘state reading values’ (open/closed, etc.), we execute all manual operations to ensure reasonable range of reported values function as claimed. We cycle 15 times between controllable states (on/off, low/high, etc.) as a consumer would during normal use. 
  • Application. If the product includes an application (i.e. app), we assess compliance with the following requirements:
    • Any fields that require entry of data must be labeled.
    • Any screen that is not immediately self-explanatory to consumer shall include some form of ‘on screen’ instructions.
    • Any action that requires more than 10 seconds must include some form of indication that it is processing (i.e. progress bar).
    • Any error message should be informative and/or instructional for the consumer as to what is wrong or what can be done to resolve what triggered the error.

Performance (2-Star)

  • Connection. Does the device function correctly within the claimed distances across multiple wireless technologies?
  • Disconnection. We assess whether a typical user can disconnect pairing with product and controlling device at any time. We perform this testing when the device is operating multiple claimed functions.
  • Reachability / Power cycle. After a product is connected to a controller, we assess to make sure it must remain reachable. We perform this test case by firstly verifying the functions work as intended in stable connection, then we turn the product’s power off, wait 10 seconds, and turn power back on. We then verify product is reachable before and after the product is power cycled. 

Localization & Usability (3-Star)

  • Once the device is connected, how easy is it for people to log in and set it up? Are there common problems that can be corrected in the next iteration?
  • Is setup smoother when users have a “quick start” guide or when the device itself walks them through the process?
  • Does the APP work in all target market countries in terms of language and Intuitiveness of APP and local wireless technology bandwith restrictions?
  • What percentage of people can complete the connection process without assistance? 
  • If they do need support, what is their preferred method? Email, live chat, phone call, etc.?
  • For those who do need help, is there a common stumbling block? What is it, and can you eliminate it or come up with a workaround?
  • How do people react to the device’s security protocols? Do they complain about them or try to circumvent them?
  • What do people try to do first? Which features are most interesting to them? Are they able to easily access those features (preferably from the home screen), or do they have to search for them? How many screens do they have to go through to find what they need?
  • Are there features that most people ignore?
  • Are most people using the IoT device in the way developers intended? What are tasks are they attempting to do that developers didn't anticipate? Are they trying to complete tasks in ways that developers didn't predict? Are there features that developers thought would be popular that are being ignored completely?
  • Are there certain types of users who are having more trouble than others? What conditions might contribute to their difficulties? Challenges with sight or hearing? Language or cultural differences? The temperature in the user environment? Finger size? Device size or orientation?
  • Guidelines on which critical failures need to be reworked to maximize market penetration