IoT is already past its technology hype cycle as it permeates our daily lives through voice assistants, smart thermostats, digital shopping carts, fitness wearables, smart kitchen appliances or smart TV. According to Gartner, there are 20 billion connected devices in use currently. Users expect these connected products to work seamlessly without any interruptions. Users switch products in the blink of an eye with the slightest discomfort. Hence, it is difficult to attain customer loyalty. In such a competitive scenario, IoT product performance becomes crucial to retain and engage customers.
Building an effective IoT infrastructure requires systems that are distributed, scalable, and reliable. Traditional application performance testing is not suitable for IoT performance testing. Testing IoT performance can be very complex. It involves testing performance across multiple components and endpoints like the device which can be a handheld, tablet, drug dispenser, video cameras or sensors; IoT protocols like MQTT, CoAP, XMPP, IoTivity, AllJoyn, Zigbee or Z-Wave; networking gateways or routers; on-premise or cloud server; web applications on different browsers or mobile applications on different operating systems; REST API services. Moreover, IoT performance testing also involves carrying out testing on new parameters that are not part of testing traditional software applications like user interface performance, network performance during high load or testing performance for high data volume.
Choosing the right performance testing tool also becomes a challenge, as it needs to support the appropriate protocol on which the IoT system is built, handle test data management, provide integration with various modules, support network simulation scenarios. In addition, the tool licensing costs should fit in the overall testing budget. Given the large number of IoT devices, which are spread across different geographical locations, it is impossible to test the performance on each of them. It is also difficult to test the real world scenarios in an IoT lab environment.
Testing the stability of an IoT system, for sensitivity to network connectivity issues, end user devices like different mobile operating systems or sheer number of users is needed as all these factors ultimately affect user experience. Hence, new testing strategies like edge/device virtualization have to be used to simulate and test a particular performance scenario from edge to cloud over different networking technologies like Wi-Fi, Bluetooth, 2G, 3G, 4G, and now 5G. A Telegesis USB stick or Raspberry Pi along with Python scripts are used to achieve and run Edge virtualization test scenarios. Performance testers need to simulate millions of devices to test real life scenarios and load on the system. Testing IoT performance involves millions of devices as against millions of users in traditional testing.
IoT devices produce huge amounts of data. Big data coming out of IoT devices is analyzed using various business intelligence and analytics tools and technologies to understand the usage patterns, to be further used in decision making for finance, marketing or supply chain, and operations. It is also important to simulate this data load on the IoT system or application under test to highlight real world scenarios. The data load is altered for normal and peak scenarios to check the IoT system performance.
Another scenario in IoT performance testing, which is very important is the request-response time between the IoT device and the server. Time taken to generate a response from the server to the IoT device to carry out a particular action ultimately affects the uptime and key monitoring metrics of IoT operations. Network plays a crucial role in testing IoT systems as smart IoT devices depend on seamless communication to perform the desired function. Network infrastructure issues related to bandwidth, networking hardware like routers, gateways, network latency, network traffic, and operator connectivity affect the performance of an IoT system. IoT gateways can be virtualized using network emulators in combination with load testing platforms to thoroughly test all these parameters and obtain desired IoT system performance.
IoT system performance testing should be an integral part of the IoT test strategy and planning. The factors affecting system performance should be clearly laid out in the test plan, as these will differ according to verticals. The cost associated with test lab setup, tools, device virtualization, etc. should be part of the initial test plan.
Performance testing both for positive and negative scenarios should be carried out. Integrating performance testing in the overall agile development and testing, in early stages helps in building a robust scalable IoT system needed to deliver optimal performance, by working on the design changes earlier in the development lifecycle. Effective performance testing of connected products can help in improving connected user experience.
eInfochips offers IoT specific unified test automation framework across various verticals that tests the performance of IoT systems right from edge to cloud, web and mobile applications. Get in touch with eInfochips Quality Engineers who can help you carve out a winning IoT performance testing strategy to achieve business benefits.
