Hardware-Software Integration Testing Solution for a Telepresence Robot PlatformThe client has successfully found a nimble partner to save 58.4% on personnel costs while their test execution time has been reduced from weeks to roughly 4 hours.

C A S E S T U D Y

As a trailblazer startup aiming to revolutionize virtual communications, our client has been transforming how people collaborate with each other in the virtual world. From New York, you can video conference with your colleagues in San Francisco through a telepresence robot which can physically “walk” around the room. These telepresence robots help remove the geographic barriers hence effectively build a global team working on the same thing at the same time.

This unique smart IoT product obviously demands a unique testing solution. Besides hardware & network configuration testing for the IoT telepresence robots, the platform also involves a set of front-end apps for the end-users to control the robots. Those front-end apps must run across platforms, including Windows, macOS, Linux (Ubuntu & CentOS), Android and iOS (both phones & tablets). On the other hand, the back-end administrators need a web app (written in Angular & NodeJS) to manage all robot settings, user groups, group policies, etc. This web app should run well on a wide range of desktop browsers (IE, Chrome, Firefox, Edge, and Safari).

Piled upon that, both front-end and back-end users must be able to switch between multiple languages, including English, French, Japanese, Spanish, Deutsch, Korean, Chinese, etc. This huge number of configurations and test environments definitely hamstrings the reproducibility and maintainability of the test cases.

To cover all of testing aspects, the client had to invest in many solution packages (manual hardware testing, network testing, automated functional testing, and mobile testing), each of which has its own set of challenges. Thus they needed one trustworthy partner with expertise in all of those areas. That’s when LogiGear came into the picture.

Business Needs

When this fast-paced startup touched base with LogiGear, their development team had to struggle with pressuring release deadlines, lots of bugs found during regression and an unmaintainable BDD test automation approach. Realizing those problems, they defined their test strategy to centralize around a solid test automation framework with maintainability as the number one priority.

Additionally, they needed professional test services provided by a reliable partner to tackle the complexity of hardware & network configuration testing. Bonus points for the potential partner that fits their startup culture by understanding their vision and knowing what it takes to deliver quality at speed. This partner should be able to cope with their pace of innovation, and aggressive product launches.

The Objectives

FAST FACTSTechnology, IoT, Virtual Communications

<50 employees (startup)

Palo Alto, CA, USA

• 6 Operating Systems

• 5 browsers

• 9 natural languages

• Maintainable Selenium automation framework

• ~400 automated functional testing on multiple platforms

• Manual hardware & network testing services

• Enabled 2-week release cadence

• Saved $338,000 per year (est.) which is 58.4% on personnel cost thanks to Vietnam’s low cost structure

• Reduced test execution from weeks to 4 hours

INDUSTRY:

COMPANY SIZE:

LOCATION:

BUSINESS NEEDS:

LOGIGEAR

DELIVERED:

BUSINESS

IMPACT:

The SolutionLogiGear ran a well-defined proof-of-concept (POC) project for the client when we first engaged. We always believe that POC is an important tool for our customers to enter a risk-free evaluation and smooth out their adoption.

Throughout this POC period, the client learned about various benefits of partnering with LogiGear. For instance, testers used to take 7 days to manually run a test suite of 142 test cases per one OS-browser-language combination. Thanks to test automation, that suite now takes only 5-7 hours depending on the browser performance. The run time can even be reduced to less than 1 hour if executed by multiple concurrent threads on clouds.

The client also got a better control of test progress as well as automation effort via detailed daily status reports provided by LogiGear engineers. Another sweet spot was Vietnam’s lower cost structure resulting a low-risk onboarding.

As long as the POC went, the in-house management and test team liked what LogiGear delivered and how LogiGear did it. Trust has been built. Consequentially, after 1 month of POC, the client finally decided to stick with LogiGear as a long-term partner.

Building a Selenium FrameworkTo accommodate the testing requirements for back-end administration web app, LogiGear leveraged the open source solution Selenium. Although Selenium is already a powerful and popular automation library, it’s just a “library”. Building a maintainable and scalable Selenium framework requires significant time and efforts. One common mistake people usually make is overlooking the long-term benefits of a good framework architecture. Just like the software we’re building, a test framework must be well-architected.

Recognizing that mistake, LogiGear has delivered a tailored-fit Selenium framework written in Python. Python is the client’s language of choice due to their familiarity with it in their in-house development. Interface definitions in this framework are stored separately in PageObjects for easier maintenance. The framework also supports writing data-driven tests with a separate layer of test data. After a few months, this nimble Selenium framework completely replaced the old and clunky BDD framework.

After the framework is up and run, LogiGear engineers have been continuously churning out new automated tests & executing those automated test suites to make sure the back-end administration web app work correctly.

Leveraged Squish for Front-end App TestingSince Squish is a strong solution for cross-platform testing, we leveraged Squish to automate the suites for the front-end apps (with Qt as the underlying GUI technology) on heterogeneous platforms including Windows, Ubuntu, CentOS, and macOS. Android and iOS apps are on the horizon. Ensuring the quality of the front-end apps is vital to the client’s business because in virtual communications, user experience is key. Malfunctions or lags in continuous video streaming may disrupt the communications thus devastatingly diminish the value of the IoT telepresence robots.

Smart Mechanism to Handle Multiple ConfigurationsUsing LogiGear’s Selenium framework, the test team doesn’t need to duplicate each test case to match a specific configuration or environment (languages, OS, browsers, etc.). For instance, they only need one test case to test all supported languages since the test steps & check points only take in a variable referencing a dictionary entry. For each run, after testers specify the language they want to run, the test runner will automatically pick the right value for the

LogiGear is a leading provider of software testing services, test automation and application development and maintenance. We help organizations deliver better products while saving time and money. Since 1994, we have completed testing projects with hundreds of companies from early stage startups to Fortune 100, across a wide range of industries and technologies.

LogiGear USA - Headquarters1850 Gateway Drive, Suite 175,Foster City, CA, 94404Tel +1 650 572 1400 Fax +1 650 572 2822www.logigear.comEmail: sales@logigear.com

About LogiGear

The Results• The client saved 58.4% (around $338,000) on personnel cost per year.

• With the nimble support from LogiGear staff, the client’s in-house team is now able to focus on what they do best: developing new code, rolling out new releases, designing new tests and other tasks that deliver value directly to their customers.

• Within 22 months, a team of 4 automation engineers based in Vietnam has churned out 399 test cases that cover almost 80% of the initial test requirements. For each release, the regression suite runs with a high level of stability.

• For the IoT robots, another manual tester works in-sync with the client’s team to explore and report hardware malfunctions, firmware bugs, and network configuration issues every sprint.

• Due to large number of possible OS-browser-language combinations, regression cycles used to take weeks to complete. Ever since the installation of test automation, regression cycles have been reduced to only 4 hours.

above-mentioned variable. This is very similar to the i18n localization standard used by Squish.

Manual Testing for Hardware & NetworkEnsuring the quality of the IoT robots involves hardware and network configuration testing which are harder to automate due to its fuzziness nature and difficulties in simulating various test scenarios. Thus the client sticks with manual testing in these testing areas.

Only one LogiGear manual test engineer covers all aspects related to hardware malfunctions, firmware testing, glitches in video streaming and exploring various network configurations such as different Wi-Fi frequencies, different authentication methods, different routers and network devices, captive portal bypassing via a helper device, etc.

With over 24 years in testing, LogiGear team has been providing the client reliable assurances and confidence. LogiGear engineers also gave valuable feedback to improve test designs and certain test practices.

Synchronized with Test Case ManagementLogiGear team manages the manual test cases on TestRail just like the client. However, automated tests are not directly associated with an equivalent TestRailt test case due to the lack of native support from this TCM tool. Thus, the team must improvise by applying their own system to monitor the execution progress, test development and defects found. Although not centralized, this monitoring system still effectively gives enough insights that inform good decision makings in the client’s fast-paced startup culture.