QW2001 Paper 2T2

Dr. Nancy Eickelmann & Mr. Allan Willey
(Motorola Labs)

An Integrated System Test Environment

Key Points

Presentation Abstract

Software and system testing is a critical activity in the development of high quality products. When testing is performed manu­ally it is highly error-prone, time consuming and costly. Automated Test Environments (ATEs) overcome the defi­ciencies of manual testing through automating the test pro­cess and integrating testing tools to support a wide range of test capabilities. Industrial use of ATEs could provide signifi­cant benefits by reducing testing costs, improving test accuracy, improving software quality, and providing for test reproducibility. Despite the critical importance of ATEs in the develop­ment of quality products, full life cycle integration of test tools is rarely achieved in practice. To understand what prevents full life cycle integration of tools with respect to data, process, platforms, user interfaces, and control, Motorola Labs is conducting focused R&D initiatives.

The Motorola Automated Test Environment (MATE) initiative provides technology to design, develop, and test high quality software and systems. The Motorola Automated Test Environment initiative addresses a subset of the product development issues focusing on testing, which remains a very costly, time-consuming phase of the product development life cycle.

The Core Process Redesign (CPR) initiative is identifying the necessary integration factors to insert an automated test environment that seamlessly inter-operates with the rest of the product lifecycle. The CPR identifies decision criteria, process input/output pairs, entry/exit criteria and common resource requirements and constraints. The overriding objectives of MATE and CPR are to radically improve time-to-market and predictability in schedules, costs, and quality of product development by:
* Development of a common process that enables consistent practices
* Build a common platform that supports interoperability and reuse
* Increase resource allocation flexibility to facilitate cost effectiveness
MATE and CPR are complementary efforts that focus on standardization, automation and integration of tools, data and processes.

A primary goal of the MATE initiative is to develop a common and automated test environment throughout the corporation. With a common environment the entire corporation would be able to take advantage of optimization improvements made to the environment based on technology advancements external to Motorola and technology acquisition internal to Motorola. Therefore the MATE architecture must accommodate legacy tools, process changes and future needs met by technology insertion. As such the MATE requirements are a negotiated construct among the identified stakeholders.

This paper examines software architectural constraints in relation to software and system test automation environments. The MATE architecture requires a mapping of multiple views of the environment including structure, functionality, process and data. An overview of the environment is represented by mapping the test functionality to hardware and software structures of the environment. An architectural representation provides a foundation for evaluating the impact of architectural and COTS choices on system test engineers and test managers. This paper discusses the integration of COTS tools into MATE including DOORS, Primavera, ClearQuest and ClearCase; the development of standardized test management support TMS; and the automation of the system test process. The MATE architecture is described and a detailed discussion of test automation and tool integration issues is undertaken. Software architectural analysis is used to determine if a specific structural decomposition and the functional allo­cation to system structures supports or impedes certain qualities. Changes to an ATE such as enhancements to system functionality, improvements to performance (space and time), and reuse of components, data representation and changes to processing algorithm are all sensi­tive to system architectural constraints.

About the Author

Dr. Nancy Eickelmann is currently a research scientist for Motorola Labs and is leading the Motorola software and system test process measurement and evaluation research initiative. Prior to joining Motorola she was program manager at the NASA/WVU Software Research Laboratory, her research focused on integrating the Balanced Scorecard into the NASA context to provide a measurement framework for software test technology improvements. Before joining NASA she was a member of the Advanced Programs Research Group at MCC where she developed a measurement framework for guiding the decision-making process in product line development. Dr. Eickelmann began her research career as a member of the technical staff at Hughes Research Laboratory (HRL) in Malibu, California while completing her doctorate at the University of California, Irvine. She was named a Hughes Doctoral Fellow while working at HRL and received several research awards while working with Dr. Debra Richardson's Formal Methods and Software Testing Group at UCI. Dr. Eickelmann has collaborated internationally on research projects for defense systems, space station applications, space shuttle and global software.

Allan Willey is a Member of Technical Staff at Motorola Labs in the Software and System Engineering Lab (SSERL). Allan leads the "Motorola Automated Test Environments" (MATE) Team. These applied researchers are developing techniques to improve the capabilities of software development groups to test new products. Projects using various advanced statistical analysis techniques, formal modeling, and simulation techniques are being carried out to assess their value for improving delivered product quality, as well as their impact on productivity and testing time. The MATE Team collaborates closely with development organizations in various Motorola product groups to transition successful results. Allan holds an AB in philosophy from the College of William and Mary, and both a BS in mathematics and an MBA in management sciences from George Washington University.