ITS Standards Testing Fact Sheets

Fact Sheet #3, June 28, 2001

What are the technical tests?

The challenge:  As explained in the testing fact sheet, "How do we test the standards", ITS standards are mainly information standards and testing them requires separating their properties, such as their correctness, completeness, and quality, from the properties of the hardware and software that incorporate them. This, in turn, requires a testing strategy that includes interviewing the developers and deployers of these systems, as well as performing formal engineering tests to ensure that the standards enable the systems to carry out all the desired operations. Generally, the formal test procedures carried out on operating systems in the field involve setting specific inputs and observing whether the outputs or resulting conditions are the expected ones.

To fully characterize the tests, the conditions under which they are carried out must be thoroughly documented. In addition, the systems must be checked to see that the standards are implemented properly, without any custom additions or other adjustments that do not strictly adhere to the specifications in the standards. The systems themselves are not tested for conformance to the standards, but any deviations, either omissions or additions, from the specifications in the standards must be known before any system is used to test standards.

A single standard cannot be tested alone because, typically, it will depend upon other standards to carry out a function. For example, standardized messages from a message set standard are composed of standardized data elements from a data dictionary standard. There-fore, formal standards tests must address suites of standards that interrelate to carry out an ITS application. Determining the dependencies among standards and developing test plans to test each of the standards that together define the functionality of the system adds complexity to the testing process.

That's the big picture. But, what about the details? ITS standards include a variety of types, such as standards for communications (including physical media), data elements, message sets, and profiles. Exactly how is each of these different types of standards tested? What are the formal engineering test strategies that are used to assure that ITS standards are rigorously reviewed and field-tested and that knowledge is gained about any deficiencies so that improvements can be made?

The test strategy

It is important that the ITS standards be tested rigorously and consistently. Consistent testing means developing common engineering test methods for each category of standard and applying these common methods to all of the standards in each category. Because the standards are tested using ITS systems under operational conditions, a consistent testing approach also reduces concerns that a particular product or service may receive preferential treatment.

The field-test approach, in particular, the engineering testing methodology, is based upon proven methods in ANSI/IEEE 829-1983, "IEEE Standard for Software Test Documentation." The actual testing procedures include inspecting software code and using data-interception equipment to collect data for analysis. (Remember that these methods are only one aspect of the overall testing process, which includes interviewing vendors and deployers and reviewing and analyzing the standards documents.) The categories of standards that have been found useful for applying consistent testing methods, their descriptions, and the specific tests that are performed are shown as follows:

Type: Application
Description: Specify the procedures for file transfers access methods and management of information for user-designed application processes.
Testing Procedures: Test messages and data capture and analysis to verify that the required information, messages and data elements used in an application are transferred and correctly interpreted; engineering review and interviews with users and vendors.

Type: Application program interface (API)
Description: Specify external system interfaces with an application for exchanging data.
Testing Procedures: Data capture and analysis to ensure that standard allows full application functionality and does not interfere with other applications.

Type: Communications (including physical media)
Description: Specify requirements for ensuring accurate information exchange between applications.
Testing Procedures: Test messages and data capture and analysis to verify error free and correctly interpreted information exchange without interference with other applications; engineering review and interviews with users and vendors.

Type: Data Dictionary
Description: Provide a repository of data definitions, including type, meaning, format, integrity rules, and other attributes
Testing Procedures: Engineering review and interviews with users and vendors to determine whether applicable data elements are accessible and do not conflict with other data.

Type: Framework,
Description: Specify a set of rules and protocols for organizing, describing and exchanging data Testing Procedures: Engineering review and interviews with users and vendors to determine whether applicable data rules allow proper application operation and do not conflict with other applications.

Type: Hardware
Description: Specify requirements for actual hardware devices, such as size weight, operating temperature range, etc.
Testing Procedures: Engineering review and interviews with users and vendors to determine whether the hardware specifications allow full application functionality.

Type: Message set
Description: Collections of messages, which are sequences and formats of data elements, for a specific application.
Testing Procedures: Engineering review and interviews with users and vendors to determine whether applicable data rules allow proper application.

Type: Objects
Description: Specify software modules that support specific applications, including the information and its format.
Testing Procedures: Reference implementation and data capture and analysis to ensure that objects are properly formatted, and allow full application functionality; engineering review and interviews with users and vendors.

Type: Profiles
Description: Prescribe particular standards and/or options necessary for accomplishing a particular function or service.
Testing Procedures: Engineering review and interviews with users and vendors to determine whether applicable data rules allow proper application operation without interference with other applications.

For more information

In effect, some standards are tested indirectly; they are tested by testing other standards that use them. For example, framework standards are tested indirectly through standards that build upon the framework, and data dictionary standards are tested indirectly through their use in message sets and application standards.

For more information on ITS standards, contact the Federal Highway Administration, ITS Joint Program Office, HOIT, 1200 New Jersey Avenue, SE, Washington, DC 20590. Phone: 202-366-8048, Fax: 202-493-2027, Web: www.standards.its.dot.gov

Produced by the Jet Propulsion Laboratory for the U.S. Department of Transportation.