ITS Standards Testing Fact Sheets

Fact Sheet #2, June 20, 2001

How do we test the standards?

The challenge:  ITS standards are information technology standards. Unlike dimensional, material, and other standards for physical properties, which may involve length units, information technology standards involve processes and quantities that are not measurable using physical units. Since testing necessarily involves some sort of measurement, testing ITS standards is not the same as testing many other kinds of standards.

A way to visualize the challenge is to consider the difference in testing hardware standards compared to testing information technology standards. Testing hardware is straightforward; it may involve measurements of geometry, material properties, and operational characteristics to determine whether the specifications in the standards are met. The measurements are made in physical units (such as length and density). The measurement conditions (such as temperature) are documented, along with the uncertainty (such as measurement accuracy and precision). The results are compared to the requirements of the standard.

ITS standards are mainly implemented in software. Software is harder to test than hardware. Physical units may not apply; neither may measurement accuracy and precision. Further-more, software must be executed in hardware; that is, some system such as a computer must be used to run the software. This complicates software testing because it is difficult to sepa-rate the hardware characteristics (how fast is the processor?) and the operating conditions (what operating system is being used and what other software is running?) from the behavior of the software under test.

If we cannot test ITS standards in isolation, but only as part of a system, how do we test the actual standards themselves and separate what we want to test from the properties of the system? In addition, since the standards are the specifications for the system or for the in-formation the system sends or receives, what exactly is it about the standards that we are measuring?

What is tested?

Generally, we test ITS standards by making measurements to show that the hardware and software developed in accordance with a standard allow a system to carry out functions that meet users' needs. Specifically, this means measuring the correctness, completeness, and quality of the standards.

Correctness means that the standard's specifications allow the system to carry out the functions properly and that systems that incorporate the standard perform as desired. In other words, all of the desired outputs occur for allowed inputs, without any unexpected problems with the system. Completeness relates to whether the standard contains specifications for all the functions that users need in the application. Finally, in the context of ITS standards testing, quality deals with such issues as whether the standard uses hardware and software efficiently and economically, references other appropriate standards, and is understandable and clear.

To ensure that the test results are applicable to standards deployed in real-world systems, the tests are carried out on operational systems and sites around the United States where standardized systems are deployed.

The testing process

Engineering Review. The first step in the testing process is to perform an engineering review and analysis of the standards document. The review looks for characteristics such as ambiguities, unclear requirements, complex requirements that could be misinterpreted, optional requirements that could lead to different interpretations, and possible conflicts in requirements. Such features in a standard could lead to indeterminate results and, as a result, different implementations of the same requirements.

User Needs / Core Functions. The next step is to work with the users, vendors, standards development organizations and other to ensure the the user's needs are clearly understood and embodied by the standard. Because exhaustive testing of every feature in the standard, including optional features, would be far too expensive and time consuming, only the most important features are tested. In the past, these "core functions" were defined by the test team and used as a measure of a standard's completeness. More recently, user needs (requirements) are being included as part of the standard itself. Either of these approaches are then used to focus analysis and testing on those features essential for the hardware and software to carry out the primary application for which the standard is intended.

Experiences of Users and Vendors. A set of questionnaires is produced and interviews are held with users and the vendors. The questionnaires build upon the engineering review. The intent is to encourage the users and vendors to describe any problems or concerns they had in developing, installing, and operating the systems that incorporate the standard.

Other Test Results. Any other tests that the standard and its implementations have already undergone are also subject to an engineering review and analysis. It may be that the results of these tests, such as manufacturers' conformance tests or procurement acceptance tests, can are sufficient and little or no further testing is required. However, additional testing is typically required and a formal engineering test plan must be produced.

Technical Test Plans. The technical (or engineering) test plan, which is based upon the core functions, the information brought out by the engineering analysis, the interview responses, and the review of other test results, lays out in detail the technical test procedures, processes and requirements that the standardized system must meet. Any suspected inconsistencies in the standard or suspect features that were determined by the interviews are also tested using engineering procedures to determine if they have caused a problem.

Technical Field Tests. The engineering field test procedure is to present an operating system with an input condition and observe whether the resulting output condition is the expected one according to the standard. The observation must be interpreted to separate the behavior of the specific hardware and software used to implement the standard from the behavior of the standard itself. As in any measurement or test procedure, the uncertainties associated with the observation of the output must be understood and the conditions under which the test was made must be thoroughly documented.

For more information

From initial interviews to the formal engineering field-test results, an extensive amount of data are collected and analyzed to draw conclusions about the correctness, completeness and quality of the tested standard. The results are issued in a formal test report. Examples of test reports can be found at: www.standards.its.dot.gov/deploy_stdsTesting.asp.

For more information on ITS standards, contact the Federal Highway Administration, ITS Joint Program Office, HOIT, 1200 New Jersey Avenue, SE, Washington, DC 20590. Phone: 202-366-8048, Fax: 202-493-2027, Web: www.standards.its.dot.gov

Produced by the Jet Propulsion Laboratory for the U.S. Department of Transportation