ITS Standards Testing Fact Sheets

Fact Sheet #1 - June 12, 2001

How do we know that the standards work?

The challenge:  Most of us agree that open ITS standards and a competitive market for standardized ITS systems can produce lasting value. However, once the standards are developed, how do we know that they will work to meet everyone's needs?

Those of us who have participated in developing the standards deserve to know that the standards do what they are supposed to do — and that they do it well. Those of us who intend to deploy standardized systems want to feel confident that they will work the way we expect them to. Those of us who build equipment or systems expect that they will work properly when they are designed and built according to the standards.

The knowledge that the standards are complete and that they enable ITS systems to do what they are supposed to do is important to us, and to future users, because the deployment of standardized systems entails effort, time, and up-front expenses. So, the question remains: How do we, as standards developers, manufacturers, systems integrators, project deployers, and other interested ITS stakeholders know that the ITS standards really work?

The answer to the question is: "By testing the standards." But, what does "testing" standards mean, who tests them and how do they test them? The following paragraphs address these questions.

The purpose of testing

Testing a standard is a continuous process that begins early in the development phase of a standard's lifecycle and does not end until systems built in conformance with the standard are examined under actual operating conditions in the field. In effect, testing is the cumulative experience gained during the lifecycle of a standard, from development, through deployment, to maintenance and even updating or replacing the standard with a newer one.

Some testing is integral to standards development and deployment, while some testing can be a separate, formal event. The purpose of testing is always to ensure that the standard does what it is supposed to do properly and that systems that correctly incorporate the standard can be built economically and will perform as desired.

The benefits of testing

Testing the standards will provide information to potential users on the reliability, interoperability, functionality, and performance of systems use the standards. The ITS standards testing efforts will "prove" the standards in actual transportation settings and give current and future users of the standards the information they need. For example, the standards testing program will provide needed information that will enable:

• Standards developers to improve the standards (if necessary);
• Systems developers and integrators to make business decisions about designing and building standardized ITS equipment; and

• Future deployers to know whether and how to specify standardized systems that will meet their needs.

Who tests the standards and how they are tested?

All interested stakeholders test the standards. They participate in different kinds of standards testing to ensure that the standards meet their needs.

Since the standards development process is open to any interested party, future users, vendors, systems integrators, and public agency professionals can take part in the work of the standards development committees. During development, they continually "test" the standard by ensuring that it meets all the requirements that were established before they even began to create the standard. This kind of standards testing is called "validation testing."

The standard is "tested" again, sometimes even before it is completed, by participating vendors and users, who examine the standard to judge whether it is practical and economical to build systems that conform to the standard. This testing may be simply a detailed examination of the standards document, or it can be an actual attempt to make software or hardware that embodies the standard. Such testing is referred to as "verification testing."

Once equipment and systems that properly incorporate the standards are produced and deployed, we can gain practical, real-world experience using the standards. The practical experience gained by deploying standardized systems is the best way to determine how well they work in the field. In this sense, early deployers are standards testers, as are the manufacturers who produce the standardized equipment and the systems integrators who set up the systems. Such "testing" can be considered "experience-based testing."

Who reports the testing results?

To complete the standards testing process, someone needs to document the experiences of manufacturers, systems integrators and other early standards deployers, analyze them, and make them available to the ITS community. In addition, someone needs to examine, in a formal, technically sound engineering sense, the behavior of standardized systems to assure that the standards enable the systems to perform as desired, without any customization. Finally, someone needs to provide a comprehensive report on all of the testing results.

To best meet all of these purposes and the different needs of stakeholders, The U.S. DOT has decided that an independent testing entity will serve best to support and accelerate the deployment of ITS systems based upon the standards. The U.S. DOT has chosen Battelle, a nationally known non-profit research organization, to carry out the comprehensive ITS Standards Testing Program that will fulfill the standards testing process.

Battelle engineers will visit "test sites" where standardized equipment is deployed. They will interview manufacturers, systems integrators and project engineers. They will conduct specialized engineering tests on the systems to observe that the electronic information is correctly performing according to the specifications in the standards. Because exhaustive testing of every feature in the standard, including optional features, would be far too expensive and time consuming, they will concentrate on the most important features. They will then issue testing reports on selected standards. In effect, they will gather the experiences of deployers and, with the addition of specialized technical tests on the systems under actual operating conditions, they will report on how well the standards work. If necessary, they will recommend improvements to the standards.

For more information

Information on other types of standards testing and related subjects that are outside the scope of the ITS Standards Program can be found on the Technology Services Web site of the National Institute of Standards and Technology (NIST) at www.nist.gov. Such subjects include conformity assessment (have vendors implemented the standard correctly?), acceptance testing (does the system meet the procurement specifications?), and certification (has someone certified that the product conforms to the standard?).

For more information on ITS standards, contact the Federal Highway Administration, ITS Joint Program Office, HOIT, 1200 New Jersey Avenue, SE, Washington, DC 20590. Phone: 202-366-8048, Fax: 202-493-2027, Web: www.standards.its.dot.gov

Produced by the Jet Propulsion Laboratory for the U.S. Department of Transportation