“This is Not a Test”

Walter Tomczykowski & Cheryl Tulkoff

Download PDF


For a program manager, resolving product acceptance or post-deployment problems is an unwelcome item on the daily to-do list. Depending on the industry (e.g., aerospace, defense, medical, telecom, transportation), potential issues include schedule delays, cost impacts, customer complaints, mission impacts, or even loss of life.

How can program managers keep such unwelcome reliability issues off their to-do list? Today, end users such as airline operators, military program managers, or telecom providers, expect a high degree of due diligence from original equipment manufacturers (OEMs) that they are designing in reliability. OEMs may accomplish this task through their own proprietary engineering practices, contractual obligations, or industry standard requirements. Excessive testing of products after they are prototyped or built is not the answer. Reliability improvement and growth tests and the classic “test, analyze, and fix” concepts were commonplace in the eighties and early nineties. These methods did improve reliability; however, they also added cost and time. The ultimate process goal is to shift the focus during engineering and manufacturing development (E&MD) from „pass test‟ to „good design‟ prior to completion of the technology development phase (Figure 1).

fig 1-11

Recognizing that full operational testing or use in a field environment ultimately proves that products operate as specified, could working in a collaborative systems engineering environment verify that reliability has been designed into the product? Could this collaborative environment that focuses on Design for Reliability (DfR) also reduce life cycle cost and minimize schedule delays?


From recalls in the automotive or medical device industry to poor weapon system reliability1, there is increased emphasis to ensure that reliability is designed into products. Discovering failures after products are designed causes schedule delays and increases life cycle cost. The foundation of a reliable product is a robust design. A robust design provides margin, mitigates risk from defects, and satisfies customers. Assessing and ensuring reliability during the design phase maximizes the return on investment (ROI) or profit for OEMs and reduces the total ownership or life cycle cost for end users. The cost comparisons for defects caught during different phases of a product life cycle are illustrated by the following points:

  • Caught during design (start of technology development): lowest cost – least disruptive impact
  • Caught during engineering and test (E&MD): higher cost than catching during design
  • Caught during production: significantly higher cost than catching during design
  • Caught at the customer - Operations and Support (O&S): highest cost – most disruptive impact

With cost multipliers ranging from 10x during a development test to 1000x once operating, the goal for both OEMs and end users is to reduce risks of discovering reliability issues after a product is prototyped, built, or delivered.

Although O&S costs are not incurred until after a product is produced and deployed, many of the major design decisions that ultimately determine these costs are made early in the life cycle, i.e., during development. This is illustrated in Figure 2.

fig 2-10


Design for Reliability (DfR): A process for ensuring the reliability of a product or system during the design stage before physical prototype.

Reliability is the measure of a product’s ability to:

  • perform the specified function
  • at the customer (with their use environment)
  • over the desired lifetime

Physics of Failure (PoF): The use of science (physics, chemistry, etc.) to capture an understanding of failure mechanisms and evaluate useful life under actual operating conditions.

Collaborative Systems Engineering

OEM‟s of electronic products today typically own their designs. They are responsible for the performance and reliability. They have to ensure that their products will perform as specified over a required period of time and when used under normal or expected operating conditions. End users acquire or purchase the products and they expect them to operate as specified over a required period of time.

Both OEM‟s and end users program managers recognize that discovering failures after products are designed and especially once operating in the field will cause an impact. In addition to catastrophic or critical impacts2, Table 1 provides some potential impacts for various industries.

table 1-1

One possible solution to reduce risk and uncertainty is to use a collaborative design environment that incorporates unbiased modeling and simulation or automated design analysis (ADA), and design reviews. This collaborative approach facilitates products that are designed for reliability (DfR).

ADA which incorporates Physics of Failure (PoF) algorithms can be used by both the OEMs to design in reliability and by the end users to verify their designs. The OEMs will increase their return on investments (ROI) and profits while end users will lower their total lifecycle costs3. Why? Because with ADA, designs can be improved before building prototypes, test time can be reduced, and overall schedule risks can be minimized. These and other benefits are summarized in Table 2.

table 2-2

The following activities describe how ADA combined with design reviews provide reliability benefits to OEMs and end users:

  • ADA modeling and simulation software provides an Independent assessment of the design before the product is built.
  • Physics of Failure algorithms are incorporated into the ADA allowing both designers and reliability engineers to determine the expected life of the design.
  • Characterization of the electronic circuit board behaviors are determined before the product is tested.
  • ADA software annotates potential problem areas within the bill of materials to help prioritize what to focus on first.
  • Reduce uncertainty and risk. For example, determine the impact of switching from Pbbased to Pb-free solder in electronic products used in the aerospace, defense, medical, transportation, or telecom industries.
  • Data from ADA can be used to assess program risks and the collaborative environment can also provide recommendations during design reviews.
  • Design can be verified (from a reliability perspective) before acceptance test or product delivery.
  • The results from ADA and the design review(s) can then determine how much testing is needed withan added goal of removing unnecessary testing.
  • Estimating the useful life under the expected operating conditions provides OEMs with better warranty cost estimates and provide lower Life Cycle Cost (LCC) to end users.


DfR Solutions believes that simple, but authoritative ADA in combination with design reviews attended by both OEMs and end users is the key to success. This combination results in more reliable products, delivered on time with lower life cycle cost than approaches with excessive focus on testing in reliability. Our Sherlock software is designed to fill this need and does so by allowing a rapid assessment of electronic systems reliability utilizing Physics of Failure (PoF). Sherlock is a reliability tool that can be used by the entire engineering design and management organization. It allows the reliability group to get involved in the design process as well, as they now can better quantify tradeoffs before the product is built.

Sherlock is the future of Automated Design Analysis (ADA): the integration of design rules, best practices and a return to a physics-based understanding of product reliability. DfR is not a test to ensure reliability; it has to be designed into products using proven physics of failure knowledge.


DfR represents that a reasonable effort has been made to ensure the accuracy and reliability of the information within this report. However, DfR Solutions makes no warranty, both express and implied, concerning the content of this report, including, but not limited to the existence of any latent or patent defects, merchantability, and/or fitness for a particular use. DfR will not be liable for loss of use, revenue, profit, or any special, incidental, or consequential damages arising out of, connected with, or resulting from, the information presented within this report.


  1. Memorandum for Principal Deputy Under Secretary of Defense (Acquisition, Technology and Logistics), June 30, 2010. Downloaded on March 8, 2011 from Defense Acquisition Portal – https://dap.dau.mil (Policy)
  2. Catastropfig 1hic: A failure which may cause death or weapon system loss (i.e., aircraft, tank, missile, ship, etc...) Critical: A failure which may cause severe injury, major property damage, or major system damage which will result in mission loss. Definitions from MIL-STD-1629 (note: MIL-STD-1629 was cancelled on March 2, 2010 however the definitions are valid for this paper)
  3. The author recognizes that for defense programs, the end user (i.e. the DoD) may also be providing the funding for development, unless they are acquiring non-developmental items (i.e., Commercial off the Shelf).