testing guidelines for new product development
TRANSCRIPT
Brigham Young University Brigham Young University
BYU ScholarsArchive BYU ScholarsArchive
Theses and Dissertations
2010-06-04
Testing Guidelines for New Product Development Testing Guidelines for New Product Development
Derek W. Egbert Brigham Young University - Provo
Follow this and additional works at: https://scholarsarchive.byu.edu/etd
Part of the Manufacturing Commons
BYU ScholarsArchive Citation BYU ScholarsArchive Citation Egbert, Derek W., "Testing Guidelines for New Product Development" (2010). Theses and Dissertations. 2529. https://scholarsarchive.byu.edu/etd/2529
This Thesis is brought to you for free and open access by BYU ScholarsArchive. It has been accepted for inclusion in Theses and Dissertations by an authorized administrator of BYU ScholarsArchive. For more information, please contact [email protected], [email protected].
Testing Guidelines for New Product Development
Derek Egbert
A thesis submitted to the faculty of Brigham Young University
in partial fulfillment of the requirements for the degree of
Master of Science
A. Brent Strong, Chair Michael Miles Charles Harrell
School of Technology
Brigham Young University
August 2010
Copyright © 2010 Derek Egbert
All Rights Reserved
i
ABSTRACT
Testing Guidelines for New Product Development
Derek Egbert
School of Technology
Master of Science
While many literary sources outline the product development process, few make mention
of the prototyping and testing stage. This thesis suggests that because of its importance in the product development process, “Testing” should be documented as a major step and not just listed as a side note. As part of the testing step, it is suggested that standardized, in-use, and market tests be used to properly evaluate a product. While many rely solely on standardized tests to validate their products, effective in-use tests can be another vital tool that can prove the performance of the product in more specific and relevant applications. In-use tests are a major focus in this thesis and the process of developing and using these in-use tests is explored.
A case study is used to prove that effective product development will follow the outlined
testing procedure. Also, it shows that in-use testing, combined with other types of testing, can be a vital tool to ensuring the successful launch of a newly developed product. As a result of the case study, the traditional new product development process is amended and a set of guidelines are proposed for use in constructing a successful testing methodology for the new product development process. Keywords: Product Development, standardized testing, in-use testing, prototyping
ii
ACKNOWLEDGMENTS
This opportunity to reach for higher education would not have happened without the love
and support of many people, including teachers, family, and friends. My parents and
grandparents have been very supportive of me through these college years and I thank them for
all of the financial and emotional support they have given me to help me through. My professors
at BYU have instilled in me a love for manufacturing and to them I am deeply indebted. I thank
Dr. Strong for giving me the chance to work in the labs as a TA and for taking me under his wing
as my committee chair. I am also very grateful for donors to the School of Technology and the
tithe-paying members of The Church of Jesus Christ of Latter-day Saints because a portion of
their tithes goes to funding this wonderful university and, without this funding, my education
would not have been so financially bearable. Lastly, I thank my wonderful wife for all of her
love and support and for giving me the motivation I need to push through and complete this
incredibly daunting task of writing this thesis and getting my Masters degree from such a great
university.
iii
TABLE OF CONTENTS
1 Introduction............................................................................................................................. 1
1.1 Background........................................................................................................................ 1
1.2 Case Study ......................................................................................................................... 7
1.2.1 Hazard Protection Systems, Inc. & NCASE............................................................... 7
1.3 Problem Statement............................................................................................................. 8
1.4 Objective............................................................................................................................ 9
1.5 Thesis Statement .............................................................................................................. 10
1.6 Case Study Null Hypothesis ............................................................................................ 10
1.7 Methodology.................................................................................................................... 10
1.8 Scope and Delimitations .................................................................................................. 11
1.9 Glossary of Terms............................................................................................................ 11
2 Review of Literature ............................................................................................................. 13
2.1 Review of Product Development Process........................................................................ 13
2.1.1 Step 1: Opportunity Selection ................................................................................... 13
2.1.2 Step 2: Concept Generation ...................................................................................... 14
2.1.3 Step 3: Concept Evaluation/Selection....................................................................... 15
2.1.4 Development ............................................................................................................. 16
2.1.4.1 Prototype Development ............................................................................................ 17
2.1.4.2 Benefits of Prototypes............................................................................................... 18
2.1.5 Launch....................................................................................................................... 19
2.1.6 Testing—Crawford & Di Benedetto’s Oversight ..................................................... 20
2.2 Testing ............................................................................................................................. 20
2.2.1 Purposes of Testing................................................................................................... 21
iv
2.2.2 The Testing Process .................................................................................................. 22
2.3 Developing Acceptance Criteria for Testing ................................................................... 24
2.3.1 Acceptance Criteria for a Specific Customer (Customer-Preference Driven).................................................................................. 25
2.3.2 Acceptance Criteria for a General Market (Specification-Driven) .............................................................................................. 25
2.3.2.1 Criteria Based on Perceived Customer Needs .......................................................... 25
2.3.2.2 Criteria Based on Competitor’s Products ................................................................. 26
2.3.2.3 Case Study: Hewlett-Packard ................................................................................... 26
2.3.2.4 Case Study: The Chemical Industry ......................................................................... 27
2.3.3 Acceptance Criteria for a Specific Customer (Customer Preference-Driven).................................................................................. 27
2.3.4 House of Quality ....................................................................................................... 28
2.4 Review of Standardized Testing ...................................................................................... 29
2.4.1 History of Standards ................................................................................................. 29
2.4.2 How Standards are Developed.................................................................................. 30
2.4.3 Recipe for a Standardized Test ................................................................................. 31
2.4.4 Standardized Testing Agencies................................................................................. 32
2.4.4.1 American Society for Testing and Materials (ASTM) ............................................. 32
2.4.4.2 America National Standards Institute (ANSI) .......................................................... 33
2.4.4.3 International Standards Organization (ISO) ............................................................. 33
2.4.4.4 German Institute for Standardization (DIN) ............................................................. 34
2.4.4.5 Japanese Standards Association (JAS) ..................................................................... 34
2.4.4.6 Consumer’s Union (CU) ........................................................................................... 34
2.5 Review of In-Use Testing ................................................................................................ 35
2.5.1 In-Use Testing and Design of Experiments .............................................................. 36
v
2.5.1.1 Step 1: Identify Critical Customer Needs ................................................................. 36
2.5.1.2 Step 2: Formulate Testing Procedures ...................................................................... 37
2.5.1.3 Step 3: Perform Testing ............................................................................................ 38
2.5.1.4 Step 4: Analyze Data ................................................................................................ 39
2.5.1.5 Step 5: Evaluate Results and Draw Conclusions ...................................................... 39
2.6 Market Tests .................................................................................................................... 39
2.6.1 Market Testing Case Study—The Gainesburger Dog Food Failure......................... 40
2.6.2 Market Testing Case Study—The Hubble Space Telescope Blunder ...................... 41
2.7 In-Use Testing Case Study—Digital Photography Review............................................. 41
2.7.1 DPReview Lens Review Procedure .......................................................................... 42
2.8 In-Use Testing Case Study—Garbage Can Testing ........................................................ 47
2.9 NCASE Case Study ......................................................................................................... 48
2.9.1 Development of NCASE........................................................................................... 48
2.9.2 Environmental and Operating Conditions ................................................................ 51
2.9.2.1 Sunlight exposure...................................................................................................... 52
2.9.2.2 Fuel Exposure ........................................................................................................... 53
2.9.2.3 Cold Weather and Crinkle Exposure ........................................................................ 53
2.9.2.4 High-Speed Impacts.................................................................................................. 54
2.9.3 Review of NCASE Materials.................................................................................... 55
2.9.3.1 Kevlar Fabric ............................................................................................................ 55
3 Methodology .......................................................................................................................... 57
3.1 NCASE Testing ............................................................................................................... 57
3.1.1 Environmental & Operating Conditions ................................................................... 58
3.1.2 Sample Preparation ................................................................................................... 58
3.1.2.1 Weathering ................................................................................................................ 59
vi
3.1.2.2 Fuel Exposure ........................................................................................................... 62
3.1.2.3 Water Exposure......................................................................................................... 63
3.1.2.4 Freezer and Crinkle Testing...................................................................................... 64
3.1.2.5 Webbing Strength ..................................................................................................... 67
3.1.2.6 Tensile Strength Testing ........................................................................................... 69
3.1.2.7 Impact Testing .......................................................................................................... 69
3.1.3 Acceptance Criteria................................................................................................... 74
3.1.4 Data Analysis ............................................................................................................ 74
4 Results of NCASE Testing ................................................................................................... 75
4.1 Liquid Exposure............................................................................................................... 75
4.1.1 Results....................................................................................................................... 75
4.1.2 Analysis..................................................................................................................... 78
4.2 Ultra Violet Exposure ...................................................................................................... 79
4.2.1 Results....................................................................................................................... 79
4.2.2 Analysis..................................................................................................................... 80
4.3 Freeze/Crinkle Exposure.................................................................................................. 81
4.3.1 Results....................................................................................................................... 81
4.3.2 Analysis..................................................................................................................... 82
4.4 Webbing Strength ............................................................................................................ 83
4.4.1 Results....................................................................................................................... 83
4.4.2 Analysis..................................................................................................................... 83
4.5 High Speed Impact Testing.............................................................................................. 84
4.5.1 Results....................................................................................................................... 84
4.5.2 Analysis..................................................................................................................... 85
5 Conclusion ............................................................................................................................. 86
vii
5.1 Case Study Review: NCASE........................................................................................... 86
5.2 New Outline for Products Development Process ............................................................ 87
5.3 Testing Progression.......................................................................................................... 88
5.4 Guidelines for Product Testing ........................................................................................ 88
5.5 Review of Hypothesis...................................................................................................... 90
5.6 Suggestions for Further Research .................................................................................... 90
References.................................................................................................................................... 92
viii
LIST OF TABLES Table 1—Statistical Significance of Liquid Exposures .........................................................77
Table 2—Statistical Significant of Liquid Exposures ...........................................................77
Table 3—Statistical Significance of UV Weathering............................................................80
Table 4—Statistical Significance of Freeze/Crinkle Testing.................................................82
Table 5—Webbing Test Results ............................................................................................83
Table 6—Old Kevlar High Speed Impact Testing.................................................................84
Table 7—New Kevlar High Speed Impact Testing ...............................................................84
Table 8—New Kevlar Endurance Impact Testing.................................................................84
ix
LIST OF FIGURES Figure 1-1—EPA Fuel Economy Window Sticker (EPA, 2010) ..........................................4
Figure 2-1—Testing Process as Outlined by Thompke (Thompke, 2008)............................22
Figure 2-2—Progression of Tests in the New Product Development Process ......................23
Figure 2-3—Specification vs. Customer-Driven Process (Reintertsen, 1997) ......................24
Figure 2-4—House of Quality Matrix (Temponi, 1999) .......................................................29
Figure 2-5—Comparison shots of different lenses (Westlake, 2008)....................................43
Figure 2-6—Macro Focus Test Chart for Lens Reviews (Westlake, 2008) ..........................44
Figure 2-7—Shots Depicting Zoom Power of Lens (Westlake, 2008)..................................45
Figure 2-8—NCASE Product Installed on Fuel Tank ...........................................................49
Figure 2-9—Sequence of Fire Testing a Bare Fuel Tank ......................................................50
Figure 2-10—Sequence of Fire Testing a Fuel Tank with NCASE Installed........................51
Figure 2-11—Average Weather Patterns in Iraq (iexplore, 2010) ........................................52
Figure 2-12—Average Weather Patterns in Afghanistan (iexplore, 2010) ...........................54
Figure 2-13—Molecular Structure of Kevlar (absoluteastronomy.com, 2010).....................56
Figure 3-1—Kevlar sample....................................................................................................59
Figure 3-2—Weatherometer in BYU Lab .............................................................................61
Figure 3-3—Sample Being Soaked in Diesel Fuel for Solvent Test .....................................63
Figure 3-4—Preparation of Crinkle Sample ..........................................................................65
Figure 3-5—Preparation of Crinkle Sample ..........................................................................65
Figure 3-6—Placement of Crinkled Samples on Plate ..........................................................66
Figure 3-7—Placement of Weight on Crinkle Samples ........................................................67
Figure 3-8—Tensile Testing a Webbing Sample...................................................................68
Figure 3-9—Impact Testing Setup at Clearplex ....................................................................70
Figure 3-10—Impact Testing Speed Chart ............................................................................71
x
Figure 3-11—Preparing Shot for Impact Test .......................................................................71
Figure 3-12—Impact Testing Projectiles (penny shown for size comparison) .....................72
Figure 3-13—Inspection of Internal Powder Baggie After Impact Testing ..........................73
Figure 4-1—Nylon Solvent Exposure ...................................................................................75
Figure 4-2—Old Kevlar Solvent Exposure............................................................................76
Figure 4-3—Kevlar Liquid Tests...........................................................................................76
Figure 4-4—Nylon UV Weathering ......................................................................................79
Figure 4-5—Old Kevlar UV Weathering ..............................................................................79
Figure 4-6—New Kevlar Freeze and Crinkle Tests ..............................................................81
Figure 4-7—Failure Points on Crinkle Samples (shown in red circles) ................................81
Figure 5-1—Proposed New Products Development Process.................................................87
Figure 5-2—Progression of Testing.......................................................................................88
1
1 INTRODUCTION
1.1 Background
Countless people throughout the world have risen from rags to riches because of their
ability to turn an idea of a product into something very real, useful, and marketable. The United
States is filled with many entrepreneurs using revolutionary product ideas to fuel their pursuit of
the American dream. Whether or not these inventors have been instructed in the traditional steps
of product development, their paths generally follow the same logical process of creating an idea
for a new product, building and testing that product, and setting up manufacturing and releasing
the product to the public. Much of the success of these entrepreneurs lies in their ability to
effectively evaluate whether or not the most recent incarnation of their product idea is suitable
for use by their intended customers. Inventors often have wonderful ideas of a product that can
definitely fill a gap in the lives of their targeted customers, but when they forgo testing to
validate their final production design, they usually meet the swift hand of failure because their
product does not hold up to the rigors of its intended use. Effective testing of products in their
anticipated environment is vital to ensure the product’s successful launch and sustainability.
There are a few different types of testing that happen throughout the development
process. Concept testing is a type of testing that is done early on in the design phase and is
important to ensure that the basic design of the product will actually work and that it fulfills the
outlined customer needs. In this type of testing, it is not critical to have a product in a final or
2
even a near-final state. For example, when designers explore options for dramatically changing
the shape of a new mobile phone, they may just want to test how it feels when holding it up to
their head; in such a test, they would most likely make a non-working model in order to simply
show the concept and allow designers to feel what the final product might be like. It would not
be essential to have all working components of a phone installed at this point in the process
because the focus would be on the shape rather than electronic functions. While concept testing
is an important part of the product development process, there is another very important
developmental process called prototype or product testing that can be done either to assist the
development process or to help validate the product after launch. This thesis will address some
challenges inherent in the prototype or product testing process.
Product testing or full-prototype testing is where a final-stage prototype or actual product
is tested to ensure that it functions and performs as desired by the person or company selling the
product. In this type of testing it is crucial to use the same exact design or materials to be used in
the actual product that will ultimately end up in the hands of the customers or, at a minimum, use
a design and materials that are so similar to the final product that they could be comparable. For
many products, customers like to see proof that the product will perform exceptionally under
similar conditions in which they intend to use them. It would be foolish to try to sell a product
without, first, verifying that it will actually work and perform in a variety of environments as
intended by the designers. This kind of validation or verification is often done by using pre-
established standardized tests.
Standardized tests are created or maintained by a governing body or oversight group that
provides information and instructions on the exact procedures for different tests. The purpose of
standardized tests is to have a base from which to compare different products throughout
3
different locations and time periods. Examples of standardized tests surround us every single
day and affect nearly all of the products we purchase and use. A good example of a common
standardized test is the Environmental Protection Agency (EPA) fuel estimates that are displayed
on all new cars (see Figure 1-1). These fuel economy (also known as gas mileage) numbers are
determined through rigorous standard tests that are performed consistently with every vehicle as
to give a solid base for comparison when people are shopping for cars. All cars are driven on a
track at the same speeds in the same conditions and fuel intake is monitored. Fuel economy
results are then calculated and posted on the windows of all new cars. People that commute or
otherwise spend a lot of time driving are usually concerned about spending too much on fuel;
these customers can compare the fuel economy estimates between different cars and have
confidence that the posted results were attained through a consistent and repeatable process.
Having confidence in these tests is important, which is why the EPA dedicates great resources to
maintain a high quality and consistent testing process. Without a set standard for testing fuel
economy, customers would be at an extreme disadvantage because they would have no solid
base from which to compare the performance of various vehicles.
4
Figure 1-1—EPA Fuel Economy Window Sticker (EPA, 2010)
While EPA fuel estimates are very useful to many car buyers, there are some limitations
to the tests. City economy and highway economy are the two metrics reported from the tests
performed, but there may be some areas where a more specific test would be appropriate. For
example, the testing procedures used by the EPA would probably not reflect the performance of
an automobile in a high-altitude area where there are many steep and windy mountain roads.
Automobiles that perform well on highway economy tests may use a greater amount of fuel in
windy and hilly conditions, making the EPA estimates irrelevant to some buyers. Customers in
such areas may desire to have a more suitable comparison of fuel economy under conditions
similar to their specific environment. Dealerships in that area could benefit from doing their own
specified testing to bring more useful information to their specific customers. This kind of
testing can include driving a pre-determined mountainous route and then measuring fuel
consumption from that route in order to get a mountain-driving mileage estimate. While this
would not be a standardized test, it could provide important and useful information to local
5
customers. Although standardized tests are very important and have many wide uses, it is
sometimes necessary in certain situations to develop specific tests that better fit the intended use
and environment of the product.
While standardized tests, in general, are appropriate for a majority of applications, it is
often beneficial to make modifications to already established standard tests, or even develop
completely new tests in order to properly evaluate a product for its intended uses and
environment. If a very specific aspect or function of a product needs to be tested, there is a
limited set of standardized tests from which to choose for testing. If no pre-established
standardized test comes close to addressing the desired performance characteristics of a specific
product, it can be advantageous to develop an in-use test to appropriately evaluate the product or
materials for the specified environment or use. An in-use test is usually a very specific test that
is set up to precisely measure the desired attributes of the product in a way that will give
confidence to the developer that the product will perform as desired. Customers also like to have
proof that the things they might purchase will perform well throughout the intended product life,
so it is important to develop pertinent and reliable tests in order to maintain significant sales of a
new product.
Sometimes, it can be very useful to combine standardized and in-use testing in evaluating
a product. Following is an example of an application where both standard tests and in-use tests
can be used together. In the motorcycle industry, there are stringent DOT and SNELL
standardized tests that a helmet must pass to be considered safe. While safety standards are the
most important consideration when buying a helmet, often motorcycle riders want to make sure
other advertised features of the helmet really work. In areas where the climate is very hot, it can
be very uncomfortable wearing a helmet if the vents on the helmet are not well designed. Since
6
there are no current standardized tests available to show the venting and cooling effectiveness of
helmets, how can a rider make sure that a helmet will be vented properly without actually buying
and trying it out? If this is a major concern for enough customers, a helmet manufacturer or
distributor can take the initiative and develop an in-use test to provide information on the
effectiveness of its helmets by developing a cooling test to measure temperatures inside a helmet
under certain wind speeds and temperatures. These tests can then be used to compare the
effectiveness of one helmet model versus another as to provide an apples-to-apples comparison.
In-use tests such as this can be a critical factor in the successful rollout and lifelong success of
many different products and can be combined with the results of standardized tests to give the
customer enough information for them to make informed purchasing decisions.
After a product has gone through standard and in-use tests to validate its performance in a
laboratory setting, it is critical to perform limited market tests. By running market tests, a
developer can see how the product will actually fare in a real-world situation with the real
customers. Even if a product performs perfectly in lab and other controlled tests, the customers
may not use the product with complete adherence to the recommended instructions and
sometimes the results can be catastrophic. Organizations that do limited market tests before a
full launch of their product often receive valuable information they could not have otherwise
gathered.
In summary, as a product developer moves from a characterization of general materials to
the verification of an actual material and design, it is necessary to move from standard tests to in-
use tests because in-use tests use concepts of standardized tests but modify them so they have
direct and immediate application to the actual product. The process of developing an effective
testing methodology for a new product is investigated throughout this thesis.
7
1.2 Case Study
To demonstrate that an effective testing methodology can be created for evaluating a new
product, this thesis provides a case study that highlights a company that was in the early stages of
development of a new product and their collaboration with the author in developing and
executing a proper testing methodology for its product. By connecting with a company that was
trying to validate its design and material choice for a new product, the process of developing
some specialized tests for the product in order to fully explore the nature of in-use testing and
whether or not it can be used as an effective tool in the new product development process could
be developed and evaluated. Hazard Protection Systems was going through the new product
development process at the time of this thesis and had connections to professors at BYU. This
provided an excellent case study for the general investigation of in-use testing.
1.2.1 Hazard Protection Systems, Inc. & NCASE
Hazard Protection Systems, Inc. (HPS) is a company that, over the last few years, has
gone through the development process for a new product by creating a new ballistic fire
protection product (NCASE). As of May 6, 2010, over 5,000 soldiers have died and over
20,000 have been wounded in action in Operation Iraqi Freedom and Operation Enduring
Freedom, otherwise known as the wars in Iraq and Afghanistan (Defense, 2010). Many of these
fatalities and horrific injuries have come when transport vehicles drive over or next to
improvised explosive devices (IED’s) that are detonated. These hidden explosive devices come
in so many shapes and forms that it is extremely difficult to identify and disarm all of them.
Since many military vehicles have external fuel tanks on the under-carriage, they are vulnerable
to a flare-up during an IED detonation because the tank can be pierced from shrapnel or gunfire.
8
In these instances, the fuel is expelled (often up to 100 gallons in a tank) and can ignite; thus
adding to the awful damage of the blast. The reduction or elimination of this fuel flare up can
save lives and major injury on the battlefield and has been addressed by HPS with the invention
of the NCASE product.
NCASE is a passive fire mitigation system that releases fire-extinguishing agent (in
powder form) when impacted. These agents absorb heat in a direct application method that
suppresses the fire in less than a second. The current version of the NCASE product contains
sealed plastic pouches of fire extinguishing powders that are sown into a Kevlar® blanket that is
then strapped around a fuel tank. Kevlar was chosen as the material for this product because of
its general strength and versatility; it is used in many military applications where strength and
durability are important product characteristics. Although Kevlar is used in many military
applications, HPS wanted to validate their product design and ensure that the Kevlar material
used on their product would stand up to the rigors of its intended environment.
In order to determine the properties of the NCASE materials in the proposed
environments, HPS connected with Brigham Young University to conduct this testing. This
product was chosen as a case study not only to determine if the materials they selected are
suitable for use in the intended harsh environments, but also to determine whether or not
modified or in-use tests can be effectively developed and utilized in the new product
development process.
1.3 Problem Statement
Testing should always play an important role in the development of new products and can
be done in various ways. In standardized testing, all of the tests are outlined with very specific
9
details with no question as to how the test is to be performed. However, some tests need to be
modified in order to gauge performance in specific environments that are different from those
allowed for standard tests. The question then arises, “How can performance in specific
conditions be reliably measured without standardized tests?”
In-use testing can be much more applicable in determining performance characteristics
of a product or its components but it is often more difficult to perform in a repeatable and
meaningful way that can be trusted by customers. Difficulties arise with in-use testing because
there are so many unknowns that must be carefully considered and addressed in the course of the
testing process. Many product developers have a hard time deciding what tests to use in order to
prove to themselves and to their customers that the products are suitable for use. Once specific
product characteristics are identified and selected for further testing, it is challenging to go
through the process of designing a testing methodology that would yield consistent and quality
results. Because in-use tests cannot generally be picked out of a book or looked up on the
internet, they require a greater deal of creativity than standardized testing alone. Developing in-
use testing can be incredibly challenging because all developed tests need to be repeatable,
accurate, realistic, doable, believable, and understandable.
1.4 Objective
The objective of this thesis is to develop a set of guidelines that can be used by
entrepreneurs and product developers to develop an effective testing methodology within the
new product development process. This thesis will also define procedures by which products
can be tested in circumstances when existing standardized tests are not suitable or applicable for
the intended product uses and/or environments. Through a case study, this thesis will document
10
the process by which the NCASE product and materials were tested and evaluated, including the
modification of existing standard tests and the development of completely new in-use tests for
the purpose of evaluating whether or not the product is suitable for use in the intended
environments. In addition to exploring the testing methodology, the integration of in-use tests is
greatly explored throughout the thesis.
1.5 Thesis Statement
Through the analysis of the product testing used in the case study, a set of guidelines can
be developed to assist in a generalized method for new product testing.
1.6 Case Study Null Hypothesis
Through the analysis of the product testing used in the case study, a set of guidelines
cannot be developed to assist in a generalized method for new product testing.
1.7 Methodology
After a thorough review of literature discussing the key points of product development,
testing methodologies, and some applications and guidelines for in-use testing, a special case
study was performed with Hazard Protection Systems’ NCASE product. HPS needed to test
their NCASE product to prove to themselves and to their customers that the product materials
would hold up to the extreme conditions experienced in Iraq and Afghanistan. After these
product uses and environments were discussed, tests were developed (either modified from
standard tests or they were developed from basic principles) for the specific purpose of analyzing
the properties of the NCASE materials. After considerable effort in creating scientific in-use
11
tests, the materials were then evaluated using these new procedures and the data was evaluated
and proclaimed suitable or unsuitable for use based on the acceptance criteria designed into the
tests. After testing and evaluation of the NCASE product and its materials, a list of suggestions
or guidelines was created in hopes to guide future product development efforts where
standardized tests do not adequately demonstrate performance characteristics of the intended
products. These guidelines were developed based on literary research, interviews with others
who have developed in-use tests, and personal experience with the development of the in-use
tests for the NCASE product.
1.8 Scope and Delimitations
Explosive field-testing will not be completed on the NCASE product. Testing of the
effectiveness of the internal powders will not be addressed. This application of standard and
modified tests will only be directly applied to the NCASE product.
1.9 Glossary of Terms
acceptance criteria—Conditions agreed upon by an organization and its customer to which
the product must meet in order for the purchase to ensue.
functional testing—Tests that are performed to evaluate the actual performance of a new
product or component of a product. These tests can be done with standard tests (if
available) or in-use tests.
in-use test—A laboratory test designed to simulate real world conditions in a controlled
environment. In-use tests are not regulated by a sanctioning body and are commonly
designed for a specific product application.
12
standardized test—A laboratory test of a product or material that has been documented and
accepted by a sanctioning body as a standard test. These tests have a wide appeal and
are fairly broad in their application.
13
2 REVIEW OF LITERATURE
2.1 Review of Product Development Process
Whether or not an inventor or product development firm has a specified product
development process, there is a common set of steps that are generally taken to bring an idea
from conception to a successful new product. Some outlined processes are very complex while
others are very simple. To maintain consistency in referring to the product development cycle,
this thesis will follow the steps in the order that Crawford and Di Benedetto have outlined in
“New Products Management” (Crawford 2008). These steps are not, necessarily, the complete or
the simplest, but they serve as a good starting point from which to consider the development
process.
2.1.1 Step 1: Opportunity Selection
The first step of the product development process is “strategic in nature and is the most
difficult to describe or define” (Crawford 2008). If an individual or firm decides to move
forward with development of a product that will have no value to the end customer, they will
ultimately fail no matter how unique their product may be. Identifying a currently unmet
customer need and then proceeding with developing a product to fill that need will yield the
greatest possibility for success. Some may simply come up with an idea on-the-fly for a new
product while others might have a structured approach to identify and then select customer needs
14
to drive their product development. While there are many different approaches to opportunity
selection, the important thing to understand is that no matter the approach, making sure there is a
legitimate need for a new product is the most important part of the development process. All
following steps would subsequently be in vain unless the product fits a real and marketable
customer need.
2.1.2 Step 2: Concept Generation
After a legitimate need has been identified, the next step in the process is to create ideas
of how to fulfill this need. Again, this is another process that can be done in many different
ways. IDEO, a California-based design company, was featured by ABC’s Nightline for a
segment that showcased their design process; their concept generation methodology shown in
that special has proven effective over time. After IDEO designers agree on customer needs, they
split up and have a limited amount of time to come up with as many designs as possible that are
aimed at satisfying those needs. One of the interesting aspects of IDEO’s process is that the
designers are highly encouraged to come up with pie-in-the-sky ideas; although such ideas may
not always be feasible for production or development, others may be able to take aspects of those
aggressive ideas and turn them into a very feasible and effective product designs (ABC, 1999).
This process of idea creation allows the designers to develop product ideas that would then be
shared and selected in the concept evaluation step described in the next section. By developing
as many product ideas as possible within a reasonable time period, there will be a greater
selection of ideas to pull from when all of the design ideas are on the table together.
In this stage of concept generation, there are many different ways to approach this
process of creating new ideas. If this process is done with the help of many different designers
and engineers (as compared to a lone-ranger type approach), it is important to appoint a leader
15
that can facilitate this idea creation process. The leader must “actuate this brainpower to
generate relevant options…he must provide a ‘safe place’; he must help the participants perceive
the elements of the problem, perhaps in a new light….The leader must also infuse electricity into
the meeting…and manage the group dynamics along the way” (Buggie, 1981).
2.1.3 Step 3: Concept Evaluation/Selection
The New Products Management textbook (Crawford, 2008) outlines a handful of
different evaluation tools that can be used in narrowing down ideas. These tools are typically
focused on quantifying how well the proposed design will fit customer needs. IDEO started its
evaluation process for the next-generation shopping cart (as seen in the abc documentary) by
bringing all of the designers together in a big room and having them place their idea sketches on
the walls around the room. Each person was given a set number of Post-it notes that they used to
identify their favorite designs. At the end of this process, the top designs with the most votes
were selected for concept testing by breaking the group into small teams to build working
prototypes in order to demonstrate the various designs.
Generally, after some designs have been selected in the design process, it is common to
build some basic functional prototypes (or concept models) in hopes of proving or disproving
their effectiveness. In IDEO’s example, the designers split up into groups on a few different
designs for a next-generation shopping cart. The groups built a mock-up of their designs and
they all brought them back together to evaluate whether the ideas would actually work in real
life. This step was highly important because it allowed everybody to physically experience the
ideas of their peers and allowed them to more effectively evaluate whether the ideas presented
were solid or not. As a result of this process, the whole group was able to come up with a final
design that incorporated individual aspects from each different prototype (ABC, 1999).
16
While IDEO’s prototypes were built to simply provide a physical incarnation of the
selected designs so everybody could see, touch, and use the product, it is possible for groups to
put their designs through a series of standardized or modified tests depending on customer
demands. For example, if a new product must withstand a specified force before breaking, it
would be important to build a prototype early in the development process that would have the
same strength characteristics as the final product so that the tests performed will be completely
applicable to the end product. The prototyping process can be used as an aesthetic tool to help
designers better visualize the product and it can also be used by engineers to make sure the
product will perform well in the intended uses and environments.
2.1.4 Development
It may be difficult to proceed with the development step because so many great ideas
have arisen in the concept evaluation and selection phase and the requirements of the product are
so varied. Crawford and Di Benedetto summed up the challenge: “Designers undergo rigorous
training to learn how to design products that function well mechanically, that are durable, that
are easy and safe to use, that can be made from easily available materials, and that look
appealing. Clearly, many of these requirements will be in conflict, and it is up to the skillful
designer to achieve all of them simultaneously” (Crawford, 2008). In the design process, it is
important to develop specific design goals or objectives, which include:
• Design for Speed to Market
• Design for Ease of Manufacture
• Design for Differentiation
• Design to Meet Customer Needs
• Design to Build or Support Corporate Identity
17
• Design for the Environment (Crawford 2008)
The development process is the step where a design is chosen based upon the previous
steps of concept generation and selection plus a synthesis and modification so that they all fit
into a single concept model. Product design is a tricky process because it must be done within
the “triangle of constraints: the end use of a product, the materials of which it is made, and the
tools and processes by which it is made” (Parsons, 1989).
2.1.4.1 Prototype Development
“Prototypes are a form of experiment in that the organization is generally testing some
aspect of the new product or of the process by which the product will be built” (Bean, 2000).
Although prototypes can be created and used in the concept phase of product development, this
later, formal prototype stage of development is highly focused on the end product rather than
concepts or ideas. After IDEO took all of their ideas and merged them together for a final
design, everybody got together to create a high quality prototype to demonstrate their finalized
product (ABC 1999). A high quality prototype at this level allows the designer to perform
critical evaluations of the final design. These prototypes can be taken to the end user to see how
actual customers would use the product and how it would perform under such conditions; this
testing is commonly referred to as product use testing, in-use testing, or market testing. “Testing
should continue until the team is satisfied that the new product does indeed solve the problem or
fill the need that was expressed in the original protocol” (Crawford, 2008). If the design proves
faulty, it is time to go back to the drawing board and make some minor or major adjustments to
the current design and then prove the design through the development and testing of another
prototype.
18
An important aspect of prototyping is to make sure the design can be economically
manufactured. “The process design must be compatible with the part design for the prototype to
have any real meaning. If the part design does not satisfy all of the criteria or design rules for the
manufacturing process that is envisioned, neither the part-design dimensions nor the performance
characteristics can be verified” (Strong, 2006). Once a designer feels like the product design
and the prototype have passed the necessary tests, it is time to move to product launch.
Bean and Radford emphasize the importance of having a prototyping plan in their book
Powerful Products: Strategic Management of Successful New Product Development. “This plan
will include tests appropriate to the product at that stage of development but will also include
tests to assure the product team that the prototype, in critical aspects, sufficiently represents the
product that production will build and customers will receive to make the tests both technically
useful and effective in communicating information to the manufacturing plants and to critical
customers” (Bean, 2000). This prototyping plan is a great activity to integrate into the
development process and it is something that many literary sources fail to address or properly
emphasize.
2.1.4.2 Benefits of Prototypes
While there are many benefits to prototyping, David S. Bent created a simple list of
benefits that show why the prototyping process is so crucial to the new product development
process:
1. Knowledge—Prototyping can help the design team overcome lack of knowledge
of how the product will perform in a real-world situation.
2. Risk Reduction—By creating and evaluating a prototype, much of the guess
work can be overcome and there can be greater confidence in the final product.
19
3. Reduced Time to Market—Effective prototyping and testing can allow the
design team to proceed with certainty instead of ambiguity.
4. Marketing to Client—An effective prototype can be touted as real physical
evidence of a successful design which may influence a client, vendor, etc. to make
investment or purchase decisions.
5. Psychological Benefit—If the prototyping process proves successful, it can boost
the confidence of the design team, which can translate into added enthusiasm for
the remaining work. (Bent, 1999)
2.1.5 Launch
There is no such thing as a perfect design, so even though a designer or engineer might
want a significant amount of time to further develop a product before launch, it is sometimes
beneficial to simply move forward with the best design because customers cannot and will not
wait forever before moving on to another product that can fulfill their needs. While there are
many details about product launch that can be covered (such as marketing, manufacturing,
distribution, etc.), the most important concept for product launch in the context of this thesis is
that once a product is launched, the design process is never over. Because a design is never
perfect, and also because customer needs change over time, it is vital to constantly re-evaluate
and improve the product after it is launched. By going through the design steps over and over,
some would argue it is a cyclical process, but if done properly, it is should be more of a helical
process because every cycle should bring an upward improvement, preventing you from going in
the same circle over and over again.
20
2.1.6 Testing—Crawford & Di Benedetto’s Oversight
The steps in the new product development process as outlined in this thesis and by
Crawford & Di Benedetto are very effective in illustrating the major steps in the process with
one exception. The prototyping process is mentioned, but there was a lack of emphasis on the
actual testing of product prototypes. Without a proper and detailed testing plan, the prototyping
process is essentially done in vain. Only by constructing a stringent and scientific testing
process, can one most effectively capitalize on the prototypes that are created. Therefore,
“Testing” should stand alone as an outlined step in the product development process because it is
very different from “Prototype Development” and also a very critical step in ensuring that newly
developed products are successful.
2.2 Testing
After prototypes are built, they must be evaluated by constructing a series of appropriate
tests to either validate the design, or identify areas for design improvements. Prototype testing at
the basic level should evaluate whether or not the product will perform as intended and also hold
up to the rigors of frequent and heavy use.
Testing of prototypes can be done with already defined standardized tests, or with in-use
tests. To determine which kinds of tests are necessary, the product development team should
group together to identify key metrics that, if measured, can prove or disprove the efficacy of the
new product. Standardized tests are great because they are pre-set procedures that can be
performed easily and the results can be trusted and compared by almost anybody. Limitations to
standardized testing exist and are covered in the following sections, along with a discussion
about in-use testing and why it can be an effective complement to standardized tests.
21
2.2.1 Purposes of Testing
Before discussing the specifics of standardized and in-use testing, it is important to
review why testing is important in the product development process. Even though the Crawford
& Di Benedetto text did not list Testing as a major step in the process, they did emphasize the
importance of product testing in an earlier edition of the New Products Management book. The
book outlined four benefits of product testing and are listed below (Crawford, 1991):
1. Fulfill Protocol—This purpose is far more important than the following three.
The aim here is to see whether or not the designers and engineers have produced a
product that fulfills the desired customer needs.
2. Obtain Ideas for Improvements—Even if there are some flaws inherent in the
design before building a prototype, it is often helpful to actually build the
prototype because it can make it easier to visualize solutions to the problem.
3. Learn Modes of Use—As most products can be used in various ways, it is
important to perform testing in order to explore other possible uses for the
product.
4. Verify Claims—While customers and competitors can often be very critical of
product claims, it is vital to ensure that the product absolutely fulfills all claims
made by the company that will produce the end product. Marketing people need
this validation before they can feel solid about the product they will promote.
“Testing enables the manufacturer to justify certain types of claim about his
product which may be made in advertisements and are likely to be challenged by
the media—or by the public. The existence of supporting technical evidence can
save valuable time” (White, 1973).
22
2.2.2 The Testing Process
While many authors value the importance of testing, there seems to be no consensus
when it comes to the sequence of steps for effective product testing. In an article titled Learning
by Experimentation: Prototyping and Testing, Stefan Thomke gives a solid approach to the
testing process as part of the product development process. The process, shown in Figure 2-1,
shows how an organization should integrate their design, prototypes, testing, and analysis. These
steps are very simple but effective in showing that the process of testing should be an iterative
process. Every time the development team goes through the process of testing as outlined above,
they must evaluate whether or not to perform another sequence of testing or move on to the next
step in the product development process.
Figure 2-1—Testing Process as Outlined by Thompke (Thompke, 2008)
23
While the chart in Figure 2-1 is very descriptive about the testing process as a whole, it
does not provide much detail about the types of tests that should be run. This thesis proposes a
sequence to help in selecting what kinds of tests to perform and when they should be used in the
testing process.
Figure 2-2—Progression of Tests in the New Product Development Process
Figure 2-2 shows the progression that tests should take in the product development cycle.
First, standardized tests should be used (as needed) to analyze the performance of specific
materials that will become a part of the product. If certain product components need to be able
to withstand a specified force or have specific performance properties, standardized tests are
generally the best fit for this situation. After the material selection is complete, various
components of the new product can be tested with customized tests aimed at evaluating various
performance factors in a controlled environment. These types of tests are commonly referred to
as “in-use” tests. Lastly, actual product testing (sometimes referred to as market tests) can be
24
used to ensure that the product can be properly and easily used by the end user with high
satisfaction.
2.3 Developing Acceptance Criteria for Testing
Before running any tests, it is critical to construct a list of acceptance criteria as to
properly evaluate the testing results. Testing should always be conducted with the purpose in
mind of accepting or rejecting the current product design based on the results of the test. When
the tests are designed, they should include the minimum criteria for accepting the product for
further development or for product launch. There are two major categories for developing
specifications or acceptance criteria for a product: customer-preference driven, and specification
driven. The general process for each method is shown in Figure 2-3 and each process is
discussed in following sections.
Figure 2-3—Specification vs. Customer-Driven Process (Reintertsen, 1997)
25
2.3.1 Acceptance Criteria for a Specific Customer (Customer-Preference Driven)
If a product is developed for a specific customer, there is often a clause in the purchasing
contract that states specific criteria that must be met in order for the purchase to be made. Many
times, the customer’s quality control department will do the actual testing on a delivered product
to ensure that it passes all of the stated requirements. If this is the case, the manufacturer should
be doing the same types of testing to ensure that all of their products will be accepted. These
criteria should be agreed upon as early as possible as to ensure that the development process is
focused around the proper performance metrics of the new product.
2.3.2 Acceptance Criteria for a General Market (Specification-Driven)
Sometimes products are released into broad markets where it would be almost impossible
to have specific acceptance criteria in place for each specific customer. The acceptance criterion
for a product that will be released in a general market can be developed in two different ways:
through perceived customer needs, or by testing against a direct competitor’s product.
2.3.2.1 Criteria Based on Perceived Customer Needs
In cases where there no specific customer contracts for a new product, it is necessary to
make a number of assumptions as to what the most important product specifications should be.
These product traits must be carefully assessed through interviews with customers in the target
market, market research, or even personal experience in the market to which the product will be
released. Listed below are two important functions of a good specification:
26
1. “First, a specification is critical input for the design process. It tells the design
team which functions the customer is likely to value in the product. As such, the
specification is the echo of the customer’s voice that is heard in the design lab.
2. The second key purpose of a specification document is as a control device for the
project. By setting a goal for the performance of the product we create a
reference point from which to measure it” (Reinertsen, 1997).
2.3.2.2 Criteria Based on Competitor’s Products
Sometimes, instead of investing in considerable resources to gather the voice of the
customer as to their needs and wants for a specific product, it may be cheaper to use an existing
product as the base for all testing criteria. If a newly developed product can out-perform an
existing market leader, there is a high probability of success in the marketplace. “Where
competition exists, it is usually right to test against the market leader. If, however, you are
aiming at a particular group in a segmented market, you should test against the brand most likely
to be used by that group, which may not be the overall market leader” (White, 1973).
2.3.2.3 Case Study: Hewlett-Packard
In Hewlett-Packard’s (HP) early years, the development team had a unique approach to
using perceived customer needs in the design and testing stages of product development. Before
development was started on a new product, the development team would get together to write the
catalog page for the product they were going to develop. This catalog page would reflect exactly
what the customer would want to see when considering purchasing a new product. By doing this
exercise, the development could directly focus on the most important product attributes that a
customer would desire. If a product attribute was not important or critical enough to make it into
27
the preliminary catalog page, it was subsequently not a major focus in the development process.
These aspects that were significant enough to make it on the catalog page were a primary focus
throughout the development and testing stages of product development for HP (Reinertsen,
1997).
2.3.2.4 Case Study: The Chemical Industry
In the chemical industry, testing is a very important piece in the development process
even after the development process is complete. Before a chemical company (e.g. Dow) releases
a new product, they test a number of batches in order to construct a range of product
characteristics that will be advertized to the customers. When a customer orders a product such
as a specific plastic, they know that the product properties be within a set range as previously
identified by the chemical company. After the chemical company actually produces each batch,
the specific tests are performed to measure the exact product characteristics to ensure they are
within the range of the overall specifications. These detailed results are given to the customer to
provide them with a more accurate report of the product upon delivery. General testing
procedures and methodologies can vary between industries but it is important to understand the
importance of testing in the product development process, no matter the industry.
2.3.3 Acceptance Criteria for a Specific Customer (Customer Preference-Driven)
Products developed under a contract to a specific customer generally outline specific
criteria that must be fulfilled before the customer will make the purchase. These types of
transactions are more common in business-to-business transactions, but can also be found in the
consumer market. Often, these contracts are binding which solidify the purchase as long as the
product meets the minimum acceptance criteria. In such instances, it is important to decide with
28
the customer as to what the minimum standards should be. The acceptance criteria should be the
primary focus during the development of the product in order to ensure the customer is getting
exactly what is desired.
2.3.4 House of Quality
One of the most common tools used in developing product specifications and product
acceptance criteria is the House of Quality (HOQ), which is one of the primary design tools of
the Quality Function Deployment (QFD) technique. This tool can be used in both the
specification-driven and the customer preference-driven methods discussed previously. HOQ’s
key function is “to translate the requirements of the upstream stage of the NPD process to the
input parameters of the downstream stage, and enable the downstream stage to understand the
trade-offs required by often conflicting requirements of the upstream stage” (Bhattacharya,
2008). In other words, the process helps take customer preferences and transform them into
ranked product specifications. Figure 2-4 shows the basic layout of the matrix that is completed
and evaluated in the HOQ process.
Although the House of Quality is typically used in the design process of product
development, it can be a great tool in the testing process. Since the HOQ matrix reveals the most
important product attributes and characteristics desired by the customer, it should be referred to
before testing. When designing the testing plan, the highest ranked product characteristics
should be on the top of the list when considering what tests to run and what test metrics and
results would be the most important to the end user.
29
Figure 2-4—House of Quality Matrix (Temponi, 1999)
2.4 Review of Standardized Testing
There are a number of procedures that are highly trusted and already established for the
purpose of measuring performance of products and materials. These pre-established testing
procedures are referred to as standardized tests and can be incredible tools in the world of
product development; they are explained in detail below.
2.4.1 History of Standards
Various engineering standards have been around for ages dating back to around 3000
B.C. with evidence of standard measurements found among the people of Egypt, Mesopotamia,
and the Indus Valley. The Indus Valley Civilization, for example, developed a standardized
30
system of weights and measures, which allowed for more precise measurement for construction
activities (Baber 1996). This standard set of measurements allowed the people to accurately and
precisely measure and build according to those measurements, resulting in much higher quality
buildings and formations. Even early Bible accounts indicate that measurements were initially
taken using the forearm, hands, and fingers; even though the length of everybody’s forearms
were not consistent, it allowed for at least some type of measurement. The biblical measurement
of the cubit (forearm length) ranged from 17.5” to 21.5” around 800 B.C.; while this
measurement was adequate for many purposes, our world we know today would not exist
without a stronger system of measurements.
The technical needs of our day have shifted dramatically from the days of old where the
only metrics used were lengths and weights. Because of these new and more intricate needs, it
has been crucial to expand the use of standards as to provide our society with the proper tools to
effectively and consistently deliver high quality goods to its citizens. In a constantly changing
world where certainty can hardly be guaranteed, solid standards can help consumers and
businesses make wise decisions by ensuring confidence in the products they both purchase and
sell. Without standards, it would be difficult to trust any claims made by companies trying to
sell their products. Standards allow consumers to make apples-to-apples comparisons between
different products in a way that establishes confidence and power in their purchasing decisions.
2.4.2 How Standards are Developed
Standards are usually developed after a special need arises. It would be foolish to spend
significant amounts of money to develop a test that is of no use to any party or individual, thus,
many tests are developed after a need is identified. If a private organization develops a test for a
specific application, it may wish to share the details of this test with a governing body in order to
31
legitimize and regulate the testing parameters. Standards agencies are very eager to work with
such labs and private parties in an effort to provide a documented standard for the rest of the
world to use.
The International Organization for Standardization (ISO) is a governing body that
regulates and controls various standards for many different industries and is explained further in
the following section. They have documented their process by which their standards are created
which is similar to the other major standards agencies. The first step in their process of creating
a standardized test is to wait for an industry or organization to propose a test for a clearly
established need. Next, “to be accepted for development, a proposed work item must receive the
majority support of the participating members of the ISO technical committee which, amongst
other criteria, verifies the global relevance of the proposed item…” (ISO, n.d.). After the
standardized test goes through this process, it can be approved as an ISO certified standard.
Another way a test may be identified is through an ISO policy development committee that
analyzes the needs of developing countries and other consumers for possible needs of standards.
Once needs are identified, it is sent to a development group that, in turn, goes through the
aforementioned development and approval process to solidify the test as an ISO standard (ISO,
n.d.).
2.4.3 Recipe for a Standardized Test
Generally, an outlined standardized test provides clear instructions on how to perform a
test so that anybody in any location can perform the test and trust the results as long as
procedures are properly followed. Standardized tests can be broken down into four parts:
1. Material or sample preparation is described in detail—Samples must be
prepared with no deviation to the instructions because any change will cloud the
32
final results and will prevent a true apples-to-apples comparison of other materials
or products that are tested in the same manner.
2. Testing procedures are carefully outlined—Included in this step, machinery
and equipment needed to perform these tests are identified in great detail. All
equipment must be calibrated and approved for use in the standardized test.
3. Proper data collection methods are outlined—These procedures must be
followed precisely so that the analysis can be trusted.
4. Testing report format must be created—All aspects of testing report must be
outlined within the text of the testing procedure. This report outline describes
exactly how the data analysis is to be conducted and how results are to be
displayed.
While some tests may vary from the above outline, this should provide a good impression
of what is contained within the text of a standardized test.
2.4.4 Standardized Testing Agencies
There are many different groups and agencies that have become respected authorities on
various types of standardized testing. This section will cover some of the major agencies and
describe their history, focus, and importance.
2.4.4.1 American Society for Testing and Materials (ASTM)
ASTM is one of the biggest standards agencies and its influence stretches into a host of
industries. Even though the name indicates it is an “American Society”, it has expanded its
reputation to be one of the most trusted standards organizations in the world. The standards
within this organization are focused primarily on “technical standards for materials, products,
33
systems, and services” (ASTM International, n.d.). ASTM has a rich history that started over
100 years ago when a group of engineers and scientists decided to create some standards in the
railroad system. Their efforts resulted in creating solid standards for rail construction which
drastically improved the safety of railroad systems (ASTM, 2010). As there were more needs
throughout the country for consistent standards, ASTM offered its expertise and expanded their
scope to include various industries. It continues to work with various groups, organizations, and
governments to create standards that ultimately establish higher quality and safer products
throughout the world.
2.4.4.2 America National Standards Institute (ANSI)
ANSI started in 1916 when the American Institute of Electrical Engineers invited four
other major engineering organizations to join together “in establishing an impartial national body
to coordinate standards development, approve national consensus standards, and halt user
confusion on acceptability” (ANSI, 2010). It’s first standard was on pipe threads and then they
started developing major safety codes to promote safety in the workplace throughout the United
States. Over the years, ANSI has joined other standards and safety organizations to create a
large and respected standards organization. Their stated corporate goal is “to lead and foster the
work of the broad-based U.S. standardization system, to protect the integrity of this system, to
promote global competitiveness of business, and to enhance the quality of life of U.S. citizens”
(ANSI, 2010).
2.4.4.3 International Standards Organization (ISO)
The ISO is the largest standards organization in the world, controlling over 18,000
international standards in its portfolio ranging from agriculture and construction standards to
34
medical devices and even management of services (ISO, 2010). The focus on international
standards has facilitated trade of goods and services between different countries because with the
standard in place, both parties can be confident that their purchased products are of consistent
and high quality. The ISO 9001 family of standards is essentially the gold standard of quality
control for almost all sectors within manufacturing and is highly regarded and well known in that
field.
2.4.4.4 German Institute for Standardization (DIN)
The German Institute for Standardization, or DIN (Deutches Institut fur Normung), has
been based in Berlin since 1917 and has primarily focused on creating national standards for
Germany, but many of these standards have carried over their borders and now over 90% of their
standards have become internationally accepted (DIN, 2010).
2.4.4.5 Japanese Standards Association (JAS)
In 1945, Japan’s Minister of Trade and Industry allowed JAS to form as a major
standards organization for the country. Their stated objective is “to educate the public regarding
the standardization and unification of industrial standards, and thereby to contribute to the
improvement of technology and the enhancement of production efficiency” (JSA, 2010). They
currently focus heavily on maintaining standards and documenting practices that can be used to
strengthen the quality control functions of Japanese based companies.
2.4.4.6 Consumer’s Union (CU)
The Consumer’s Union is much like the previously mentioned standards organizations
except the focus of the Consumer’s Union is geared heavily towards end-user or consumer
35
products. CU’s mission “is to work for a fair, just, and safe marketplace for all consumers and to
empower consumers to protect themselves” (Consumers, 2010). Reports are created based on
product research, customer surveys, and laboratory research. The combinations of all aspects of
their reports have helped Consumer’s Union become one of the most trusted sources in regards
to consumer products.
2.5 Review of In-Use Testing
With so many different standards agencies controlling thousands of different standards
each, it is tempting to suppose that any aspect of product or prototype testing that needs to be
done will have a related standardized test somewhere in the world. Although many procedures
are carefully defined and maintained by oversight organizations, there will always be needs for
customized testing that does not fit any pre-defined procedure because products and consumers
are constantly changing.
An in-use test is much like a standardized test, except it generally is a more specific test
to measure the performance of a material or product in a way that is not currently standardized
and documented by a governing body. If an in-use test is created in the proper manner, it can
actually be more useful than a standardized test, except it will not have the big-name
organization to validate the strength of the test. While there are multitudes of standardized tests
to choose from when conducting product testing, sometimes the available tests do not adequately
measure performance in a way that is valuable.
As in the example in Chapter 1, if a fuel mileage estimate for very mountainous driving is
desired, there are no available tests for this specific application. If a dealership in the area took
the initiative to develop a specific test consisting of mountainous driving to provide his
36
customers with better information for purchasing his automobiles, he may sell more cars or at
least take pride in providing his customers with more pertinent data. Even though this newly
developed test may not necessarily be immediately adopted as an EPA standard, if the test is
documented and executed well, it could provide far better information than any readily available
standardized test.
2.5.1 In-Use Testing and Design of Experiments
A product test is essentially an experiment that is set up to determine specific
characteristics of the product. In-use tests are very similar to experiments but they are typically
less known and usually appeal to a much smaller audience than standardized tests. During the
course of this literature review, surprisingly few sources were encountered that specifically
addressed the topic of in-use testing as it pertains to product development. In order to provide a
clearer approach to the process of constructing and administering an in-use test, the classic
scientific process is a great tool and is clearly defined in texts specifically addressing “design of
experiments” or DOE. Following is a description of major steps that should be taken throughout
in-use testing. Many of the DOE steps have been integrated into the following process.
2.5.1.1 Step 1: Identify Critical Customer Needs
Virgil L. Anderson outlines “Formulation of the Problem” as one of the major
experimentation steps in his book Design of Experiments: A Realistic Approach. This can relate
to identifying critical customer needs for the product. Even though customer needs are assessed
when designing the product, the most critical customer needs must be identified as to provide a
basis for testing. Once all of the major needs are identified, then the organization doing the
testing can proceed to the next step of the process.
37
For example, if a marketing claim says that a specific product can be used for 1000 hours
without needing service, this would obviously be something that should be substantiated through
rigorous testing before being advertized. Tests should not be arbitrarily devised simply for the
sake of testing, rather, all those involved with the development and launch of the product should
have the chance to suggest product characteristics that, if proven through testing, would make the
product more marketable because customers can better trust the claims.
2.5.1.2 Step 2: Formulate Testing Procedures
After the critical needs are identified, engineers can proceed with designing a test that
will effectively measure the performance of the outlined product characteristics. It is important
to identify which variables should be measured and analyzed when constructing various tests.
“It is usually quite obvious what the variable should be; however, the means of measurement is
sometimes quite difficult” (Anderson, 1974). For example, if endurance testing of a spring is
desired, it is not possible to develop “endurance factor” as the final reported metric because it is
not something that is easily measured and communicated. Rather, it would be much more
applicable to measure the “number of compressions to failure” as it is something that can be
attained easily in a lab and it will be a result that anybody can understand. Although this was a
simple example, all factors in testing should be easily measureable and understood by a wide
audience.
A crucial point to consider in devising a test is to ensure that the results can be adequately
analyzed through proven statistical practices. Even if tests are performed perfectly, if the design
of the test does not allow for a solid statistical analysis, the test would be done in vain. It is
necessary to ensure that the results can be easily analyzed before proceeding with the test.
38
Another important step in this process is that of documentation. If a testing procedure is
not properly documented, three major problems can arise:
• Difficulty in Performing Test—If the procedures are not carefully outlined, it
may be difficult for the person performing the tests to maintain consistency
throughout the tests, resulting in skewed, biased, or unreliable results.
• Lack of Consistency—If there is not a solid written procedure in place, there is a
risk that the next person or group who conducts similar tests will not have the
information they need to perform the test exactly as it was performed previously.
• Lack of Trust—If procedures are not carefully outlined, there is a risk that
people outside of the organization may not trust the results of the tests because
they cannot analyze and see exactly what was done to produce those results.
Because of the above factors, it is critical to have a keen focus on clear documentation of
all testing procedures, no matter how simple they may be.
2.5.1.3 Step 3: Perform Testing
As the tests are being performed, it is critical that they are executed precisely as they are
documented. Paul Berger, in his book Experimental Design with Applications in Management,
Engineering, and the Sciences, addressed this point in the running of experiments, “It is vital that
the experiment that was designed is the experiment that is run” (Berger, 2002). Throughout
testing, it is also important to ensure that the data is recorded in a consistent manner and stored
properly as to eliminate the risk of digital file corruption or data loss.
39
2.5.1.4 Step 4: Analyze Data
As stated in the second step, it is necessary to devise a test that produces analyzable
results. After results are gathered, they should be analyzed precisely in the way that was outlined
in the design of the test. Solid and trusted statistical analysis procedures and tools should be
used so customers and peers have no question as to how the results are calculated. Results
should be formatted with clear explanations and charts so as to make them easy to understand
and trust.
2.5.1.5 Step 5: Evaluate Results and Draw Conclusions
A solid data analysis should provide a good basis for drawing conclusions and devising
future action plans. If there are any questions or doubts about the validity of the results, the tests
should be adjusted, or performed again under a more watchful eye. Once an organization is
comfortable with the results from the testing, it must carefully evaluate the results. This
evaluation can sometimes prompt a product modification or redesign, or the organization can
proceed forward with further in-use testing, limited market testing, or it can jump ahead to
product launch.
2.6 Market Tests
While the focus of this thesis is on the combined effects of standardized and in-use
testing, it would be a mistake to overlook the potential importance of a solid market testing
practice within the product development process. Product testing can take the shape of a market
test where products are given to the actual customers in order to see the response of the end
users. At this point, the products used for the market tests should either be a production run, or
40
very similar to it because it is valuable to see how potential customers will interact with the final
product. Measurement of results in market testing will often be very different than in previous
testing stages. Surveys with results on the Likert scale will provide a measureable base from
which results can be analyzed.
2.6.1 Market Testing Case Study—The Gainesburger Dog Food Failure
A classic example of a product that was not properly market tested was that of a new dog
food product named Gainesburger. General Foods was the developer of the new dog food that
contained more nutrition in less volume, so pet owners would not have to feed their dogs as
much food (volume-wise) as they normally do. An extensive amount of testing was done in
laboratory settings where doctors and scientists carefully measured out proper amounts of food
for dogs in the lab. The lab results turned out great and General Foods decided to roll forward
with the sales of its new product. Many customers bought the little packets of food and began to
feed their dogs. To the surprise of General Foods and all of the dog owners, many pets eating
the Gainsburger pet food were getting sick! After an investigation, researchers found that many
of the pet owners were feeding their dogs much more than one packet of food simply because
they just didn’t think that one little packet was enough for their cherished pet. Even though the
product would perform perfectly in lab conditions, General Foods should have done limited
market tests to ensure that the end users would properly use the product as intended (Crawford,
1991).
This example proves that even though the final product was proven completely effective
in laboratory environments, things can still go wrong! Customers do not always use products
exactly as directed, so market tests can be a valuable tool to analyze potential customer behavior
and to also see how severe the effects of product misuse can be.
41
2.6.2 Market Testing Case Study—The Hubble Space Telescope Blunder
Testing of prototypes of the final product in its intended use is critical to success of the
product. NASA got to learn this lesson the hard way when they launched their Hubble Space
Telescope. While the telescope was being built, the head of NASA, James Beggs, proposed that
the fully assembled telescope be further tested on the ground before being launched into space,
but his idea of more testing was quickly dismissed. The rest of the Hubble team thought that a
fully assembled test would not reveal any more information than the separate component tests
had already revealed. NASA launched the telescope without heeding the counsel of Beggs,
which turned out to be a major error because the first images sent back to Earth were blurry and
distorted which rendered the telescope in its current form as useless. This was an extremely
costly mistake that could have easily been corrected had the team decided to heed Beggs’
direction and perform further fully assembled tests before launching the flawed telescope into the
sky (Slade, 1993).
2.7 In-Use Testing Case Study—Digital Photography Review
In the world of digital photography, the Digital Photography Review website has been
one of the most trusted sources for objective and professional equipment reviews for many years.
The site was created in 1998 by Phil Askey and was focused on not only providing a database of
general camera information and news, but it focused heavily on providing intense high-quality
structured camera reviews which has been a cornerstone in its ability to bring viewers to the site.
Authors of the site believe in providing “unbiased content with as much detail as [they] can
provide…” (Digital Photography Review, n.d.). At the time Digital Photography Review was
created, there was a lack of solid equipment testing standards for digital equipment in the
42
photography industry and Askey took this opportunity to develop a scientific approach in
creating detailed, reliable, and useful reviews of new photography equipment. These reviews are
now a solid resource for millions of photography enthusiasts and consumers who want to make
informed decisions before spending their hard-earned money on new equipment. This unbiased
content can be provided because of the extensive series of highly objective tests they have
created. In regards to his employment history with Digital Photography Review and relationship
with Phil Askey, Simon Joinson wrote, “As the cameras got more sophisticated and image
quality started to improve the value of Phil’s approach became more and more apparent, and it’s
no accident that by the time I met Phil, he was considered the only authority on digital camera
image quality by every Japanese manufacturer I ever spoke to” (Joinson 2008). By being a first
mover in developing testing procedures, Askey was able to have a major influence on the entire
digital photography industry.
2.7.1 DPReview Lens Review Procedure
While there are many different types of equipment that Digital Photography Review has
developed tests for, the following is a walk-through of the testing procedures for the camera lens
reviews. This camera lens test that DPReview designed is a great example of a test that started
out as an in-use test that has transformed into an industry standard. The first step in DPR’s lens
review is to take a series of pictures of the brand new lens in predetermined angles; these
pictures are taken before any other testing procedures because the lens is free from smudges and
blemishes that are commonly found after the intensive testing begins. Mulitple pictures of the
lens are taken from each angle to ensure the picture used in the review will be perfect; while only
10-15 views are used per review, the total number of pictures taken of the lens can be close to
50. Figure 2-5 shows images from two different lens reviews where you can see that the angles,
43
sizes, lighting, etc., are all exactly the same between the two images, allowing the viewer to
perfectly compare the size and appearance of the two lenses. This image is important because it
showcases the consistency with which DPReview performs its lens reviews (the lenses are from
two different lens reviews from different times). This attention to detail is consistent throughout
all of its tests.
Figure 2-5—Comparison shots of different lenses (Westlake, 2008)
Step two in the lens testing process is a studio shooting process where test charts are
placed at a pre-determined distance under controlled lighting and the lens being reviewed is used
to take many pictures of each chart. “Test charts have to be very precisely aligned and focused,
and the data shot at least twice to assure reproducibility” (Westlake, 2008). These studio tests
44
measure things such as resolution, distortion, falloff, bokah (background blur), macro
performance, and image stabilization. These metrics are chosen for testing because poor
performance in any of these areas will likely affect the ability of the user to capture excellent and
clear photos in the field. Figure 2-6 shows a test chart that is used for evaluating macro (close-
up shooting) performance of the lens. By taking a high resolution shot of this chart from a set
distance and lighting, certain performance factors can be exposed and compared. This controlled
studio shooting process is vital to the equipment reviews because it allows a quantifiable
comparison of performance on factors that are nearly impossible to quantify by field testing
alone. Results from this process are often used to narrow the focus on different perfomance
characteristics in the third step of the lens testing process.
Figure 2-6—Macro Focus Test Chart for Lens Reviews (Westlake, 2008)
45
The final step in the review process is real-world shooting where the tester takes the
camera and lens outside and starts shooting pictures in the same manner that many of the end
users of the product would do. An example of this real-world testing is shown in Figure 2-7
where a camera with the tested lens was placed on a tripod and two pictures were taken. One
picture was a full wide-angle (zoomed all the way out) and the next picture was taken at full
telephoto (zoomed all the way it). These images are very useful in helping a custoemr
understand the zoom power of a lens.
Figure 2-7—Shots Depicting Zoom Power of Lens (Westlake, 2008)
In talking about this process, one of the regular lens reviewers, Andy Westlake, stated “I
actually prefer to shoot with the lens fairly extensively before processing all of the studio data, to
get an initial impression of how it performs which isn’t colored by those results” (Westlake
2008). This insightful statement shows that the process is highly planned and many factors are
taken into consideration before the testing even begins. This reviewer realizes that if he
processed all of the lab-testing results first, he would probably try to compensate for some of the
weaknesses of the lens which is something many of the end users would not do, so he decided to
46
do some shooting before reviewing the lab results to better simulate using the camera as the
average user would. After doing a fair amount of non-biased real-world shooting (before seeing
lab results), and once the lab data is processed, the reviewer then goes out for more real-world
shooting with the purpose of illustrating any weaknesses identified in the lab tests. If the lab-
based weaknesses are not replicated in the real-world shooting session, the reviewer goes back
out and does more shooting to validate his findings. In relation to the real-world tests, Westlake
says, “It’s also important to realize that a full lens review simply can’t be done purely in a studio
environment: the technical data is hugely valuable but it says nothing about autofucus speed
under various lighting conditions, or flare, or bokeh; for these we need to go out and take ‘real’
pictures. (Westlake 2008)” In an effort to maintain the integrity of the testing results, real-world
pictures are taken with the same lens on multiple camera bases in order to identify and eliminate
any deficiencies that may have been caused by the camera base rather than the lens itself.
This lens testing process shows that it is vital to be as scientific as possible but also
provide real information that is exactly what the customer wants to see. DPReview’s camera and
lens reviews provide incredible detail that only the most intense photography enthusiast would
understand, but they also have very clear summaries of their testing that almost anybody can use
and understand. From this example, we learn about not only attention to detail, but also the
importance of clarity and understanding the audience as to provide them with the most
understandable and valuable information possible. This process utilizes a great mix of subjective
tests and objective tests and combines them to create a lens review that can be held up as a strong
standard in the photography industry.
47
2.8 In-Use Testing Case Study—Garbage Can Testing
Dr. A. Brent Strong has been a leader in the waste-container industry when it comes to
setting standards for rolling waste carts. Years ago, the city of Provo, UT commissioned Dr.
Strong to devise a series of tests to be used in analyzing the waste-containers before making a
hefty purchase. The containers were to be compatible with the automatic gripping arm dump
trucks where a mechanical arm grips the can and hoists it to the top of the truck before spilling
its contents into the back of the truck. The city was very interested to see if the carts would be
able to withstand 10 years of use.
In devising tests, Dr. Strong had to consider all of the concerns of the city such as tipping
force, ability to withstand high winds when empty, strength of lid, and more. One of the major
tests that was devised was a grip and lift test that would put the cart through an endurance test of
sorts to see if it could handle ten years worth of gripping, lifting, and dropping. If the carts were
dumped 52 times a year (once a week) for ten years, then the life of the cart would experience
the automatic loading process 520 times! Instead of using a dump truck for this testing, a gripper
arm was obtained and used on a special rig for this testing. Each garbage can to be tested was
filled with the maximum specified load which was over 300 pounds for the large carts, and then
the carts were gripped, lifted, and set down 520 times or until failure.
This test really puts the same stresses on the cart that it would see in normal use, yet it
had the advantage of being done in a controlled environment. Another advantage of this test was
that it could simulate ten years of use in the matter of a few hours of testing; this benefit alone is
so valuable that many garbage can manufacturers consult with Dr. Strong to perform such work
so the companies can speed up their development process. The tests that were devised long ago
48
for the city of Provo are now industry-wide standards because the tests were designed with
practicality, repeatability, and efficiency in mind.
2.9 NCASE Case Study
To illustrate the application of in-use tests in the real world, this thesis will look at the
process by which a new military product was developed and evaluated. In this situation,
standardized tests could not adequately test the desired performance characteristics of the
product because the product uses and requirements did not relate to any currently available
standardized test. Because of this, new tests were carefully created and documented in an effort
to effectively evaluate the product design and its materials. This section will describe the
development process and some of the major characteristics that the creators of the NCASE
wanted to have tested.
2.9.1 Development of NCASE
Development of the NCASE product started when one of the owners of Hazard
Protection Systems, Inc. approached Dr. Brent Strong with a product idea because of his solid
background and experience with new product development. HPS’s product idea consisted of
covering that could be filled with a flame-extinguishing powder that could be wrapped around an
external fuel tank. This product was conceived in the hopes that it would reduce injuries and
fatalities associated with incidents of rocket propelled grenades (RPGs) and roadside bombs,
commonly called improvised explosive devices (IEDs), which often strike military vehicles.
49
After the development of the NCASE had started, HPS and Dr. Strong decided to develop
a different design that used a fabric blanket with plastic bags of the flame-retardant powder sown
into the middle layers of the fabric, much like a quilt. Candace Jesclard, CEO of HPS, had a lot
of experience with sewing and figured out a way to cut and sew fabric so it would fit like a snug
jacket over the fuel tank. HPS also realized at this point that a zipper could be sewn into the
product so the powder baggies filled with the flame retardant powder could be installed after the
jacket was sewn, thus avoiding accidental punctures of plastic bags during manufacturing. The
new blanket design is shown in Figure 2-8 as a finished product.
Figure 2-8—NCASE Product Installed on Fuel Tank
Although both the original and the blanket designs would effectively dispel flare-ups on
the fuel tanks, preliminary conversations with the military led HPS to shift their focus solely on
the blanket design because it would be possible to use materials like Kevlar that are very familiar
50
and trusted by the military in applications where the operating and environmental conditions can
be quite rugged. Since using standard and trusted materials like Kevlar was important to the
military, HPS and Dr. Strong decided to use a Kevlar fabric that was coated with a urethane
polymer coating so as to reduce any possible ultra-violet radiation damage that would come from
prolonged sun exposure, something that would be very common for the intended trucks in Iraq
and Afghanistan.
After a few months of developing both ideas, HPS performed a live explosive test in
Alaska to determine whether or not this concept would prove itself in real-world conditions. For
the testing, a bare metal tank filled with fuel was shot with a large caliber rifle on a range and the
result was a massive fireball that shot out of one side of the tank (see Figure 2-9). Next, another
fuel tank was wrapped with the NCASE product and also shot with a large caliber rifle. There
was a cloud of smoke/steam that came out of the can, no flame (see Figure 2-10)! The flame was
dispelled because the bullet broke the powder bags and the fuel was infused with the flame
extinguishing powder, preventing a dangerous flare up.
Figure 2-9—Sequence of Fire Testing a Bare Fuel Tank
51
Figure 2-10—Sequence of Fire Testing a Fuel Tank with NCASE Installed
The successful field tests of the NCASE product gave HPS the confidence they needed to
present their product to the military (Strong, private conversation). HPS produced some full-size
products to send to the military for testing at the official military testing center. After meeting
with the military, HPS provided a full-scale prototype to be tested by the military’s Aberdeen
Test Center near Baltimore, Maryland. This full-size product testing was deemed successful and
the Army decided to order NCASE wraps for one type of transport truck that is used heavily in
Iraq and Afghanistan. Even though the military had indicated they would order many NCASE
products for their trucks, they requested that extensive environmental tests be performed on the
product to prove that the product will be suitable for use in the intended harsh environments.
From this point on, the focus went from the product design to a validation process to ensure that
the materials chosen for the final product would be adequate for the intended conditions. This
validation process consisted of a variety of tests that were either newly developed or adapted
from existing tests and are the focus of the case study of this thesis.
2.9.2 Environmental and Operating Conditions
Following are descriptions of the environmental and operating conditions that were
considered when constructing the testing procedures for the NCASE materials.
52
2.9.2.1 Sunlight exposure
As the NCASE product will be used in desert environments, it is critical that the product
be resistant to decay resulting from exposure to ultra-violet rays. As the product is to be used on
an external fuel tank, it will be subject to sunlight exposure as long as it is outside. According to
the World Weather Information Service, there are 4 months out of the year in Iraq where the
daily high averages over 100 degrees Fahrenheit (WMO, 2010). Figure 2-11 shows the average
yearly weather patterns in Iraq. These high temperatures come from intense sunlight exposure,
which contains harmful ultra-violet rays that can cause decay in a multitude of materials,
especially plastic materials. Unless vehicles are stored in a covered parking area, there is little
by way of trees and vegetation to shield the vehicles from damaging UV rays.
Figure 2-11—Average Weather Patterns in Iraq (iexplore, 2010)
53
2.9.2.2 Fuel Exposure
The army indicated to HPS that during the filling operation of the tactical vehicles, fuel is
commonly spilled around the fill cap region. JP-8, commonly known as jet fuel, is the most
commonly used fuel in the military because of the US Army Single Fuel Forward Policy that has
been enacted in hopes of simplifying the fuel storage and distribution practices of the military
(Fernandez, 2005). Military mechanics stated this occurs on numerous tactical vehicles as a
result of improper filling procedures, rush of refueling under combat conditions, or as the result
of a design limitation. On many tanks stain marks were observed where fuel had leaked out of
the fill cap or was spilled during refueling. HPS immediately recognized this potential
endurance threat to NCASE and commissioned a study to determine the long-term exposure
threat to the materials used to form NCASE. Within the labs at BYU, specific tests have been
developed to adequately measure the performance of the Kevlar material after similar fuel
exposures. These tests are explained in the “Test Protocol” section.
2.9.2.3 Cold Weather and Crinkle Exposure
As the product will be used in cold environments like the mountains of Afghanistan, the
HPS product needs to be able to perform well after exposure to freezing temperatures. Even
though Afghanistan is at the same latitude as the South-Central USA, the high mountains and
elevation make it a much colder climate (iexplore.com, 2010). Not only are there harsh winters,
but the southern lowlands of the country produce very hot summers as well (iexplore.com,
2010). Figure 2-12 shows the average weather patterns in Kabul, Afghanistan
54
Figure 2-12—Average Weather Patterns in Afghanistan (iexplore, 2010)
There is also the possibility of the product getting pinched or crinkled during normal
operating conditions or when the product is being removed or installed for inspection and
maintenance. Some tests have been developed to determine whether or not the strength of the
material is compromised through freezing, crinkling, or a combination of the two.
2.9.2.4 High-Speed Impacts
Lastly, as this product is designed to be an external safety component on various military
vehicles, it will be prone to high-speed flying rocks and pebbles. Fuel tanks on larger transport
vehicles are usually mounted underneath the cabin of the vehicle and are very close to the
55
ground, making it more exposed to rocks that are kicked up by tires on both paved and unpaved
roads. These rocks can travel over 100 miles per hour and have enough force to break the cars of
windshields up to 100 feet away. The product is close to the tires and must be able to withstand
multiple severe high-speed impacts without compromising the integrity of the materials.
2.9.3 Review of NCASE Materials
2.9.3.1 Kevlar Fabric
Kevlar fabric is the material used as the outer component on the NCASE product. Kevlar
(commercialized by Du Pont Company in 1972), is an aramid fiber that possesses a “high
tenacity and modulus, thermal stability and low elongation, stress decay and creep associated
with inorganic fibers while retaining the low density, fatigue resistance and wear
resistance…characteristic of organic fibers” (Schaefgen, 1983). Kevlar is a highly versatile
material that is used in a variety of applications from car tires to bullet-proof vests where
strength and toughness are desired product characteristics. “Aramid is a generic term for a
‘manufactured fiber in which the fiber-forming substance is a long chain synthetic polyamide in
which at least 85% of the amide linkages are attached directly to two aromatic rings’, as defined
by the US Federal Trade Commission” (Yang, 1993). Kevlar can be produced in a variety of
forms such as yarns, short fibers, pulp, felt, woven fabric, cord, etc. The molecular composition
of Kevlar can be seen in Figure 2-13.
56
Figure 2-13—Molecular Structure of Kevlar (absoluteastronomy.com, 2010)
The Kevlar used for the NCASE product is a woven fabric that is actually coated with a
UV-resistant urethane material. Kevlar fabric is considered a high performance fabric because of
its high strength and modulus at low to high temperatures (Yang, 1993). One of the reasons a
coated Kevlar was chosen was because it is sensitive to radiation from ultraviolet light, which
comes from direct sunlight exposure; however, this deteriorization is not enhanced by moisture
or other atmospheric contaminants (Yang, 1993). UV exposure can lead to a loss of mechanical
properties and should be considered when selecting Kevlar as a material for a new product.
2.9.3.2 Flame Retardant Powder
The flame retardant powder is branded as Purple K. This well-known flame retardant
powder is supplied in bulk to the flame retardant industry (for use in portable flame suppression
devices and other applications). Purple K is a slightly modified potassium bicarbonate.
57
3 METHODOLOGY
3.1 NCASE Testing
To demonstrate whether or not in-use testing can effectively be applied to a newly
developed product, a case study was needed for this thesis. Dr. Brent Strong had connections to
Hazard Protection Systems, Inc. (HPS) and a relationship was established for this thesis after
HPS indicated their need for some customized testing. The NCASE product was in early
development stages at the start of this thesis and was, therefore, used as the subject for this
research.
Testing of the NCASE materials was done in two separate phases at different time
periods. The first phase was conducted in 2008 and two materials were tested: a Kevlar fabric
(referred to as Old Kevlar) with a urethane coating and an uncoated nylon fabric. The Kevlar
was intended for use in exposed conditions on outer fuel tanks and the nylon was intended for
use on fuel tanks that were fully contained in an external box and would not be exposed to
sunlight. These materials were to be tested to evaluate performance under (separately) extended
sunlight exposures, hydrocarbon (fuel exposure), and impact resistance (for Kevlar).
The second phase of testing was performed in 2010 on different Kevlar fabric (referred to
as New Kevlar). This new Kevlar material required an evaluation similar to the previous tests.
Separate tests were done to evaluate exposure to hydrocarbons, water, cold weather, impact, and
58
crinkling. Due to time constraints and the long-term nature of sunlight or UV testing, the UV
testing was not performed in this run of tests during the 2nd phase of testing.
3.1.1 Environmental & Operating Conditions
Chapter 2 contains a detailed description of various critical operating conditions that were
considered in the design of the product and in the selection of materials used to manufacture the
product. To effectively evaluate the NCASE product in regards to its performance in the
intended environments, a number of tests were designed to effectively simulate specific
operating or environmental conditions in a controlled environment. These tests were either
developed by slightly adjusting standardized procedures or by creating an entirely new
procedure.
3.1.2 Sample Preparation
Test samples for fuel exposure, water exposure, cold weather, and crinkling were cut from
the same Kevlar material that was used to construct the NCASE product. The samples were cut
in 7/8” x 8” strips because this size allowed for accurate testing in the clamps on the Instron
tensile-testing machine at BYU. While dog-bone samples are usually used for tensile testing of
solid materials (ASTM D 638-00, 2001), rectangular samples were used in this test because only
the fibers in the long direction would receive any load (see Figure 3-1). Careful consideration
was placed in cutting the fabric as straight as possible because an off-axis cut would reduce the
amount of fibers effectively bearing a load during testing, which could possibly skew test results.
It is quite possible that some of the variance in the testing results can be attributed to the angle at
which the samples were cut as it is extremely difficult to precisely line up the thin fibers;
however, every effort was made to ensure that the samples were cut at the right angle. The
59
specimens were cut using a fresh razor blade to avoid any stress to the material. The specimens
were labeled and divided into groups for testing. Control groups were isolated apart from any
exposure contamination. Five specimens were exposed in each type of test then subsequently
tested on the Instron machine to determine the maximum strength of the material.
Figure 3-1—Kevlar sample
3.1.2.1 Weathering
As the NCASE product will be exposed to great amounts of heat and ultra-violet
radiation in the desert areas of Iraq and Afghanistan, it is crucial to know whether or not the
product will survive the test of time. One of the most important reasons organizations perform
weathering testing is to determine the service life of their product (Atlas, 2002). The ASTM G7
standardized test outlines conditions for testing the “Atmospheric Environmental Exposure
Testing for Nonmetallic Materials.” The results of this test would show how the NCASE would
perform throughout time in exposed conditions, but it would take months or years to obtain
usable results. While it would be ideal to outfit a military vehicle with the NCASE product and
place it in service for a few years to see how the product will perform, it would not make
business sense to take so much time to only evaluate one aspect of the product’s performance.
After the product is sold, it would certainly be beneficial to track performance through time, but
in the product development process, time comes at a high premium and all efforts to reduce the
60
development cycle should be made. There are ways to accelerate the results from natural
weathering tests by placing mirrors near the product and angling them properly as to increase the
amount of sunlight exposure, but the testing time would still be very significant.
“The three primary factors of weather are light, moisture and temperature” (Atlas, 2002).
Since Kevlar can only be degraded by light exposure and not moisture and normal testing
temperatures (Yang, 1993), UV degradation was the only focus for testing of the Kevlar samples.
Kevlar and nylon samples were prepared for artificial weathering and were placed in the
weatherometer machine at BYU on April 7, 2008 (see Figure 3-2). The samples were held on
racks with the appropriate fixtures as outlined in the ASTM “Standard Practice for Fluorescent
UV Exposure of Plastics” (ASTM D 4329-99, 2001). The fluorescent UV bulbs can accelerate
the natural UV exposure by a factor of about three, thereby reducing testing time considerably
compared to non-artificial testing. The first set of specimens was labeled and removed from the
machine on May 7, 2008 then placed in a plastic bag which was then stored in a dark desk
drawer for the next month. On June 7, 2008, the remaining samples were labeled and removed.
All of the weathered samples were then tested on June 20, 2008.
61
Figure 3-2—Weatherometer in BYU Lab
Maximum tensile strength was used as the metric in comparing exposed samples with
control samples. The fabric samples did not have a consistent mode of failure as some would
break all at once while others would break one strand at a time. While it may be appropriate in
determining the properties of some materials, measuring the pull strength at failure would not be
a good metric in giving consistent results because the failure modes were so inconsistent.
Instead of measuring failure strength, it was determined that maximum tensile strength of the
material would be the most consistent and pertinent metric for this testing because of the
consistency the results would provide.
62
3.1.2.2 Fuel Exposure
The materials used in the NCASE system were exposed to certain hydrocarbon-based
liquids (kerosene and diesel) for various time intervals to simulate the exposure that the NCASE
product might encounter during fueling operations on a tactical vehicle. In a worst case scenario
during product use, fuel spills might flow into the recess between the tank and the NCASE panel
and be retained there until evaporated. It is possible that the presence of this fuel would weaken
the NCASE product.
JP-8 fuel is currently used in the military for the transport vehicles. As JP-8 fuel was not
accessible for the solvent testing, diesel and kerosene were used because they closely resemble
the JP-8 fuel and share similar chemical properties. If there is concern regarding the actual effect
of JP-8 fuel, further testing may include solvent testing with the actual JP-8 fuel.
The purpose of this test was to establish the effects of several common fuels used in
military and commercial vehicles on the material used to produce NCASE. In conformance with
standard ASTM testing (ASTM D 543), the material was immersed in the solvent (fuel). As the
exposure time of the fuel to the NCASE material can be limited by the spill being cleaned or
evaporation, the times for exposure were adjusted accordingly. Furthermore, because the danger
in actual use is the loss of physical properties, the test used to evaluate the effect of the solvent
was tensile properties (ASTM D 638) rather than the weight gain measurement used in the
immersion test (ASTM D 543).
The specimens were dipped into fuels for various exposure times; see Figure 3-3 for an
example of how the materials were exposed to the hydrocarbons. The exposure times were
chosen to simulate reasonable, worst case exposures that might be encountered in actual use.
The exposure times chosen were: 3 seconds, 1 minute, 30 minutes, 1 hour, and 2 hours. After
63
exposure, the specimens were dried and then tensile tested. The hydrocarbon tests were actually
conducted at different times to ensure that the diesel and kerosene samples would not be
inadvertently interchanged. All Phase 1 and Phase 2 tests were conducted in an identical
manner.
Figure 3-3—Sample Being Soaked in Diesel Fuel for Solvent Test
3.1.2.3 Water Exposure
Samples were exposed to water in the same manner as the hydrocarbon exposure as
described in the section above. Again, the samples were dried before testing the tensile strength
on the Instron machine.
64
3.1.2.4 Freezer and Crinkle Testing
Although the NCASE product will likely not experience freezing climates on products in
Iraq, there are many fuel tank wraps on vehicles in Afghanistan, which has a climate much like
Salt Lake City, UT. Because of the likely exposure to cold-weather climates, a freeze test was
devised to determine if cold weather will compromise the strength of the Kevlar material.
Regular Kevlar samples were cut and then placed into a 0 degree Fahrenheit freezer for various
amounts of time. These samples were then removed from the freezer, allowed to come to room
temperature, and then subsequently tensile tested on the Instron machine.
Another concern expressed regarding the Kevlar material is its performance after being
exposed to forces that fold and bend the material. A test was devised that puts consistent forces
on samples that have been folded in a standardized manner. The samples were prepared by
folding them like an accordion, folding 1-inch over at a time, creating eight 1-inch sections (see
Figure 3-4). After this, the entire folded sample was folded in half and then wrapped with clear
tape to maintain the folds for testing (see Figure 3-5). The samples were then exposed to a
weight by sandwiching five samples between two metal plates that measured 2-feet x 2-feet
each. The samples were placed as follows: one sample in each corner and one in the middle (see
Figure 3-6).
65
Figure 3-4—Preparation of Crinkle Sample
Figure 3-5—Preparation of Crinkle Sample
66
Figure 3-6—Placement of Crinkled Samples on Plate
After the samples were set between the plates, a 150-pound weight was placed on top for
a period of 24 hours (see Figure 3-7). 150 pounds was chosen because that is the maximum
weight of an actual section of an NCASE kit. If the product is removed from the tank and left on
the ground, the weight would be dispersed over many contact points on the bottom surface of the
product, so by spreading this weight over five samples this test actually provides more force than
will be encountered in normal use. Therefore, if the material is not compromised through this
test, it should logically not be compromised through normal use. After the 24-hour exposure
period, the samples were removed from between the plates and tensile tested on the Instron
machine in the same manner as previous tests.
67
Figure 3-7—Placement of Weight on Crinkle Samples
In addition to freeze tests and crinkle tests, we performed some combined testing to see if
both freezing and crinkling the samples would lead to a further decrease in strength. Samples
were frozen for a period of one, two, and three days and then subsequently subjected to the
above-mentioned crinkle test for the standard period of 24 hours after which they were also
tensile tested on the Instron machine.
3.1.2.5 Webbing Strength
The NCASE product contains multiple sections that are strapped onto the fuel tanks. The
straps are stitched to the Kevlar fabric with a specific stitching, or webbing, pattern that must be
tested to ensure that the product can withstand the adequate forces in service. To determine the
maximum force, the pieces were tested for maximum tensile strength on the Instron machine.
68
For the testing samples, nylon straps were sewn to strips of the Old Kevlar material with the
same webbing pattern to be used on the actual product. The Kevlar section was much wider than
the testing clamps, so metal tabs were placed on either side to sandwich the Kevlar to ensure the
entire width of the sample was gripped (see Figure 3-8). The nylon strap was thick enough that
slipping in the clamps was not an issue and tabs were subsequently unnecessary. Ten samples
were pulled at a rate of 1.5 inches per minute until failure and the maximum tensile strength of
all the samples in the set were then averaged.
Figure 3-8—Tensile Testing a Webbing Sample
69
The military requires that the product be capable of withstanding forces of two times the
force of gravity (2G). The weight of the product depends on the size of the fuel tank, but no
single section will be over 75 lbs. Consequently, to be deemed acceptable for use, the webbing
sections on the product must withstand at least 150 lbs of pulling force (75 lbs x 2G). A safety
factor was then calculated by dividing the test results by the required force (Actual/Required).
3.1.2.6 Tensile Strength Testing
The test apparatus for the tensile tests was an Instron 4204, model 201. It was installed at
BYU in 1988. The 5 kilonewton load cell was used for all of the testing as none of the forces
approached the 1,000 lb threshold. The machine was balanced and calibrated each time the load
cell was changed.
For all, the pull rate was set at 1.5 inches per minute and the test was run until the
material snapped. For all tests, the maximum force required to pull the sample was the metric
recorded and used for the statistical analysis.
3.1.2.7 Impact Testing
The purpose of this test was to establish the effects of rock impacts on the for the NCASE
product. Because the velocity of debris impact is much higher for vehicles than the normal
falling dart impact test (ASTM D 5628), the testing procedure was modified to be able to
simulate the type of damage that might occur in off-road service. The setup that was used for
this test is shown in Figure 3-9. The device was calibrated using an electronic speed gun. The
speeds of the projectiles were known to approximately ±1 mph with some additional minor
70
variation because of the inability of the operator to exactly match the calibration chart during the
actual testing.
Figure 3-9—Impact Testing Setup at Clearplex
This apparatus is a slingshot type device that propels projectiles onto the surface to be
tested. The user must place the projectile in the slingshot and then pull back to the
corresponding mark for the desired speed (see Figure 3-10 & Figure 3-11); the user then releases
the projectile, which hits the test sample. A 3/8-inch ball bearing and a small sharp rock were
used as projectiles for this test (see Figure 3-12).
71
Figure 3-10—Impact Testing Speed Chart
Figure 3-11—Preparing Shot for Impact Test
72
Figure 3-12—Impact Testing Projectiles (penny shown for size comparison)
In Phase 1, tests were conducted on a sample product that was constructed with the Old
Kevlar material. In Phase 2, tests were conducted on a sample product with the New Kevlar
material. The samples were tested at the following speeds: 50mph, 75mph, 100mph, and
125mph. The samples were tested five times at each speed. Speeds were started at 50mph and
progressively increased by 25mph after each set of tests. After each set of tests, the sample was
inspected on the surface for marks or punctures, and then the bag of powder was removed and
inspected to check for leaks (see Figure 3-13).
73
Figure 3-13—Inspection of Internal Powder Baggie After Impact Testing
In Phase 2, an endurance test was constructed in order to more stringently test the
performance of the NCASE product. After these progressive tests were performed, endurance
tests were conducted to ensure that the product would be able to withstand continued exposure to
its intended harsh environment. For this endurance testing, each sample was shot in the same
general spot 100 times with a ball bearing and 100 times with various small rocks at the
maximum speed of the testing device of 125mph. At the end of the test, each sample had been
shot over 200 times in the same location at full speed. Inspections of the Kevlar material and
powder packets were performed every 25 shots to assess any possible damage.
74
3.1.3 Acceptance Criteria
In working with the Army, HPS decided that a material must retain at least 50% of its
original strength after exposure to any testing condition. Therefore, if any material does not pass
the minimum criteria of retaining 50% of its strength after exposure, it will not be acceptable for
use in the intended conditions. All accept/reject suggestions shown in the results section are
based on this acceptance criterion that was ultimately defined by the customer.
3.1.4 Data Analysis
Even though a material might pass the minimum criteria of retaining 50% of its strength,
it is still important to understand the performance characteristics of the product, which is why a
statistical analysis was conducted. The statistical significance of the results was obtained by
taking all of the results from testing to the statistics department on campus at BYU. All of the
testing was explained to Dr. Dennis Eggett, professor in the Statistics Department, and he then
wrote code for the SAS software that he used to analyze the data from the testing. The results
were constructed to show whether or not the test results of exposed materials were statistically
different than the control set that was tested in the same manner. In this circumstance, a 95%
significance level (p < .05) was used to indicate whether or not the test set was significantly
different from the control set.
75
4 RESULTS OF NCASE TESTING
4.1 Liquid Exposure
4.1.1 Results
For every condition, the exposure and subsequent tensile test was performed on five
samples and those results were averaged for each data point shown below.
Figure 4-1—Nylon Solvent Exposure
76
Figure 4-2—Old Kevlar Solvent Exposure
Figure 4-3—Kevlar Liquid Tests
77
Table 1—Statistical Significance of Liquid Exposures
Statistical Significance of Liquid Exposures α Value for Pr>|t| Exposure Time New Kevlar Diesel 3 Seconds 0.193 1 Minute 0.219 30 Minutes 0.110 1 Hour 0.251 2 Hours 0.426 Kerosene 3 Seconds 0.159 1 Minute 0.878 30 Minutes 0.293 1 Hour 0.112 2 Hours 0.106 Water 3 Seconds 0.012 1 Minute 0.006 30 Minutes 0.005 1 Hour 0.020 2 Hours 0.009
BOLD results are significant based on 95% confidence level (p < .05)
Table 2—Statistical Significant of Liquid Exposures
Statistical Significance of Liquid Exposures α Value for Pr>|t| Exposure Time Nylon Old Kevlar Diesel 3 Seconds 0.634 0.795 1 Minute 0.462 0.5232 30 Minutes 0.676 0.8731 1 Hour 0.237 0.692 2 Hours 0.001 0.25 Kerosene 3 Seconds 0.869 0.1761 1 Minute 0.191 0.453 30 Minutes 0.362 0.134 1 Hour 0.058 0.0418 2 Hours 0.472 0.2745 BOLD results are significant based on 95% confidence level (p < .05)
78
4.1.2 Analysis
All materials in the fuel exposure tests passed the minimum criteria of maintaining at
least 50% of their original strength after exposure to the different hydrocarbons. The Kevlar
strips had no significant loss of strength.
All materials in the water exposure tests passed the minimum criteria of maintaining 50%
of their original strength properties. The Kevlar strips that were soaked in water experienced a
slight loss of strength that was found to be statistically significant (based on a 95% confidence
interval). Although the samples were patted dry before testing, the actual Kevlar fibers were
likely damp because the fabric is sandwiched between thin protective layers of urethane coating
and it would be difficult to remove all moisture before testing. As Kevlar is an aromatic
polyamide material (also known as aramid), it is expected to absorb up to 1-2% of its weight in
water, which can temporarily soften the material and reduce its strength. This condition appears
to have occurred as the Kevlar experienced around a 20% decrease in strength throughout all
exposure times; however, the loss of strength had no correlation to the amount of time it was
submerged in the water. If the samples had been left out to dry for long periods of time and all
of the moisture was evaporated, we would expect them to regain all of their strength because
aramid materials typically regain all of their strength when they are completely dried. In the
intended environments, the NCASE product will likely be exposed to cycles of wet and dry
conditions and while the strength may be decreased by 20% at times, it is expected to regain it
back when dry. In consideration of these factors, the Kevlar material is fully suitable and
acceptable for use in environments where water and moisture are prevalent.
79
4.2 Ultra Violet Exposure
4.2.1 Results
Figure 4-4—Nylon UV Weathering
Figure 4-5—Old Kevlar UV Weathering
80
Table 3—Statistical Significance of UV Weathering
Statistical Significance of UV Weathering α Value for Pr>|t| Weathering Time Old Kevlar Nylon 1 month 0.0421 <.0001 2 months 0.0008 <.0001
BOLD results are significant based on 95% confidence level
4.2.2 Analysis
Weathering testing was performed in Phase 1 to evaluate the Old Kevlar material and the
nylon material. As expected, the nylon material did not hold up well to UV exposure and its
strength was severely compromised. HPS had indicated they were skeptical about the
performance of the Nylon and simply wanted to do tests to confirm that the material is not
suitable for use, which it definitely is not.
The UV tested Kevlar samples had a mean that was statistically different from the control
set, simply meaning that the UV tested Kevlar samples were statistically different than the
control. A look at Figure 4-5 shows that after one month of UV exposure, the strength was
slightly lower, but after 2 months, the strength had increased above that of the control set! These
results show that while weathering may have a slight effect on the Kevlar, it is not anything that
would significantly compromise the performance of the materials and should be of no concern.
It may be of interest to perform tensile tests on material pulled from a product that has been in
service for an extended period of time in the intended environment to see if longer exposures
result in a decreased strength.
81
4.3 Freeze/Crinkle Exposure
4.3.1 Results
Figure 4-6—New Kevlar Freeze and Crinkle Tests
Figure 4-7—Failure Points on Crinkle Samples (shown in red circles)
82
Table 4—Statistical Significance of Freeze/Crinkle Testing
Statistical Significance of Freeze/Crinkle Testing α Value for Pr>|t| Freeze Time Crinkle Tested New Kevlar 1 Day NO 0.357 1 Day YES 0.124 2 Day YES 0.024 3 Day YES 0.069
BOLD results are significant based on 95% confidence level
4.3.2 Analysis
All crinkle and freeze samples passed the minimum criteria by retaining at least 50% of
their strength throughout each test and in most samples there was no statistically significant
difference between the samples and the control. The “crinkle” and “2-day freeze and crinkle”
samples were the only sets that had strengths that were statistically different than the control
(based on a 95% confidence level); however, the strength lost in each set was not more than
20%. Although there was not a consistency of reduced strength throughout the samples, it is still
important to note that material that is crinkled and under pressure may become slightly
weakened. An important observation during testing was that all breaks in the material occurred
precisely on one of the fold lines from the crinkle testing (see Figure 4-7 with break spots
outlined in red). While the material consistently maintained over 80% of its strength after being
folded and under the test weight, it would be wise to avoid situations where the NCASE product
is subject to sharp folds and excess force on those folds.
83
4.4 Webbing Strength
4.4.1 Results
Table 5—Webbing Test Results
Webbing Strength Test Results Minimum Strength Requirement (at 2G) 150 lbs Average Max. Tensile Strength 874 lbs
Standard Deviation 34 lbs Safety Factor 5.83
4.4.2 Analysis
In this specific application, the Army indicated that the product should be able to
withstand at least 2 G’s of force (twice the force of gravity). While any one section of the
product will not exceed 75 pounds, twice the force of its own weight will be 150 pounds, which
was the minimum acceptance criteria for this test. As the results show, the webbing that holds
the straps to the product was able to withstand a force of 874 pounds, thereby far exceeding the
acceptance criteria.
84
4.5 High Speed Impact Testing
4.5.1 Results
Table 6–Old Kevlar High Speed Impact Testing
Old Kevlar High Speed Impact Testing Ball Bearing Sharp Rock Speed (5 shots @ each speed) Control Weathered Control Weathered 50 mph ND ND ND ND 75 mph ND ND ND ND 100 mph ND ND ND ND 125 mph ND ND ND ND ND=No Damage
Table 7—New Kevlar High Speed Impact Testing
New Kevlar High Speed Impact Testing Ball Bearing Sharp Rock
Speed (5 shots @ each speed) Kevlar Kevlar
50 mph ND ND
75 mph ND ND
100 mph ND ND
125 mph ND ND
ND=No Damage
Table 8—New Kevlar Endurance Impact Testing
New Kevlar Endurance Impact Testing Ball Bearing Sharp Rock
Speed Kevlar Kevlar
100 shots @ 125 mph ND ND
(total of 200 shots per sample) ND=No Damage
85
4.5.2 Analysis
The impact testing samples did exceptionally well and there were no instances of the
internal powder packets rupturing or leaking throughout the entire set of tests. In the Phase 1
tests (Table 6), there were no indications of any damage throughout the entirety of the testing. In
Phase 2 (Table 7 & Table 8), the Kevlar material, again, did very well. In fact, it was very hard
to even notice that the Kevlar sample had been shot over 200 times with projectiles at speeds of
over 125mph.
86
5 CONCLUSION
5.1 Case Study Review: NCASE
After testing of the NCASE product was complete, full testing reports were created and
submitted to Hazard Protection Systems, Inc. These reports were not only used as internal
guides to validate the material selection and product design, but they were also submitted to their
main customer (the United States Army) to give them confidence in the NCASE product. The
fact that the materials, in general, far exceeded the Army’s minimum requirements was a great
selling point and further strengthened the trust that the Army had in the product. If there is any
doubt as to the validity of the results, the tests are documented in such a way that any other lab
can study the reports and replicate the tests.
These in-use testing reports can only help HPS in securing more contracts for outfitting
military vehicles with the NCASE product. Competitors with similar products do exist and HPS
must compete with them to win the bids from the military. Having highly documented tests on
the product and its materials can give HPS the edge over their competitors. HPS has secured
contracts for outfitting select vehicles that are deployed in Iraq and Afghanistan and they will
continue to work through the product development process in order to improve their product.
Throughout this cyclical, or helical, process, they will continue to rely on the testing process
outlined in this thesis as it has proven successful in their business model thus far.
87
5.2 New Outline for Products Development Process
A detailed investigation of proposed sequences for the new product development process
was conducted in Chapter 2. While the existing steps outlined in various literary sources had
solid principles, none of the cited sources adequately emphasized the importance of and
sequence of the testing process as a part of new product development. In Chapter 2, it was
suggested that the new product development steps as outlined by Crawford and Di Benedetto
should be modified to place more emphasis on the testing aspect of the process. Figure 5-1
shows the proposed process with the testing modification shown in a dashed box. This new
outline better emphasizes the importance testing should play in the new products development
process.
Figure 5-1—Proposed New Products Development Process
88
5.3 Testing Progression
While many sources stated the importance of prototyping and testing, few sources
actually discussed the logical steps in the testing process. This process was addressed in Chapter
2 and is shown again in Figure 5-2. It is important to note the sequence of steps from
standardized testing to in-use testing to actual product or market testing.
Figure 5-2—Progression of Testing
5.4 Guidelines for Product Testing
The objective of the thesis was as follows: “Through the analysis of the product testing
used in the case study, a set of guidelines can be developed to assist in a generalized method for
new product testing.” After a thorough literature review, and through looking at the case study,
certain guidelines were developed. These guidelines can be summarized as follows.
89
1. Decide on critical properties that need to be tested and develop a set of
acceptance criteria for evaluating the product. It is important to first decide
which product aspects are most important before moving forward with designing
tests. Once the most relevant product or material traits are identified, then it is
appropriate to select or develop a list of acceptance criteria either through market
research or through direct communication with the customer.
2. Begin by using standardized procedures, if possible, for your product tests.
With thousands of pre-established standardized tests from a variety of agencies,
there is a possibility that testing needs can be satisfied by a procedure that is
already established, saving time and money while maintaining the highest level of
credibility in the testing results.
3. When developing in-use tests, integrate components of standard tests to
maintain trust and credibility. If no standardized procedures fit your needs, use
aspects of standardized tests for the in-use testing procedures. Using key pieces
of standardized tests will strengthen the value of the in-use tests because the
standardized aspects of the testing procedures bring a higher level of credibility to
the results.
4. Decide on an appropriate statistical analysis for the results. Before
performing tests, it is critical to outline what metrics are wanted and how they
will be analyzed; it is critical that everybody who will see and use the results of
the test easily understands the results. Once the metrics and procedure for the
statistical analysis are identified, then it is appropriate to finalize testing design
and actually perform the tests.
90
5. Perform tests with accuracy and precision and document thoroughly. If tests
are not executed as they are documented, all trust in the testing procedures may be
lost. Results should also be analyzed exactly as specified. If tests are
documented properly and performed exactly as stated, others should be able to
obtain the same results even if performed in a separate lab.
6. Seek consensus within the industry and pursue standardization of the new
testing procedures. Once the tests are established, it may be of value to speak
with others in the industry to validate the procedures, get advice on how to
improve the process, or to see if others have developed similar tests. If desired,
contact a standards agency (e.g., ANSI) to have the test become an industry
standard. Standardizing an in-use test will provide a solid reputation for the
procedure, resulting in a much higher level of trust of the test.
5.5 Review of Hypothesis
This thesis has shown that a set of guidelines can be developed from a case study that will
improve the method of testing for new products.
5.6 Suggestions for Further Research
While this thesis focuses primarily on the testing aspect of the new development process,
it would be valuable to further explore the following:
• The process by which in-use tests become accepted and adopted by a sanctioning
body as a legitimate standardized test.
91
• A detailed investigation into the success of products that were previously tested
before launch versus products that were not tested.
92
REFERENCES
Abadir, M.S., and Reghbati, H.K. "Functional Testing of Semiconductor Random Access Memories." ACM Comput. Surv. 15, no. 3 (1983): 175-198.
ABC. "Deep Dive." In Nightline, 1999. absoluteastronomy.com, "Kevlar" http://www.absoluteastronomy.com/topics/Kevlar (accessed
May 3, 2010 2010). ANSI. "Ansi: Historical Overview", ansi.org
http://www.ansi.org/about_ansi/introduction/history.aspx?menuid=1 (accessed May 3, 2010 2010).
ASTM. "About Astm International", ASTM International
http://www.astm.org/ABOUT/aboutASTM.html (accessed April 14, 2010 2010). ASTM. "Standard Test Method for Breaking Strength and Elongation of Textile Fabrics (Grab
Test)." In Textiles, D 5034-95, 655-662. West Conshohocken, PA: ASTM, 2001. ASTM. "Standard Test Method for Impact Resistance of Flat, Rigid Plastic Specimens by Means
of a Falling Dart (Tup or Falling Mass)." In Plastics 8.03, D 5628-96, 629-638. West Conshohocken, PA: ASTM, 2001.
ASTM. "Standard Practices for Evaluating the Resistance of Plastics to Chemical Reagents." In
Plastics 8.01, D 543-95, 24-30. West Conshohocken, PA: ASTM, 2001. ASTM. "Standard Test Method for Tensile Properties of Plastics." In Plastics 8.01, D 638-00,
44-56. West Conshohocken, PA: ASTM, 2001. ASTM. "Standard Practice for Fluorescent Uv Exposure of Plastics." In Plastics 8.03, D 4329-
99, 1-5. West Conshohocken, PA: ASTM, 2001. ASTM. "Atmospheric Environmental Exposure Testing for Nonmetallic Materials." In
Laboratory Apparatus; Degradation of Materials; SI; Oxygen Fire Safety 14.04, G7. West Conshohocken, PA: ASTM, 2001.
93
Atlas. Laboratory Weathering. Powerpoint Presentation. 2002. Atlas. Fundamentals of Weathering. Powerpoint Presentation. 2002. Baber, Z. The Science of Empire: Scientific Knowledge, Civilization, and Colonial Rule in India.
Albany, NY: State University of New York, 1996. Di Benedetto, A., Crawford, M.. New Products Management, 9th Edition. New York, NY:
McGraw-Hill, 2008. Bent, D.S. "Characterizing Learning Prototype Factors within the Early Stages of the Product
Development Process." Brigham Young University, 1999. Bhattacharya, S. "Delivering the Product: Defining Specifications." In Handbook of New
Product Development Management, edited by Stylianos Kavadias Christoph H. Loch, 377-399. Burlington, MA: Elsevier Ltd., 2008.
Buggie, F.D. New Product Development Strategies. New York, NY: AMACOM, 1981. consumerreports.org, "About Us", Consumer's Union
http://www.consuerreports.org/cro/aboutus/mission/overview/index.htm (accessed April 14, 2010 2010).
Crawford, M. New Products Management. Third Edition ed. Homewood, IL: Irwin, 1991. Department of Defense. Dod Casualty Report, 2010. DIN, "About Us", German Institute for Standardization http://www.din.de/ (accessed April 14,
2010 2010). dpreview.com, "About the Site", Digital Photography Review
http://www.dpreview.com/misc/about.asp (accessed April 8, 2010 2010). EPA, "Fuel Economy Label" http://www.epa.gov/fueleconomy/label.htm (accessed May 7, 2010
2010). Fernandes, G., Fuschetto, J., Filipi, Z., Assanis, D., McKee, H. "Impact of Military Jp-8 Fuel on
Heavy-Duty Diesel Engine Performance and Emisions." Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering 221, no. 8 (2005): 957-970.
HPS, "Products and Services", Hazard Protection Services, Inc.
http://www.hazardprotection.com/#Products (accessed May 4, 2010 2010).
94
iexplore.com, "Afghanistan Weather"
http://www.iexplore.com/dmap/Afghanistan/Weather+and+Climate (accessed May 3, 2010 2010).
Iepxlpore.com, "Iraq Weather" http://www.iexplore.com/dmap/Iraq/Weather+and+Climate
(accessed May 3, 2010 2010). ISO, "How Iso Develops Standards", ISO
http://www.iso.org/iso/about/how_iso_develops_standards.htm (accessed April 15, 2010 2010).
ISO, "Discover Iso: The Scope of Iso's Work" http://www.iso.org/iso/about/discover-iso_the-
scope-of-isos-work.htm (accessed April 14, 2010 2010). Joinson, S. "It Doesn't Seem Like Yesterday...", Digital Photography Review
http://blog.dpreview.com/editorial/2008/12/a-decade-in-digital-cameras.html#more (accessed April 8, 2010 2010).
JSA, "History and Organization", Japanese Standards Association
http://www.jsa.or.jp/eng/about/about.asp (accessed May 3, 2010 2010). Parsons, L. J. "Product Design." In New-Product Development and Testing, edited by Michael
Menaso Walter Henry, Hirokazu Takada, 51-75. Lexington, Massachusetts: Lexington Books, 1989.
Berger, P.D., Maurer, R.E. Experimental Design with Applications in Management, Engineering,
and the Sciences. Belmont, CA: Wadsworth Group/Thompson Learning, 2002. Reinertsen, D. G. Managing the Design Factory: A Product Developer's Toolkit. New York, NY:
The Free Press, 1997. Bean, R., Bradford, R. Powerful Products: Strategic Management of Successful New Product
Development. New York, NY: AMACOM, 2000. Schaefgen, J. R. "Aramid Fibers: Structure, Properties, and Applications." In The Strength and
Stiffness of Polymers, edited by Roger S. Porter Anagnostis E. Zachariades, 327-355. New York, NY: Marcel Dekker, Inc., 1983.
Schilling, M. A., Hill, C.W.L. "Managing the New Product Development Process: Strategic
Imperatives." The Academy of Management Executive (1993) 12, no. 3 (1998): 67-81. World Meteorological Organization, "Bagdad, Iraq",
http://worldweather.wmo.int/154/c01464f.htm#climate (accessed April 7, 2010 2010).
95
Slade, B.N. Compressing the Product Development Cycle from Research to Marketplace. New
York, NY: Amacom, 1993. Strong, A.B. Plastics: Materials and Processing. 3rd ed. Upper Saddle River, NJ: Pearson
Prentice Hall, 2006. Strong, A.B. "History of Hps and Ncase." edited by Derek Egbert, 2010. Temponi, C., Yen, J., Tiao, W.A. "House of Quality: A Fuzzy Logic-Based Requirements
Analysis." European Journal of Operational Research 117 (1999): 340-354. Thomke, S. "Learning by Experimentation: Prototyping and Testing." In Handbook of New
Product Development Management, edited by Stylianos Kavadias Christoph H. Loch, 401-420. Burlington, MA: Elsevier, 2008.
Anderson, V.L., McLean, R.A. Design of Experiments: A Realistic Approach. New York, NY:
Macel Dekker, Inc., 1974. Westlake, A. "What's in a (Lens) Review?", Digital Photography Review
http://blog.dpreview.com/editorial/2008/12/whats-in-a-revi.html (accessed April 8, 2010 2010).
White, R. Consumer Product Development. London: Longman Group Limited, 1973. Yang, H. H. Kevlar Aramid Fiber. New York, NY: John Wiley & Sons, 1993.