cots metrics and modeling working group
DESCRIPTION
COTS Metrics and Modeling Working Group. Chris Abts, Vic Basili, Jim Cannon, Brad Clark, Nancy Eickelmann, Peter Hantos, Keun Lee, Gary Thomas, Lori Vaughn. What should I measure?. In what activities did I spend my time, money? In what phases did I spend my money? What problems are we having? - PowerPoint PPT PresentationTRANSCRIPT
COTS Metrics and Modeling Working Group
Chris Abts, Vic Basili, Jim Cannon, Brad Clark, Nancy Eickelmann, Peter Hantos, Keun Lee, Gary Thomas, Lori Vaughn
2© 2001 USC-CSE
What should I measure?
In what activities did I spend my time, money?• In what phases did I spend my money?• What problems are we having?• What advantages does the vender give us over our
internal staff?• What are the vender capabilities?• How do I characterize the environment?• How do I characterize the volatility of the system?
– How do I characterize the volatility of each COTS product?– How do I aggregate that volatility?
3© 2001 USC-CSE
SEI Activity Sets
• Engineering Activity Area Evaluation
– Construction
– Configuration Management
– Deployment and Sustainment
– System Context
– Architecture and Design
– Marketplace
• Business Activity Area Vendor Relationship
– Intergovernmental Supplier Relationships
– COTS Business Case
– COTS Cost Estimation
Ref: "An Activity Framework for COTS-Based Systems,” CMU/SEI-2000-TR-010, by Oberndorf, Brownsword, Sledge.
4© 2001 USC-CSE
Possible Effort Model Outputs
• Amount of time we should spend on evaluation– Product (effort)– Vendor (rating)
• Confidence in our assessment• Risk exposure associated with the decision
5© 2001 USC-CSE
Product Evaluation Effort Model Inputs(In what activities did I spend my time, money?)
How much effort does it take?– Number of potential products available– Technology maturity / volatility– Number of features to be evaluated– Number / complexity of interfaces– Number of other COTS products it has to interface to– Criticality to system– Organization maturity with COTS based-development– Vendor Rating
How much confidence do we have in our answers
6© 2001 USC-CSE
Vendor Relations Effort Model (What is the Vendor’s influence on my evaluation effort?)
Inputs– What size of the vendor’s market share does your need
represent?– Vendor’s market share– Vendor’s product maturity– Criticality of COTS product to system– Vendor responsiveness– Trade exposure– Vendor level of support (training, help desk, tech. support)
How much confidence do we have in our answers
7© 2001 USC-CSE
Risk Model for Exposure(How much resource should I invest?)
Effort Spent in Evaluation
Ris
k E
xpo
sure
8© 2001 USC-CSE
Evaluation Effort Model Discussion
Effort to evaluate = F (#potential products to evaluate, # interfaces, #features)
But most important factor is organization maturity with COTS
Meta Model (what have you got available)
9© 2001 USC-CSE
Priority Chart for Models
Difficulty
Imp
ort
an
ce
L M H
M
H
L
XRisk Exposure
X Product Evaluation XVendor Evaluation
XMeta Model
XSIZE
10© 2001 USC-CSE
Conclusion
• There is a lot we don’t know about how to conduct a COTS based project so that it ends successfully
• Modeling activity is very important to project success– How much time should I spend doing an activity?– What happens when I spend more or less time doing an
activity
• Building activity models is very hard– Hard to identify a primary relationship between effort and
something else (e.g. size and effort in COCOMO II)– Lots of different things affect effort in different activities– Little information is collected in addition to effort (if effort is
collected at all down to the activity level)