national institute of standards and technology 1 testing and validating oagi ndrs puja goyal salifou...
TRANSCRIPT
National Institute of Standards and
Technology
1
Testing and Validating OAGi NDRs
Puja Goyal
Salifou Sidi
Presented to OAGi
April 30th, 2008
National Institute of Standards and
Technology
2
Outline
• Naming and Design Rules• Quality of Design Tool (QOD)• OAGi NDR v9.0 Encoding
– Testability
– Observations
– Demo
• NDR Authoring/Sharing• What’s Next
National Institute of Standards and
Technology
3
NDR: Naming and Design Rules
• Rules applied when creating standards for the exchange of information– Any of a series of rules documents since 2003
that restrict XML Schema flexibility– Indicate features which should, may, may not,
should not, must, or must not be used.
National Institute of Standards and
Technology
4
What’s in an NDR?
• Naming Rules: – Names (or tags) represent information to be
exchanged– Subject names to consistent naming practices
for clarity and consistency in an XML schema.
• Design Rules– Address how information is structured
• Superstructures: namespaces
• Microstructures: global vs. local elements
National Institute of Standards and
Technology
5
Benefits of Encoding Rules
• NDRs written in English– Easy to misinterpret
– Schema testing and validation not automated
• Encoded NDRs– Result in better rules
• Verifies rule is clear
• Validates rule is being used as intended
– Can be computationally enforced
– Cycle time from requirements for a schema to production XML is reduced
National Institute of Standards and
Technology
6
QOD – Apply rules directly
• Interface to write Test Requirements (rules) and associate with Test Cases (encoded rules)– Schematron– Java-Expert System Shell (JESS)
• Automate testing and validation against schema(s)– Displays and stores results from execution against
Schematron/JESS
• Repository to store rules– Easy to reuse– Group rules to create Test Profile
National Institute of Standards and
Technology
9
OAGi NDR v9.0 Encoding
• Rule Testability
• Additional observations
• Demo
National Institute of Standards and
Technology
10
Rule Testability
• Fully Testable – Rule can be encoded without any assumptions
• Partially Testable– Some assumptions are needed
• Not Testable– unable to encode the rule
National Institute of Standards and
Technology
11
Testability of OAGi Rules
OAGi Rules Testability
53%
13%
34%
Fully Testable
Partially Testable
Not Testable
National Institute of Standards and
Technology
12
Not Testable Rules – From Our Perspective
• QOD Limitations– Related to XML instance
• Lack of clarity
• Nondeterministic– Lack of information in the OAGi schemas
• Documentation or Guidance– Supplementary information
National Institute of Standards and
Technology
13
Examples: OAGi-76, OAGi-70
- Ambiguous: OAGi-76 : “xsd:notation MUST NOT be used.”
– Which notation must not be used? xsd:notation element or xsd:NOTATION type ?
- Nondeterministic: OAGi-70
“Minor version schema MUST incorporate all XML constructs from the immediately preceding major or minor version schema” - No minor version information in OAGi schemas
National Institute of Standards and
Technology
14
Additional Observations
• Overlaps– “Fully Overlapped”
• Generalization vs. Restriction
• Generic rule may be removed for same results
– “Partially Overlapped”• Occasionally can combine rules if same type of schema
• Links (one rule refers to another)– For testing purposes, all referenced rules have to be
included
AB
Rule ARule B
National Institute of Standards and
Technology
15
Overlaps between rules
• Example :
OAGi-11 “XML element, attribute and type names constructed from dictionary entry names MUST NOT include periods, spaces, or other separators; or characters not allowed by W3C XML 1.0 for XML names.”
OAGi-10 “Element, attribute and type names MUST be drawn from the following set: a – z and A – Z.”
• OAGi-10 is restriction of OAGi-11
• OAGi-11 can be deleted from the NDR without affecting the original intention
National Institute of Standards and
Technology
17
Demo
• Overview of the QOD main functionalities (Public vs Private)
• Demo Test Profile :
- Rules related to the BOD
- Rules related to the usage of annotation and documentation
elements
• Schema to be tested:
- OAGi Developer BOD version 9.0
• Full Test Results of the execution of OAGi Developer BOD schemas version 9.0, 9.1 and 9.2 against the rules that we have encoded.
National Institute of Standards and
Technology
18
OAGi Developer BOD Testing :
Materials
• Schemas tested- Developer BOD schemas version 9.0 : 434 files
- Developer BOD schemas version 9.1 : 434 files
- Developer BOD schemas version 9.2 : 494 files
• Profiles tested- General Profile : 25 rules
- BOD Profile : 5 rules
National Institute of Standards and
Technology
19
General Profile BOD Profile TotalDeveloper BOD v9.0 errors
Developer BOD v9.1 errors
Developer BOD v9.2 errors
Rule OAGi 90 : 1 error found
- GetReceiveItem.xsd
Rule OAGi 98 : 6 errors found
- ShowCustomerPartyMaster.xsd
- ShowPartyMaster.xsd
- ShowPayable.xsd
- ShowSupplierPartyMaster.xsd
- SyncCatalog.xsd
- SyncSupplierPartyMaster.xsd
Rule OAGi 100 : 6 errors found
- GetLocationService.xsd
- ShowLocationService.xsd
Error details
Rule OAGi 98 : 6 errors found
- ShowCustomerPartyMaster.xsd
- ShowPartyMaster.xsd
- ShowPayable.xsd
- ShowSupplierPartyMaster.xsd
- SyncCatalog.xsd
- SyncSupplierPartyMaster.xsd
Error details
13
6
0
2
1
0
Rule OAGi 31 : 2 errors found
- GetReceiveItem.xsd
- GetRouting.xsd
Rule OAGi 31 : 1 error found
- GetRouting.xsd
15
7
0
OAGi Developer BOD Testing : Results
National Institute of Standards and
Technology
20
NDR Sharing Environment
• Facilitate reuse of rules– Different groups have different requirements
but some overlap– Group relevant tests to create customized test
profile– Upload new tests to share with others
National Institute of Standards and
Technology
21
NDR Authoring Environment
• Capability to export/import rules– Edit rules offline – Display in multiple ways
• HTML, XML, PDF, Spreadsheet, DocBook, etc
– Customize NDR to fit your needs– Based on NDR Profile Schema
• Standard structured vocabulary for describing NDR rules
• Available: http://ecc02.eccnet.com/ndr/
National Institute of Standards and
Technology
26
What’s next
• Create stand alone system
• Enhance support for rule sharing by including rule traceability
• Pursue similar support for developing tests for constraints on instance data
• Incorporate feedback into our tools!
National Institute of Standards and
Technology
28
Contacts• KC Morris , NIST – [email protected] • Simon Frechette, NIST – [email protected]• Josh Lubell, NIST – [email protected] • Puja Goyal, NIST –[email protected]• Salifou Sidi, NIST – [email protected]• Betty Harvey, ECC, Inc. -- [email protected]
Websites
• QOD onlinehttp://syseng.nist.gov/b2bTestbed/projects/QOD/
• Developer’s site for QOD tool and executable rules: http://qod.sourceforge.net/
• XML Tools available from MSID: http://www.mel.nist.gov/msid/xml_related.htm
• DAS XSI Working Group: https://www.collab.core.gov/CommunityBrowser.aspx?id=2234