![Page 1: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/1.jpg)
The Art and Science ofGrammar Coverage Analysis
Dominique BoucherNu Echo [email protected]
SpeechTEK 2009New York, USA
![Page 2: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/2.jpg)
The grammar development processEnter / get initial set of sentences in coverage set
Write initial grammar to cover
utterances
Run coverage tests
Fix grammaryesProblems?no
Enrich coverage set
Generate sentences
Objective: To obtain:(a) A complete coverage set (b) A grammar that covers the coverage set
and produces the correct semantic result
![Page 3: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/3.jpg)
The importance of coverage analysis
• Ensure grammars conform to their specification• Semantic tags testing• Ensure grammars conform to their specification• Semantic tags testing
Design and development
• Grammars evolve over the life of an application• Provides an effective tool for testing that a grammar
isn’t accidentally broken by a grammar change
• Grammars evolve over the life of an application• Provides an effective tool for testing that a grammar
isn’t accidentally broken by a grammar change
Maintenance and optimization
• Ensure proper conversion between grammar formats
• Ensure proper conversion between grammar formats
Conversion projects
![Page 4: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/4.jpg)
Grammar coverage challenges
Provide exhaustive coverage of all sentence patterns… … with the smallest possible set of sentences.
Otherwise:– Analysis of generated sentences will be time-consuming; and– Errors will go undetected
![Page 5: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/5.jpg)
Two complementary techniques
#1 Sentence
Generation
#2 Sentence patterns
Exploration
![Page 6: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/6.jpg)
Technique #1Sentence generation
Tools usually provided by the ASR engine SDK– Operate on the source or compiled grammar
Commonly used generation strategies– Exhaustive generation– Generation of a fixed number of random sentences
![Page 7: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/7.jpg)
Technical difficulties
Some grammars generate an infinite number of sentences– Exhaustive generation not possible
Semantic tags not all tested– Errors may remain undetected until application run time
All interesting cases may not be covered by the generated sentences
Uninteresting patterns are generated over and over
![Page 8: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/8.jpg)
Sentence generation best practices
Avoid generating redundant sentences– Powerful sentence generation tools makes this possible– Too many sentences increases the risk of errors going
undetected
Carefully examine generated sentences– If a sentence doesn’t look right, it probably shouldn’t be in
there (although, to be sure, look at the parse tree)
Make sure the coverage test is as complete as possible– Should include all semantic tags and all sentence patterns– Full coverage is best (whenever possible)
![Page 9: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/9.jpg)
Sentence generation revisited
Individual rule configuration
More effective strategies – Tags coverage– All grammar paths– Pick from @examples– Use fixed sentence
Generation can be started from any set of sentence patterns
![Page 10: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/10.jpg)
Sentence generation tool
Interactive Demo
![Page 11: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/11.jpg)
Technique #2Exploring sentence patterns
Interactive expansion of grammar rules
Derive sentence patterns– Useful to generate
sentences for a specific pattern
Derive complete sentences– Can be debugged, etc.
Ideal to understand the structure of a grammar
![Page 12: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/12.jpg)
Sentence explorer tool
Interactive Demo
![Page 13: The Art and Science of Grammar Coverage Analysis](https://reader034.vdocuments.site/reader034/viewer/2022052619/555c0377d8b42a56448b53d8/html5/thumbnails/13.jpg)
Three typical problems…… and how to fix them!
Interactive Demo