the art and science of grammar coverage analysis
Post on 20-May-2015
1.144 Views
Preview:
DESCRIPTION
TRANSCRIPT
The Art and Science ofGrammar Coverage Analysis
Dominique BoucherNu Echo Inc.dominique.boucher@nuecho.com
SpeechTEK 2009New York, USA
The grammar development processEnter / get initial set of sentences in coverage set
Write initial grammar to cover
utterances
Run coverage tests
Fix grammaryesProblems?no
Enrich coverage set
Generate sentences
Objective: To obtain:(a) A complete coverage set (b) A grammar that covers the coverage set
and produces the correct semantic result
The importance of coverage analysis
• Ensure grammars conform to their specification• Semantic tags testing• Ensure grammars conform to their specification• Semantic tags testing
Design and development
• Grammars evolve over the life of an application• Provides an effective tool for testing that a grammar
isn’t accidentally broken by a grammar change
• Grammars evolve over the life of an application• Provides an effective tool for testing that a grammar
isn’t accidentally broken by a grammar change
Maintenance and optimization
• Ensure proper conversion between grammar formats
• Ensure proper conversion between grammar formats
Conversion projects
Grammar coverage challenges
Provide exhaustive coverage of all sentence patterns… … with the smallest possible set of sentences.
Otherwise:– Analysis of generated sentences will be time-consuming; and– Errors will go undetected
Two complementary techniques
#1 Sentence
Generation
#2 Sentence patterns
Exploration
Technique #1Sentence generation
Tools usually provided by the ASR engine SDK– Operate on the source or compiled grammar
Commonly used generation strategies– Exhaustive generation– Generation of a fixed number of random sentences
Technical difficulties
Some grammars generate an infinite number of sentences– Exhaustive generation not possible
Semantic tags not all tested– Errors may remain undetected until application run time
All interesting cases may not be covered by the generated sentences
Uninteresting patterns are generated over and over
Sentence generation best practices
Avoid generating redundant sentences– Powerful sentence generation tools makes this possible– Too many sentences increases the risk of errors going
undetected
Carefully examine generated sentences– If a sentence doesn’t look right, it probably shouldn’t be in
there (although, to be sure, look at the parse tree)
Make sure the coverage test is as complete as possible– Should include all semantic tags and all sentence patterns– Full coverage is best (whenever possible)
Sentence generation revisited
Individual rule configuration
More effective strategies – Tags coverage– All grammar paths– Pick from @examples– Use fixed sentence
Generation can be started from any set of sentence patterns
Sentence generation tool
Interactive Demo
Technique #2Exploring sentence patterns
Interactive expansion of grammar rules
Derive sentence patterns– Useful to generate
sentences for a specific pattern
Derive complete sentences– Can be debugged, etc.
Ideal to understand the structure of a grammar
Sentence explorer tool
Interactive Demo
Three typical problems…… and how to fix them!
Interactive Demo
top related