Introducing Natural Language Program Analysis
Lori Pollock, K. Vijay-Shanker, David Shepherd,
Emily Hill, Zachary P. Fry, Kishen Maloor
NLPA Research Team Leaders
Lori Pollock“Team Captain”
K. Vijay-Shanker“The Umpire”
ProblemModern software is large and complex
object oriented class hierarchy
Software development tools are needed
Successes in Software Development Tools
object oriented class hierarchy
Good with local tasks
Good with traditional structure
object oriented class hierarchy
Scattered tasks are difficult
Programmers use more than traditional program structure
Issues in Software Development Tools
public interface Storable{...
activate tool
save drawing
update drawing
undo action
public void Circle.save()
//Store the fields in a file....
object oriented system
Key Insight: Programmers leave natural language clues that
can benefit software development tools
Observations in Software Development Tools
Studies on choosing identifiers
Impact of human cognition on names [Liblit et al. PPIG 06] Metaphors, morphology, scope, part of speech hints Hints for understanding code
Analysis of Function identifiers [Caprile and Tonella WCRE 99] Lexical, syntactic, semantic Use for software tools: metrics, traceability, program understanding
Carla, the compiler writer Pete, the programmer
I don’t care about names.
So, I could use x, y, z. But, no one
will understandmy code.
Our Research Path
[MACS 05, LATE 05]
[AOSD 06]
[ASE 05, AOSD 07, PASTE 07]
Motivated usefulness of exploiting natural language (NL) clues in toolsDeveloped extraction process and an NL-
based program representationCreated and evaluated a concern
location tool and an aspect miner with NL-based analysis
pic
Name: David C ShepherdNickname: Leadoff HitterCurrent Position: PhD May 30, 2007Future Position: Postdoc, Gail Murphy
StatsYear coffees/day redmarks/paper draft2002 0.1 5002007 2.2 100
Aspect Mining
Aspect-Oriented Programming
Aspect Mining TaskLocate refactoring
candidates
Applying NL Clues for
Molly, the Maintainer
How can I fix Paul’s
atrocious code?
Timna: An Aspect Mining Framework [ASE 05]
Uses program analysis clues for mining Combines clues using machine learning Evaluated vs. Fan-in Precision (quality) and Recall (completeness)
P R 37 2 62 60
Fan-InTimna
iTimna (Timna with NL) Integrates natural language cluesExample: Opposite verbs (open and close)
P R 37 2 62 60 81 73
Fan-InTimna iTimna
Integrating NL Clues into Timna
Natural language information increases the effectiveness of Timna[Come back Thurs 10:05am]
Concern Location
60-90% software costs spent on reading and navigating code for maintenance*
(fixing bugs, adding features, etc.)
*[Erlikh] Leveraging Legacy System Dollars for E-Business
Applying NL Clues for
Motivation
Key Challenge: Concern Location
Find, collect, and understand all source code related to a particular concept
Concerns are often crosscutting
State of the Art for Concern Location
Mining Dynamic Information [Wilde ICSM 00]
Program Structure Navigation [Robillard FSE 05, FEAT, Schaefer ICSM 05]
Search-Based Approaches RegEx [grep, Aspect Mining Tool 00]
LSA-Based [Marcus 04]
Word-Frequency Based [GES 06]
Reduced to similar problem
Slow
Fast
Fragile
Sensitive
No Semantics
Limitations of Search Techniques
1. Return large result sets
2. Return irrelevant results
3. Return hard-to-interpret result sets
The Find-Concept Approach
concept
Find-ConceptConcrete query
Recommendations
Source Code
Method a
Method bMethod c
Method d Method e
NL-basedCode Rep
Result GraphNatural
Language Information
1. More effective search
2. Improved search terms
3. Understandable results
Underlying Program Analysis
Action-Oriented Identifier Graph (AOIG) [AOSD 06] Provides access to NL information Provides interface between NL and traditional
Word Recommendation Algorithm NL-based
Stemmed/Rooted: complete, completing Synonym: finish, complete
Combining NL and Traditional Co-location: completeWord()
Experimental Evaluation
Research Questions Which search tool is most effective at forming and
executing a query for concern location? Which search tool requires the least human effort to form
an effective query?
Methodology: 18 developers complete nine concern location tasks on medium-sized (>20KLOC) programs
Measures:Precision (quality), Recall (completeness), F-Measure (combination of both P & R)
Find Concept, GES, ELex
Overall Results
Effectiveness FC > Elex with statistical
significance FC >= GES on 7/9 tasks FC is more consistent than GES
Effort FC = Elex = GES
FC is more consistent and more effective in experimental study without requiring more effort
Across all tasks
Natural Language Extraction from Source Code
Key Challenges:Decode name usageDevelop automatic extraction
processCreate NL-based program
representation
Molly, the Maintainer
What was Pete thinking
when he wrote this code?
Natural Language: Which Clues to Use?
Software MaintenanceTypically focused on actionsObjects are well-modularized
Maintenance Requests
Natural Language: Which Clues to Use?
Software MaintenanceTypically focused on actionsObjects are well-modularized
Focus on actions Correspond to verbsVerbs need Direct Object
(DO)
Extract verb-DO pairs
Extracting Verb-DO Pairs
Two types of extractionclass Player{ /** * Play a specified file with specified time interval */ public static boolean play(final File file,final float fPosition,final long length) { fCurrent = file; try { playerImpl = null; //make sure to stop non-fading players stop(false); //Choose the player Class cPlayer = file.getTrack().getType().getPlayerImpl(); …}
Extraction from comments
Extraction from method signatures
public UserList getUserListFromFile( String path ) throws IOException {
try {
File tmpFile = new File( path );
return parseFile(tmpFile);
} catch( java.io.IOException e ) {
throw new IOrException( ”UserList format issue" + path + " file " + e );
}
}
Extracting Clues from Signatures
1. POS Tag Method Name
2. Chunk Method Name
3. Identify Verb and Direct-Object (DO)
get<verb> User<adj> List<noun> From <prep> File <noun>
get<verb phrase> User List<noun phrase> From File <prep phrase>
POS Tag
Chunk
pic
Name: Zak FryNickname: The RookieCurrent Position: Upcoming seniorFuture Position: Graduate School
StatsYear diet cokes/day lab days/week2006 1 22007 6 8
Developing rules for extraction
For many methods: Identify relevant verb (V)
and direct object (DO) in method signature
Classify pattern of V and DO locations
If new pattern, create new extraction rule
verbDO
verb DO
verbDO
Our Current Extraction Rules
4 general rules with subcategories:
URL parseURL()
void mouseDragged()
void Host.onSaved()
Left Verb
Right Verb
Generic Verb
Unidentified Verb
void message() message-
hostsaved
mousedragged
URLparse
DOVerb
Example: Sub-Categories for Left-Verb General Rule
Look beyond the method name:
Parameters, Return type, Declaring class name, Type hierarchy
Subcategory1) Standard left verb 2) No DO in method name; has parameters; non object return type3) No DO in method name; no parameters; no return type4) Creational left verb; has return type5) No DO in method name; has parameters; return type is more specific than parameters in type hierarchy6) No DO in method name; parameters are more specific than parameters in type hierarchy
2) No DO in method name; has parameters; non object return type
Verb-DO pair:
<remove, UserID>Left
Verb
Representing Verb-DO Pairs
Action-Oriented Identifier Graph (AOIG)
verb1 verb2 verb3 DO1 DO2 DO3
verb1, DO1 verb1, DO2 verb3, DO2 verb2, DO3
source code files
use
use
use
use
use
use
useuse
Action-Oriented Identifier Graph (AOIG)
play add remove file playlist listener
play, file play, playlist remove, playlist add, listener
source code files
use
use
use
use
use
use
useuse
Representing Verb-DO Pairs
Evaluation of Extraction Process
Compare automatic vs ideal (human) extraction 300 methods from 6 medium open source programs Annotated by 3 Java developers
Promising Results Precision: 57% Recall: 64%
Context of Results Did not analyze trivial methods On average, at least verb OR direct object obtained
pic
Name: Emily Gibson HillNickname: Batter on DeckCurrent Position: 2nd year PhD StudentFuture Position: PhD Candidate
StatsYear cokes/day meetings/week2003 0.2 12007 2 5
Program Exploration
Purpose: Expedite software maintenance and program comprehension
Key Insight: Automated tools can use program structure and identifier names to save the developer time and effort
Ongoing work:
Dora the Program Explorer*
* Dora comes from exploradora, the Spanish word for a female explorer.
DoraDora
Natural Language Query• Maintenance request• Expert knowledge• Query expansion
Natural Language Query• Maintenance request• Expert knowledge• Query expansion
Relevant Neighborhood
Program Structure• Representation
• Current: call graph• Seed starting point
Relevant Neighborhood• Subgraph relevant to query
Query
State of the Art in Exploration
Structural (dependence, inheritance) Slicing Suade [Robillard 2005]
Lexical (identifier names, comments) Regular expressions: grep, Eclipse search Information Retrieval: FindConcept,
Google Eclipse Search [Poshyvanyk 2006]
Motivating need for structural and lexical information
Program: JBidWatcher, an eBay auction sniping program
Bug: User-triggered add auction event has no effect
Task: Locate code related to ‘add auction’ trigger
Seed: DoAction() method, from prior knowledge
ExampleScenario
DoNada() DoNada() DoNada() DoNada() DoNada()DoNada() DoNada()DoNada()DoNada() DoNada() DoNada()
DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
DoNada() DoNada()DoNada() DoNada() DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
Using only structural information
DoAction() has 38 callees, only 2/38 are relevant Relevant
Methods
Irrelevant Methods
Looking for: ‘add auction’ trigger
DoAction()
DoAdd()
DoPasteFromClipboard()
And what if you wanted to explore more than one edge away?
Locates locally relevant items, but many irrelevant
Using only lexical information
50/1812 methods contain matches to ‘add*auction’ regular expression query
Only 2/50 are relevant
Locates globally relevant items, but many irrelevant
Looking for: ‘add auction’ trigger
DoNada() DoNada() DoNada() DoNada() DoNada()DoNada() DoNada()DoNada()DoNada() DoNada() DoNada()
DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
DoNada() DoNada()DoNada() DoNada() DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()DoNada()
Combining Structural & Lexical Information Structural: guides exploration
from seed
Looking for: ‘add auction’ trigger
RelevantNeighborhood
DoAction()
DoPasteFromClipboard()
DoAdd()
Lexical: prunes irrelevant edges
The Dora Approach
Determine method relevance to queryCalculate lexical-based relevance score
Low-scored methods pruned from neighborhood
Recursively explore
Prune irrelevant structural edges from seed
Calculating Relevance Score:Term Frequency Score based on query term frequency of the method
6 query term 6 query term occurrencesoccurrences6 query term 6 query term occurrencesoccurrences
Only 2 Only 2 occurrencesoccurrences
Only 2 Only 2 occurrencesoccurrences
Query: ‘add auction’
Weigh term frequency based on location: Method name more important than body Method body statements normalized by length
Calculating Relevance Score:Location Weights Query: ‘add auction’
?
Dora explores ‘add auction’ trigger
From DoAction() seed:Correctly identified at 0.5 threshold
DoAdd() (0.93)DoPasteFromClipboard() (0.60)
With only one false positiveDoSave() (0.52)
Summary
NL technology usedSynonyms, collocations, morphology, word frequencies, part-of-speech tagging, AOIG
Evaluation indicatesNatural language information shows promise for improving software development tools
Key to successAccurate extraction of NL clues
Our Current and Future Work
Basic NL-based tools for softwareAbbreviation expanderProgram synonymsDetermining relative importance of words
Integrating information retrieval techniques
Posed Questions for Discussion
What open problems faced by software tool developers can be mitigated by NLPA?
Under what circumstances is NLPA not useful?
