exploiting context in dealing with programming errors and exceptions
DESCRIPTION
Exploiting Context in Dealing with Programming Errors and Exceptions. Mohammad Masudur Rahman Department of Computer Science University of Saskatchewan. Exception triggering point. Exception: a common experience!!. Exception Debugging. Web search!!. - PowerPoint PPT PresentationTRANSCRIPT
EXPLOITING CONTEXT IN DEALING WITH PROGRAMMING ERRORS AND EXCEPTIONSMohammad Masudur Rahman
Department of Computer Science
University of Saskatchewan
2
Softw
are
Rese
arch
Lab, U
of S
Exception: a common experience!!
Exception triggering point
3
Softw
are
Rese
arch
Lab, U
of S
EXCEPTION DEBUGGING
Not a helpful message for understanding or solving the
exception
Web search!!
4
Softw
are
Rese
arch
Lab, U
of S
SOLVING EXCEPTION (STEP I: WEB SEARCH)
The browser does not know the context (i.e., details) of the exception.
Not much helpful ranking Hundreds of search results Forces the developer to switch back and forth
between IDE and browser. Trial and error in searching
Switching is often
distracting
5
Softw
are
Rese
arch
Lab, U
of S
SOLVING EXCEPTION (STEP II: QUERY SELECTION)
Selection of traditional search query
Switching to web browser for web search
This query may not be sufficient enough for most of the exceptions
6
SOLVING EXCEPTION (STEP III: POST-SEARCH ANALYSIS)
• Only the most relevant section needs to be checked to determine the relevance of a page.• Frequent switching between IDE and web browser for content analysis• Manual analysis of a number of web pages is non-trivial and time-consuming
1
2
3
4
7
SOLVING EXCEPTION (STEP IV: HANDLING EXCEPTIONS)
Only adds a generic handler (i.e., printStackTrace()) for the exception. Not much helpful for effective handling of the exception
8
Softw
are
Rese
arch
Lab, U
of S
OBSERVATIONS ON TRADITIONAL/AD-HOC APPROACH FOR EXCEPTION SOLVING
Step I: Web search may not be much effective or reliable
Step II: Support for search query formulation is not enough
Step III: Support for post-search analysis is insufficient
Step IV: Support for exception handling is not enough
9
DEALING WITH ERRORS & EXCEPTIONS:
PROPOSED APPROACH
10
CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013, CSMR/WCRE 2014)
(2) QueryClipse (ICSME 2014)
(3) ContentSuggest
(4) SurfExample (SCAM 2014)
(5) ExcClipse
Web search
Query formulation
Post-search analysis
Exception handling
User study
11
Softw
are
Rese
arch
Lab, U
of S
TECHNICAL DETAILS & PROGRAMMING CONTEXT OF AN EXCEPTION
Programming context (i.e., context code)
Error message
Stack trace
1
2
3
12
SURFCLIPSE: IDE-BASED CONTEXT-AWARE META SEARCH ENGINE
13
Softw
are
Rese
arch
Lab, U
of S
MOTIVATIONAL EXPERIMENT
75 programming exceptions (details later) Individual engine can provide solutions for at
most 58 exceptions & each has some unique results.
Combination of content & context is always better than content only
Search Query
Common for All
Google Unique
Yahoo Unique
Bing Unique
Content Only 32 09 16 18
Content and Context
47 09 11 10
14
Softw
are
Rese
arch
Lab, U
of S
THE KEY IDEA !! META SEARCH ENGINE
Fig: Meta search engine
15
Softw
are
Rese
arch
Lab, U
of S
PROPOSED IDE-BASED META SEARCH MODEL
Start
Results
Web page
16
Softw
are
Rese
arch
Lab, U
of S
PROPOSED IDE-BASED META SEARCH MODEL
Distinguished Features (5) IDE-Based solution
Web search, search result and web browsing all from IDE No context-switching needed
Meta search engine Captures data from multiple search engines Also applies custom ranking techniques
Context-Aware search Uses stack traces information Uses context-code (surroundings of exception locations)
Software As A Service (SAAS) Search is provided as a web service, and can be
leveraged by an IDE. http://srlabg53-2.usask.ca/wssurfclipse/
17
Softw
are
Rese
arch
Lab, U
of S
PROPOSED IDE-BASED META SEARCH MODEL
Two Working Modes Proactive Mode
Auto-detects the occurrence of an exception Initiates search for exception by client itself Aligned with Cordeiro et al. (RSSE’ 2012) & Ponzanelli et
al. (ICSE 2013)
Interactive Mode Developer starts search using context menu Also facilitates keyword-based search Aligned with traditional web search within the IDE
18
Softw
are
Rese
arch
Lab, U
of S
PROPOSED METRICS & SCORES
Content Matching Score (Scms) Cosine similarity based measurement
Stack trace Matching Score (Sstm) Structural and lexical similarity measurement of
stack traces
Code context Matching Score (Sccx) Code snippet similarity (code clones)
StackOverflow Vote Score (Sso) Total votes for all posts in the SO result link
19
Softw
are
Rese
arch
Lab, U
of S
PROPOSED METRICS & SCORES
Site Traffic Rank Score (Sstr)-- Alexa and Compete Rank of each link
Search Engine weight (Ssew)---Relative reliability or importance of each search engine. Experiments with 75 programming queries against the search engines.
Heuristic weights of the metrics are determined through controlled experiments.
21
Softw
are
Rese
arch
Lab, U
of S
EXPERIMENT OVERVIEW
75 exceptions
Eclipse plug-in & Java development
Gold set solutions Peers
22
Softw
are
Rese
arch
Lab, U
of S
RESULTS ON DIFFERENT RANKING ASPECTSScore Components
Metrics
Proactive Mode (Top 30)
Interactive Mode (Top 30)
Content MPTEFR
0.037156 (75)74.66%
0.048165 (75)86.66%
Content +Context
MPTEFR
0.037655 (75)73.33%
0.051466 (75)88.00%
Content + Context + Popularity
MPTEFR
0.038156 (75)74.66%
0.051966 (75)88.00%
Content +Context + Popularity +Confidence
MPTEFR
0.038056 (75)74.66%
0.053868 (75)90.66%
[ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]
23
Softw
are
Rese
arch
Lab, U
of S
COMPARISON WITH EXISTING APPROACHES
Recommender Metrics Top 10 Top 20 Top 30
Cordeiro et al. (only stack traces)
MPTEFR
0.020215 (75)20.00%
0.012818 (75)24.00%
0.008518 (75)24.00%
Proposed Method (Proactive Mode)
MPTEFR
0.088651 (75)68.00%
0.052955 (75)73.33%
0.038056 (75)74.66%
Ponzanelli et al. (only context-code)
MPTEFR
0.02437 (37)18.92%
0.01357 (37)18.92%
0.00997 (37)18.92%
Proposed Method (Proactive Mode)
MPTEFR
0.100030 (37)81.08%
0.062132 (37)86.48%
0.045032 (37)86.48%
[ MP = Mean Precision, R = Recall, TEF= Total Exceptions Fixed]
24
Softw
are
Rese
arch
Lab, U
of S
COMPARISON WITH TRADITIONAL SEARCH ENGINESSearch Engine Metrics Top 10 Top 20 Top 30
Google MPTEFR
0.157157 (75)76.00%
0.086457 (75)76.00%
0.058057 (75)76.00%
Bing MPTEFR
0.101355 (75)73.33%
0.053358 (75)77.33%
0.036458 (75)77.33%
Yahoo! MPTEFR
0.098654 (75)72.00%
0.053957 (75)76.00%
0.036957 (75)76.00%
StackOverflow Search
MPTEFR
0.022614 (75)18.66%
0.014017 (75)22.66%
0.009717 (75)22.66%
Proposed Method (Interactive mode)
MPTEFR
0.122959 (75)78.66%
0.073664 (75)85.33%
0.053868 (75)90.66%
25
CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013, CSMR/WCRE 2014)
(2) QueryClipse (ICSME 2014)
(3) ContentSuggest
(4) SurfExample (SCAM 2014)
(5) ExcClipse
Web search
Query formulation
Post-search analysis
Exception handling
User study
26
QUERYCLIPSE: CONTEXT-AWARE SEARCH QUERY RECOMMENDER
27
Softw
are
Rese
arch
Lab, U
of S
MOTIVATING EXAMPLE
Stack traceContext code
QueryClipse
21
Recommended search queries
28
Softw
are
Rese
arch
Lab, U
of S
PROPOSED CONTEXT-AWARE QUERY RECOMMENDATION APPROACH
Distinguishing Features (4) Context-Aware Query
Exploits both stack trace and context code Extract search keywords carefully and systematically
Ranked List for Queries Ranked list based on keyword importance in queries Automatic suggestion through auto-completion
Custom Search Query Stack trace graph based on implied relationships Keyword importance based on network connectivity
Query Length Customization Number of query keywords customizable Search-friendly & easily applicable for any search
engines
29
CONTENTSUGGEST: CONTEXT-AWARE PAGE CONTENT RECOMMENDER
30
MOTIVATING EXAMPLE
1
23
Only the most relevant page section displayed Less information overhead, less effort required No need to browse the page for relevance checking
Softw
are
Rese
arch
Lab, U
of S
31
Softw
are
Rese
arch
Lab, U
of S
PROPOSED CONTEXT-AWARE PAGE CONTENT SUGGESTION APPROACH
Distinguishing Features (3) Relevant section(s) suggestion
Analyzes both quality and relevance of the content Exploits stack trace & context code for relevance
checking Partial automation in post-search analysis
Less content, less overhead Need to analyze less content for page relevance
checking Displayed content more useful than meta description
Noise-free version of the page Removes advertisements, irrelevant widgets and so on Applies link-based heuristics Returns a noise-free version of the web page
32
CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013, CSMR/WCRE 2014)
(2) QueryClipse (ICSME 2014)
(3) ContentSuggest
(4) SurfExample (SCAM 2014)
(5) ExcClipse
Web search
Query formulation
Post-search analysis
Exception handling
User study
33
SURFEXAMPLE: CONTEXT-AWARE CODE EXAMPLE RECOMMENDER FOR
EXCEPTION HANDLING
34
Softw
are
Rese
arch
Lab, U
of S
MOTIVATING EXAMPLE
Context code
Recommended code example
SurfExample
35
Softw
are
Rese
arch
Lab, U
of S
PROPOSED CONTEXT-AWARE CODE EXAMPLE RECOMMENDER FOR EXCEPTION HANDLING
Distinguishing Features (3) Graph-based structural relevance
Static relationship and data dependency graph Graph structure matching
Handler quality Paradigm Novel idea to ensure quality of exception handlers Based on readability, amount & quality of the handler
actions Seamless integration of dataset
Exploits GitHub API for data collection Hundreds of popular and mature open source projects
from Eclipse, Apache and others
36
CONTRIBUTIONS OF THIS THESIS
(1) SurfClipse (WCRE 2013, CSMR/WCRE 2014)
(2) QueryClipse (ICSME 2014)
(3) ContentSuggest
(4) SurfExample (SCAM 2014)
(5) ExcClipse
Web search
Query formulation
Post-search analysis
Exception handling
User study
37
EXCCLIPSE: COMPARATIVE ANALYSIS BETWEEN PROPOSED APPROACHES
AND TRADITIONAL ONES
38
Softw
are
Rese
arch
Lab, U
of S
USER STUDY OVERVIEW
Six participants Four exceptions
Four tasks
39
Softw
are
Rese
arch
Lab, U
of S
USER STUDY OVERVIEW
Questionnaire Observation checklist
Training Execution
Evaluation
41
Softw
are
Rese
arch
Lab, U
of S
EVALUATION FEATURES
Tool Feature Functionality Notation
Support for query formulation Web search F1
Accuracy & effectiveness of results Web search F2
Post-search content analysis Web search F3
Support for query formulation Code search F4
Relevance & accuracy of results Code search F5
Usability Overall F6
Efficiency Overall F7
Visualization support Web search F8
Visualization support Code search F9
42
Softw
are
Rese
arch
Lab, U
of S
FEATURE-WISE RATINGS (WEB SEARCH)
43
Softw
are
Rese
arch
Lab, U
of S
FEATURE-WISE RATINGS (CODE SEARCH)
44
Softw
are
Rese
arch
Lab, U
of S
FEATURE-WISE RATINGS (NON-FUNCTIONAL)
45
Softw
are
Rese
arch
Lab, U
of S
OVERALL RATINGS
46
Softw
are
Rese
arch
Lab, U
of S
THREATS TO VALIDITY
SurfClipse--Search engines constantly evolving, same results may not be produced at later time.
QueryClipse-- Long query generation due to lengthy error message in the stack trace.
ContentSuggest-- Changes in the look and feel of the page due to removal of <style> and <script> tags
SurfExample-- Subjective bias in gold set development
ExcClipse--Limited number of participants
47
CONCLUSION
48
Softw
are
Rese
arch
Lab, U
of S
CONCLUDING REMARKS
SurfClipse – context-aware meta search engine IDE-based complete web search solution Outperforms two relevant existing approaches More recall than three search engines with
precision comparable to Google, the best performing engine.
QueryClipse– context-aware query recommender More effective than traditional queries and
queries by existing approaches Highly applicable in terms of pyramid score
49
Softw
are
Rese
arch
Lab, U
of S
CONCLUDING REMARKS
ContentSuggest– context-aware page content recommender Exploits exception details and recommends
relevant section(s) from the page Less information, less overhead for relevance
check Great potential for problem solving
SurfExample– code example recommender for exception handling Graph-based structural relevance matching Handler Quality paradigm Outperforms four existing approaches in all
metrics
50
Softw
are
Rese
arch
Lab, U
of S
FUTURE WORK
SurfClipse – DOM-based element extraction & topic modeling
QueryClipse– Semantic and customized query recommendation
ContentSuggest– More focused content recommendation (e.g., paragraph of interest) for problem solving
SurfExample– More directed support (e.g., applicability of an example) for exception handling
51
Softw
are
Rese
arch
Lab, U
of S
THANK YOU!!
52
Softw
are
Rese
arch
Lab, U
of S
REFERENCES[1] J. Cordeiro, B. Antunes, and P. Gomes. Context-based Recommendation to Support Problem Solving in
Software Development. In Proc. RSSE, pages 85 –89, June 2012.
[2] L. Ponzanelli, A. Bacchelli, and M. Lanza. Seahawk: Stack Overflow in the IDE. In Proc. ICSE, pages 1295–1298, 2013
[3] J. Brandt, P. J. Guo, J. Lewenstein, M. Dontcheva, and S. R. Klemmer. Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code. In Proc. SIGCHI, pages 1589–1598, 2009.
[4] F. Sun, D. Song, and L. Liao. DOM Based Content Extraction via Text Density. In Proc. SIGIR, pages 245–254, 2011.
[5] T. Gottron. Content Code Blurring: A New Approach to Content Extraction. In Proc. DEXA, pages 29–33, 2008.
[6] S. Bajracharya, J. Ossher, and C. Lopes. Sourcerer: An Internet-Scale Software Repository. In Proc. SUITE, pages 1–4, 2009
[7] E. A. Barbosa, A. Garcia, and M. Mezini. Heuristic Strategies for Recommendation of Exception Handling Code. In Proc. SBES, pages 171–180, 2012
[8] R. Holmes and G. C. Murphy. Using Structural Context to Recommend Source Code Examples. In Proc. ICSE, pages 117–125, 2005
[9] W. Takuya and H. Masuhara. A Spontaneous Code Recommendation Tool Based on Associative Search. In Proc. SUITE, pages 17–20, 2011.
[10] M. M. Rahman, S. Yeasmin, and C. K. Roy. Towards a Context-Aware IDEBased Meta Search Engine for Recommendation about Programming Errors and Exceptions. In Proc. CSMR-WCRE, pages 194–203, 2014
[11] M. M. Rahman and C.K. Roy. On the Use of Context in Recommending Exception Handling Code Examples. In Proc. SCAM, 10 pp., 2014 (to appear)
[12] M. M. Rahman and C.K. Roy. SurfClipse: Context-Aware Meta Search in the IDE. In Proc. ICSME, 4 pp., 2014 (to appear)
[13] Slide 13, Meta Search Engine, http://en.wikipedia.org/wiki/Metasearch_engine
53
APPENDICES
54
MAPPING BETWEEN PROBLEM SOLVING STEPS & PROPOSED APPROACHES
Step II: Associated with QueryClipse
Step I: Associated with SurfClipse
Step III: Associated with ContentSuggest
Step IV: Associated with SurfExample
Proposed Approach
55
Softw
are
Rese
arch
Lab, U
of S
THREATS TO VALIDITY (SURFCLIPSE)
Search not real time yet, generally takes about 20-25 seconds per search. Multithreading used, extensive parallel processing needed.
Search engines constantly evolving, same results may not be produced at later time.
Experimented with common exceptions, which are widely discussed and available in the web.
56
Softw
are
Rese
arch
Lab, U
of S
TRADITIONAL SEARCH QUERIES
Popular queries by search engines may not be relevant all the time
Preparing a suitable query is non-trivial Trial and error approach in query
formulation Ad-hoc (e.g., error message) queries may
not reflect the context of the exception
57
Softw
are
Rese
arch
Lab, U
of S
PROPOSED CONTEXT-AWARE QUERY RECOMMENDATION APPROACH
58
Softw
are
Rese
arch
Lab, U
of S
PROPOSED METRICS (3)
Trace Token Rank (TTR) Trace graph developed based on implied
relationship Calculated using Graph-based term-weighting An adaptation from Google’s PageRank algorithm
Degree of Interest (DOI) Heuristic proximity of token to the exception
location Associated with call references in the stack trace
Trace Token Frequency (TTF) Frequency of trace token in context code Associated with method call and object
instantiation
59
Softw
are
Rese
arch
Lab, U
of S
EXPERIMENT OVERVIEW
50 exceptions, their technical details and context code segments collected from our first study.
Recommended queries evaluated by searching with Google, Bing and Yahoo!
Queries compared with existing approaches Query ranks validated with experiments Applicability of query validated using a user
study Performance metrics– precision, recall, % of
exceptions solved
60
RESULTS ON DIFFERENT RANKING ASPECTS
Rank Aspects
Metrics
Google Bing Yahoo!
Top 10 Top 20 Top 10 Top 20 Top 10 Top 20
{DOI, TTR}
MAPKRPTCS
36.36%15.19%48.00%
36.36%15.19%48.00%
49.24%27.84%70.00%
49.24% 30.68% 76.00%
51.70% 34.09% 76.00%
51.61% 35.23% 78.00%
{TTR, TTF}
MAPKRPTCS
38.23%15.34%46.00%
38.23%15.34%46.00%
50.18%29.55%70.00%
50.09% 31.25% 74.00%
45.46% 30.68% 68.00%
44.60% 32.39% 70.00%
{DOI, TTF}
MAPKRPTCS
37.26%17.61%50.00%
37.26%17.61%50.00%
49.53%27.84%72.00%
48.23% 30.11% 74.00%
53.49% 30.68% 78.00%
51.35% 32.95% 78.00%
{DOI, TTR, TTF}
MAPKRPTCS
34.06%13.64%42.00%
34.06%13.64%42.00%
51.85%27.84%72.00%
50.44% 31.25% 76.00%
55.31% 31.82% 76.00%
53.40% 35.23% 80.00%
[ MP = Mean Average Precision at K, R = Recall, PTCS= % of Exceptions Solved]
61
Softw
are
Rese
arch
Lab, U
of S
COMPARISON WITH EXISTING APPROACHES
Approach Metrics
Google Bing Yahoo!
Top 10 Top 20 Top 10 Top 20 Top 10 Top 20
Traditional (Only error message)
MAPKRPTCS
38.97%19.88%52.00%
38.97%19.88%52.00%
44.11%24.43%58.00%
43.82% 26.14% 60.00%
43.18% 25.00% 56.00%
43.18 25.00% 56.00%
Cordeiro et al.
MAPKRPTCS
21.33%10.80%36.00%
21.17%11.93%38.00%
19.22%11.93%34.00%
19.22% 13.07% 36.00%
15.94% 10.80% 32.00%
16.60% 13.06% 40.00%
Ponzanelli et al.
MAPKRPTCS
14.36%9.09%24.00%
14.36%9.09%24.00%
30.27%12.50%38.00%
29.98% 13.07% 38.00%
28.12% 12.50% 38.00%
28.12% 12.50% 38.00%
Proposed approach
MAPKRPTCS
34.06%13.64%42.00%
34.06%13.64%42.00%
51.85%27.84%72.00%
50.44% 31.25% 76.00%
55.31% 31.82% 76.00%
53.40% 35.23% 80.00%
[ MP = Mean Average Precision at K, R = Recall, PTCS= % of Exceptions Solved]
62
Softw
are
Rese
arch
Lab, U
of S
FINDINGS FROM USER STUDY
Query No. 1 2 3 4 5 APS MAPS
PS (Rank I) 0.75 0.89 1.00 1.00 1.00 0.93
PS (Rank II) 0.67 0.72 0.93 1.00 0.63 0.79 0.84
PS (Rank III) 0.67 0.72 1.00 0.93 0.63 0.79
[ PS = Pyramid Score, APS = Average Pyramid Score, MAPS= Mean Average Pyramid Score]
63
Softw
are
Rese
arch
Lab, U
of S
THREATS TO VALIDITY
Long query generation due to lengthy error message in the stack trace.
Less user-friendly query due to complex program tokens– class name, method name in the trace information.
64
Softw
are
Rese
arch
Lab, U
of S
PROPOSED CONTEXT-AWARE PAGE CONTENT SUGGESTION APPROACH
65
Softw
are
Rese
arch
Lab, U
of S
PROPOSED METRICS & SCORES (3)
Content Density (CTD) Text Density
Density of any textual content within a tag Link Density
Density of link-based content (i.e., <a>, <input>) Code Density
Density of code related content (i.e., <code>, <pre>)
Content Relevance (CTR) Text Relevance
Relevance of any textual content within a tag Code Relevance
Relevance of code related content within a tag
Content Score (CTS) Combines both Content Density and Content
Relevance
66
Softw
are
Rese
arch
Lab, U
of S
EXPERIMENT OVERVIEW
500 web pages,150 exceptions and their details (i.e., stack trace, context code) as dataset
40% of the pages from StackOverflow Q & A site
Evaluated against manually prepared gold sets
Evaluated for both relevant and noise-free content recommendation
Compared with four existing approaches Performance metrics– precision, recall,
F1-measure
67
Softw
are
Rese
arch
Lab, U
of S
RESULT ON DIFFERENT ASPECTS OF PAGE CONTENT (RELEVANT CONTENT SUGGESTION)
Content Aspect
Metrics
SO Pages
Non-SO Pages
All Pages
Content Density (CTD)
MPMRMF
50.91%91.74%62.32%
49.50%75.71%53.76%
50.07% 82.18% 57.22%
Content Relevance (CTR)
MPMRMF
86.63%52.17%61.07%
69.17%57.66%55.88%
76.23% 55.44% 57.98%
{CTD, CTR}
MPMRMF
89.91%74.90%80.07%
74.12%80.76%73.91%
80.50% 78.39%76.40%
[ MP = Mean Precision, MR = Mean Recall, MF= Mean F1-measure]
68
Softw
are
Rese
arch
Lab, U
of S
COMPARISON WITH EXISTING APPROACHESApproach Metrics SO Pages Non-SO Pages All Pages
Sun et al. MPMRMF
80.61%86.41%83.14%
78.70%75.67%75.48%
79.49% 80.14% 78.67%
ACCB [50] MPMRMF
90.65%77.32%83.07%
93.07%79.98%84.64%
92.06% 78.87% 83.99%
Proposed(noise-free content extraction)
MPMRMF
91.27%89.27%90.00%
88.90%86.20%85.76%
89.88% 87.48%87.53%
Sun et al. (adapted)
MPMRMF
52.63%86.49%62.57%
38.89%41.84%34.49%
44.44%59.88%45.84%
Proposed (relevant section suggestion)
MPMRMF
89.91%74.90%80.07%
74.12%80.76%73.91%
80.50%78.39%76.40%
[ MP = Mean Precision, MR = Mean Recall, MF= Mean F1-measure]
69
Softw
are
Rese
arch
Lab, U
of S
THREATS TO VALIDITY
Changes in the look and feel of the page due to removal of <style> and <script> tags
Lack of a fully-fledged user study
70
Softw
are
Rese
arch
Lab, U
of S
PROPOSED METRICS (3)
Structural Relevance (Rstr) API Object Match (AOM) Field Access Match (FAM) Method Invocation Match (MIM) Data Dependency Match (DDM)
Lexical Relevance (Rlex) Cosine Similarity Code Clone Measure
Quality of Exception Handler (Qehc) Readability (RA) Average Handler Actions (AHA) Handler to Code Ratio (HCR)
71
Softw
are
Rese
arch
Lab, U
of S
EXPERIMENT OVERVIEW
65 exceptions and context code segments 4400 code examples from 700+ repositories
of Eclipse, Apache, Facebook and Twitter Evaluated against manually prepared gold
set Compared with four existing approaches Performance metrics– precision, recall, # and
% of exceptions handled
72
Softw
are
Rese
arch
Lab, U
of S
RESULT ON DIFFERENT RANKING ASPECTSRanking Aspects Metrics Top 5 Top 10 Top 15
Structure (Rstr) MAPKRPEH
38.07%50.00%69.23%
33.84%61.93%75.38%
32.64%69.32%81.54%
Content (Rlex) MAPKR PEH
35.00%45.45%66.15%
33.85%63.63%75.38%
33.08%70.45%81.54%
{Structure (Rstr),Content (Rlex)}
MAPKRPEH
43.08%51.70%69.23%
38.69%66.48%75.38%
37.33%74.43%81.54%
{Structure (Rstr), Content (Rlex), Quality (Qehc)}
MAPKRPEH
41.92%57.39%73.85%
39.92%68.75%81.54%
38.64%76.70%86.15%
[ MAPK = Mean Average Precision at K, R = Recall, PEH= % of exceptions handled]
73
Softw
are
Rese
arch
Lab, U
of S
COMPARISON WITH EXISTING APPROACHESRecommender Metrics Top 5 Top 10 Top 15
Barbosa et al. MAPKRPEH
16.15%16.47%27.69%
14.69%25.57%38.46%
13.72%31.25%44.62%
Holmes & Murphy MAPKR PEH
4.62%11.36%24.62%
2.31%21.59%38.46%
2.31%27.84%47.69%
Takuya & Masuhara MAPKRPEH
21.54%15.34%33.85%
20.51%27.27%47.69%
19.74%30.68%47.69%
Bajracharya et al. MAPKRPEH
8.46%10.80%18.46%
7.95%15.91%27.69%
6.41%19.32%30.77%
Proposed approach MAPKRPEH
41.92%57.39%73.85%
39.92%68.75%81.54%
38.64%76.70%86.15%
[ MAPK = Mean Average Precision at K, R = Recall, PEH= % of exceptions handled]
74
Softw
are
Rese
arch
Lab, U
of S
THREATS TO VALIDITY
Subjective bias in gold set development Limited size of dynamic corpus for
recommendation Limited number of exceptions for
experiments
75
Softw
are
Rese
arch
Lab, U
of S
TRADITIONAL WEB SEARCH
No ties between IDE and web browsers
Does not consider problem-context
Environment-switching is distracting & time-consuming
Often not much productive (trial & error approach)
76
Softw
are
Rese
arch
Lab, U
of S
TRADITIONAL SUPPORT FOR POST-SEARCH CONTENT ANALYSIS
Keyword highlighting in title based on search query
Very limited meta description using keywords or phrases.
Page URL, Page visit statistics Little clues for actual content in the web page Forces one to browse the page
77
Softw
are
Rese
arch
Lab, U
of S
TRADITIONAL CODE SEARCH ENGINES No ties with the IDE Returns hundreds of
result pages Keyword matching search No support for search
query formulation
78
Softw
are
Rese
arch
Lab, U
of S
PROPOSED CONTEXT-AWARE CODE EXAMPLE RECOMMENDER FOR EXCEPTION HANDLING
79
USER STUDY GROUPING
P1,P2,P3,P4,P5,P6
A (P1,P4,P5) B (P2,P3,P6)
EC1, EC2, EC3, EC4
S1 (EC1, EC2)
S2 (EC3, EC4)
T/S1 + E/S2
T/S2 + E/S1
I (P1,P5) II (P4) I (P2) II (P3,P6)
TE ET TE ET