troubleshooting jira & confluence
DESCRIPTION
With Atlassian Support as your guide learn how to diagnose your JIRA and Confluence installation like a pro, where to look for solutions, and tips for getting a quick response from us when all else fails.TRANSCRIPT
Troubleshooting JIRA and Confluence!
Jeff Curry!!
While keeping your sanity!
Atlassian Support!
Agenda!• Troubleshooting process!• Reviewing logfiles!
• Reading thread dumps!
• Taking it further!• Getting support ( quicker )!!
Case Review!
Case Review!• Entitled ‘Error Message’!• Daily crashes, poor
performance!• Larger instance!
Case Review – Reviewing Logfiles!
logs/atlassian-jira.log!( NOT catalina.out )!
QuartzWorker-0 ERROR ServiceRunner xxx [service.services.mail.MailFetcherService] xxx[10030]: Error connecting to host ‘xxx' as user ‘xxx' via protocol 'pop3s': javax.mail.MessagingException: Connect failed;nested exception is:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failedjavax.mail.MessagingException: Connect failed;nested exception is:javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed!
org.apache.jk.core.MsgContext action!WARNING: Error sending end packet!java.net.SocketException: Software caused connection abort: socket write error!at java.net.SocketOutputStream.socketWrite0(Native Method)!at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)!at java.net.SocketOutputStream.write(SocketOutputStream.java:136)!at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:518)!at org.apache.jk.common.JkInputStream.endMessage(JkInputStream.java:112)!at org.apache.jk.core.MsgContext.action(MsgContext.java:293)!at org.apache.coyote.Response.action(Response.java:182)!at org.apache.coyote.Response.finish(Response.java:304)!at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:204)!at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)!at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)!at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)!at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)!at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)!at java.lang.Thread.run(Thread.java:534)!
DefaultIndexManager - Lock attempt timed out - waited 30000 millisecondscom.atlassian.jira.issue.index.IndexException: Lock attempt timed out - waited 30000 milliseconds
!at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:463)!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:286)!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:252)!at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:260)!at sun.reflect.GeneratedMethodAccessor608.invoke(Unknown Source)!at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)!at java.lang.reflect.Method.invoke(Method.java:618)!at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:71)!at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)!at $Proxy4.reIndexIssueObjects(Unknown Source)!at com.atlassian.jira.event.listeners.search.IssueIndexListener.reIndex(IssueIndexListener.java:138)!at com.atlassian.jira.event.listeners.search.IssueIndexListener.issueCommented(IssueIndexListener.java:78)!at com.atlassian.jira.event.issue.AbstractIssueEventListener.workflowEvent(AbstractIssueEventListener.java:118)!at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:125)!at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:83)!at com.atlassian.jira.web.action.issue.AddComment.alertSystemOfComment(AddComment.java:101)!at com.atlassian.jira.web.action.issue.AddComment.doExecute(AddComment.java:85)!at webwork.action.ActionSupport.execute(ActionSupport.java:153)!at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:54)!at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:132)!at com.atlassian.jira.web.dispatcher.JiraServletDispatcher.service(JiraServletDispatcher.java:178)!at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
!
Case Review – Reviewing Logfiles!
Garbage collection ( GC ) logs!-verbose:gc -Xloggc:<home>/logs/gc.log - XX:
+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintGCDateStamps!
![ http://www.oracle.com/technetwork/java/javase/tech/index-jsp-140228.html ]!
Case Review – Reviewing Logfiles!• Plain!48.630: [GC [PSYoungGen: 112384K->5460K(131072K)] 112384K->5460K(1029888K), 0.0158160 secs] [Times: user=0.03 sys=0.01, real=0.03 secs] !
93.865: [GC [PSYoungGen: 117844K->16468K(131072K)] 117844K->16468K(1029888K), 0.0329740 secs] [Times: user=0.06 sys=0.03, real=0.03 secs] !
105.728: [GC [PSYoungGen: 128852K->18302K(131072K)] 128852K->18302K(1029888K), 0.0287090 secs] [Times: user=0.07 sys=0.01, real=0.02 secs] !
122.928: [GC [PSYoungGen: 130686K->18679K(131072K)] 130686K->21123K(1029888K), 0.0221030 secs] [Times:
user=0.06 sys=0.01, real=0.02 secs]!• Filtered! grep 'Full GC' gc.log | awk '{print $1, $16}'!
9285.005: real=7.80!
9474.161: real=10.99!
10106.270: real=5.57!
10355.367: real=12.84!
10758.017: real=15.17!
Case Review - Workaround!• GC tuning for stability!• Concurrent !
• -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:SurvivorRatio=8!
• 2 GB allocation!
• Added vCPU!
!
Case Review – Thread dumps!"QuartzWorker-0" prio=10 tid=0x0a2ac400 nid=0xe3e runnable [0x72f04000]!
java.lang.Thread.State: RUNNABLE!
at java.net.SocketInputStream.socketRead0(Native Method)!
at java.net.SocketInputStream.read(SocketInputStream.java:129)!
at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293)!
at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331)!
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798)!
- locked <0x744f7a38> (a java.lang.Object) !
Case Review – Thread dumps!at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)!
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)!
- locked <0x74501140> (a java.io.BufferedInputStream)!
at java.io.DataInputStream.readLine(DataInputStream.java:496)!
at com.sun.mail.pop3.Protocol.simpleCommand(Protocol.java:360)!
at com.sun.mail.pop3.Protocol.<init>(Protocol.java:104)!
at com.sun.mail.pop3.POP3Store.getPort(POP3Store.java:214)!
- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!
at com.sun.mail.pop3.POP3Store.protocolConnect(POP3Store.java:157)!
- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!
Case Review – Thread dumps!at javax.mail.Service.connect(Service.java:288)!
- locked <0x744f6ce8> (a com.sun.mail.pop3.POP3SSLStore)!
at com.atlassian.jira.service.services.mail.MailFetcherService.run(MailFetcherService.java:146)!
at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:60)!
at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)!
at org.quartz.core.JobRunShell.run(JobRunShell.java:195)!
at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72)!
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)!
Case Review – Thread dumps!"IssueIndexer:thread-3" prio=10 tid=0x08ec1000 nid=0x1b64 runnable [0x71f44000]!
java.lang.Thread.State: RUNNABLE!
...!
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java)!
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:386)!
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)!
at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:92)!
at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)!
Case Review – Thread dumps!at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:605)!
at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)!
at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:790)!
at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:775)!
at org.ofbiz.core.entity.GenericDelegator.getRelated(GenericDelegator.java:1211)!
at org.ofbiz.core.entity.GenericDelegator.getRelatedByAnd(GenericDelegator.java:1169)!
at org.ofbiz.core.entity.GenericValue.getRelatedByAnd(GenericValue.java:260)!
at com.atlassian.jira.issue.managers.DefaultIssueManager.getEntitiesByIssue(DefaultIssueManager.java:318)!
Case Review – Thread dumps!at com.atlassian.jira.issue.comments.DefaultCommentManager.getComments(DefaultCommentManager.java:135)!
at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:40)!
at com.atlassian.jira.issue.index.DefaultCommentRetriever.get(DefaultCommentRetriever.java:29)!
at com.atlassian.jira.issue.index.DefaultIssueIndexer$CommentDocumentBuilder.get(DefaultIssueIndexer.java:460)!
at com.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:315)!
at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:371)!
!
!
Case Review - Resolution!
JIRA POP service
Issue Key
Database Mail inbox
Case Review - Resolution!
Taking it further!
Taking it further - VisualVM!• http://visualvm.java.net!• Monitor performance!
• Profile memory usage!
• Review thread dumps!
• Browse heap dumps!
Taking it further – OS Monitoring!Linux 2.6.18-164.11.1.el5PAE (test.jira.com) !05/17/2012!
!
12:00:01 AM CPU %user %nice %system %iowait %steal %idle!
12:10:01 AM all 5.22 0.00 1.50 6.50 0.00 86.79!
12:20:01 AM all 3.42 0.00 0.90 0.08 0.00 95.60!
12:30:01 AM all 4.77 0.00 1.01 0.29 0.00 93.92!
12:40:01 AM all 3.47 0.00 0.90 0.07 0.00 95.56!
12:50:01 AM all 3.43 0.00 1.13 0.07 0.00 95.38!
01:00:01 AM all 3.48 0.00 1.12 0.06 0.00 95.33!
01:10:01 AM all 4.27 0.00 0.95 1.74 0.00 93.04!
01:20:01 AM all 3.48 0.00 1.14 0.06 0.00 95.33!
01:30:01 AM all 3.80 0.00 0.95 0.06 0.00 95.19!
01:40:01 AM all 3.43 0.00 0.90 0.07 0.00 95.60!
01:50:01 AM all 3.41 0.00 0.92 0.07 0.00 95.61!
02:00:01 AM all 3.42 0.00 0.91 0.19 0.00 95.49!
02:10:01 AM all 3.71 0.00 0.94 3.59 0.00 91.76!
02:20:01 AM all 3.38 0.00 0.90 0.08 0.00 95.65!
02:30:01 AM all 3.77 0.00 0.94 0.07 0.00 95.23!
02:40:01 AM all 3.38 0.00 0.89 0.07 0.00 95.66!
02:50:01 AM all 3.37 0.00 0.89 0.06 0.00 95.69!
03:00:01 AM all 3.37 0.00 0.89 0.07 0.00 95.67!
03:10:01 AM all 3.78 0.00 1.04 1.56 0.00 93.62!
03:20:01 AM all 3.41 0.00 0.89 0.06 0.00 95.65!
03:30:01 AM all 3.82 0.00 0.95 0.06 0.00 95.17!
03:40:01 AM all 5.94 0.00 1.43 1.33 0.00 91.30!
03:50:01 AM all 3.42 0.00 0.89 0.05 0.00 95.63!
04:00:01 AM all 3.47 0.00 0.89 0.07 0.00 95.57!
04:10:01 AM all 29.31 0.00 3.65 22.59 0.00 44.45!
Taking it further - SAR!http://www.ibm.com/developerworks/aix/library/au-unix-perfmonsar.html!
Taking it further – HTTP inspection!• Fiddler!• http://www.fiddler2.com/fiddler2/!
• Live HTTP Headers ( Firefox addon )!• https://addons.mozilla.org/en-US/firefox/addon/live-http-headers/!
Taking it further – HTTP inspection!
Taking it further – HTTP inspection!
Getting support ( quicker )!
Getting support – self-help!
Getting support – self-help!• Knowledge base articles!• https://confluence.atlassian.com/display/KB/Knowledge+Base+Home!
• Answers.atlassian.com!• #atlassian on irc.freenode.com!
Getting support – self-help!
!!
Getting support – Atlassian view!• Initial response time SLA!• 1 – 1 hour!
• 2 – 4 hours!
• 3 – 8 hours!
• 4 – 24 hours!
!
Getting support – Atlassian view!
Open Waiting for Customer
Atlassian Investigating
Waiting for Customer
IRT
Getting support – file a ticket!• When I… I expect… however…!• Application log files !
• Screenshots!
• Steps to reproduce!
Getting support – file a ticket!• When I use JIRA and try to login!• I expect JIRA to display a login page!• however I get this 404 page!• log files attached!
Getting support – Atlassian view!
Open Atlassian Investigating
Waiting for Customer
IRT
Summary!• Basic!• Application and GC log files!
• Thread dumps!
• Advanced!• VisualVM!
• HTTP inspection!
#atlassiansummit
I am my own Atlassian support engineer AMAA
Thank you!