new troubleshooting apache cloudstack · 2014. 10. 13. · #ccceu14 ! working @ schuberg philis...
TRANSCRIPT
![Page 1: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/1.jpg)
#CCCEU14
TROUBLESHOOTING APACHE CLOUDSTACK
![Page 2: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/2.jpg)
#CCCEU14
! Working @ Schuberg Philis since 2010
! Mission Critical
! +3 year IaaS team
! Part of the initial CS vs OS
! Started with CloudStack 2.2.14
JORIS VAN LIESHOUT
![Page 3: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/3.jpg)
#CCCEU14
! Reading the logs
! Understanding System VMs
! Use the source, Luke!
! DB API, hands off?
! Employee Cloud
! Questions
! Side notes • Have you worked with ACS? • CloudStack 4.4.1 • XenServer 6.2.0 SP1 • Default log settings
AGENDA
![Page 4: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/4.jpg)
#CCCEU14
! Less is more <=> • less /var/log/cloudstack/management/management-
server.log
! Keep management server ids at hand • select * from cloud.mshost where removed is null;
! Stack traces • Look at the first instead of the last
! Search for API key
! Log4j 1.2 EnhancedPatternLayout • /etc/cloudstack/management/log4j-cloud.xml
READING THE LOGS
![Page 5: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/5.jpg)
#CCCEU14
! (Date, Time), Log priority
! Class • Will match a .java file in the source
READING THE LOGS
![Page 6: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/6.jpg)
#CCCEU14
! Thread Name
! Thread context
! Optional Job id
READING THE LOGS
![Page 7: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/7.jpg)
#CCCEU14
! When forwarding commands • Host_id-Sequence_nr • MgmtId • Can be the other server
• Via • Command
READING THE LOGS
![Page 8: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/8.jpg)
#CCCEU14
! Find the first call • API call • API key • Name
! Thread name and first context
! Async creates a job
READING THE LOGS
![Page 9: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/9.jpg)
#CCCEU14
! Find the first call
! Thread name and first context
! Might creates a job
! Picked up by…
READING THE LOGS
![Page 10: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/10.jpg)
#CCCEU14
! Thread name and first context
! Sending… • Track sequence id • Keep an eye on MgmtID
! Executing…
READING THE LOGS
![Page 11: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/11.jpg)
#CCCEU14
! ssh to SVM • From hypervisor • Port 3922 • /root/.ssh/id_rsa.cloud
!xensourse.log, SMlog and cloud/vmops.log
! XAPI call to vmops plugin
UNDERSTANDING SYSTEM VMS
![Page 12: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/12.jpg)
#CCCEU14
! GitHub • https://github.com/apache/cloudstack
! Eclipse for Java EE • https://cwiki.apache.org/confluence/display/
CLOUDSTACK/Using+Eclipse+With+CloudStack
! DevCloud4 • https://github.com/imduffy15/devcloud4
USE THE SOURCE, LUKE!
![Page 13: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/13.jpg)
#CCCEU14
! Read only!
! Unless… • After code review • Bug solution also changes db
! Why? • Change state => data mismatch • Incorrect value => ACS start fails • DB data is not always leading • Effects of DB change can stick around
! Marvin and CloudMonkey • And find a way out without DB change
DB API, HANDS OFF!
![Page 14: New TROUBLESHOOTING APACHE CLOUDSTACK · 2014. 10. 13. · #CCCEU14 ! Working @ Schuberg Philis since 2010 ! Mission Critical ! +3 year IaaS team ! Part of the initial CS vs OS !](https://reader033.vdocuments.site/reader033/viewer/2022051809/6010d90d5caa2a55ff2c186c/html5/thumbnails/14.jpg)
#CCCEU14
! Realistic workload
! Use as UAT environment
! To reproduce bugs and workarounds
! Technology test ground
EMPLOYEE CLOUD