copyright 2009 trend micro inc. classification 9/19/2015 1 troubleshooting tmsp marks shen senior...
TRANSCRIPT
![Page 1: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/1.jpg)
Copyright 2009 Trend Micro Inc.Classification 04/21/23 1
Troubleshooting TMSP
Marks Shen • Senior Engineer – QAEvan Wang • Engineer - QA
![Page 2: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/2.jpg)
Copyright 2009 Trend Micro Inc.Classification 04/21/23 2
Agenda |
Frequent Case
Debug log and information
Troubleshooting
Q&A
![Page 3: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/3.jpg)
Copyright 2009 Trend Micro Inc.
No report was generated
• Logs need to collect on Daemon Server– /opt/TrendMicro/tdss/tdes/log/iae_log.txt– /var/log/cron
• Information – Customer expiration date– Device register to TMSP
![Page 4: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/4.jpg)
Copyright 2009 Trend Micro Inc.
No report was generated Cont.
• Normal debug log for log correlation and report generation
• Crontab task
cron_iae.sh will be executed at 2:15 am every day
![Page 5: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/5.jpg)
Copyright 2009 Trend Micro Inc.
No report was generated Cont.
Report will not be generated if
• Customer service get expired
• Customer without device registered
![Page 6: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/6.jpg)
Copyright 2009 Trend Micro Inc.
No incident in report• Logs need to collect on Daemon Server
– /opt/TrendMicro/tdss/tdes/log/iae_log.txt
• Information– Check if TDA log has uploaded
Latest log time
![Page 7: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/7.jpg)
Copyright 2009 Trend Micro Inc.
Cannot access Admin console (err 404)
• Logs need to collect on Daemon Server– /var/log/httpd/access_log– /var/log/httpd/error_log
• Information – ps –ef | grep httpd– Netstat –anp | grep httpd
![Page 8: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/8.jpg)
Copyright 2009 Trend Micro Inc.
No Rsync log uploaded
• Logs need to collect on Access Server– Log receiver
• /home/tdalog/log/pre-post-exec.log /home/tdalog/log/db_import_tda.log
• /home/tdalog/log/db_import_tdm.log
– Authentication (describe in next sides)
• Information – Ps –ef | grep tmsshd– Netstat –anp | grep tmsshd
• Listen on port 22
![Page 9: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/9.jpg)
Copyright 2009 Trend Micro Inc.
No Rsync log uploaded Cont.
• Normal procedure debug log of TDA log processing– /home/tdalog/log/pre-post-exec.log
![Page 10: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/10.jpg)
Copyright 2009 Trend Micro Inc.
CAS server caseProblems caused by CAS failure:
– Device register to TMSP fail– Customer portal login fail– Log uploading fail through RSYNC
Logs need to collect on Access Server:– /var/log/messages– /var/log/cas_8000.log– /var/log/cas_8001.log– /var/log/cas_8002.log
Information:• Ps –ef | grep pound• Ps –ef | grep rubcasd
Normal log of CAS authentication– /var/log/cas_8000.log
![Page 11: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/11.jpg)
Copyright 2009 Trend Micro Inc.
CAS server case – Service down
• TDA register fail
• Check /var/log/messages
• Recover– If pound or cas service is down
• /etc/init.d/pound start• /etc/init.d/rubcasd start
![Page 12: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/12.jpg)
Copyright 2009 Trend Micro Inc.
Data Gateway Case
Problems caused by Data Gateway failure:– OCS Heartbeat / OCS log real-time cannot upload– T2 / T3 mitigation request cannot deliver to TMTM– SIC sample cannot upload from TMTM
Logs need to collect on Access Server:– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/dg/WEB-
INF/logs/dg.log– $APACHE_HOME/logs/ssl_request_log_dg– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/catalina.out
Information:• Ps –ef | grep httpd• Ps –ef | grep tomcat• Netstat –anp | grep 443• Netstat –anp | grep 8009
![Page 13: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/13.jpg)
Copyright 2009 Trend Micro Inc.
Data Gateway Case – DB disconnect
• TDA register fail
• Check apache error log: /usr/apache/logs/error_log
![Page 14: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/14.jpg)
Copyright 2009 Trend Micro Inc.
Customer portal cannot login
• Logs need to collect on Access Server:– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/
tms2/WEB-INF/logs/tms.log– $APACHE_HOME/logs/ssl_request_log_portal– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/
catalina.out
• Information• Ps –ef | grep httpd• Ps –ef | grep tomcat• Netstat –anp | grep 443• Netstat –anp | grep 8009
![Page 15: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/15.jpg)
Copyright 2009 Trend Micro Inc.
Cannot get eMail notification
• Exclude Mail server problem, collect debug logs:– /root/infomation.log
– /var/log/cron
No Subscription
DB connection fail
![Page 16: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/16.jpg)
Copyright 2009 Trend Micro Inc.
![Page 17: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/17.jpg)
Copyright 2009 Trend Micro Inc.
FAQ
Q: How to re-generate report manually?Login TDES machine, change dir to "/opt/TrendMicro/tdss/tdes/";
– 1. Daily Report#php gendailydata.php user_id YEAR MONTH DAYexample: "php gendailydata.php trend 2009 01 04" generate daily report of 2009.01.04 for customer "trend“
– 2. Executive Report (Weekly / Monthly)#php genexecdata.php user_id START_DATE START_DATE yes m/w START_DATE: report start data with format “YYYY-MM-DD” START_DATE: report end data with format “YYYY-MM-DD” yes: if this report will be imported into DB m/w: monthly or weeklyexample: "php genexecdata.php trend 2009-01-01 2009-01-31 yes m" to generate monthly report for customer "trend" of 2009-01
– 3. Upsell Report# php genupselldata.php user_id START_DATE END_DATE noexample: " php genupselldata.php trend 2009-01-01 2009-01-31 no" to generate upsell report for customer "trend" from 2009-01-01 to 2009-01-31Note: Upsell report will not be imported into DB and cannot download from admin console
![Page 18: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/18.jpg)
Copyright 2009 Trend Micro Inc.
FAQ
Q: What the difference between two service type “TDS” and “TLMS”?
A: “TDS” service is set to the customer, who has TDA deployed. “TLMS” service is set to the customer, who has both TDA and TMTM deployed.
TMSP provide different report for these 2 service type, and for TLMS, TMSP provide additional UI for abnormal endpoints handling process.
![Page 19: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/19.jpg)
Copyright 2009 Trend Micro Inc.
FAQ
Q: Why AE status show N/A on Admin console?
A: AE (Abnormal Endpoint) status is designed for the customers, who has TMTM deployed, to show the mitigation failed client on TMSP UI.
That means, only the customer with service type “TLMS” will show the AE details in TMSP admin UI as link.
![Page 20: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/20.jpg)
Copyright 2009 Trend Micro Inc.
FAQ
Q: Whether the IAE will be updated?
A: Yes, TMSP IAE rules will be updated timely to detect new threats. Currently, the rule is updated monthly. In TMSP side, we get new threat rule from threat team and deploy to each site after verification.
![Page 21: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/21.jpg)
Copyright 2009 Trend Micro Inc.
FAQ
Q: Can TMSP generate DOC format report?
A: No, currently, TMSP can only generate report in PDF format.
![Page 22: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/22.jpg)
Copyright 2009 Trend Micro Inc.
FAQ
Q: How to backup Database?
A: Here are two type of DB backup, Full Backup and Lightweight Backup.
• Full backup:Backup full database of TMSP
#mysqldump -uroot -p"$password" tdors_new > tdors_new_backup.sql
![Page 23: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/23.jpg)
Copyright 2009 Trend Micro Inc.
FAQ
• Lightweight DB backup: Backup customer info, configuration and some correlation result.
#mysqldump -h $host -u$account -p"$password" tdors_new access_code admin_log admins contacts customer customer_expiration_conf customer_expiration_notif notification ocs_ast_event ocs_malware scsd_case scsd_case_followup scsd_case_status scsd_pattern scsd_pattern_status scsd_sample_info scsd_sample_summary t2t3_dce_result tb_devprofile> SJDC2.5.sql
After backup entire database, it is recommended to store the backup file into another physical storage.
![Page 24: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/24.jpg)
Copyright 2009 Trend Micro Inc.
Q&A
![Page 25: Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA](https://reader036.vdocuments.site/reader036/viewer/2022081519/56649e5f5503460f94b59a6c/html5/thumbnails/25.jpg)
Copyright 2009 Trend Micro Inc.Classification 04/21/23 25
THANK YOU!