zenworks configuration management 11.2: core ... · copying all or part of this manual, or dist...

www.novel l .comNovell Training Services

AT T L I V E 2 0 1 2 L A S V E G A S

ZENworks Configuration Management 11.2: Core TroubleshootingLecture

Z E N 1 3

Novell, Inc. Copyright 2012-ATT LIVE-1-HARDCOPY PERMITTED. NO OTHER PRINTING, COPYING, OR DISTRIBUTION ALLOWED.

Copying all or part of this manual, or distributing such copies, is strictly prohibited.To report suspected copying, please call 1-800-PIRATES.

Version 12

Legal Notices

Novell, Inc., makes no representations or warranties with respect to the contents or use of this documentation, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to revise this publication and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes.

Further, Novell, Inc., makes no representations or warranties with respect to any software, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. Further, Novell, Inc., reserves the right to make changes to any and all parts of Novell software, at any time, without any obligation to notify any person or entity of such changes.

Any products or technical information provided under this Agreement may be subject to U.S. export controls and the trade laws of other countries. You agree to comply with all export control regulations and to obtain any required licenses or classification to export, re-export or import deliverables. You agree not to export or re-export to entities on the current U.S. export exclusion lists or to any embargoed or terrorist countries as specified in the U.S. export laws. You agree to not use deliverables for prohibited nuclear, missile, or chemical biological weaponry end uses. See the Novell International Trade Services Web page (http://www.novell.com/info/exports/) for more information on exporting Novell software. Novell assumes no responsibility for your failure to obtain any necessary export approvals.

Copyright © 2012 Novell, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the publisher.

Novell, Inc., has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more of the U.S. patents listed on the Novell Legal Patents Web page (http://www.novell.com/company/legal/patents/) and one or more additional patents or pending patent applications in the U.S. and in other countries.

Novell, Inc.

404 Wyman Street, Suite 500

Waltham, MA 02451

U.S.A.

www.novell.com

Online Documentation: To access the latest online documentation for this and other Novell products, see the Novell Documentation Web page (http://www.novell.com/documentation).

Novell Trademarks

For Novell trademarks, see the Novell Trademark and Service Mark list (http://www.novell.com/company/legal/trademarks/tmlist.html).

Third-Party Materials

All third-party trademarks are the property of their respective owners.


ZENworks 11 SP2Core Troubleshooting

Jason BlackettSenior Product [email protected]


mailto:[email protected]

© Novell, Inc. All rights reserved.2

IntroductionIn this course you:

– Review Core Architecture – Identify key log files– Identify troubleshooting tools– Configure and use troubleshooting tools– Understand key processes– Troubleshoot common problems



Core Server Architecture

Java Virtual Machine 1.6 (32-bit or 64-bit)

ZENworks Loader

Tomcat 6TCP 80/443

(ZENworks Server)

Tomcat 6 Instance #2(TCP 2638)

Tomcat 5(TCP 8443)

Content Im

port

Quicktasks

…System

Up date

CASA Authentication Token Service(CASA-ATS)

ZEN

works (ZC

C)

CA

SA

-ATS

Zenworks-c oreadm

in

…

ZENworks Report Server Web Services

Hibernate

ZEN DB Content Repo MySQL ZRS DBeDir/AD

DB specific JDBC driver



Core Server Architecture Key TakeawaysMultiple instances of Tomcat are present on any ZENworks Server

– The ZENworks agent typically contacts only the first instance– The CASA tomcat instance is available for fallback when the

ZENworks Tomcat server runs out of threads for authentication

– The ZENworks server instance of Tomcat has a default value of 200 http and https threads

– If you have a large number of devices using this server, it is 64-bit, and has extra memory

– Increase ZENserver Java Heap Size– Increase HTTP/HTTPS threads

– Refer to best practices guide for more information

– If you often are seeing 503 – Server Busy responses you want to do this



Core Server Architecture Key TakeawaysZENworks Loader is responsible for any scheduled server tasks

– Responsible for persisting collected data to the database– Responsible for applying system update to the database– Responsible for content packaging– If you have more memory and this server is a collection

server, consider increasing the Java heap size and the number of queue runner tasks



Windows Core Agent Architecture

.NET Framework 3.5.1

SQLLite ZEN FS Cache

ZEN

worksU

ser

ZMD ZESSERVICE

TES

S

Content M

gr

Location Mgr

…

ZESZID

ZESU

SER

ZESC

MD

ND

ISFW D

river

ZESFS

FD D

river

Winlogon / Core Windows Auth

ZEN GINA / Cred

Provider

3rd Party

GINA / Cred

Provider

ZEN CredMgr

3rd Party Cred Mgrs

CASA ATS Client



Windows Core Agent Architecture Key TakeawaysZENworks 11 core agent includes the ZESM core agent

– Includes NDIS FW driver used for location awareness– FSFD included for Client Self Defense and Secure Store– Currently investigating “Lite Location Awareness”

• ZENworks 11SP2 features a network credential manager

– Allows seamless login when another credential provider is used to login on Windows 7

– Seamless if the credentials are the same as ZENworks– When using the credential manager Dynamic Local User,

Roaming Profile policies and ZENworks Windows Group Policies are not supported



Windows Satellite Architecture

.NET

Content S

erver

ZMD

Collection S

ever

Image P

roxy Server

Proxy D

HC

P Server

TFTP Server

ZMG

PrebootPolicy

Java

Jetty

CASA ATS

Configuration M

gr.

Content/Collection

Imaging Authentication



Java Core Agent Architecture

Java 1.6

SQLLite ZEN FS Cache

ZEN

worksU

ser

ZMD

TES

S

Content M

gr

Location Mgr

Jetty

Location Decider

…

Native Helpers



Java Satellite Architecture

Content S

erver

ZMD

Collection S

ever

Java

Jetty

CASA ATS

Configuration M

gr.

Image P

roxy Server

Proxy D

HC

P Server

TFTP Server

ZMG

PrebootPolicy


Identify log files



Agent Login Related Log Files• Core Agent Log

– %ZENWORKS_HOME%\logs\zmd-messages.log– Log level determined by zcc/z-icon/zac setting– Contains information about most ZENworks agent

components

• GINA log (Windows XP only)– %WinSysDir%\Nwgina.log– Enabled by setting agent log level to debug– Contains information related to the initial login to the Windows

system and the calls out to the ZENworks agent



Agent Login Related Log Files• Credential Provider log (Vista/7)

– %WinSysDir%\zennotify.log– Enabled by setting agent log level to debug– Contains information related to the initial login to the Windows

system and the calls out to the ZENworks agent

• Credential Manager (Vista/7 – 11SP2+)– %WinSysDir%\zcredmgr.log– Enabled by setting agent log level to debug– Contains information related to seamless authentication when

a credential provider other than the ZENworks credential provider is used to login to Windows



Agent Login Related Log Files• ZENworks login

– %WinSysDir%\zenlgn.log– Enabled by setting agent log level to debug– Contains information about the ZENworks login process, not

including the portion performed by CASA

• CASA Authentication Token Client log– Casaauthtoken.log– \Program Files\Novell\casa\etc\auth\client.conf– DebugLevel needs to be set to 3– Contains information related to CASA authentication



Server Login Related Log Files• ZENworks Server and CASA Logs

– %ZENWORKS_HOME%\logs\services-messages.log– Contains logs from most of the ZENworks server servlets

• To enable CASA logging for 443 instance– /opt/novell/zenworks/share/tomcat/conf/log4j.properties – ZENWORKS_HOME\tomcat\conf\log4j.properties– Logs to services-messages.log and ats.log

• To enable CASA logging for 2645 instane– etc /CASA/authtoken/svc/log4j.properties– %ZENWORKS_HOME%\share\ats\etc\svc\log4j.properties– Logs to /opt/novell/zenworks/share/tomcat/logs/catalina.out– Logs to %ZENWORKS_HOME%\logs\catalina.out and ats.log



Other Agent Log Files

• NAL Shell Log– Nalshell.txt– HKEY_LOCAL_MACHINE\SOFTWARE\NetWare\Nal\1.0\debug– EnableFile=0x00000001 (DWORD)– Level=0x0000000F (DWORD)– Contains information about the ZENworks explorer shell extension

• System update Logs– System-update.log & msi log files for each agent MSI– Used to determine the progress and errors related to installing a

system update



Other Server Log Files

• ZENworks Loader Log– Loader-messages.log– Contains messages from all registered ZENworks loader modules and the

queue

• CASA ATS Log– Ats.log– Contains information related to CASA connections to the agent and to the

LDAP user source– Verboseness is controlled through configuration file

• ZCC Log– Zcc.log– Contains information related to operations performed in ZCC



Java Agent Log Files

• Core agent log file– zmd-messages.log– Available at /var/opt/novell/zenworks/log/LocalStore– Contains information about most ZENworks agent operations– Controlled via zcc/zicon/zac

• Agent startup log file– novell-zenworks-xplatzmd.out– Available at /var/opt/novell/zenworks



Key Server Configuration Files

• Data Model to Database Connectivity File– /etc/opt/novell/zenworks/datamodel/zdm.xml– Controls what database server the Primary Server talks to

• Alternate LDAP Servers– /etc/opt/novell/zenworks/datamodel/authsource/alt-

servers.properties.xml– Allows you to have more than one server for a given user

source– Currently only works for non-SSL servers



Key Agent Configuration Files• Initial web service configuration file

– initial-web-service– Provided as a part of the initial deployment package– Contains the IP/DNS of the device to register with– Contains the certificate of the associated device– Used if configuration is lost (such as when zac cc is executed)

• Mac XPLAT agent startup parameters– xplat_startup.ini– /var/opt/novell/zenworks– Controls what options are passed to the agent by the launch daemon

startup script


Identify troubleshooting tools



zac zeninfocollect

• Collects all information that NTS may need to help diagnose and resolve your problem

• Makes easy to read HTML files that show the information in cache and the contents of that information

• Creates a zip file that can be easily emailed to other individuals in your company or to Novell



Other zac commands• zac cache-clear (cc)

– Clears the agent cache, requiring the device to get any information from the server on the next refresh– Use with caution as this fully removes the cache on the device

• zac logger (log)– Allows you to change the local device logging level from the CLI

• zac refresh (ref)– Initiate a refresh for both the logged in user and device– bypasscache flag can be use used to ignore cache during refresh

• zac zc– Shows the current zone configuration servers.

• zac reg– Initiate registration

• zac rereg <GUID>– Register the device as an existing device

• zac unreg– Un-register the device from the zone– -f can be used to unregister locally only



Z-icon

• Can be used to:– Initiate a refresh– View information about the device– View inventory information and kick off an inventory scan– Initiate remote management requests and view remote

management information– View bundle and policy assignments– Allows you to launch the ZESM About box to configure logging

and view configuration



zman

• Provides a command line interface for interacting with ZENworks server

• Some useful commands:– zman get-database-credentials– Get DB username and password for connecting with third party ODBC tools

– zman zenserver-backup | zenserver-restore– Allows you to backup and restore server configuration information

– zman queue-list– View scheduled, running and past ZENworks Loader tasks

– zman system-update– Provides commands for monitoring and managing system updates



Web Services Test Pages

• Most ZENworks administrative web services provide a test page

• Useful for determining if a given service is running properly

• Can sometimes provide access from the browser not surfaced in ZCC

• Samples:– https://<primary server>/zenworks-useradmin– https://<primary server>/zenworks-coreadmin



Windows Troubleshooting Tools• Windows Event Viewer

– Application log will contain information about ZENworks events– System log will contain information about services– Group Policy log will contain information about group policy application

• Networking Tools– Ping, ipconfig, tracert, nslookup, etc

• Powershell– Execute remote commands from a central point outside of ZENworks

• Certificate Manager MMC Plug-in– View and manage Windows certificates– Certmgr.msc

• User Environment Log– Shows information regarding Windows login process and profiles– Userenv.log



Linux Troubleshooting Tools

• Networking tools– ifconfig, traceroute, nslookup, etc

• Text manipulation tools– Cat, tail, grep, etc



Microsoft SysInternals Suite

• Provides a suite of tools that are useful for troubleshooting problems on the Windows agent and on Windows servers

– http://technet.microsoft.com/en-us/sysinternals/default.aspx

• Key tools of interest:– Autoruns (autoruns.exe)– Shows and allows you to configure all startup processes that may be slowing down the login or boot process

– Process Explorer (procexp.exe)– Task manager replacement with significantly more information

– Process Monitor (procmon.exe)– Monitor file, registry, handle, thread, etc activity by any process on the system

– Debug Monitor– View debugger messages being output by running applications

– TCPView– View process and the TCP/UDP sockets they have open


http://technet.microsoft.com/en-us/sysinternals/default.aspx


Wireshark

• Wireshark is an open source network packet analyzer – Available at http://www.wireshark.org

• Can be used to capture traffic between Primary Servers, Satellites, Agents, LDAP source and Database

• Primary to Agent Connectivity uses SSL encryption– Packets will be encrypted SSL or LDAPS

• Wireshark allows you to decode SSL packets if you have the Private Key for the certificate being used

– Useful for troubleshooting and identifying the progress through a given process– Requires that you provide the SSL private key in the properties for the SSL protocol– Can only decode the SSL traffic if it saw the connection process in the trace


http://www.wireshark.org/


Other Tools

• LDAP Browser– Can be used to verify connectivity to LDAP sources outside of

ZENworks– http://www.ldapbrowser.com

• Database Tools– Sybase Interactive ODBC tool (included with ZENworks)– MS-SQL tools (osql.exe, SQL Enterprise Manager, etc)– Oracle tools (sqlplus)

• WINHTTP Trace Utility– Allows you to capture agent traffic before it is sent on the wire– http://msdn.microsoft.com/en-us/library/aa384119.aspx


http://www.ldapbrowser.com/

http://msdn.microsoft.com/en-us/library/aa384119.aspx


Exercise 1

• Run ‘zac zeninfo’ and review the contents on a working system

• Export the SSL private key from a ZCM Primary Server and use it to decode SSL traffic with Wireshark

• Use LDAP Browser to verify LDAP connectivity


Understand core processes



Core Processes

• This section helps you understand how the following core processes should work:

– ZENworks Agent Registration– ZENworks Agent Login– ZENworks System Update



ZENworks Adaptive Agent Registration

1. Agent attempts to contact the zenworks-registration servlet and retrieve the ostargets.xml file

2. Agent gathers important information from the device– OS Target String Generated Password– Hostname IP address– DNS info– Get the GUID from ISD, file system guid, or deployguid.txt

3. Compares the OS Version string against the entries in OSTargets.xml to verify a supported OS and set the OS information

4. Sends a registration request to the server

5. Server parses the XML

6. If enabled, device reconciliation occurs to ensure the device does not already exist

7. Using the information in the registration key or rule the device is created if not present

8. Server sends back the name of the object and a return code



ZENworks Adaptive Agent Login

1. On bootup the Windows Logon process calls either NWGINA.DLL or the Credential Provider

2. The user provides a username and password

3. GINA, CP or CredMgr hand the credentials off to zenlgn

4. Zenlgn verifies finds a configuration server that is available

5. Zenlgn hands the credentials off to the CASA client

6. CASA client creates an encrypted token with the credentials and sends them to the CASA server

7. CASA server validates the credentials and returns back a CASA token that ZENworks then uses to validate user authentication against the user source

8. Agent then issues a RegisterUser which looks up key attributes which are then stored in cache and used for finding assignments



System Update

1. Based on license, available system updates are downloaded into the system

2. When deployed, first Pre-Global actions are applied– Typically this updates database schema and information

3. Queue handler then creates assignments for all devices in a given stage

4. On refresh the agent finds a system update assignment and hands it to the system update module

5. System update module confirms the update is applicable and then requests the update commands for the device by GUID

6. ZENUpdater then applies the update using the update_commands.xml file

7. Reboot occurs as configured if needed

8. Agent deployment packages on the updated device are updated if appropriate

9. Process continues for other devices and stages until All Devices stage

10. After all devices are updated Post Global actions occur

11. Baselining deletes files that are no longer needed


Troubleshooting Common Problems



General Troubleshooting

Start by checking the basics:1. Are the right services running on the agent and the server?

2. Is the firewall configured properly?

3. Does the workstation have an IP address?

4. Can the workstation resolve the server’s name?

5. Can the workstation ping the server?

6. Is time in sync between the workstations and server(s)?

7. Are the closest server rules configured properly?

8. Does the server have multiple DNS names?

9. Are the right agent features enabled?

Then use the logs, tools, and process information to dig in and find the problem



Slow Login Troubleshooting

1. Understand what you define as slow login

2. Ensure the device meets the pre-requisites

3. Make sure you have proper AV execlusions

4. Try a clean machine and compare performance

5. Verify that you have setup locations and closest server rules properly

6. Verify servers are available and certificates are correct

7. Try removing assignments or agent features to see if you can narrow down the problem

8. Use the log files to determine where the slowness is

9. Use Process Monitor to help narrow it down



System Update Troubleshooting

1. Verify that the server has the appropriate license to find the update and can access NCC

2. Verify that the zenworks-systemupdate URL is accessible

3. Try using the manual zip file and using the zman sui command if you are having download problems

4. Verify standard infrastructure is working – DNS, certificates, IP, etc.

5. Verify all devices you want to patch are powered on and checking in on a regular basis so the assignment can be discovered

6. Verify the assignment using the zenworks-systemupdate test service for getAssignedSystemUpdates and specify the device’s GUID

7. Check the loader-messages.log file for queue handler messages including download, import, pre-global actions, etc

8. Check the system-update.log file for messages about applying the update, look at update_commands.xml in conjunction with the log to see where the update is failing

9. Check the zmd-messages.log file for messages from the system update module



Exercise 2

• Troubleshoot agent login problemsScenario: Nobody can login to the zone

Objective: Get Ghanley logged into the zone on XP-WS

• Steps– Change snaps of DA-ZCM and XP-WS to Exercise 1-2

Starting Snapshot– Boot-up DA-ZCM then XP-WS– Troubleshoot and resolve problems until Ghanley can

successfully login



Summary

• ZENworks 10 Configuration Management provides many tools and log files that help when you need to troubleshoot problems

• Wireshark is your friend• Having a good understanding of the normal process, will help you troubleshoot problems when something bad happens

• Use the quick troubleshooting checklists before diving too deep

• Use the appendices to compare info from your environment


This document could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein. These changes may be incorporated in new editions of this document. Novell, Inc. may make improvements in or changes to the software described in this document at any time. Copyright © 2011 Novell, Inc. All rights reserved. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.


zenworks configuration management 11.2: core ... · copying all or part of this manual, or dist...

Documents