diagnosticsagenttroubleshooting-1

SAP Solution Manager 7.0 EHP1, 7.10

End-to-End DiagnosticsDiagnostics Agent Troubleshooting GuideVersion 3.2, Januar 2014

END-TO-END DIAGNOSTICSDIAGNOSTICS AGENT TROUBLESHOOTING

Icon Meaning

Caution

Example

Note

Recommendation

Syntax

Type Style RepresentsExampleText

Words or characters quotedfrom the screen. Theseinclude field names, screentitles, pushbuttons labels,menu names, menu paths,and menu options.

Cross-references to otherdocumentation.

Exampletext

Emphasized words orphrases in body text, graphictitles, and table titles.

EXAMPLETEXT

Technical names of systemobjects. These include reportnames, program names,transaction codes, tablenames, and key concepts ofa programming languagewhen they are surroundedby body text, for example,SELECT and INCLUDE.

Exampletext Output on the screen. Thisincludes file and directory

names and their paths,messages, names ofvariables and parameters,source text, and names ofinstallation, upgrade anddatabase tools.

Exampletext Exact user entry. These arewords or characters that you

enter in the system exactlyas they appear in thedocumentation.

Variable user entry. Anglebrackets indicate that you

replace these words andcharacters with appropriateentries to make entries in thesystem.

EXAMPLETEXT

Keys on the keyboard, forexample, F2 or ENTER.

END-TO-END DIAGNOSTICSDIAGNOSTICS AGENT TROUBLESHOOTING

Contents1 Document Status ................................................................................................................................. 4

1.1 New Document Location .............................................................................................................. 41.2 Location of Intermediate Document ............................................................................................ 41.3 Old Document Location ................................................................................................................ 4

2 Pre-requisites ...................................................................................................................................... 52.1 JDK compatibility .......................................................................................................................... 52.2 SLD production system ................................................................................................................ 5

3 Agent installation strategy .................................................................................................................. 83.1 Managing system scaling ............................................................................................................. 83.2 Managed System environment ..................................................................................................... 83.3 Managed Systems HA/Failover environment .............................................................................. 8

4 Agent installer...................................................................................................................................... 94.1 SAP Host Agent ............................................................................................................................. 9

4.1.1 E2E RCA configuration for Solution Manager 7.0x or higher .................................................... 94.1.2 E2E RCA configuration for Solution Manager 7.1x only............................................................ 9

4.2 JVM parameters .......................................................................................................................... 104.3 Startup scripts ............................................................................................................................. 10

5 Agent connectivity ............................................................................................................................ 145.1 Managing system ........................................................................................................................ 145.2 Managed system ......................................................................................................................... 16

Connection configuration for managed systems .................................................................................. 195.3 Self-Management ........................................................................................................................ 195.4 SAP Router (moved to Setup Guide) ......................................................................................... 20

6 Agent Maintenance ............................................................................................................................ 216.1 Configuration............................................................................................................................... 21

6.1.1 Change Path of the JDK......................................................................................................... 216.1.2 Change JVM Parameters ...................................................................................................... 216.1.3 Replacing JDKs 1.4.2 with SAP JVM 4 .................................................................................. 216.1.4 Verifying Logical hostname .................................................................................................... 226.1.5 Usage of smdsetup script (moved to Setup Guide) ................................................................ 226.1.6 Change Agent log level (moved to Setup Guide) .................................................................... 22

6.2 Managing J2EE user ................................................................................................................... 227 Tracking Agent issues ...................................................................................................................... 24

7.1 OSS message pre-requisites ...................................................................................................... 247.2 Agent crash investigation........................................................................................................... 247.3 Activate P4 logs .......................................................................................................................... 26

8 Telnet SMD server commands .......................................................................................................... 279 Full Qualified Name resolution issue ............................................................................................... 28

End-to-End Diagnostics 4Diagnostic Agent Troubleshooting Guide

SAP AG 2014

1 Document Status

1.1 New Document LocationThe new location for the troubleshooting content originally covered by this document is on the SDN WIKI

x Here http://wiki.sdn.sap.com/wiki/display/SMSETUP/Diagnostics+Agent+Troubleshooting

(also accessible via this tiny link http://wiki.sdn.sap.com/wiki/x/aQSAF)

1.2 Location of Intermediate DocumentWhile we are reviewing the content of this document, and migrate it to the SDN WIKI, this intermediate version of thisdocument will be available as an attachment to SAP Note 1833501, as DiagnosticsAgentTroubleshooting.pdf.

1.3 Old Document Location

The old location of this document is on the SMP (both this SPM location and the document on SMP aredeprecated/obsolete)

x https://service.sap.com/~sapdownload/002007974700000409092009E/DiagAgent_TroubleShooting.pdf


SAP AG 2014

2 Pre-requisites2.1 JDK compatibility

The following table describes the list of different JDKs supported by the Diagnostics Agent for each OS platform:

OS Diagnostics Agent7.00 / 7.01

Diagnostics Agent7.11

Diagnostics Agent7.20

DiagnosticsAgent 7.30 SP2

WindowsServer

2003/2008 Note 716604

SAPJVM5 SAPJVM6 SAPJVM6

AIX

5.3 64bit onPower 4, 5 Note 716927

6.x 64bit onPower 4, 5 Note 716927

HP

11.31 on IA6464bit Note 716926

11.11,PA-RISC 64bit Note 716926

Solaris 10 on SPARC64bit / x86_64 Note 716604

Linux 2

IA32 32bit /IA64 64bit Note 716604

x86_64 64bit Note 861215Power 64bit Note 810008SLES9 orhigher on

zSeries 64bitNote 746299

I5/OS V5R3, V5R464bit Note 717376

Z/OS 1.6, 1.7, 1.8,1.9 64bit - -

-

The above listed SAP notes could be helpful to determine the adequate JDK version but do not apply JVMparameters proposed in these SAP notes for the Diagnostics Agent. (They are only intended to SAP J2EEEngines).

2.2 SLD production systemThe SLD CIM Model version must be 1.5.26 or higher to be able to register a Diagnostics Agent.

How to determine if I need to register my agent in SLD or not? Refer to the SAP note 1365123 and its attached document

How to determine which version of CIM Model is installed? Go in SLD with url http://sldhostname:port/sld Logon with J2EE user with role SAP_SLD_ADMINISTRATOR Click on link About, the CIM Model version is displayed in a window popup.

How to upgrade the CIM Model? Refer to SAP Note 669669 / Updating the SAP Component Repository in the SLD


SAP AG 2014

How to check if the Diagnostics Agent is known in a given SLD in order to connect it to Solution Manager? Go to the SLD with URL http://sldhostname:port/sld Logon with J2EE user with role SAP_SLD_ADMINISTRATOR Go to the Administration section For SLDs based on NW 7.10 and higher, click on the link Maintenance / CIM Instances For SLDs based on earlier NW releases, click on the link Content Maintenance Select the subset: All with Instances and Class: SAP_DiagnosticsAgentInstance

N.B.: No Diagnostics Agent is currently registered, in case the class Diagnostics Agent Instance does notappear in drop down list.

In column Name of table, check if the desired Diagnostics Agent is present.Or since Solution Manager 7.1 use rather the step Connect Diagnostics Agent in transaction solman_setup,scenario System PreparationThis step displays for the selected SLD the list of known Diagnostics Agents

Remind that in case the landscape includes several SLDs, having an SLD bridge forwardingin place, the above approach will ONLY allow validating that a specific Diagnostics Agent isregistered in one of those SLDs. But it does NOT necessarily mean that this Agent is directlyregistered in the precise SLD, where the look-up is been performed, like explained above.

This means, to be able to connect Diagnostics Agents using the Connect DiagnosticsAgents feature in solman_setup transaction / System Preparation or in Managed SystemConfiguration / Assign Diagnostics Agent, it is important to select the SLD in which theDiagnostics Agent is directly registered. This direct registration target SLD is decided andconfigured in most cases at Diagnostics Agent installation time.

N.B.: Remind to have a look at the runtime.properties file of the Diagnostics Agent, in orderto find out in which SLD the Agent is directly registered in.

Which J2EE Group is assigned to the SLD user used by the Diagnostics Agent? The J2EE SAP_SLD_DATA_SUPPLIER group or the security role DataSupplierLD should be assigned to the

user used by the Diagnostics Agent.

What is the SLD registration frequency of Diagnostics Agent? Every 12 hours, the Diagnostics agent sends an SLD request to update the agent model information.

What is the frequency on which the Diagnostics Agent checks within its associated SLD for SolutionManager connectivity data updates? If the Diagnostics agent is registered in an SLD and not yet connected to a Solution Manager System, it will

check every 20 minutes if an association has been created in that SLD, using the Connect DiagnosticsAgents feature in solman_setup transaction / System Preparation or in Managed System Configuration /Assign Diagnostics Agent.

If the Diagnostics agent registered in an SLD and currently connected to a Solution Manager System, it isgoing to check every hour if a new / modified association has been created in that SLD.


SAP AG 2014

Why is the Diagnostics Agent not visible in the Agent Administration UI? Verify the below listed items, in case the Diagnostics Agent is not connecting to the Solution Manager system

after you used the Connect Diagnostics Agents feature, via solman_setup transaction / scenario: ManagedSystem Configuration / Step: Assign Diagnostics Agent, or scenario: System Preparation.N.B.:Make sure to also search the Agent entry in the Agent Administration UI, using the filter function on the Hostname column (visible when the More details button is enabled).Also, neither use the Server name (as it might be not assigned yet), nor the Fully Qualified Hostname (ashostname resolution issues will for example make the UI display the physical host) as filter criteria.You can access the Agent Administration UI in general via a hyperlink displayed in the above mentionedsolman_setup steps.

o Has the adequate SLD been selected when performing the Connect Diagnostics Agents operation? Seealso above Q&A item.

o Did you wait long enough? See also above Q&A item.o Is the Diagnostics Agent still able to interact with its associated SLD?

Check for SLD CIM or SLD credentials errors in the Diagnostics Agent SMDSystem.?.log log fileso Is the Diagnostics Agent detecting a Solution Manager connectivity data change in the SLD?

Check for other kind of issues in the Diagnostics Agent SMDSystem.?.log log files.Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the scriptsmdsetup (see part 6.1.5 Usage of smdsetup script) instead of using indirectly the SLD to do so.

Note that in case the following message appears in former Agent Candidates UI :

2010.03.17 14:50:26, Managing System '.SystemHome.' is registered in the SLD butthe CIM declaration is not valid, the agents cannot be attached correctly. Check SLDManaging system declaration. (detail: CIM_ERR_FAILED: No such instance:sld/active:SAP_J2EEEngineCluster.CreationClassName="SAP_J2EEEngineCluster",Name=".SystemHome.")

This error simply reports that Solution Manager instance is not found in SLD with the mentionedhostname BUT it does not prevent the agent attachment process to succeed. The UI will ask forthe hostname and port of Solution Manager during the attachment sequence.

This error message is solved in 7.01 SP7 P1 or higher.


SAP AG 2014

3 Agent installation strategy

3.1 Managing system scalingUse the SAP Solution Manager Sizing Toolkit to check if is necessary to configure your solution manager system ascluster environment. If it is necessary we strongly recommend using additional Dialog instances instead of adding Javaserver nodes to already existing Instances (like 1 dispatcher / 2 or more server node). Because the dispatcher is thecritical component to connect the Agents to Solution Manager, it is more interesting to have many dispatchers (andtherefore instances) for failover/load balancing capabilities.Last but not least also consider the currently known limitation documented in SAP Note 1480419 / Limitations inAlerting and Monitoring

3.2 Managed System environmentTo determine the Agent version and the amount of Agents to install (on Managed System hosts) refer to SAPNote 1365123

The following check list must be validated before installing a new Diagnostics agent on a managed host: Check whether a Diagnostics Agent is already installed: Verify if the managed system has been installed

with a SAPInst version greater than NW04S SR3 (see part 1.1 ) or check on the file system whether a folderlike /usr/sap/DAA or /usr/sap/SMD already exists. In both cases an agent is already installed on the system.

If the Diagnostics Agent(s) is/are already installed on the Managed system host(s), check if theAgent(s) is/are present in Agent Administration UI.If not, double check whether the Agent(s) is/are known in the SLD where the managed system isregistered. In this last case connect the Agent to the Solution Manager System using the ConnectDiagnostics Agents feature (or the Agent candidate Management with Solution Manager 7.0 EHP1).Alternatively simply connect the Diagnostics Agent to the Solution Manager system with the scriptsmdsetup (see part 6.1.5 Usage of smdsetup script) instead of using indirectly the SLD to do so.

In case no Diagnostics Agent is installed, first find out the logical hostname or physical/virtual hostname of themanaged system.

Install the Diagnostics Agent following the rules defined in the document of SAP Note 1365123, on

3.3 Managed Systems HA/Failover environment

With Solution Manager 7.1 SP05 and higher refer to the following article to put in place the so called Agents On-the-fly:Diagnostics+Agent+and+HA+support.In the context of earlier Solution Manager Releases, it is recommended to install the Diagnostics Agent onshared/cluster file system. The goal is to move the Agent each time together with the Managed System (only forApplication Server Instances, not ASCS/SCS), but adding the Agent stop, file system un-mounting, file system re-mounting and re-start in the cluster fail-over scripts.In case the Managed instance and Agent are installed on different file system partitions, it is important when theManaged SAP instance and the Diagnostics Agent are moved to another cluster node to mount the AgentInstance file partition before the SAP Instance partition, to allow the Introscope Byte Code Adapter to start.This is due to the fact that the Wily Setup wizard makes the SAP java instance classpath depending on some jar filescontained in the Diagnostics agent folder, especially if the Introscope Byte Code Adapter has been setup with SolutionManager (prior to Solution Manager 7.1 SP05). See also Diagnostics+Agent+and+HA+support for further details.


SAP AG 2014

4 Agent installer

4.1 SAP Host Agent

Startup commands

The SAPHOSTAGENT is located in /usr/sap/hostctrl/exe (on UNIX) On Windows it is usually located in C:\Progam Files\SAP\hostctrl\exe In general you need to work with the root or Administrator user to execute the SAPHOSTAGENT

commandsAction Command

restart SAPHOSTAGENT saphostexec -restart

Check its status saphostexec -status

Check its version saphostexec version

On Windows it is usually installed as a service, meaning that it can be started or stopped via WindowsServices

4.1.1 E2E RCA configuration for Solution Manager 7.0x or higherTo allow the Diagnostics Agent to retrieve CPU, Memory and disk usage via the SAP Host Agent, it is necessary tocheck the SAP Host Agent profile is configured as follow:

Location : "C:\Program Files\SAP\hostctrl\exe\host_profile" or /usr/sap/hostctrl/exe/host_profile

Property to enable the different webservice port type:

service/porttypes = SAPHostControl SAPOscol SAPCCMS SAPControl

4.1.2 E2E RCA configuration for Solution Manager 7.1x onlyFor some E2E RCA applications like LogViewer it is necessary to update the SAP Host Agent profile to add the OSuser of Diagnostics Agent as user authorized to call specific protected methods via dedicated authentication.

Therefore add or update the property service/admin_users in the SAP Host Agent profile "C:\ProgramFiles\SAP\hostctrl\exe\host_profile" or /usr/sap/hostctrl/exe/ host_profile to list all Diagnostics Agent OS users thatare running on that Physical or Virtual host:

For Windows: service/admin_users= ex:service/admin_users= MY_DOMAIN\SAPServiceDAA MY_DOMAIN\SAPServiceDAB

For UNIX:service/admin_users= ex:service/admin_users= daaadm dabadm


SAP AG 2014

4.2 JVM parametersIn this section, you will find the recommended JVM parameters for each JDK vendor 1.4 of older Diagnostics Agents7.00/7.01:

JDK JVM Parameters SAP Note

IBM JDK -DP4ClassLoad=P4Connection -Xj9 -Xgcpolicy:gencon -Xmn50m -Xmx256m-Xms256m 1093831SUN JDK 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128m

n.aSUN JDK 64 bits -DP4ClassLoad=P4Connection Xmx256m Xms256mHP JDK 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128mHP JDK 64 bits -DP4ClassLoad=P4Connection Xmx256m Xms256m

With agent 7.1x or higher an SAP JVM is used to execute the Diagnostics Agent, the recommended JVMparameters are:

JDK JVM ParametersSAP JVM4/5/6 32 bits -DP4ClassLoad=P4Connection -Xmx128m -Xms128m -XX:MaxPermSize=64mSAP JVM4/5/6 64 bits -DP4ClassLoad=P4Connection Xmx256m Xms256m -XX:MaxPermSize=128m

Dont put the java parameter verbose:gc in the Diagnostics Agents JVM parameters, because thisparameter will steadily increase the size of the file work/std_SMDAgent.out

4.3 Startup scripts

Information about Diagnostics Agents 7.11 or higher

With Diagnostics Agents 7.11, 7.20 & 7.30 the startup system is compliant to the SAP startup mechanism. It meansthat the former commands smdstart.sh and smdstop.sh are no longer available.

Under Windows the SAP MMC allows to start or stop the Diagnostics Agent. The Windows service of theDiagnostics Agent is configured to start automatically during the Windows startup.

The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does howeverNOT mean that the Agent is operational / connected to the Solution Manager system. Only the SolutionManager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected tothe Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation)

Under UNIX, while being logged with the relevant Diagnostics Agent OS User, use the following commands tooperate the Diagnostics Agent.

Command Actionstartsap SMDA [-v ] Start the Diagnostics agent. stopsap SMDA [-v ] Stop the Diagnostics agent.

The sapstartsrv process of Diagnostics Agent is automatically started during the boot sequence of system if the sapinitprogram has been installed on the system. The Diagnostics Agents could later be also stopped and started remotelywith the help of the SAP MC or SAP MMC.

Information about former Diagnostics Agents 7.0x


SAP AG 2014

On Windows platforms the SAP MMC allows to start or stop the Diagnostics Agent. The Windows service ofthe Diagnostics Agent is configured to start automatically during Windows startup.

The green status in SAP MMC only indicates that the Diagnostics Agent process is running. It does howeverNOT mean that the Agent is operational / connected to the Solution Manager system. Only the SolutionManager Agent Administration UI allows to confirm that the Diagnostics Agent is successfully connected tothe Solution Manager system and operational. (In the status is gray see 7.2 Agent crash investigation)

On UNIX platform, the commands smdstart.sh, smdstop.sh and smdadmin.sh allow to operate the DiagnosticsAgent. Since 7.0 release of the Diagnostics Agent SAPInst, those commands are included in the PATHvariable of the OS agent user. In that case, the commands can be called from every folder (not only in theAgent script subfolder).

Command Actionsmdstart.sh Starts the Diagnostics agent and the sapstartsrv process.smdstop.sh Stops the Diagnostics agent and the sapstartsrv process

smdadmin.sh Launches jcmon tool to administrate the Diagnostics agentinstance

On Solaris platform, an error (/usr/sap/SMD/exe/smdstart.sh: usage: ulimit [ -HSacdfnstv ] [ limit ])occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.

On HP-UNIX platform, an error (!! ERRROR !! current user is not the owner of this instance )occurs when the smdstart.sh is executed. Check the solution in sap note 1109366.

Exit code of smdstart.sh Reason

0 Success to launch the agent process (This is not the confirmation that the agent isconnected to SMD).2 Input Parameters are invalid.5 Agent is already running.4 User limit are not set correctly.3 Current OS user is invalided to start the agent.


SAP AG 2014

For UNIX system, the Diagnostics Agents do not start automatically when the system reboots because the DiagnosticsAgent startup scripts are not included in the system startup scripts.

The following command line can be used in the system startup scripts:

And in the shutdown sequence:

The following procedure describes how to insert the agent in the boot sequence for a Linux SUSE 10 system:

Create a file diagnostics_agents in folder /etc/init.d Determine the init run level in file /etc/inittab Create a symbolic link in folder /etc/init.d/rc.d/ to file

diagnostics_agents with name S80diagnostics_agents Create a symbolic link in folder /etc/init.d/rc6.d to file diagnostics_agents with name

S80diagnostics_agents. (runlevel 6 is System reboot)

When smdstart.sh and smdstop.sh commands are embedded in the system startup scripts, the exit code ofthe smd scripts cannot be retrieved. Further more it is important to redirect the standard error output into thestandard output with a syntax like 2>&1 to avoid that the smdstart.sh and smdstop.sh are blocking the parentscript.

su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm

su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null 2>&1 -s /bin/sh - smdadm

#!/bin/sh### BEGIN INIT INFO# Provides: diagnostics_agents# Required-Start: $network# Required-Stop: $network# Default-Start: 2 3 5# Description: start diagnostics agents### END INIT INFO

case "$1" in'start') su -c /usr/sap/SMD/J98/script/smdstart.sh > /dev/null 2>&1 -s /bin/sh - smdadm ;;'stop') su -c /usr/sap/SMD/J98/script/smdstop.sh > /dev/null -s /bin/sh - smdadm ;;'status') su -c /usr/sap/SMD/J98/script/smdadmin.sh -s /bin/sh - smdadm

;;*) echo "Usage: $0 { start | stop | status }" ;;esacexit 0


SAP AG 2014

About sapstartsrv process

SAP Note Description927637 - Web service authentication insapstartsrv as of Release 7.00

The Web service interface of sapstartsrv requires a user authenticationfrom the client (MMC, SAP MC, sapcontrol and so on) for protectedoperations such as starting and stopping the SAP instance.

Note 823941 - SAP Start Service on Unixplatforms

After an SAP upgrade, the SAP Start Service (sapstartsrv) on UNIXplatforms is not automatically started when the operating system isstarted.


SAP AG 2014

5 Agent connectivity

5.1 Managing systemThis section describes the different primary connection modes used by the Diagnostics Agents to connect to theSolution Manager System.

Two modes are available:x Connection through the Solution Manager SAPJ2EE Message serverx Direct connection with the Solution Manager SAPJ2EE P4 dispatcher.

Connection through the Solution Manager SAPJ2EE Message server

It is RECOMMENDED to connect the Diagnostics Agent to Solution Manager Java stacks using the J2EEMessage Server. To establish this connection the hostname and port of the Solution Manager J2EE Messageserver (Java SCS instance) have to be found out using for example the SCS instance profile.

The Agent connection mechanism can be described mainly as follow:

A. During the installation of a Diagnostic Agent the SAPJ2EE Message server connection mode has to beselected.

B. This information is stored in the Agent configuration (file runtime.properties) and the credentials are storedin the Agent secure store (file secstore.properties).

C. When the Diagnostics Agent starts, the P4 connection framework will retrieve from the SCS MessageServer (on HTT port ex: 81xx) all J2EE entry points that are available.

D. Once the list of J2EE entry points is collected, one P4 entry point is chosen from the list.E. The Diagnostics Agent will then establish a P4 connection to the selected J2EE entry point with the host

name and the port corresponding to that P4 connection.


SAP AG 2014

Troubleshooting Message server connectionBefore starting the Agent installation or before executing the smdsetup script, you can double check thecontent of the Java SCS Message Server with the following description:1. Open the URL: http://msgserver:81xx/msgserver/text/logon (xx corresponding to instance number

of the SCS instance)2. Check if a P4 entry point is available with the adequate hostname3. On the managed system host, open a shell as adm OS user (from Managed system or Agent)4. Execute the command ping .

Double check that the host is accessible5. Execute also the command telnet .

If no error message occurs (blank screen) the connection is OK.

Most common agent connection problems

All connection problems are logged in a file under SMDAgent/log/SMDSystem.X.log. Check this log file tofind the root exception of the issue: Once the error exception is found also locate the root cause of theexception by looking at the last caused by sentence.

No Entry point found

Exception: com.sap.engine.services.jndi.persistent.exceptions.NamingException:Exception while trying to get InitialContext. [Root exception iscom.sap.engine.interfaces.cross.DestinationException: there are no access points forservice: P4 registered on the message server]

No P4 entry point has been reported by the Message Server. This is probably due to the fact that it is notthe appropriate Message Server. It has to be the Solution Manager Java stack and not the ABAP stackMessage Server. Also check the text display of the Message server content using the URLhttp://msgserver:81xx/msgserver/text/logon. Verify if a P4 entry point is available (as shown below)

Unknown host

Exception: [sapms:// yupxxxxx.domain.com:8136] Checking server availability...Connecting to SMD server sapms://yupxxxxx.domain.com:8136 failed - error counter: 0- com.sap.engine.services.jndi.persistent.exceptions. NamingException:Exception while trying to get InitialContext. [Root exception iscom.sap.engine.interfaces.cross.DestinationException: cannot establish connection with any of the available instances #Nested exceptions are: #java.net.UnknownHostException: yupxxxxx: yupxxxxx ]

It is a network resolution problem due to the fact that the selected P4 hostname retrieved from the SCSMessage server list cannot be resolved from the Managed system host. Retrieve the list of the P4 entriesin the Message server which looks like:

version 1.0J2EE933894500J2EE NCED6018XXXXA.dhcp.ncel.sap.corp 59300 LB=1J2EES NCED6018XXXXA.dhcp.ncel.sap.corp 59301 LB=1P4 NCED6018XXXXA.dhcp.ncel.sap.corp 59304 LB=1P4S NCED6018XXXXA.dhcp.ncel.sap.corp 59306 LB=1P4HTTP NCED6018XXXXA.dhcp.ncel.sap.corp 59305 LB=1

Execute the command ping in a Managed system shell.If the Solution Manager host cannot be reached, then change the host definition in your DNS or in the/etc/hosts. You should ideally fix you engine installation to report the right host in the Message Server.


SAP AG 2014

Agent Maintenance Mode (required during LM-SERVICE deployment)

As documented in various notes and guides, the Maintenance mode need to be enabled before deployinga new LM-SERVICE Release, Support Package or Patch, as described also in SAP note 1483508.

If this operation is not performed the Diagnostics Agent runtime behavior will in most cases be affected.As a consequence the Diagnostics Agent logs might for example contain errors like:

[SMDJNDILookupAgentSideImplementation] lookup/smdserverctx/ObjAutoRegistered/SMD_DataEnrichment_Registrator failed[]Object not found in lookup of SMD_DataEnrichement_Registrator

Solution Manager SAPJ2EE P4 dispatcher (SAP does NOT recommend to use this mode)

Solution Manager in cluster environment

If the Solution Manager system is installed as a cluster (having Dialog Instances), the direct P4 Agentconnection should not be used as the Diagnostics Agent would else always establish a connection tothe same Solution Manager Java instance corresponding to the selected P4 host/port. Also from asomehow High Availability point of view, you have to use a connection through Solution Manager J2EEMessage server. In that situation, the connection of the Diagnostics Agent will be each time load-balanced to the different available Java instances.

5.2 Managed systemAll connections from the Diagnostics Agent to the Managed systems are executed locally on the same host. It is thereason why it is really imperative to install the Diagnostics Agent on the same Physical or Virtual host than the SAPManaged system.

The Agent connects to different SAP System layers as follow: Java stack on port P4 (5xx04) Sapstartsrv http port of Managed System (5xx13) J2EE Message server (39xx and 81xx) Local RFC call to Managed ABAP stack Webservice call to SAP host agent on port 1128 for agent 7.1x


SAP AG 2014

Troubleshooting P4 connection problems with Managed system

1. Check on which host the P4 socket is bound on the Managed systema. Run the OS command netstat an to find the entry line containing the P4 port of the

managed system:

TCP 0.0.0.0:59304 0.0.0.0:0 LISTENING

Ortcp 0 0 *.50304 *.* LISTEN

In this example we see that the P4 port is bound for all hosts where a p4 request can beemitted (* or 0.0.0.0), if the port is bound like 10.45.192.204:50304 only the p4 requestsemitted from the host 10.45.192.204 will be accepted.It explains why the Diagnostic Agent must be installed on Logical hostname 10.45.192.204 withSAPInst. The situation can be fixed by changing the profile property SAPLOCALHOST (see6.1.4 Verifying Logical hostname).

b. After verifying the port in the OS configuration, check in the J2EE dispatcher configuration thatthe property bindHost is set to 0.0.0.0. This should be visible with the J2EE configuration toolof the managed system->Instance_xxx->dispatcher->services->p4. If the value is notconfigured to 0.0.0.0 (which accepts all hosts), add the Agent IP or install the Agent usingon one of the listed IP corresponding to logical hostnames running on the SAMEunderlying physical or virtual host.

2. Check if the Agent can access the J2EE Message Server (SCS) of the Managed system with the hostand the http port stored in the Agent configuration (see the next section). For example :

Open a shell with the agent OS user (adm) on the Managed System Execute the command msprot -l -mshost -msserv 39

or 36 -d J2EE s, the values of host and port must be retrieved from the Agentconfiguration (see the next section)

The command must return the following content :

SAP Message Server Protocol Program, Version 1.3 (built: Jan 24 2008 01:27:19)

INFO CLUSTER-ID HOST GROUP-ID STATE J2EE-SERVICE-MASK

----------------------------------------------------------------------------------------------


SAP AG 2014

---> LIST SERVICE:00000 NAME:ConfigurationManager

---> LIST SERVICE:00001 NAME:_CacheManager

---> LIST SERVICE:00003 NAME:{service_manager_deploy_distributor}

---> LIST SERVICE:00004 NAME:{service_manager_internal_connection}

---> LIST SERVICE:00005 NAME:shell

---> LIST SERVICE:00006 NAME:jmx_notification

---> LIST SERVICE:00007 NAME:p4

---> LIST SERVICE:00008 NAME:webservices

---> LIST SERVICE:00010 NAME:iiop

---> LIST SERVICE:00011 NAME:r3startup

---> LIST SERVICE:00012 NAME:telnet

---> LIST SERVICE:00013 NAME:log_configurator

---> LIST SERVICE:00014 NAME:jmx

---> LIST SERVICE:00016 NAME:http

---> LIST SERVICE:00022 NAME:com.sap.security.core.ume.service

---> LIST SERVICE:00023 NAME:security

---> LIST SERVICE:00024 NAME:tc~sec~certrevoc~service

---> LIST SERVICE:00025 NAME:servlet_jsp

---> LIST SERVICE:00026 NAME:webdynpro

LIST 0004105250 ncevhost.mydomain 0000041052 RUNNING XXXXXXXXOXXOXXXXXOXXXXXXXXXXXX

LIST 0004105200 ncevhost.mydomain 0000041052 RUNNING XXXXXXOXXXXXXXXXXOOOOOOOOOOOOO

In this content, the P4 service is listed as well as the J2EE nodes with hostnames (e.g.ncevhost.mydomain).Check if those hostnames can be resolved on your network by executing the command:ping ncevhost.mydomain

3. Open the J2EE Visual admin tool of the Managed system and connect with the J2EE user used by theDiagnostics Agent (see in the next section the Agent configuration) to check if the user can beauthenticated and if it has the Administrator role.

4. Check also the Managed system defaulttrace file to find some exception related to P4 connections.


SAP AG 2014

Connection configuration for managed systems

The following table lists all properties generated and updated by the Managed Setup Wizard or solman_setup withSolution Manager 7.0 EHP1 and higher. These properties allow the Diagnostics Agent to connect to the Managedsystem in order to complete the E2E RCA operations. They are accessible using the Agent Administration UI /Application configuration.

Those properties are stored by server name (scope) of the Diagnostics Agent.Connection Type toManaged system

Port Configuration stored in agelet Property

P4 port 5xx04 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/P4/porthttp port 5xx00 com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/portJ2EE Message server InternPort

39xx com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/port

J2EE Message server Httpport

81xx com.sap.smd.agent.application.global.configuration [SID]/[INB]/sapj2ee/http/port

J2EE Message server host com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/msgserver/host

For the credentials :Connection Type toManaged system

Configuration stored in agelet Property user Property password

J2EE credentials com.sap.smd.agent.application.global.configuration [SID]/sapj2ee/com/user [SID]/sapj2ee/com/pwd

Connection Typeto Managedsystem

Configuration stored in agelet Propertyclient

Property user Property password

ABAP credentials7.0x

com.sap.smd.agent.application.global.configuration

[SID]/its/client [SID]/abap/com/user [SID]/abap/com/pwd

ABAP credentials7.1x

com.sap.smd.agent.application.global.configuration

[SID]/abap/client [SID]/abap/com/user [SID]/abap/com/pwd

It is recommended to change those values in the Managed system setup wizard or solman_setup with SolutionManager 7.0 EHP1 and higher instead of directly editing them in the Agent Admin tool. Those values could however bechanged using the Agent Administration tool in case of troubleshooting/debugging.

5.3 Self-ManagementDuring a Self-management setup of Solution Manager, some special configurations in the different setup steps areneeded due to some restrictions about the local Diagnostics Agent connection sharing.

Possible error in Managed system Configuration

End of stream is reached unexpectedly during input from Socket

Exception:com.sap.smdagent.plugins.connectors.p4.exceptions.P4AuthorizationException: Access isdenied: Please make sure the and parameters are correct.; nestedexception is:com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exception duringgetInitialContext operation.No server is running. [Root exception iscom.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating newRemoteLoginContext instance.]atcom.sap.smd.agent.plugin.connectors.p4.P4ConnectionService.getP4Connection(P4ConnectionService.java:69)...at java.lang.Thread.run(Thread.java:838)Caused by: com.sap.engine.services.jndi.persistent.exceptions.NamingException: Exceptionduring getInitialContext operation.No server is running. Root exception is


SAP AG 2014

com.sap.engine.services.security.exceptions.BaseLoginException: Exception in creating newRemoteLoginContext instance.atcom.sap.engine.services.security.remote.login.RemoteLoginContextExt.(RemoteLoginContextExt.java:34)...at java.lang.Thread.run(Thread.java:838)Caused by: java.io.EOFException: End of stream is reached unexpectedly during input fromSocket [addr=/56.zzz.xx.xx,port=50204,localport=22326]at com.sap.engine.services.rmi_p4.Connection.run(Connection.java:423)

It is a problem with a Java server level. Two possibilities to fix the issue: (See also SAP Note 1473530)

1. Apply the fix (see SAP note ) on java server and use at least the LM-SERVICE 7.0 EHP1 SP7 P1

2. Use the workaround describes in SAP note (adapt the property bindHost).

In a Self-Management scenario there exists another solution

Since Solution Manager 7.0 EHP1 SP21, the Diagnostics Agent configuration property enabling the self-management scenario is added automatically by transaction solman_setup / Managed System Configurationto avoid such a problem. It is named /selfmonitoring/enabled (value: true) and visible in the AgentAdministration under comglobal.configuration.

5.4 SAP Router (moved to Setup Guide)Information was moved to Diagnostics Agent Setup Guide for SWPM 1.0 SP4 (and higher).See SAP Note 1858920: Diagnostics Agent installation with SWPM.


SAP AG 2014

6 Agent Maintenance6.1 Configuration

This section describes all procedures to update Diagnostics Agent configurations that could not be changed/updatedcentrally using the Solution Manager system.

6.1.1 Change Path of the JDKIn the eventuality that the JDK installation path changes after the Diagnostics Agent has been installed, you might needto update this JDK path information within the Agent profile file, as follow.

For 7.00 and 7.01

Open the profile _J_ in folder/usr/sap//sys/profile.

Update the parameter jstartup/vm/home by putting the absolute path to the JDK home. Remove the file /usr/sap//JXX/profile/smd.properties.vmprop. Restart the Diagnostics agent.

For 7.1x , 7.2x and 7.3x

As the Diagnostics Agent 7.1x and higher run using their own SAPJVM, it is not recommended to change thepath of the JDK.

6.1.2 Change JVM ParametersIf the JVM parameters need to be changed, you have to update the Diagnostics Agent instance properties file.

For 7.00 and 7.01

Open the file smd.properties in folder /usr/sap//J/profile. Update the parameter smdagent.javaParameters (see 4.2 JVM parameters). Remove the file /usr/sap//JXX/profile/smd.properties.vmprop. Restart the Diagnostics Agent.

For 7.1x, 7.2x and 7.3x

Open the file smdagent.properties in the folder/usr/sap//SMDA/SMDAgent.

Update the parameter smdagent.javaParameters (see 4.2 JVM parameters). Remove the file /usr/sap//JXX/profile/smd.properties.vmprop. Restart the Diagnostics Agent.

6.1.3 Replacing JDKs 1.4.2 with SAP JVM 4

Some Partner JDK 1.4.2 implementations are going out of maintenance. In this context SAP proposes to replace thoseJDKs with the SAP JVM 4. The manual procedure to be followed concerning the Diagnostics Agents is described in the


SAP AG 2014

SAP JVM Switch Tool guide available under https://service.sap.com/instguides. Especially, consider the chapter 5.Performing the Switch to SAP JVM 4 for the Diagnostics Agent.

6.1.4 Verifying Logical hostnameVerify the logical hostname that was used during the Diagnostics Agent installation, in case a Diagnostics Agent is notreporting the expected logical hostname within the Agent Administration UI.

For 7.01, 7.1x, 7.2x and 7.3x

Open the profile _J_ in the folder/usr/sap//sys/profile.

The parameter SAPLOCALHOST has to be present and contain a short hostname (not the FQN)

6.1.5 Usage of smdsetup script (moved to Setup Guide)Information was moved to Diagnostics Agent Setup Guide for SWPM 1.0 SP4 (and higher).See SAP Note 1858920: Diagnostics Agent installation with SWPM.

6.1.6 Change Agent log level (moved to Setup Guide)Information was moved to Diagnostics Agent Setup Guide for SWPM 1.0 SP4 (and higher).See SAP Note 1858920: Diagnostics Agent installation with SWPM.

6.2 Managing J2EE userInformation about the Solution Manager J2EE user needed by the Diagnostics Agent to connect to the SolutionManager System, is described: For Solution Manager 7.0 / 7.0 EHP1 in the User Administration guide, section 2.3.3 [SOLMAN.AGT.COM]:

Diagnostics agent Communication User. For Solution Manager 7.1 in the Solution Manager Security Guide.

Starting with Solution Manager 7.1 SP5, the Diagnostics Agent can connect to Solution manager server using anauthentication via username / certificate, instead of using a basic authentication (user/password).Note that with the certificate authentication the user is not getting locked in case of multiple attempts (performed byseveral Diagnostics Agents).

Pay attention to the following situation

For Diagnostics Agents that have not been connected so far to a Solution Manager 7.1 SP05 system orhigher, remind that the Agent will stop trying to connect/login on the Solution Manager system in case wronguser credentials have been provided (or the user is locked).In that context also be aware that in case the SMD_ADMIN user credentials are changed, offlineDiagnostics Agent will not get notified of that change. At the time these Agents will go online again they willlock the SMD_ADMIN user, in case at least 3 or more Agents are in this situation.

Since Solution Manager 7.1 SP05, consider using the Non-authenticated Agent list, as described by:http://wiki.sdn.sap.com/wiki/display/SMSETUP/Non-Authenticated+Diagnostics+Agents

Most common agent connection problems

Authentication failedall authentication problems are logged in file SMDSystem.log.

Exception: at com.sap.engine.core.thread.impl3.SingleThread.run(SingleThread.java:170)Caused by:com.sap.engine.services.security.exceptions.BaseLoginException:Authentication did not succeed.


SAP AG 2014

atcom.sap.engine.services.security.login.ModulesProcessAction.run (ModulesProcessAction.java:175) ... 13 more

Before restarting the agent It is important to execute some checks about the J2EE user as follow:

1. Connect to the ABAP transaction SU01, to check if the J2EE user (SMD_ADMIN) is not locked and if theRole assigned is compliant to SAP recommendations done in the User Administration guide.2. Logon to the Agent administration tool with the J2EE user (SMD_ADMIN) to check that the logon module ofthe Java stack works correctly.3. Call the smdsetup to update only the credentials to be sure the password is up-to-date.4. Restart the Diagnostic agent.

In dev_smdagent.log, if you find the exit code -11112 it is a authentication problem as described above[Thr 880] Wed Feb 04 09:28:47 2009[Thr 880] JLaunchIStartFunc: end of main method (rc = 0)[Thr 6000] JLaunchIExitJava: exit hook is called (rc = -11112)[Thr 6000] **************************************************************** ERROR => The Java VM terminated with a non-zero exit code.*** Please see SAP Note 943602 , section 'J2EE Engine exit codes'*** for additional information and trouble shooting.**********************************************************************[Thr 6000] JLaunchCloseProgram: good bye (exitcode = -11112)


SAP AG 2014

7 Tracking Agent issues

7.1 OSS message pre-requisitesBefore forwarding any OSS messages to the Dev support organization, it is important to check the following points (asfar as concerning Diagnostics Agents):

The Diagnostics agent has been installed with SAPinst 7.01 EHP1, 7.11, 7.20, 7.3x or Solution Manager 7.0EHP1 SR1 or 7.1.

The OS limit for Unix platform are compliant with the agent OS user (adm) JDK vendor and version is supported by the Diagnostics Agent JVM parameters of Diagnostics Agent are compliant to recommendations The Diagnostics Agent connection to the Solution Manager system is of type J2EE Message Server connection The Solution Manager Java dispatcher is correctly sized to avoid Agent disconnections (see part 1 .)

In case the check points cannot solve the problem, follow the SAP Note 1076573 to create the OSS message.Also in case the LM-SERVICE version is at least NW04S SPS17 Patch 3, attach the support logs Zip filegenerated by the smdsetup script (see part 6.1.5 Usage of smdsetup script).

7.2 Agent crash investigationWhen the Diagnostics agent process shutdowns abnormally, the procedure to determine the cause is the following:

At OS level login as adm OS user Go in folder /usr/sap//J/work Open the log file dev_jcontrol (This is the log file of the jcontrol process which is controlling the Diagnostics

Agent java process) to check if the jcontrol has started the jlaunch process as described below:

[Thr 3288] JControlICheckProcessList: process SMDAgent started (PID:3616)

If the process jlaunch cannot be start, the error is probably located in SAP profile parameters

The following message is NOT a critical error:JStartupICheckFrameworkPackage: can't find framework package C:\usr\sap\SMD\exe\jvmx.jar

The Diagnostics Agent does not use specific SAP kernel functions and this message can be ignored.

The process jlaunch is started, open the file dev_SMDAgent which corresponds to the jlaunch process output :

[Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Initial (0)][Thr 3860] JLaunchISetState: change state from [Initial (0)] to [Waiting for start (1)]

[Thr 3860] Thu Dec 25 12:11:35 2008[Thr 3860] JLaunchISetState: change state from [Waiting for start (1)] to [Starting (2)][Thr 3860] JLaunchISetState: change state from [Starting (2)] to [Running (3)]

In this file, all information about agent java process is stored like JVM loading, process state, crash of VM orbad JVM parameters. It is important to have a look this file to find out Exit Codes or Error messages.

JVM Loading problem

all JVM loading problems are logged in file dev_SMDAgent.


SAP AG 2014

Error: [Thr 1084238144] *** ERROR => DlLoadLib: dlopen()= libjvm.so: cannot open sharedobject file: No such file or directory -> DLENOACCESS (0, Success) [dlux_mt.c 314][Thr 1084238144] *** ERROR => Can't load VM shared library (libjvm.so) (rc=-2)[jhvmxx_mt.c 1576][Thr 1084238144] *** ERROR => Cannot load DLL for Java VM [jlnchxxi_mt.717]

Here you can find potential JDK installation issues like : Wrong JDK version for Agent JDK is not compatible with SAP Kernel (32bits/64bits) Diagnostics Agent OS user have not rights to access to JDK Libraries

The SAP Note 940893 - Analysis of errors within the startup of J2EE Engine 6.40can help to troubleshoot JVM loading problems.

If the JVM is loaded correctly, now check the file std_SMDAgent.out and jvm_SMDAgent.out to find someerrors like OutOfMemory, StackOverflow, Nullpointer which stop the agent process.


SAP AG 2014

7.3 Activate P4 logsIf a p4 connection problem (like timeout, connection reset) persists with the Solution Manager system or the ManagedSystem after applying the different checks of this document, you should enable the P4 logs before sending the OSSmessage to the Development Support organization.

Connection problem to Managing System or Managed system

The procedure to enable the P4 logs on the Diagnostics agent is the following:

Stop the agent Clean the log folder under SMDAgent directory Set the log level (effective-severity) of SMDSystem location to ALL (see 6.1.6 Change Agent log level) Set the log level (effective-severity) of P4 location to ALL (see 6.1.6 Change Agent log level) Add the Java parameter -Ddebug=true in agent configuration (6.1.2 Change JVM Parameters)

Once the p4 log is enabled on the Agent, go to the Solution Manager system, or the Managed system to execute thefollowing instructions:

Open the Visual Administrator of Java stack Select the server node Under SID node Select the service Log Configurator Go in tab Locations Select the location com.sap.engine.services.rmi_p4 Increase the log level to DEBUG and save the modification.

Once all P4 locations are configured, restart the agent to reproduce the problem and attach the agent logs (use thecommand smdsetup supportlogs) and the current defaultrace.log of the Managing or the Managed system to the OSSmessage.

Performance Impact

It is important to restore All log locations changed to their default level because the above describedprocedure has a high performance impact on the Diagnostics Agent and therefore indirectly on the Managedsystem or the Solution Manager system.


SAP AG 2014

8 Telnet SMD server commandsSome telnet commands are available in the Solution Manager J2EE to check the installation and configuration of theSMD server.

Connect with telnet program to SAP J2EE server of Solution Manager (default port 5xx18) and go on server node withcommand jump .Add the group smdserver to enable the commands (use the command add smdserver).Once the group smdserver added,execute the command man g smdserver to have access to all available smd server commands.

SMD server telnet command DescriptionSmdstatus Display the version and running mode of SMD server.smdstatus a Check each agent connectivity to detect some connection problem with one

or several agents.list_agents Display all Diagnostics Agents connected to SMD server with their status.find_agent or Retrieve an Agent using a servername, fully qualified domain name or IP.mm ON Enable the maintenance mode of SMD servermm OFF Disable the maintenance mode of SMD serverkill_agent or Reboot the Diagnostics Agent processstart_agent or Start the Agent applications of selected Diagnostics agentstop_agent or Stop the Agent applications of selected Diagnostics agenthost_mapping Display the mapping between servername and network names for all agents.known_agents -c Displays the list of known Agents. (Agents that have been connected at least

once to the Solution Manager system)known_agents -n Displays the list of known Agents by FQN.As Provides ping statistics for all Diagnostics Agents.Es Provide method statics for the Solution Manager server.


SAP AG 2014

9 Full Qualified Name resolution issue

The list of SAP notes following can help to solve some network resolution problems in different part of SAP products.

SAP Note ResumeNote 611361 - Hostnames of SAPservers

The use of hostnames that are too long or contain illegal characters mayresult in undetermined errors

Note 962955 - Use of virtual TCP/IPhost names

You want to use one or more logical host names for SAP software withinan SAP server landscape to hide the physical network identities for thepieces of software from each other.

Note 773830 - FQHN determination inICM

It is not clear which fully qualified host name (FQHN) the InternetCommunication Manager (ICM) is using.

Note 1282975 - Use of virtual TCP/IPhost names in Windows

You want to use one or more logical host names for SAP software within anSAP server landscape to hide the physical network identities for the piecesof software from each other.

Note 129997 - Hostname and IPaddress lookup

Information about IP address lookup.

Note 1052122 - Host names in SLD You wonder if a hostname that occurs in the System Landscape Directory(SLD) is correct. In the SLD, hostnames primarily occur in technical systemsand destinations

Note 979277 - Incorrect registrationof computer systems in the SLD

In computer systems registered by the ABAP data supplier in the SLD,attributes may be missing in certain cases (for example, details of theoperating system).

diagnosticsagenttroubleshooting-1

Documents

menu names

names ofinstallation

names ofvariables

program names

exampletexttechnical

agent installer

sap host agent

theseinclude field names