cover front cover · student exercises july 2011 edition the information contained in this document...

V11.0

cover

IBM Systems Software Education

Front cover

Course Exercises Guide

AIX Internals & Performance I: Process and Thread ManagementCourse code AHQV471 ERC 1.1

Student Exercises

Trademarks

IBM® and the IBM logo are registered trademarks of International Business Machines Corporation.

The following are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide:

Intel is a trademark or registered trademark of Intel Corporation or its subsidiaries in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Java™ and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

VMware and the VMware "boxes" logo and design, Virtual SMP and VMotion are registered trademarks or trademarks (the "Marks") of VMware, Inc. in the United States and/or other jurisdictions.

Other product and service names might be trademarks of IBM or other companies.

AIX 6™ AIX® DB2®DS8000® Express® FlashSystem™GPFS™ IBM FlashSystem® Power Systems™Power® PowerHA® PowerSC™PowerVM® POWER6® POWER7+™POWER7® POWER8® PurePower System™SystemMirror® Systems Director VMControl™

July 2011 edition

The information contained in this document has not been submitted to any formal IBM test and is distributed on an “as is” basis without any warranty either express or implied. The use of this information or the implementation of any of these techniques is a customer responsibility and depends on the customer’s ability to evaluate and integrate them into the customer’s operational environment. While each item may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar results will result elsewhere. Customers attempting to adapt these techniques to their own environments do so at their own risk.

© Copyright International Business Machines Corporation 2011.This document may not be reproduced in whole or in part without the prior written permission of IBM.Note to U.S. Government Users — Documentation related to restricted rights — Use, duplication or disclosure is subject to restrictions set forth in GSA ADP Schedule Contract with IBM Corp.

Student ExercisesV5.4.0.3

TOC
Contents
Trademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

Exercises description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Exercise 1. Processor Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1

Exercise 2. Process Management Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1

Exercise 3. Scheduling and Dispatching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1

Exercise 4. Monitoring Processor Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1

Exercise 5. Synchronization Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1

Course materials may not be reproduced in whole or in part without the prior written permission of IBM.

© Copyright IBM Corp. 2011 Contents iii

Student Exercises


iv AIX Internals I © Copyright IBM Corp. 2011


TMK
Trademarks
The reader should recognize that the following terms, which appear in the content of this training document, are official trademarks of IBM or other companies:

IBM® and the IBM logo are registered trademarks of International Business Machines Corporation.

The following are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide:

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates.

Other product and service names might be trademarks of IBM or other companies.

AIX 5L™ AIX 6™ AIX®BladeCenter® EnergyScale™ Express®Micro-Partitioning® POWER Hypervisor™ Power Systems™POWER® PowerPC® PowerVM™POWER4™ POWER5™ POWER6+™POWER6® POWER7® pSeries®Tivoli®


© Copyright IBM Corp. 2011 Trademarks v

Student Exercises


vi AIX Internals I © Copyright IBM Corp. 2011


pref
Exercises description
This course includes the following exercises:

• Processor Resources

• Process Management Framework

• Scheduling and Dispatching

• Monitoring Processor Utilization

• Synchronization Mechanisms

In the exercise instructions you will see each step prefixed by a line. You may wish to check off each step as you complete it to keep track of your progress.

Most exercises include required sections which should always be completed. These may be required before performing later exercises. Some exercises may also include optional sections that you may wish to perform if you have sufficient time and want an additional challenge.

This course includes two versions of the course exercises, “with hints” and “without hints”.

The standard “Exercise instructions” section provides high-level instructions for the tasks you should perform. You need to apply the knowledge you gained in the unit presentation to perform the exercise.

The “Exercise instructions with hints” provide more detailed instructions and hints to help you perform the exercise steps.


© Copyright IBM Corp. 2011 Exercises description vii

Student Exercises

Text highlighting

The following text highlighting conventions are used throughout this book:

Bold Identifies file names, file paths, directories, user names, principals, menu paths and menu selections. Also identifies graphical objects such as buttons, labels and icons that the user selects.

Italics Identifies links to web sites, publication titles, is used where the word or phrase is meant to stand out from the surrounding text, and identifies parameters whose actual names or values are to be supplied by the user.

Monospace Identifies attributes, variables, file listings, SMIT menus, code examples and command output that you would see displayed on a terminal, and messages from the system.

Monospace bold Identifies commands, subroutines, daemons, and text the user would type.


viii AIX Internals I © Copyright IBM Corp. 2011


EXempty
Exercise 1. Processor Resources
What this exercise is about

This exercise provides you with experience obtaining processor resources information in the AIX partition. In addition, this exercise contains activities in which you will display and examine information regarding the kernel structures and hypervisor calls.

What you should be able to do

At the end of the exercise, you should be able to:

• Obtain information regarding the processor resources of your partition

• Use kdb to examine processor-related information

• Display and examine information regarding hypervisor calls

Requirements

In the lab environment for this class, each lab team will be assigned a logical partition (LPAR) on a managed system. The assigned logical partition should be running AIX 7 on a POWER6 or POWER7 processor-based system.

You will not be sitting directly in front of your lab system. Instead, you be using your personal PC to connect to your lab system.

Your instructor will need to assign you to a lab team and assign a lab system to your lab team before you can start the exercise. Your instructor will also need to provide you with any required passwords, addresses, system names, and additional procedures required to access your lab system.


© Copyright IBM Corp. 2011 Exercise 1. Processor Resources 1-1

Student Exercises

Exercise instructions

Preface

This exercise includes information for you to read, and exercise steps for you to perform. The following examples illustrate the numbered checklist format used to identify exercise steps:

__ 1. (This is example step one.) Login to ...

__ 2. (This is example step two.) Execute the following ...

Two versions of these instructions are available: one with hints and one without. You can use either version to complete this exercise (or flip back and forth between the two versions). In other words, use these two versions of the exercise in whatever way best aids your learning. Also, please don’t hesitate to ask the instructor if you have questions.

Note: Unless otherwise specified, all the command and output examples were captured on a partition running AIX 7 on a POWER7 processor-based system.


1-2 AIX Internals I © Copyright IBM Corp. 2011


EXempty
Section 1 - Getting started with exercises
__ 1. Obtain the following information from your instructor:

__ a. Lab system hostname: ___________________________________

__ b. Lab system IP address: __________________________________

__ c. root password of the lab system: ___________________________

__ 2. Login to your assigned lab system as the root user.

__ 3. Use the prtconf command to obtain the CPU type, kernel type, LPAR number, LPAR name, and processor clock speed.

__ a. CPU type: _______________________________________

__ b. Kernel type: ______________________________________

__ c. LPAR number: ____________________________________

__ d. LPAR name: ______________________________________

__ e. Processor clock speed: _____________________________



Student Exercises

Section 2 - Obtaining kernel-related information

__ 4. Start the kdb command, and answer the following:

__ a. Starting address of the pvproc table: ______________________________

__ b. Starting address of the pvthread table: ____________________________

__ 5. Use the status subcommand of kdb to obtain information about what is currently running on each processor. This subcommand displays information about the current selected thread, and the process that owns the thread, for each logical CPU. This is particularly useful on a multiprocessor system because it allows you to see what each CPU has selected as the currently running thread. However, you should note that this output could be misleading since the thread that is shown in the output of the status subcommand may in fact have been displaced by an interrupt handler.

__ a. Logical Processor 0: ____________

__ b. Logical Processor 1: ____________

__ c. Logical Processor 2 (if running on POWER7 processor-based system): ______

__ d. Logical Processor 3 (if running on POWER7 processor-based system): ______

__ 6. Use the q subcommand of kdb to quit (exit) from kdb.

__ 7. Use the ls command to verify that /unix and /usr/lib/boot/unix are symbolic links. Which file do the links refer to? ______________________________




EXempty
Section 3 - Obtaining information regarding processor resources
__ 8. The lparstat command reports LPAR-related information and statistics. When the lparstat command is invoked with no flags or other parameters, the resulting output will show partition configuration information and average utilization statistics from when the system was last booted.

Run the lparstat command and answer the following:

__ a. Is your partition a dedicated processor partition or a shared processor partition? ____________________

__ b. How many physical processors are there in the shared processor pool? ______________

__ c. What is the entitled processing capacity of your partition? _________________

__ d. Is your partition processor capacity capped or uncapped? ______________

__ e. Is your partition running with simultaneous multi-threading enabled or disabled? ____________

__ f. How many online logical processors are available for the entire partition? _____

__ 9. You can list the physical or virtual processors for your partition using the command lsdev -Cc processor. When a partition is using dedicated processors, lsdev shows physical processors. When a partition is using shared processors, lsdev shows virtual processors (that is, the number of physical processors the partition thinks it has).

Run the lsdev command to list the physical or virtual processors for your partition. Does proc0 represent a physical processor or a virtual processor? ________



Student Exercises

Section 4 - Obtaining hypervisor-related information

__ 10. The lparstat command reports LPAR-related information and statistics. When the lparstat command is invoked with the -h flag (and no other flags or other parameters), the resulting output will show the percentage of available processing time spent in hypervisor mode and the number of hypervisor calls since the last time the partition was booted (as well as partition configuration information and other utilization statistics). Invoke the lparstat command with the -h flag (and no other parameters) to obtain this summary of hypervisor call activity for your system, and answer the following:

__ a. How many hypervisor calls have been made since your partition was rebooted? ___________________

__ b. What percentage of available processing time has been spent in hypervisor mode since your system was rebooted? ___________

__ 11. When the lparstat command is invoked with the -H flag (and no other flags or other parameters), the resulting output will show detailed information regarding hypervisor calls since the last time the partition was rebooted.

Invoke the lparstat command with the -H flag (and no other parameters) to obtain detailed hypervisor call information for your system, and answer the following:

__ a. What was the most frequently invoked hypervisor call in your partition since it was rebooted? ________________________________

__ b. What percentage of hypervisor time has been spent in calls of the most frequently invoked type since your system was rebooted? ____________

__ 12. Let your instructor know that you have completed the exercise.

End of exercise




EXempty
Exercise 2. Process Management Framework

This exercise provides you with experience obtaining information related to the use of kernel extensions on your system. You will also be asked to locate and interpret the contents of system header files. In addition, you will use the ps and kdb commands to look at process and thread related information. You will also use a variety of kdb subcommands to examine process and thread related data from a running kernel.



• Obtain information related to the use of kernel extensions on your system

• Locate and interpret system header files

• Display process-related information from the command line

• Locate and examine the process table and the thread table

Requirements





© Copyright IBM Corp. 2011 Exercise 2. Process Management Framework 2-1

Student Exercises


Preface









EXempty
Section 1 - Obtaining kernel extension-related information

__ 2. The genkex command can be used to display a list of kernel extensions currently loaded in the kernel on your system, detailing the kernel address, size, and path name for each loaded extension.

Invoke the genkex command with no parameters.

__ 3. Examine the output of the genkex command. For kernel extensions loaded onto the system, the kernel maintains a linked list consisting of data structures called loader entries. A loader entry contains the name of the extension, its starting address, and its size. Observe the name of the file of each kernel extension. It is in fact a full path name from where the system loads the extension.

Use the file command to examine the aio kernel extension. Is it 32-bit or 64-bit executable? ________________________

__ 4. As you observed in the previous step, all files presented by the genkex command are 64-bit executable modules. Why do you not see any 32-bit kernel executables if AIX 7 provides support for both 32-bit and 64-bit user processes? _______________________________________________________________

__ 5. Determine the name of the fileset that owns the kernel extension /usr/lib/drivers/nfs.ext by using the lslpp command with the -w flag. _______________________________

__ 6. Display the description of the fileset by using the lslpp command with the -L flag. _________________________________________________

__ 7. What type of functionality is implemented by the /usr/lib/drivers/hd_pin file? ___________________________________________

__ 8. In a previous step, you used the genkex command to list the currently loaded kernel extensions. The kdb subcommand lke can also be used to list the loaded kernel extensions. When invoked with no arguments, the lke subcommand shows a one line summary of each loader_entry structure in the kernel load list. Enter the lke subcommand at the kdb prompt and examine the output of this subcommand. Remember that kdb has a built-in pager; press the <Enter> key to obtain a new page of output.

__ 9. Use the q subcommand of kdb to quit (exit) from kdb.



Student Exercises

Section 2 - Using system header files

__ 10. Change your working directory to /usr/include.

__ 11. The /usr/include directory contains a significant number of subdirectories. Determine the number of subdirectories in the /usr/include directory on your system. ______________________________

__ 12. Use the view command to open the proc.h header file that is located under the sys subdirectory. This header file contains definitions of the proc structure and the pvproc structure, two important structures used by the kernel to hold information about processes.

__ 13. Locate the definition of the proc structure (struct proc) in the proc.h header file.

Examine the proc structure and answer the following questions:

__ a. What is the first member of the structure? What data type is it? ____________

__ b. What structure member contains the number of active threads? ____________

__ c. What data type is used in the definition of p_sig? _______________________

Take some time and examine the other members of the proc structure.

__ 14. Locate the definition of the pvproc structure (struct pvproc) in the proc.h header file.

Examine the pvproc structure and answer the following questions:

__ a. What is the first member of the structure? ____________________________

__ b. What information is stored in the pv_ppid member? ____________________

Take some time and examine the other members of the pvproc structure.

__ 15. Quit view by typing :q!<Enter>.

__ 16. Use the view command to open the thread.h header file that is also located in the sys subdirectory. This header file contains definitions of the thread structure and the pvthread structure, two important structures used by the kernel to hold information about threads managed by the kernel.

__ 17. Locate the definition of the thread structure (struct thread) in the thread.h header file.

Examine the thread structure. One of the members of the structure is a pointer called t_procp. What type of data structure does this field point to? _________________________

Take some time and examine the other members of the thread structure.

__ 18. Locate the definition of the pvthread structure (struct pvthread) in the thread.h header file.

Take some time and examine the pvthread structure.




EXempty
__ 19. Exit view by typing :q!<Enter>.


Student Exercises

Section 3 - Displaying process and thread related information

__ 20. What command is most often used to view processes running on a specific system? _____________

__ 21. Use the ps -Af command to display information about all processes.

__ 22. What is the PID of the srcmstr process? ___________________________

__ 23. Use the following command to list the active processes for which srcmstr is the parent process:

# ps -Af | grep <PID of srcmstr> | grep -v srcmstr | grep -v grep

__ 24. Now, use the -T flag of the ps command to display a tree showing the process hierarchy rooted at the PID of srcmstr. All of the processes listed in the previous lab step should appear in the tree displayed. Use the following command:

# ps -T <PID of srcmstr>

__ 25. Now, display the hierarchy of the init process (PID=1).

__ 26. Use the -k flag of the ps command to list the current kernel processes on your system.

Locate and examine the following processes in the ps -k output:

__ a. wait: If AIX finds no other thread that is ready to run on a particular CPU, it will run a thread of a wait process. There will probably be several wait processes listed in the ps -k output (one for each logical CPU).

__ b. swapper: This kernel process runs the scheduler code that will be discussed later in this course.

__ 27. Use the -e flag of the ps command to list information about all processes, except kernel processes.

Does the ps -e output list kernel processes such as wait and swapper? _______




EXempty
Section 4 - Displaying process and thread information using kdb
__ 28. Invoke the kdb command.

__ 29. Use the status subcommand of kdb to obtain information about what is currently running on each logical processor.

__ 30. A process is a self-contained entity that consists of the information required to run a single program, such as a user application. A process is divided into two components:

• A collection of resources • A set of one or more threads

The p subcommand of kdb displays process table entries. With the * argument, p displays a summary of all entries in the process table. Detailed information about a single process can be displayed by specifying a slot number or the effective address of a process table entry. Without any arguments, p displays details for the current process.

Use kdb to display a summary of all entries in the process table.

The p subcommand displays many values, including process ID (PID) and parent process ID (PPID) in hexadecimal. The table slot number (SLOT) is shown in decimal.

__ 31. Use the p subcommand without any arguments to display the current process. Then use the cpu subcommand to change context to each logical processor and run the p subcommand again to display its current process. When you are finished switch back to processor 0.

__ 32. As explained before, a process has a set of one or more threads. A thread can be thought of as a path of execution through the instructions of the process. The kernel maintains tables that contain information regarding processes and threads on the system.

The thread subcommand (thread/th) of kdb displays thread table entries. If no argument is specified, details for the current thread are displayed. The * argument displays a summary of the entire thread table. Details for a specific thread can be displayed by specifying a thread_slot or an address.

Display a summary of the entire thread table.

Do you see several threads with gil in the column labeled NAME? ________

If so, note the slot number of two of these threads: ____________________

__ 33. Use the command th <thread_slot> to examine the details for the two gil threads you noted in the previous lab step.

Look for the “DATA.........pvprocp :” field. This is a pointer back to the process table. Are the two threads (from gil) that you have identified in fact owned by the same process? ____________________



Student Exercises

__ 34. Use the p * subcommand to re-examine the process table. Locate the entry for gil and examine the last column of the entry, which gives the number of threads for the process (column heading #THS).

Does this column indicate that gil has more than one thread? ______

How many threads? _________

__ 35. Exit out of the kdb command.




EXempty
Section 5 - Correlating output from the ps and kdb commands
__ 36. Invoke the ps -ekmo THREAD command. Locate the syncd process in the output of this command. How many threads does the syncd process have? _________

__ 37. Record the PID of the syncd process, and the TID value for two of its threads.

PID = _______________________

TID = _______________________

TID = _______________________

Note: These values are in decimal.

__ 38. Start the kdb command, and turn off the hexadecimal_wanted switch to have the PID and TID information displayed in decimal format. Use the following kdb subcommand:

(0)> set hexadecimal_wanted false

__ 39. Use kdb to locate the process slot number for the syncd process.

Process slot number = __________________________

Note: This value is in decimal.

__ 40. Use the th subcommand to get the slot number for the two threads of the syncd process whose TIDs you recorded in a previous lab step.

Thread slot number = __________________________

Thread slot number = __________________________

Note: These values are in decimal.

__ 41. Use the kdb command to examine the syncd process and thread data structures using the slot numbers obtained in the previous steps.

We do not expect you to be able to read these structures or to even become totally familiar with everything in them. It is the goal of this step just to show you how to find them and then be able to tie them back together.



Student Exercises

Section 6 - Using additional kdb subcommands

__ 42. The ppid subcommand of kdb displays the process table entry for a specified PID. Use the ppid subcommand to examine the process table entry of the syncd process. Utilize the PID value you recorded in Step 37.

__ 43. The ptid subcommand of kdb displays the process table entry for the process that owns the thread with a specified TID. Use the ptid subcommand to examine the process table entry of the syncd process. Utilize one of the TID values value you recorded in Step 37.

__ 44. Is there any difference between the output of the ppid and ptid subcommands?

__ 45. The ttid subcommand of kdb displays the thread table entry that has a specified TID value. Use the ttid subcommand to examine the thread table of the two threads of the syncd process you noted in Step 37. The -d parameter allows you to specify the TID in decimal.

__ 46. The tpid subcommand will display a one line summary of all of the threads that belong to a specified PID. Use this subcommand to display the threads of the syncd process you noted in Step 37. The -d parameter allows you to specify the PID in decimal.

__ 47. Is there any difference between the output of the two subcommands you ran in the two previous steps? __________________

__ 48. End your kdb session.


End of exercise




EXempty
Exercise 3. Scheduling and Dispatching

This exercise provides you with experience using the ps, vmstat, mpstat, sar and kdb commands to view and observe AIX process and thread management mechanisms in action. In addition, you will be asked to examine signal-related information stored in structures used by the kernel.



• Use the ps, vmstat, mpstat, and sar commands to view general process and thread related information

• Use kdb to locate and examine process-related and thread-related information used by the kernel

• Modify thread priorities and use schedo to modify the scheduler algorithms

• Examine run queue and context switches

• Examine signal-related information stored in structures used by the kernel

Requirements





© Copyright IBM Corp. 2011 Exercise 3. Scheduling and Dispatching 3-1

Student Exercises


Preface









EXempty
Section 1 - Examining process and thread scheduling information

__ 2. Use the ps command to display all of the processes running on the partition, along with their associated priority values.

Note: The priority value displayed for each process is the “most favored” priority found for any of the threads owned by the process.

__ 3. What is the numerically lowest (most favored) priority value in use on the running system? ___________________________________

__ 4. You will observe that some processes have a “--” in the NI column of the ps output. What does that tell us about these processes? ___________________

__ 5. Use the ps command to view all of the threads on the system and their associated priority values.

__ 6. Change directory to /u/QV471.

__ 7. Run the rundifft program. This program creates several threads running with different priorities, scheduling algorithms, and processor bindings. The threads do not do anything; they are just for you to look at and analyze.

The rundifft program will print out its PID and the TIDs of the threads. Leave the program running.

__ 8. Open another window, and login to your assigned system as the root user.

__ 9. Make sure the rundifft program is still running in one window, then in the other window, use the ps command to verify that rundifft is running and to view the threads that are associated with this program.

__ 10. Record the PID and TID values of the rundifft process and any threads it starts. Remember that when you use the ps command the process and thread ID’s are displayed as decimal values.

PID = ____________________________

TID = ____________________________

TID = ____________________________

TID = ____________________________

TID = ____________________________

__ 11. Start the kdb command, and then use the appropriate kdb subcommand to display process table information to locate the entry for the rundifft process.

Record the slot number of the rundifft process. You will need it later in this exercise.

rundifft process table slot number = __________



Student Exercises

__ 12. Display the detailed process table entry for the rundifft process using the appropriate kdb subcommand.

Note: Process-related kdb subcommands show information from both the pvproc and proc structures.

__ 13. Examine the kdb output you obtained in the previous step, and answer the following questions:

__ a. What is the state of the process? ________________________________

__ b. How many threads does the process have? _________________________

__ c. Does the process have any children? _______________________________

__ d. What is the address of the pvthread structure used by the first thread of the process? _____________________

__ 14. Use the value you obtained in the previous step, and display detailed information for this thread. Use the th <pointer> subcommand of kdb.

Note: Thread-related kdb subcommands show information from both the pvthread and thread structures.

__ 15. Using the tv_nextthread pointers (labeled simply nextthread by kdb), follow the list of threads for this process. Use the same command you used in the last step substituting the tv_nextthread pointer value for the address. The tv_nextthread pointer in the last thread in the list will point back to the first thread.

Fill in the following table using the data you gathered by following the list of threads of this process:

__ 16. Use the information that you entered into the table in the previous step to answer the following questions:

__ a. What threads are running with a fixed priority? ________________________

__ b. What thread is bound to a CPU? How did you know? ___________________

__ c. What thread is running at a fixed priority and ignoring time slicing? ________

__ 17. Use kdb to display the thread priority for the non-fixed priority threads. Repeat the display several times. Are any of the thread’s priorities changing? ____________

__ 18. There is an easier way to see the threads associated with a process. The subcommand th -p <process slot number> displays the threads for the process in the specified slot number. This subcommand can also allow you to view the

Addressof pvthread

Slot State TID Priority CPUID Scheduling Policy




EXempty
thread priorities as they change (for those non-fixed priority threads) by simply repeating the subcommand several times. Using the process table slot number you recorded in a previous lab step, display the threads for the rundifft process.
Is the thread priority for any of the threads owned by the rundifft process changing? __________

You should run the th -p <process slot number> subcommand several times and look that the PRI column to see if the thread priority values are changing.

__ 19. In most situations there are multiple different methods that can be used to view the same data in kdb. Each person develops a personal preference for what subcommands they use and for the specific strategy they use to find data they need.

__ a. A different way to get the same information you obtained in the previous step is to use the tpid subcommand of kdb. So, use this subcommand to display the summary of threads using the PID of the rundifft process.

__ b. Using grep can save you some time and can make looking at the information a little easier. You still have to look at the thread detail for each thread but the following subcommands make this a little easier as well. Use the following command to obtain the policy value of one of the rundifft threads:

th <slot> | grep policy

__ 20. Terminate rundifft by entering a Ctrl-C in the window in which this program is running.



Student Exercises

Section 2 - Examining process and thread relationships

__ 21. In this course we have talked about process and thread relationships. Sometimes it gets confusing when talking about processes and threads because some of the output of different commands can be misleading. For example, we might say that a process is running when in fact it is the threads of that process that are actually running. We might say that a process has a priority value but the reality is that the process does not have priority values assigned. The steps in this section are designed to help clarify the roles of the process and thread and the relationship between them.

__ 22. In following steps of the exercise, you will be running a program called runrrt. This program creates several threads that use the SCHED_RR (round-robin) scheduling policy with different priority values. The threads do not do anything, they are just for you to look at and analyze.

Run the runrrt program. It will print out the hexadecimal TIDs of several threads it starts. Leave the program running.

__ 23. In your second terminal window, use the ps -ekmo THREAD command to verify that runrrt is running and to view the threads that are associated with this process.

Answer the following questions:

__ a. How many threads does the process runrrt own? ________________

__ b. Are the priorities for the runrrt threads the same? ________________

__ c. Is the relationship between the priority displayed for the process and the priority being displayed for the threads what you expected? ________________

__ 24. Terminate runrrt by entering a Ctrl-C in the window in which the process is running.




EXempty
Section 3 - Using schedo to affect scheduler algorithms
__ 25. Make sure you are in the /u/QV471 directory.

__ 26. Execute the run4procs.sh shell script.

The run4procs.sh shell script runs four identical CPU bound single-thread runproc processes, in background. If you are interested to investigate the code of this program, the source file is available under the current directory.

__ 27. Use the ps command to check to see if the runproc processes are running and record the PIDs of those processes. You will need these PIDs later in this exercise.

PID = _______________________

PID = _______________________

PID = _______________________

PID = _______________________

__ 28. Verify the priority and processor utilization information of the runproc processes using ps -el | grep runproc. Run this ps command again in a few seconds and notice these values again. Repeat after a few more seconds.

To make it easier, you can run the ps command in a loop in one window while you proceed to the next step to do some tuning in a separate window. An example command to run ps in a loop is shown below. Note that the > character is the prompt displayed by the shell when input is continued on a subsequent line. You do not need to type this character as it is displayed automatically.

# while true> do> ps -el | grep runproc | grep -v grep> echo ---> sleep 3> done

__ 29. In the window where the looping ps is not running, use the schedo command to make sure the scheduler short term CPU usage decay rate (sched_D) is set to the default. The default value (16) decays the short-term CPU usage by 1/2 (16/32) every second. Decreasing this value enables foreground processes to avoid competition with background processes for a longer time.

__ 30. Change the scheduler decay rate (sched_D) to a value of 31.

Look at the output of ps again. What do you observe about the priority? ________________________________________________________

__ 31. Change the scheduler decay rate (sched_D) to a value of 1.

Look at the output of ps again. What do you observe about the priority and CPU utilization? ________________________________________________________

__ 32. Set the scheduler decay rate (sched_D) back to its default value.



Student Exercises

__ 33. In the window where the looping ps is not running, use the renice command to change the nice values of three runproc processes to 40, leave one process with its initial nice value (20).

__ 34. Use the schedo command to make sure the weighting factor for short-term CPU usage in priority calculations (sched_R) is set to the default. The default value (16) will result in half (16/32) of the short term CPU usage being factored in to the priority calculation. Decreasing this value makes it easier for foreground processes to compete. This value is also known as the penalty factor.

__ 35. Change the scheduler CPU usage penalty factor (sched_R) to a value of 1, so the CPU penalty ratio will be1/32, almost no penalty.


__ 36. Change the scheduler CPU usage penalty factor (sched_R) to a value of 32, so the CPU penalty ratio will be 32/32, maximum CPU penalty.


__ 37. Reset the sched_R value back to its default value.

__ 38. Kill all runproc processes.




EXempty
Section 4 - Observing the run queue and context switches
__ 39. Display the run queue and context switch statistics using the mpstat command over a three second monitoring interval for three iterations, and answer the following questions:

__ a. How many context switches were observed per second during the monitoring period? ________

__ b. How many involuntary context switches were observed per second during the monitoring period? ________

__ c. How many threads are waiting to run on the run queue during the monitoring period? ________

__ 40. Run the run8t program.

The run8t program creates eight identical CPU bound threads. If you are interestedto investigate the code of this program, the source file is available under the current directory.

__ 41. In another terminal window, use the sar command to display the run queue and context switches statistics over a three second monitoring interval for three iterations, and answer the following questions:

__ a. On average, how many threads are on the run queue now? _______________________

__ b. What is the percentage of the time the run queue is occupied? ___________

__ c. How many context switches are observed? ________

__ 42. Use the mpstat command again over a three second monitoring interval for three iterations to display the run queue and context switch statistics. Are the results similar from what you got from the sar output? __________________

__ 43. Now, use the vmstat command over a three second monitoring interval for three iterations to obtain the number of threads on the run queue. Is this number similar to what you obtained in the previous steps? ____________

__ 44. Terminate run8t by entering Ctrl-C in the window in which the process is running.



Student Exercises

Section 5 - Examining signal-related information with kdb

__ 45. Run the runsig program.

The runsig program creates several threads. Each thread sets up a signal handler for several signals and blocks several others. The signal handler simply prints the number of the signal being handled and the TID of the thread running the handler.

__ 46. In another terminal window, start the kdb command and determine how many threads belong to the runsig process. _______________

__ 47. Examine the process table and thread table entries for runsig. Using data from these structures, answer the following questions:

__ a. What signals are being handled by the process? ______________________

__ b. If a SIGCHLD is sent to the process what will happen? _________________

__ 48. If a SIGUSR1 is sent to the runsig process what thread(s) could run the signal handler for this signal? Explain your answer. ____________________

__ 49. Confirm your answer to the previous question by using the kill command to send the SIGUSR1 signal to the runsig process three times. What do you observe? ________________________________________________________________

__ 50. End the kdb session.

__ 51. Terminate the runsig process by entering a Ctrl-C in the window in which it is running.


End of exercise




EXempty
Exercise 4. Monitoring Processor Utilization

This exercise provides you with experience using kernel trace data and trace based curt data to analyze CPU related performance issues. Also, you will run the ps, vmstat, mpstat, and sar commands to view and observe AIX process and thread management mechanisms in action. In addition, you will be asked to examine performance in a simultaneous multi-threading environment.



• Monitor and analyze CPU activity using the ps, vmstat, mpstat, and sar commands

• Enable and disable simultaneous multi-threading

• Monitor and analyze CPU activity with simultaneous multi-threading enabled and disabled

• Analyze CPU utilization using the curt command

• Analyze trace data

Requirements





© Copyright IBM Corp. 2011 Exercise 4. Monitoring Processor Utilization 4-1

Student Exercises


Preface









EXempty
Section 1 - Using monitoring tools
__ 1. Login to your assigned system as the root user.

__ 2. Display simultaneous multi-threading threads utilization information using the mpstat command with the -s flag. Invoke the following command and leave it running for the remainder of this part of the exercise:

# mpstat -s 3

What do you observe in terms of CPU utilization? _______________________

__ 3. Open another window, login to your assigned system as root user, and change directory to /u/QV471.

__ 4. Make sure the mpstat command is still running in one window, then in the other window run the run8t program.

The run8t program creates eight identical CPU bound threads. If you are interested to investigate the code of this program, the source file is available under the current directory.

__ 5. Look now to the window running the mpstat command. What do you observe in terms of CPU utilization? ____________________________________________

__ 6. Kill the mpstat command with Ctrl-C.

__ 7. Use the sar command to display the physical processor consumed and entitlement consumed for all logical processors of the partition.

Invoke the following command to generate 100 reports at 3 second intervals. Leave it running for the remainder of this part of the exercise:

# sar -u -P ALL 3 100

__ 8. Open a third window, login as the root user, and run the bindprocessor command with -q flag to query the available processors.

The bindprocessor command is used to bind and unbind the kernel threads of a process to a processor, and also to list the available processors.

__ 9. Make sure the sar command and the run8t program are still running.

Run the bindprocessor command to bind the threads of the run8t process to logical processor 0, and observe the sar report.

The first parameter to the bindprocessor command is the process ID of the process whose threads you wish to bind to a CPU. The second parameter is the logical CPU number you wish to bind the specified process to. An example command is shown below.

# bindprocessor 7077940 0

__ 10. After binding the run8t process to logical processor 0, is this reflected in the output of the sar command, showing the breakdown of the logical processor utilization?



Student Exercises

__ 11. Kill the sar command with Ctrl-C.

__ 12. You can also monitor the activity of each individual logical processor using the topas command. Run the topas command and observe the utilization of the logical processors. When you are finished viewing the topas output, type ‘q’ to exit.

__ 13. Kill the run8t program with Ctrl-C.




EXempty
Section 2 - Enabling and disabling simultaneous multi-threading
__ 14. Use the smtctl command to check the status of simultaneous multi-threading. Check how many processors are listed. ______________________________

__ 15. Use the time command to run the batch command and measure the execution time. Make a note of the program execution time.

__ 16. Disable simultaneous multi-threading in the partition using the smtctl command.

__ 17. Invoke the smtctl command to display the status of simultaneous multi-threading. Notice the difference in the output when SMT is disabled.

__ 18. Time the execution of the batch command once again. How does the execution time compare with that obtained when SMT was enabled? _________________

__ 19. Enable simultaneous multi-threading in the partition using the smtctl command.



Student Exercises

Section 3 - Analyzing curt data

__ 20. Change directory to /u/QV471/perf71/perfdata. There you will find files from a PerfPMR testcase.

__ 21. Invoke the following curt command to read the raw system trace file from the PerfPMR testcase and produce the CPU utilization summaries:

# curt -i trace.raw -m trace.nm -est -o curt.out

__ 22. Look at the “System Summary” section in the generated curt.out report. From this report, is the system processor capacity being well utilized? ____________

__ 23. In the “System Summary” section of the report, what are the top three categories of processor resource utilization? ________________________________________

__ 24. Examine the generated curt.out report, and look at the “Processor Summary” section. This section shows detailed information for each logical processor.

Answer the following questions:

__ a. Which processor is handling most of the system calls? ____________

__ b. Which processor is spending the most amount of time running kernel processes (kprocs)? ______________

__ c. Which processor is almost totally running in user mode? ___________

__ 25. Examine the generated curt.out report. Look at the “Application Summary (by process type)” section of the report, and answer the following questions:

__ a. What are the three most predominant processes? ____________________

__ b. Which of these predominant processes consumes the most CPU resources? _______________________________

__ c. How many threads of this process ran during the trace data collection period? __________

__ d. Which processes spent the largest amount of time processing system calls? ______________

__ 26. Look at the “Application Summary (by Tid)” section of the report. What is the TID of the predominant thread? ______

__ 27. Look at the “Kproc Summary (by Tid)” section of the report. What is the name of the predominant kernel process? _______

» .

__ 28. Look at the “System Calls Summary” section of the report. What are the top five system calls in terms of CPU utilization? ____________________________________________




EXempty
__ 29. Look at the “Threads Summary” section of the report. This section shows detailed information for each thread. What are the thread identifiers (TID) of the first three threads reported in this section? __________________
__ 30. Take a close look at the detail report for thread 15335497 of the app3 application. Do you see any problem on the execution of the kopen system call? _____________



Student Exercises

Section 4 - Analyzing trace data

__ 31. Change directory to /u/QV471/perf71/perfdata. There you will find files from a PerfPMR testcase.

__ 32. Invoke the command shown below to read the raw system trace file from the PerfPMR testcase and produce a formatted trace report. The trace.sh shell script is one of the scripts provided by the PerfPMR tool.

# ../trace.sh -r

__ 33. A formatted trace report (trace.int) has been generated in the previous step. You will be using this trace report to investigate the kopen problem that occurred during the execution of the application app3.

What is the name of the file that the application app3 is trying to open and fails? ___

Note: Try to use a combination of grep commands to filter the trace events.


End of exercise




EXempty
Exercise 5. Synchronization Mechanisms

This exercise provides you with experience using kernel traces and splat reports to analyze event synchronization activities that use kernel locks.



• Examine event synchronization management activities

• Interpret trace report data to examine synchronization activities

• Interpret splat reports

Requirements





© Copyright IBM Corp. 2011 Exercise 5. Synchronization Mechanisms 5-1

Student Exercises


Preface









EXempty
Section 1 - Analyzing inode lock with trace
__ 1. Login to your assigned lab system as the root user and change your working directory to /u/QV471/lockcase.

__ 2. Use the more command to view the file lockname.h located in /usr/include/sys.

Lock operations can sometimes cause performance problems. In this exercise you will investigate a performance problem caused by inode lock contention. You will be using PerfPMR files to investigate an application accessing a DB2 database server. The partition on which the data was recorded has 15 CPUs which are seen as 30 logical CPUs, since simultaneous multi-threading is enabled. The original complaint was that the server was having performance problems. As you will see in this part of the exercise, the main issue was caused by lock contention.

Note: For the next steps of this exercise, try to use combinations of the more, awk, and grep commands to filter the trace events.

__ 3. Invoke the following command to determine the names of the lock classes used for inode locks:

# grep INODE /usr/include/sys/lockname.h

__ 4. Use the grep command to extract all lines from the trace report (PerfPMR file called trace.int) that contain the term wakeup_lock. Redirect the output to a new file called waiters.temp.

Note: It would be hard to use the grep command to find all lines where the threads_awoken field contained a large value.

__ 5. Browse waiters.temp and look at the values displayed in the “lockaddr=” field. There are a variety of lock addresses. In fact, there are too many to analyze by hand.

__ 6. You will first use basic commands and utilities to examine the trace data that you might use to create your own script for doing lock analysis. Later, you will compare this with reports generated by the splat command.

Your first goal will be to count, for each lock, the number of wakeup_lock trace events. To do this you need a list of these events sorted by the lock address value. Because this event can have a variable number of fields between the first field and the lockaddr field, you will use the fact that the lockaddr value is preceded by an “=” to parse it out, using awk.

Use the following command sequence to generate a file called waiters.sorted which contains a sorted list of just the lockaddr values in each wakeup_lock event:

# awk -F"=" '{print $2}' waiters.temp | awk '{print $1}' | sort > waiters.sorted

__ 7. Given this sorted list, a very simple awk script can be used to count how many of these events there are for each lock address. You have been provided this awk script, named waitcount.awk. The basic function of the script is to read the sorted



Student Exercises

list on STDIN and generate a report to STDOUT with one line for each lock address where the first field is the count of events for that lock and the second field is the lock address.

Briefly examine the awk script to see what it does.

__ 8. Run the command sequence shown below to generate a new file called waiters.ranked, which lists the most frequently used lock addresses first.

# ./waitcount.awk < waiters.sorted | sort -rn > waiters.ranked

__ 9. Examine the waiters.ranked file to identify the locks with the greatest number of wakeup_lock events. Note that some lock addresses are 15 hex digits long starting with “78” (leading digit of zero is not shown) while others are 16 hex digits long and start with “F1”.

Write down the top two lock addresses which start with “78” and their associated frequencies. ___________________________________________________

Write down the top two lock addresses which start with “F1” and their associated frequencies. ___________________________________________________

__ 10. Browse the provided trace.inode (PerfPMR) file, which lists information about the inodes that were in use when the trace data was collected.

Given the addresses of these inodes, which of the two types of lock addresses (“78” or “F1” locks) are more likely to be matched to an inode lock? ___________

__ 11. For the locks that have been identified as being top ranked (file waiters.ranked) and which start with “F1”, you will want to try to match these to inodes listed in trace.inode.

For each of the two top ranked lock addresses starting with “F1”, write down the corresponding address of the memory page that contains that lock. To do this, simply copy all of the digits of the lock address except the last three hex digits (these last three digits are the offset of the lock within the memory page).

Lock address: _______________ Memory Page: ________________________

Lock address: _______________ Memory Page: ________________________

__ 12. Use grep on the trace.inode file to extract the inodes that are on the same memory page as the lock with the address that ends with 6B8. Then, sort them by the inode address (1st field). Since the addresses in trace.inode are in lower case while the addresses in the trace event are in uppercase, make the grep execution case insensitive.

__ 13. Compare the selected lock address to the list of inodes on the same memory page, and identify the inode which has an address which is closest to but less than the lock address. Record the address, major and minor numbers and the inode number for the matching inode. Note that the values are in hexadecimal:

Address: _______________ Inode: ________________________




EXempty
Major: _________________ Minor: ________________________
__ 14. Browse the trace.maj_min2lv (PerfPMR) file which contains a list of the opened directories and files, along with the information about mode, owner, group, size, and time of last modification for each one.

__ 15. Convert the previously recorded hexadecimal inode major and minor numbers to decimal. You can either use a calculator which is on your workstation, or you can use an AIX facility such as the bc command. Record the equivalent decimal values.

Decimal Major: _____________ Decimal Minor: ____________________

__ 16. Search the trace.maj_min2lv (PerfPMR) file for a device name which matches the recorded decimal major and minor number. Record the device name: ________________________

__ 17. Look for the logical volume qp1data16 in the config.sum (PerfPMR) file for the mount command listing. Locate the logical volume in the mounted file systems listing and record the mount point: ____________________________________

__ 18. It would be useful to see if there was a pattern to lock contentions on the various inodes, but this can be tedious to do by hand. To allow you to identify the file systems for the numerous locks in your waiters.ranked file, you have been provided a script that will lookup the inode information for provided lock addresses. The script is locklookup.sh. If given a file via STDIN with the format of the waiters.ranked file, it will create a waiters.out file which lists those locks with the inode number and file system identified (along with the offset of the lock field within the inode).

Note: This script was constructed specifically for this lab and is not an AIX supported tool. As such we are not providing it as a tool for you to take and use on your system. It is here just as an example of one way that a script could be constructed to assist in the analysis.

Obtain the inode and file system information for the top 10 lock addresses which start with “F1000100”. Use head and grep to filter the input to the locklookup.sh script.

__ 19. Examine the generated waiters.out file to see the inode information for these locks. Is there pattern to the files involved? _________________________________



Student Exercises

Section 2 - Analyzing simple locks with splat

__ 20. Make sure you are in the /u/QV471/lockcase directory.

__ 21. Having used basic commands to analyze the provided trace files, you will now examine a splat report. Browse the file splat.default and search for the lock address starting with “78” that you determined was mentioned most frequently in the wakeup_lock events in the trace data.

__ 22. Search further down in the report for the details of the identified lock and answer the following:

__ a. How many times did threads spin on this lock? _____________

__ b. How many times did threads immediately acquire this lock when requested? ___

__ c. How many times did threads exhaust the spin timeout and need to wait (sleep) until awoken? ______

__ d. What was the average length of the wait queue for this lock? ___________

__ e. What was the maximum length of the wait queue for this lock? __________

__ 23. Search further down in the report to find the thread detail for this lock. Looking at the thread detail for this lock, what are the names of the processes which are using the lock? _____________________________________________________________

__ 24. You can use the listed process IDs (5148882 for example) to examine the svmon.before (PerfPMR) file to determine where the lock contention is occurring within the application address space.

__ 25. The waiters.ranked file shows the locks with the greatest number of wakeup_lock events. Note that the “F1” lock addresses refer to inode locks while the “78” addresses refer to application locks. For each type of lock, note the address and the associated lock count in the table below.

__ 26. Going further in generic lock analysis, we have provided several versions of the splat report, as described in the table below. Depending on what aspect of locking is a concern, you could use a different sort criteria.

Search the available splat reports and write down which of the locks has the highest spin count: _________________________________

Type of Lock Lock Address Associated CountApplication Lock

Inode Lock

splat report file Sort criteria splat.spincount -s s (sort by spin count)splat.holdtime -s S (sort by percent processor spin hold time)splat.averwait -s W (sort by average wait queue)splat.percwait -s w (sort by % real wait time)




EXempty
__ 27. Search the available splat reports and write down which of the locks has the highest percent hold time: _________________________________
__ 28. Examine the inode lock 0xF10001004C30C6B8 and fill in the table below:

__ 29. Scroll further down in the report for the thread detail report for this lock. Looking at the thread detail for this lock, what are the names of the processes that were using the lock? _________________________________________________________


End of exercise

How many times did threads spin on this lock?How many times did threads immediately acquire this lock when requested?How many times did threads exhaust the spin timeout and need to wait (sleep) until awoken? What was the average length of the wait queue for this lock?What was the maximum length of the wait queue for this lock?How many seconds did threads stay sleeping (voluntarily) after exceeding the thread's spin threshold?



Student Exercises



V5.4.0.3

backpg
Back page

cover front cover · student exercises july 2011 edition the information contained in this document...

Documents