the case of the unexplained…
DESCRIPTION
The Case of the Unexplained…. Mark Russinovich Technical Fellow Windows Azure. WCA-B306. Outline. Introduction Sluggish Performance Error Messages Application Crashes Blue Screens. Case of the Unexplained…. This is the 2013 version of the “case of the unexplained” talk series - PowerPoint PPT PresentationTRANSCRIPT
The Case of the Unexplained…Mark RussinovichTechnical FellowWindows Azure
WCA-B306
OutlineIntroductionSluggish PerformanceError MessagesApplication CrashesBlue Screens
Case of the Unexplained…• This is the 2013 version of the “case of the
unexplained” talk series• Previous versions covered different cases• Can view webcast on Sysinternals->Mark’s webcasts
• Based on real case studies• Some of these have been written up on my blog
Troubleshooting• Most applications do a poor job of reporting
unexpected errors• Locked, missing or corrupt files• Missing or corrupt registry data• Permissions problems
• Errors manifest in several different ways• Misleading error messages• Crashes or hangs
Purpose of Talk• Show you how to solve these classes of
problems by peering beneath the surface• Interpreting process, file and registry activity• Interpreting call stacks
• You’ll learn tools and techniques to help you solve seemingly unsolvable problems
Tools We’ll Use• Sysinternals: www.microsoft.com/technet/sysinternals (\\redmond
\files\SYSINTERNALS\LBI\Latest) • Process Explorer – process/thread viewer• Process Monitor – file/registry/process/thread tracing• Procdump – process memory dumper• Autoruns – displays all autostart locations• SigCheck – shows file version information • PsExec – execute processes remotely or in the system account• TcpView – shows TCP/IP endpoints• Strings – dumps printable strings in any file• Zoomit – presentation tool I’m using
• Microsoft downloads:• Debugging Tools for Windows: Windbg application and kernel debugger:
www.microsoft.com/whdc/devtools/debugging (//dbg)
The Sysinternals Administrator’s Reference• The official guide to the Sysinternals
tools• Covers every tool, every feature, with tips• Written by markruss and aaronmar• Full chapters on the major tools:• Process Explorer• Process Monitor• Autoruns• Other chapters by tool group• Security, process, AD, desktop, …
OutlineSluggish PerformanceError MessagesApplication CrashesBlue Screens
The Case of the Sluggish Explorer• Explorer was very slow to display the
contents of local files• Captured a Process Monitor trace
Process Monitor• Process Monitor is a real-time file, registry, process
and thread monitor• When in doubt, run Process Monitor!
• It will often show you the cause for error messages• It many times tells you what is causing sluggish performance
Process Monitor Enhancements: Bookmarks• Bookmarking enables you to save markers
in the trace:
• Use F6 to find the next one, Shift+F6 to search up
The Case of the Sluggish Explorer (Cont)• Noticed numerous references to network
path with NAME NOT FOUND:
The Case of the Sluggish Explorer (Cont)• Checked account mapping and saw that
home drive was network share:
• Confirmed in Process Explorer current directory was the network share:
The Case of the Sluggish Explorer: Solved• Opened command prompt, killed Explorer,
and restarted from local directory: no delays• Contacted IT and had them change home
directory: problem solved
Process Explorer• Process Explorer is a Task Manager replacement• You can literally replace Task Manager with Options-
>Replace Task Manager• Hide-when-minimized to always have it handy• Hover the mouse to see a tooltip showing the process
consuming the most CPU• Open System Information graph to see CPU usage
history• Graphs are time stamped with hover showing biggest
consumer at point in time• Also includes other activity such as I/O, kernel memory
limits
Process Explorer v15: GPU Monitoring and Windows 8• Captures GPU utilization and memory
usage• System-wide• Per-Process
Process Explorer v15.2• Process timelines
• Autostart locations
Process Explorer v15.3• Process view heat map columns
• Bit-matching dump files• Disk counters fixed for Windows 8
The Case of the Spiking ICQ Process• System was sluggish• Opened Process Explorer and saw Instant
messaging application consuming CPU:
Processes and Threads• A process represents an instance of a running program• Address space• Resources (e.g., open handles)• Security profile (token)
• A thread is an execution context within a process• Unit of scheduling (threads run, processes don’t run)• All threads in a process share the same per-process address space
• The System process is the default home for kernel mode system threads• Functions in OS and some drivers that need to run as real threads• E.g., need to run concurrently with other system activity, wait on
timers, perform background “housekeeping” work• Other host processes: svchost, Iexplore, mmc, dllhost
Viewing Threads• Task Manager doesn’t show
thread details within a process• Process Explorer does on
“Threads” tab• Displays thread details such as
ID, CPU usage, start time, state, priority
• Start address is where the thread began running (not where it is now)
• Click Module to get details on module containing thread start address
Thread Start Functions and Symbol Information• Process Explorer can map the addresses
within a module to the names of functions• This can help identify which component within a process is
responsible for CPU usage• Configure Process Explorer’s symbol
engine:• Download the latest Debugging Tools for Windows from Microsoft
(free)• Use dbghelp.dll from the Debugging Tools• Point at the Microsoft public symbol server (or internal symbol
server if you have access)
The Case of the Spiking ICQ Process (Cont)• Looked at threads tab• Start address of spiking threads were
generic:
Viewing Call Stacks• Click Stack on the
Threads tab to view a thread’s call stack• Note that start address
on Threads tab is different than first function shown in stack• This is because all threads created by
Windows programs start in a library function in Kernel32.dll which calls the programmed start address
The Case of the Spiking ICQ Process (Cont)• Looked at thread stack and saw video DLL:
The Case of the Spiking ICQ Process (Cont)• Opened DLL view and saw that each time
issue occurred, four video DLLs were loaded:
• Noted that DLLs were not loaded on program startup
• Opened Add/Remove programs, but saw no software from Viscom
Autoruns• Shows every place in the system that can
be configured to run something at boot & logon• Standard Run keys and Startup folders• Shell, userinit• Services and drivers• Tasks• Winlogon notifica`tions• Explorer and IE addins (toolbars, Browser Helper Objects, …)• More and ever growing…
• Each startup category has its own tab and all items display on the Everything tab• Startup name, image description, company and path
Identifying Malware Autostarts• Zoom-in on add-ons (including malware) by
selecting these filter options:• Verify Code Signatures• Hide Microsoft Entries
• Select an item to see more in the lower window• Online search unknown images• Double-click on an item to look at where its configured in the Registry
or file system• Has other features: • Can also show empty locations (informational only)• Includes compare functionality• Includes equivalent command-line version, Autorunsc.exe
The Case of the Spiking ICQ Process: Solved• Opened Autoruns and disabled all Viscom
DLLs:
• Problem solved
The Case of the Company 15-Minute Logons• Logons were taking 5-15 minutes across a
company for one OS image• The other OS image had normal logon time• Execs started to complain• Admins tried everything:• Examined event viewer• Captured Wireshark traces• Checked for malware• Finally, another admin that had seen “Case
of the Unexplained” decided to help:• Captured a Process Monitor boot trace
The Case of the Company 15-Minute Logons (Cont)• Looked through process tree and noticed
Framepkg taking four minutes:
• Command-line showed it was McAfee A/V reinstallation
The Case of the Company 15-Minute Logons: Solved• Question was: why was script reinstalling
A/V?• Contacted server admins:• They discovered that Active Directory OU for OS image had GPO• GPO should have been removed from domain after Win7 rollout
• Removed GPO: problem solved
OutlineSluggish PerformanceError MessagesApplication CrashesBlue Screens
The Case of the File in Use Error• Tried deleting PowerPoint deck I had been
editing, but got an error:
• No sign of PowerPoint running,not even in Task Manager’s applications tab
The Case of the File in Use Error: Solved• Did a Handle Search in Process Explorer
• Found hidden PowerPoint process:
• Terminated process: problem solved
The Case of the Mistaken Windows-to-Go• User tried to use the Windows 8 AppStore,
but got an error message:
• Had installed Windows on a Maxtor SATA 2 disk, but for some reason Windows thought it was a portable USB disk
The Case of the Mistaken Windows-to-Go: Solved• Captured a Process Monitor trace when trying to
access the store and scanned the log• Found this reference:
• Deleted the PortableOperatingSystem value: problem solved
The Case of the Default Win8 RDP Lockscreen Background• Set Windows 8 lockscreen
to custom background
• When RDP’ing into the system, default lockscreen showed• Needed to find out where default was coming from• Captured a Process monitor trace of locking system and RDP’ing into
system
The Case of the Default Win8 RDP Lockscreen Background (Cont)• Set a filter to “Path Begins With .jpg”:
RDP
Lock
The Case of the Default Win8 RDP Lockscreen Background: Solved• Had to replace system version of lockscreen
background• Launched command prompt as system using PsExec:
• Copied background over RDP lockscreen file:
• Problem solved
OutlineSluggish PerformanceError MessagesApplication CrashesBlue Screens
The Case of the Crashing IE Favorites• IE was crashing every time
Favorites was clicked• Ran IE8 without Add-ons• Still Crashed• Upgraded to IE9• Still Crashed• Ran IE9 without Add-ons• Still Crashed
The Case of the Crashing IE Favorites (Cont)• Used Process Explorer to get the PID of
iexplore.exe:
The Case of the Crashing IE Favorites (Cont)• Used ProcDump to capture a unhandled
exception (-e) mini dump
The Case of the Crashing IE Favorites (Cont)• Changed to the Exception Context Record
(.ecxr) and looked at the stack (k) – looked like it was sorting
The Case of the Crashing IE Favorites (Cont)• Suspected one favorite that had bullets:
• Moved all of the Favorites to another folder• Still Crashed
The Case of the Crashing IE Favorites (Cont)• Captured a Process Monitor trace• Set a filter for IE, ‘Action’ is SUCCESS and
‘Path’ contains ‘Favorites’, and Registry
The Case of the Crashing IE Favorites (cont.)• Saw Order REG_BINARY value read from
HKCU\Software\Microsoft\Windows\CurrentVersion\Explorer\MenuOrder\Favorites:
The Case of the Crashing IE Favorites: Solved• Used “Jump to” context menu to open
Regedit• Renamed ‘Order’ value to ‘xOrder’
• Restarted IE: problem solved• Moved the favorites back• Looked at RegEdit, new ‘Order’ value made
OutlineApplication HangsSluggish PerformanceError MessagesBlue Screens
Blue Screen Crashes• Windows has various components that run in
Kernel Mode, the highest privilege mode of the OS• OS components: Ntoskrnl.exe, Hal.dll• Drivers: Ntfs.sys, Tcpip.sys, device drivers
• Kernel-mode components are privileged extensions to the OS have to adhere to various rules• Not accessing invalid memory• Accessing memory at the right “Interrupt Request Level”• Not causing resource deadlocks
• When a kernel-mode component performs an illegal operation, Windows crashes (blue screens)• Crashing helps preserve the integrity of user data• A resource deadlock can hang the system
Online Crash Analysis• When you reboot after a crash, Windows
offers to upload it to Microsoft Online Crash Analysis (OCA)• Automated server generates a thumbprint of the crash and
uses it as a key in a database• If the database has an entry, the user is told the cause and
directed at a fix
Basic Crash Dump Analysis• Many times OCA doesn’t know the cause:
• Basic crash dump analysis is easy and it might tell you the cause• Requires Windbg and symbol configuration• Dump files are in either:• \Windows\Memory.dmp: Vista+ and servers• \Windows\Minidump: Windows 2000 Pro, Windows XP, Vista+
The Case of the Windows 8 IE BSODs• Users complained that Windows 8 would
crash when running IE • Administrator analyzed dump file:
The Case of the Windows 8 IE BSODs: Solved• Did a web search and found that Avast had
just posted an update:
• Applied update: problem solved
Summary and More Information• A few basic tools and techniques can solve
seemingly impossible problems• I learn by always trying to determine the root cause
• Resources:• Sysinternals Administrator’s Reference• Webcasts of two previous “Case of the Unexplained “ talked• Sysinternals->Mark’s Webcasts• My blog• Windows Internals: understand the way the OS works
• If you’ve solved one, send me a description, screenshots and log files!
Windows 8 Bluescreens