msg323 exchange transport monitoring and troubleshooting max ciccotosto program manager - exchange...
TRANSCRIPT
MSG323
Exchange Transport Monitoring and TroubleshootingMax CiccotostoProgram Manager - Exchange Server Microsoft Corporation
Why Are We Here?
Learn about Transport in Exchange 2003
Common ScenariosMessages are not routed
<X> queue keeps growing
NDRs are returned for unidentifiable reason
Client has problems with messages/content
Authentication failures
Monitoring is essential to preventing problems, catching them before they happen
AgendaTransport in Exchange
What’s new in Exchange 2003Exchange Mailflow
Common Troubleshooting Scenarios DNS and ConnectivityWorking with QueuesRouting and ConnectorsContent and Message PropertiesDealing with NDRs Tracking Down Messages Setting up Counters and Logging
Monitoring Best PracticesQ&A
New in Exchange 2003Easier to support:
More Queues (Hidden Queues)Improved Message TrackingImproved Logging (DSN Logging)New Internet Email Connection WizardNew DNS Resolver tool
Routing Performance and Stability improvementsNew filtering and antispam featuresQuery-based Distributed Groups (QDGs)Journaling is BCC enabled
Transport in Exchange
What Is The Transport?
Handles message delivery and routing details, examples:
Look up users in AD, expand DLs
Dynamic routing logic
Picks up and delivers from/to Store
Handles SMTP protocol
Handles queuing of messages
Touches every message, even local-to- -local user!
Transport Essentials
Server-Server transportIn Exchange 2003: SMTP native transport
InteroperabilityExchange 2003 can talk to Exchange 5.5 via RPC
Support X.400 connectors, EDK (foreign) gateways
No IMC/IMS needed for Exchange 2000 – Exchange 5.5 interoperability
MTA still thereUsed for X.400 and RPC interoperability
Exchange 2000 within an Exchange 5.5 site
5.5 Server 2003 Server
2003 Server
RPC
SMTPRPC
RPC
RPC
RPC
5.5 Server
SMTP Service
Uses Windows® 2000 SMTP Service
Protocol events - extend the SMTP protocol
Transport events - extend the function of the Transport Core
Multi-threaded, high-performance
Transport Core Categorizer
Categorizer: Component that resolves sender and recipients against Active Directory
Limited Categorizer ships with Windows 2000, disabled
Can do some Directory access, such as expanding mail-enabled Groups
Enhanced Categorizer ships with Exchange 2000Adds Exchange features, such as Recipient Limit checking, reading Home-MDB
Transport Core Routing
Advanced link-state based routing engine replaces RID Server and GWART used by Exchange 5.5 MTA
MTA uses same engine when it needs to compute next hop, so X.400/RPC connectors get benefit
Size, priority, sender of message; cost and state of links used to compute path
Transport Core Store Driver
The interface between Transport and the Store
Uses “ExIPC” for inter-process communication, IFS for large data transfer (message body)
Windows 2000 SMTP Service: NTFS store driver
Exchange 2000: Exchange Web Storage store driver
Transport Dependencies
Check these as necessary: Transport looks up User info in ADRouting configuration read from AD on startup, link state info kept in memoryDNS used to resolve names, even internal servers!DS2MB replicates info to MB for Core SMTP Transport queues messages in Store, delivers to/from storeEnd client – e.g. auth problems. NOTE: Client-DC issues are not Exchange!
Queues – Basic Flowchart
MAPI / OWA MAPI / OWA ClientClient
InformationInformationStoreStore
MTA (X400)MTA (X400)
MAPI / OWA MAPI / OWA ClientClient
InformationInformationStoreStore
MTA (X400)MTA (X400)
SMTPSMTPProtocolProtocol
SMTPSMTPProtocolProtocol
Remote Remote Delivery Delivery QueueQueue
Remote Remote Delivery Delivery QueueQueue
Local Local Delivery Delivery QueueQueue
Active Active DirectoryDirectory
RoutingRoutingEngineEngine
CategorizerCategorizer QueueingQueueing
NTFSNTFS
A More Advanced View…
CategorizerAqueue.dll & Phatcat.dll
Routing Engine (reapi.dll)
MAPI OWA CDO
MTAX.400RPC
EDK Gateway
Another Exchange2000 Server
25
143
SMTPConnection
NTFS
PickupDirectory
Formattedfile
imsg
imsg
PreCatQ
DL expansionRecipient & Sender name resolutionDestination determinationApplies limits (max msg size etc)Determines Format for conversionBifurcation
DSAccessCache
GC / DCServer
ldap 3268
Q
DynamicQueues
Q
Internet
SMTP
LocalQ
RemoteDelivery
Q
RPC
IIS
pop3 imap
SMTP
110
Next hopInformation
Advanced Queuing Engine (AQ)
By Paul Flaherty, v1.3 11 Dec 2001
OnMessageSubmission
PostCatQ
Storesubmission
OnSyncMessagePost
Categorize OnPostCategorize
IIS
pop3 imap
SMTP
Client
QueueManager
Exchange 5.5
Link Statetable
a Archive sink
a
a
b Catdbg sink
bb
Regtrace (Q238614)
c
d ESM q’s/AqAdmcli
d e
ee
f Mdbvu
h Replay sink
SMTP SystemMailboxMTS-IN
MTS-OUTTemptable i
imsg
ImailMsg is a memory structure containing headerinformation of the email plus a pointer to theactual file that is either on NTFS or in the storedriver mbx
Troubleshooting Tools
j AV sink
j
X.400Connector
Intrasite or SiteConnector
25
ldap 389 (config info)
Rest of the messageis retrieved using thepointer in theImailMsg, usuallythrough IFS
Another Exchange2000 Server
GetMessageRouter
g
LocalDelivery
Internet
c winroute
e Protolog.dll
g Resvc.dll
i Q311737
h
k Q297700
k
Exchange 2000 Message Flow
MTS-OUT
MTS-IN
Mailbox Store
f
StoreDriver
exsmtp
.dlldrvi
is.d
ll epoxy
PreSubmission Q
Pre-Routing
Q
d
d
d
d
..This boxshowswhereevent sinksfires
MTS-OUT
MTS-IN
f
Gwart.dll
x.400/EDKConnector
Q
Localdelivery
l Remonitor & Q303518
l
Topic 1:
Internet Connectivity
Email From InternetPossible reasons:
Internet DNS is mis-configured
Recipient Policy does not contain the domain
You maybe filtering the specific domain/IP
Tips:Use a tool such as www.dnsreport.com
Check the type of NDR that sender gets
Specific domains or everyone?
Check SMTP Logs
There is a lot of information in the KBs!!!Search for Shared Domain, SMTP and DNS
SMTP Greenbook
Send Mail to The InternetPossible reasons:
You cannot reach the Internet DNS
Smarthost mis-configured (permissions, IP)
Domain not properly registered with DNS (Reverse Lookup enabled on the recipient SMTP)
Tips:Use a tool such as www.dnsreport.com
Check the type of NDR that sender gets (5.7.1)
Verify info/status with your ISP
Verify if you can connect to remote server
Check Real-time denied lists
There is a lot of information in the KBs!!!Search for Shared Domain, SMTP and DNS
SMTP Greenbook
Topic 2:
Working with Queues
Queue Problems
Symptoms:Queue growth ‘abnormal’ – monitoring tools report queue grows beyond threshold
Messages “stuck” in Queue – these messages stay in queue, do not get delivered
Messages waiting to be delivered to external domains (DNS problems)
Internal queues grow (waiting AD lookup, local delivery)
TroubleshootingQueues – What are they for?
TroubleshootingQueues – Local Delivery
Local Delivery QueueMessages awaiting delivery to the Information Store
Make sure store is mounted
Could indicate a performance issue
“Poison” Message
TroubleshootingQueues – Pre-Submission
Pre-Submission Queue Messages waiting to be processed by Transport
Exposed for event sink developers
Could indicate a store performance issue or issue with 3rd party event sink
TroubleshootingQueues – Pre-Categorization
Pre-Cat Queue Messages waiting to be processed by Categorizer
Categorizer resolves addresses
Could indicate an issue when talking to Global Catalog
Could indicate a permissions issue
TroubleshootingQueues – Pre-Routing
Post-Cat / Pre-Routing Queue Messages waiting to be routed
Slowdowns usually due to expensive restrictions
Messages w/ Unreachable Destination (not shown)
Indicates Routing failed to find a path for the message
TroubleshootingQueues – Remote Delivery
Remote Delivery Queue Messages being sent to a remote location
Note: Messages may be physically on disk or in the Information Store! – It depends on where the message originated
Use the error message to help focus your troubleshooting (netmon, nslookup, etc.)
TroubleshootingQueues – Remote Delivery Queue Error Message
TroubleshootingQueues – New in Titanium
Goal was to expose “hidden” queues
QueuesDSN Awaiting Submission
NDR messages that are being submitted
Failed Message Retry QueueMessages that failed conversion
Deferred DeliveryMessages that have deferred delivery specified by Outlook clients
Standard actions are exposed
Fixing Queue Issues
To monitor and Troubleshoot: use WMI or ESM queue viewer
To manipulate queues: use ESM
Verify Dependencies:DNS external/internal
GC Availability
Store
Routing
Ensure there are no “stuck” messages, if so freeze or delete
Tool: Queue Viewer In ESM
Shows queue state and performs actions through ESM, under:
SMTP Protocol
X.400 Service
Actions: Freeze, Delete, Disable queue
WMI access
“Stuck” messages can be frozen or deleted
Queue ViewerQueue Viewer
Topic 3:
Routing and Connectors
Routing Problems
Symptoms: Links / Connectors are marked “down”Topology changes / breaks message path“Currently unreachable” queue growsExternal mail is not routedNDRs, delayed deliveryRouting specific errors in event logs Mail “disappears” or gets queued upOnly some mail gets delivered to end-user
Fixing Routing Problems
Check topology status:Confirm routing configuration has not changed (Did you uninstall IIS?)Ensure master is up, routing service is running Monitor queues, set up countersAre bridgeheads up?Are routing groups connected?
DNS internal: check Network-Address AD attribute of destination server, try to resolve nameUse WinRoute to debug topology, link state informationNew “Routing and LinkState Whitepaper”
Tool: WinRoute
Available:On CD in SUPPORT\UTILS (Exchange 2000)Now on “Exchange Tools Site”
Read Q281382: “How to use WinRoute”Connects to specified host, acts as read-only slaveDisplays Link State info packet decodedResolves GUIDs against AD
If no AD access, see GUIDs and statesConfigure DC hostname if running outside the domain
Can save link state information to file (*.rte files)
WinRouteWinRoute
Topic 4:
Content and Message Properties
Content And PropertiesSymptoms:
Messages do not appear correctly to clientContent is not preserved outbound/inboundNeed to check advanced message properties (X-Headers, FROM, TO)Mail message fidelity is lost
Hard to monitor, rely on user infoTo troubleshoot, investigate original message(s), use Archive Sink tool Don’t forget “Global Settings”
SMTP vs. MAPI Submission
SMTP submission happens on protocol level (port 25)
MAPI submission through MAPI client (Outlook), directly to store
Both submission paths go through Transport (Archive Sink)
SMTP mail is MIME encoded, MAPI is MS-TNEF encoded
Archive Sink captures both
Tool: Archive Sink
Available:On CD in SUPPORT\UTILS (Exchange 2000)
Now on “Exchange Tools Site”
Hooks on 2 possible Transport Events and dumps message properties (regkey)
New version:three files per message: xml, .eml, p1 stream
Works on multiple Virtual SMTP servers
Archives both MAPI and SMTP messages
Not to be confused with Journaling – meant to be troubleshooting tool
Archive SinkArchive Sink
Topic 5:
Dealing with NDRs
Non Delivery Reports It is a symptom
Always check returned NDRs when troubleshooting
Sent to end-user – but can have NDRs also sent to another account(s)
We added many codes
In Exchange 2003New log category for DSN (Sev 0-5).
X-Header “Error number”
Added regkey to enable pre-E2K behavior
TroubleshootingDelivery Status Notifications
Server reporting the problem
DSN error code
Original message
(may get “Send Again” form in Outlook)
Recipient
Check NDR Check NDR onlineonline
NDR Troubleshooting
General steps to follow: Is it permanent or transient? 4.x.x or 5.x.x?
Check specific diagnostic code (e.g. 5.4.0)
Reference cause/solution KB Q284204
Is it a client or server problem?
What’s the reporting server?
Can you reach the reporting server?
Can other users send messages?
Worst case: use tools such as Queue viewer, Message tracking or Winroute
Topic 6:
Tracking down messages
Message Tracking“I sent an email to John two days ago, he as not received it yet”
Useful for: Diagnosing “missing” or “lost” messages
Discovering the message path – so you can increase logging / tracing
Recording successful / failed deliveries
Gathering statistical data from tracking logs
Tools available:Message Tracking Center in ESM
Do-it-yourself scripts
Third party products
Tracking DetailsPer-server
Writes plain text logs to share \\servername\servername.log
Enabled on server object, option to log subject
Turned off by default
In Exchange 2003 we added extra logs
ReferenceUse KB Q246959
Make sure NOT to manually modify logs – can lead to corrupt data
Message Tracking UIMessage Tracking UIand Logsand Logs
Monitoring and Troubleshooting Best
Practices
SMTP Protocol Logging
Per-SMTP Virtual Server
Common logging interface for all IISSame formats, ODBC
Automatic rolloverHourly, Daily, Weekly, Monthly, File size
Extended Logging tabCannot log all DATA
Default location\Winnt\system32\Logfiles
Perfmon Counters - General
Processes - Working Set Memory, CPU utilization (% Processor Time, Working Set Bytes, Pool Nonpaged Bytes)
Overall memory (Available Mbytes)
CPU – Overall CPU utilization
Disk – PhysicalDisk component
Perfmon Counters - SMTP
SMTP Server component
QueuesCategorizer Queue Length
Local [Retry] Queue Length
Messages Currently Undeliverable
Messages Pending Routing
Remote [Retry] Queue Length
Performance: Msgs/sec
Best Practices
Set up perfmon counters to monitor key areas, use WMI to centralize monitoringCheck queue state when something goes wrong!Use NDRs to narrow down and diagnose condition (Q284204)Check global, server settings for mis-configuration and/or changesUse advanced troubleshooting tools as necessary Introductory KB: Q281800
ResourcesSMTP Greenbook: http://www.microsoft.com/technet/treeview/default.asp?url=/technet/prodtechnol/exchange/exchange2000/proddocs/onlinebooks/confsmtp/confsmtp.asp Tools: http://www.microsoft.com/exchange/2003/updates Documentation: http://www.microsoft.com/exchangeNewgroups: dev team answers questions (see Exchange transport and connectivity)
Ask The ExpertsGet Your Questions Answered
Talk one-on-one with a community of your peers
Community Experts: Microsoft product teams, consultants and Tech*Ed speakers
Resources: whiteboards, internet, etc.
Location: in the middle of the Exhibit Hall
Hours: at least 12-3:30p every day
I will be at the ATE after this session
Community Resources
Community Resourceshttp://www.microsoft.com/communities/default.mspx
Most Valuable Professional (MVP)http://www.mvp.support.microsoft.com/
NewsgroupsConverse online with Microsoft Newsgroups, including Worldwidehttp://www.microsoft.com/communities/newsgroups/default.mspx
User GroupsMeet and learn with your peershttp://www.microsoft.com/communities/usergroups/default.mspx
Suggested Reading And Resources
The tools you need to put technology to work!The tools you need to put technology to work!
TITLETITLE AvailableAvailable
Microsoft® Exchange Server 2003 Microsoft® Exchange Server 2003 Administrator's Companion: 0-Administrator's Companion: 0-7356-1979-47356-1979-4
9/24/039/24/03
Active Directory® for Microsoft® Active Directory® for Microsoft® Windows® Server 2003 Windows® Server 2003 Technical Reference: 0-7356-Technical Reference: 0-7356-1577-21577-2
TodayToday
Microsoft Press books are 20% off at the TechEd Bookstore
Also buy any TWO Microsoft Press books and get a FREE T-Shirt
Community Resources
Community Resourceshttp://www.microsoft.com/communities/default.mspx
Most Valuable Professional (MVP)http://www.mvp.support.microsoft.com/
NewsgroupsConverse online with Microsoft Newsgroups, including Worldwidehttp://www.microsoft.com/communities/newsgroups/default.mspx
User GroupsMeet and learn with your peershttp://www.microsoft.com/communities/usergroups/default.mspx
evaluationsevaluations
© 2003 Microsoft Corporation. All rights reserved.© 2003 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY.