Advanced Troubleshooting and Performance Testing

#SPSBE SPSBE31

Donald Hessing

April 26th, 2014

Who am I?

Donald Hessing Principal Consultant | Thought Leader SharePoint @Capgemini

Netherlands Microsoft Certified Master (MCM) - SharePoint (Virtual) Technology Solution Professional for Microsoft Work full time on SharePoint since 2007 | #DEV | #ITPRO | #STRATEGY donald.hessing@capgemini.com | @dhessing | #SPSBE SPSBE31

It’s not all about SharePoint!

Load Balancer

Understand response time

Response Time

1414 KB

1024 KB (10Mb)

20

ms

(44

/2)300 ms 300 ms 2.6 sec.

1414KB

100 KB (1Mb)

150

ms

(44

/2)300 ms 300 ms

15.42

sec

IE8, FireFox 3 – 6

connections max.

TCP Slow Start

SQL IO Characteristics

Operation Random / Sequential Read/Write Size Range

OLTP – LOG Sequential Write 512 bytes – 64KB

OLTP- Data Random Read / Write 8KB

Bulk Insert Sequential Write 8KB – 128KB (in

multiples of 8KB)

Read Ahead Sequential Read 8KB – 128KB (in

multiples of 8KB)

Backup Sequential Read/Write 1MB

Restore Sequential Read/Write 64KB

Validate configurations prior to deployment

Customer Example – IOMonitor.exe

Design TEST - 2 KB Random - Read/Write (67%/33%)

Drive

Expected

IO

Total

IO

Average

Read Time

(ms)

Average Write

Response Time

(ms)

Maximum

Read Time

(ms)

Maximum

Write

Time(ms)

C:\ 60 314 3 1 197 6

L:\ 300 419 2 3 31 11

S:\ 600 386 3 3 115 28

T:\ 1200 51 35 52 89 2501

U:\ 1000 43 5 59 41 1817

Monitoring Pending Disk IO

http://henkvandervalk.com/sql-under-the-hood-part-1-monitoring-current-pending-disk-ios

Let’s investigate….

Who

What

What

When when

Where

Why Why

…defines the scope

Lab\User8

30 seconds render time

Mon-Fri between 0900-1100 CST, but not on weekends

WFE1

Typically achieves results between 5-7 seconds

When user is searching for “Widget” and hits WFE1

ULS Viewer

Fiddler

Fiddler – Troubleshooting WCF services

http://127.0.0.1:8888

Fiddler – Troubleshooting Search

LOGPARSER 2.2

LOGPARSER

IIS LOGS

Example – Find slow pages

Customer Case

Questions?

LOGPARSER "SELECT QUANTIZE(TO_TIMESTAMP(date, TO_LOCALTIME(time)), 3600) AS Hour, DIV(SUM(sc-bytes),1048576) AS TotalBytesSent, DIV(SUM(cs-bytes),1048576) AS TotalBytesReceived FROM *.log GROUP BY Hour ORDER BY Hour" -i:w3c

Determine network capacity..

Number of users 5.000 10.000 20.000 50.000

Bandwidth (Mbit/s) 25 50 100 250

Response times

SC-STATUS AVGTIME MAXTIME HITS

200 215 33882 30581

302 56 3104 2949

400 58 78 13

401 16 3166 14811

403 31 31 1

404 81 2745 1472

500 63 109 12

Response Times Web Frontend Servers

Response times per DOMAIN

Response times for DOMAINA per WFE

Why are my responses for DOMAINA users SLOW on WFE2?

Win32Status

SC-WIN32-STATUS

Win32-Status Description Hits

1 Incorrect function. 17

2 The system cannot find the file specified. 4788

5 Access is denied. 1

64 The specified network name is no longer available. 1663

121 The semaphore timeout period has expired. 964

995The I/O operation has been aborted because of either a thread exit or an application

request.85

1236 The network connection was aborted by the local system. 3934

1330 The password for this account has expired. 6

2148074248 The token supplied to the function is invalid 120

2148074252 The logon attempt failed 271

2148074254 No credentials are available in the security package 117831

2148074257 No authority could be contacted for authentication. 9379

Performance Testing

Custom Web Test Request Plugin

Valid testing data is essential for performance tests

Content Generation

• Download a copy of the wiki databasehttp://en.wikipedia.org/wiki/Wikipedia:Database_download#English-language_Wikipedia

• English ~40GB XML Data

• Dutch ~10GB XML Data

• Wrote custom wiki parser for generating

Load Testing

Performance Counters

% CPU SharePoint Servers Threshold

% CPU SQL Server <75%

% CPU SharePoint Servers <75%

# Available Mbytes SharePoint Servers < 20%

Avg. Disk Write Queue Length < 2 Per Spindel

Avg. Disk Read Queue length < 2 Per Spindel

Avg. Disk Sec/Read SQL Server Disks < 12 ms – 0.012

Avg. Disk Sec/Write SQL Server Disks < 12 ms – 0.012

Load Results

Avg=48%

Avg=29%

Avg=11%

Avg=0.013

Avg=6.18

Thank you!

Thanks to our sponsors!

Gold

Silver