network performance monitoring
TRANSCRIPT
-
8/13/2019 Network Performance Monitoring
1/37
-
8/13/2019 Network Performance Monitoring
2/37
Topic
Level
Presenter(s):
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Technical
Client
Network Performance
Monitoring
-
8/13/2019 Network Performance Monitoring
3/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Things our lawyers make us say...
Neither NextGen Healthcare nor any presenter at the Users
Group Meeting is engaged in rendering legal or other
professional advice and this presentation is not a substitute
for the advice of your attorney, accountant and/or other
professional advisor.
-
8/13/2019 Network Performance Monitoring
4/37
-
8/13/2019 Network Performance Monitoring
5/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
5
BJC HealthCare Hospital Locations
Broad Missouri/Illinois Geography (only 13% from St Louis City)
13 Hospital Facilities
BJC Medical Group
Largest employer in
state of Missouri
Long-time
relationship with
Washington
University School of
Medicine
-
8/13/2019 Network Performance Monitoring
6/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
National Leader Local Brand Identity
-
8/13/2019 Network Performance Monitoring
7/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
400+ Providers
100+ Locations
1200+ Concurrent Users
Growth Strategy
Tier 1 Physicians employed by BJC using NextGen
Tier 2 Private Physicians not employed by BJC and using
NextGen
Tier 3
Private Physicians not employed by BJC and notusing NextGen. Leverage Community Health Connection
(CHC)
-
8/13/2019 Network Performance Monitoring
8/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
How issues where handle in the past
CustomerCalls
Mobilize ISTeams
CaptureData
AccessData
DetermineRoot
Cause
ImplementFix
Challenges
Communication between teams
Ability to capture data to accessand implement fix quickly
How to reduce customer impact
-
8/13/2019 Network Performance Monitoring
9/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Performance Monitoring Project
CustomerCalls
Or
ALERTS
Mobilize ISTeams
CaptureData
AccessData
DetermineRoot
Cause
ImplementFix
Performance Monitoring Objectives
Proactive monitoring
Identify critical data
Application
Network
Hardware Resolve problems before customer
impact
Keys to Success
Relationship with customers Relationship with vendors
Relationship with internal teams
-
8/13/2019 Network Performance Monitoring
10/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Things our lawyers make us say...
Neither NextGen Healthcare nor any presenter at the Users
Group Meeting is engaged in rendering legal or other
professional advice and this presentation is not a substitute
for the advice of your attorney, accountant and/or other
professional advisor.
-
8/13/2019 Network Performance Monitoring
11/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Sample Encounters
Track encounter performance at different times of the
day
Identify system hangs, slowness before input from
endusers
Compare performance from different locations in the
environment
Can be as simple or as complicated as you want
-
8/13/2019 Network Performance Monitoring
12/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Sample Encounter
-
8/13/2019 Network Performance Monitoring
13/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Database Monitoring
SQL Counters
SQL DMVs/Tables
SQL Notifications
SCOM Data
-
8/13/2019 Network Performance Monitoring
14/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Page Life Expectancy
Page Life- The amount of time a page stays in memory
before being replaced
Constant low values indicate memory pressure
Track values continuously to compare over time
-
8/13/2019 Network Performance Monitoring
15/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Page Life Expectancy
Page Life- The amount of time a page stays in memory
before being replaced
0
200
400
600
800
1000
1200
1400
1600
10:00
10:05
10:10
10:15
10:20
10:25
10:30
10:35
10:40
10:45
10:50
10:55
Microsoft Recomended Value
Microsoft Warning Value
BJC Value
-
8/13/2019 Network Performance Monitoring
16/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Page Life Expectancy
Page Life- The amount of time a page stays in memory
before being replaced
0
5000
10000
15000
20000
25000
30000
35000
Microsoft Recomended Value
Microsoft Warning Value
BJC Value
-
8/13/2019 Network Performance Monitoring
17/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Cache Hit Ratio
Buffer Cache Hit Ratio-Percent of pages found in cache
without having to read from disk
Recommended Value of >99%
-
8/13/2019 Network Performance Monitoring
18/37
-
8/13/2019 Network Performance Monitoring
19/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Database Backup Completion
Real Life Example
0:00
2:24
4:48
7:12
9:36
12:00
14:24
We
d
Thur
Mo
n
Tue
s
We
d
Thur
F
ri
Mo
n
Tue
s
We
d
Thur
F
ri
Mo
n
Tue
s
We
d
Thur
F
ri
Mo
n
Tue
s
We
d
Thur
F
ri
-
8/13/2019 Network Performance Monitoring
20/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Indexes
Missing Indexes
Impacts performance as SQL needs to read more
data than necessary to return results to user
Covering Indexes
-
8/13/2019 Network Performance Monitoring
21/37ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Indexes
Unused Indexes
Unnecessary writes to the database
-
8/13/2019 Network Performance Monitoring
22/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Expensive Queries
Find procedures/queries that can be tuned to
increase performance or scheduled to run during
non-production hours
Find recent reports run that had large datareads
-
8/13/2019 Network Performance Monitoring
23/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Daily/Weekly Maintenance
NextGen Weekly maintenance updates defragments
indexes on a weekly basis
There are some indexes that fragment too quickly for
once a week rebuilds
Consider running weekly maintenance daily or daily
defrag on a subset of tables
-
8/13/2019 Network Performance Monitoring
24/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Database Blocking
SQL Agent Job that sends email to support staff to alert
database blocking
Also writes blocking history (head blocker, blocked
queries) to table for review
Support response changes based on EHR, EPM,
support application blocking
-
8/13/2019 Network Performance Monitoring
25/37
-
8/13/2019 Network Performance Monitoring
26/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Database Storage Monitoring
0
50000
100000
150000
200000
250000
6/30/2013
7/1/2013
7/2/2013
7/3/2013
7/4/2013
7/5/2013
7/6/2013
7/7/2013
7/8/2013
7/9/2013
7/10/2013
7/11/2013
7/12/2013
7/13/2013
7/14/2013
7/15/2013
7/16/2013
7/17/2013
7/18/2013
7/19/2013
7/20/2013
7/21/2013
7/22/2013
7/23/2013
7/24/2013
7/25/2013
7/26/2013
7/27/2013
7/28/2013
7/29/2013
Core
Index
LOB
-
8/13/2019 Network Performance Monitoring
27/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Image File Storage
0
20000
40000
60000
80000
100000
120000
6/30/2013
7/1/2013
7/2/2013
7/3/2013
7/4/2013
7/5/2013
7/6/2013
7/7/2013
7/8/2013
7/9/2013
7/10/2013
7/11/2013
7/12/2013
7/13/2013
7/14/2013
7/15/2013
7/16/2013
7/17/2013
7/18/2013
7/19/2013
7/20/2013
7/21/2013
7/22/2013
7/23/2013
7/24/2013
7/25/2013
7/26/2013
7/27/2013
7/28/2013
7/29/2013
NextgenRoot
ICS Images
-
8/13/2019 Network Performance Monitoring
28/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Server Storage
-
8/13/2019 Network Performance Monitoring
29/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Application Server Monitoring
0
10
20
30
40
50
60
70
80
0
500
1000
1500
2000
2500
3000
3500
Citrix Server Utilization
Min Free Memory
Mem Warning
Max CPU
CPU Warning
-
8/13/2019 Network Performance Monitoring
30/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Application Server Monitoring
0
10
20
30
40
50
60
70
80
90
100
41199.3
5328
41199.3
6022
41199.3
6717
41199.3
7411
41199.3
8105
41199.3
88
41199.3
9494
41199.4
0189
41199.4
0883
41199.4
1578
41199.4
2272
41199.4
2984
41199.4
3696
41199.4
439
41199.4
5085
41199.4
5779
41199.4
6474
41199.4
7168
41199.4
7862
41199.4
8557
41199.4
9251
41199.4
9946
41199.5
0658
41199.5
1352
41199.5
2046
41199.5
2741
41199.5
3435
41199.5
413
41199.5
4824
41199.5
5519
41199.5
6213
41199.5
6908
41199.5
7602
41199.5
8297
41199.5
9008
41199.5
9703
41199.6
0397
41199.6
1092
41199.6
1786
41199.6
2481
41199.6
3175
41199.6
3869
41199.6
4564
41199.6
5258
41199.6
5953
41199.6
6647
41199.6
7376
41199.6
8088
41199.6
8783
41199.6
9477
41199.7
0172
41199.7
0866
BJCNXXAP01 CPU Usage 10/17/2012
-
8/13/2019 Network Performance Monitoring
31/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Application Server Monitoring
Users on each server
Load caused by new applications
Environment Changes that will change the amount of
users supported
-
8/13/2019 Network Performance Monitoring
32/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Interface Server Monitoring
Intrf_Queue Table
Record Cleanup
Temp_Rec_Trfr
-
8/13/2019 Network Performance Monitoring
33/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Network Monitoring
Alert Support on Node Down (Wan/Primary Switch)
Track utilization on WAN links (inbound/outbound)
Track network latency
-
8/13/2019 Network Performance Monitoring
34/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
BBP
Track completion times on time sensitive jobs
Disable Job steps that require data when data is not
present
Track completion rates
Notify users before job completes
-
8/13/2019 Network Performance Monitoring
35/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Tools
SQL Server Management Studio
Perfmon
SCOM
Orion Report Server
Citrix Management Console
In-house development
-
8/13/2019 Network Performance Monitoring
36/37
ConfidentialProprietary InformationFor use by authorized NextGen clients and employees only. Not for public distribution. Copyright 2013
Monthly Review with LeadershipSummary
Action Plan
We will be adding indexes suggested by the SQL Server to improve performance of database reads after removing the indexes that were not used previously. In addition we discovered that the patient images on the templates add a significant amount of time to the template loadtime when using the fat client environment. The EHR team is working on removing the image where they can. after initially believing that only hiding the image was necessary. We have migrated the disk space monitoring from SCOM to the Orion data and resumed the display of
all of the storage drives to the SQL and File servers. This was necessary due to a change made by Microsoft to the method that data is captured into SCOM.
Object Notes Impact
Workstation MonitoringWorkstation monitoring time was up slightly to 143s this month. We continue to have issues with the process completing on a consistent basis. The change in the
workflow for the new KBM caused an increase in the amount of work inside the templates which is giving us more points of f ailure within the script. We may need to revisitthe script to simplify the steps that it takes.
UsersWe saw no change in the total user capacity in the environment which was expected as there were no major changes. The amount of users continue to trend upwards at
a slight pace. The eight o'clock hour was up to 678 and the fi ve o'clock hour was up to 307 users.
Blocking
We are working with the DBA team to remove the blocking emails that are sent out when only the BBP machines are impacted, that will eliminate some of the extraneous
emails that are sent when users are not impacted. We had a major blocking issue on 11/6. We cleared the blocking by stopping the CHS service. In working with
NextGen, they have offered us updates on triggers, that will lesson the possibility of blocking caused by CHS i n the future. It is not known if CHS was the root cause ofthe block or another process, just that the interaction within CHS and the rest of the system caused blocking. We also found that information was being sent to CHS that
was not necessary and removed that. Deadlocks Deadlocks were down slightly to 25/day during business days and 27.21 overall.
Network Outages Steelville clinic experienced a three hour outage caused by a router failure. The router was replaced by the network team. The overall down time was up due to overnightmaintenance that took multiple sites down.
Jobs The Report Server restore job failed multiple times at the end of the month due to human error. Programming has been put in place to prevent these types of failures inthe future. All other job success rates were consistent with the previous month.
Citrix
The results from the sample of the five servers is complete. We did see periods of 100% usage on one of the servers caused by the DR. Watson program that generatesthe dump files when an application crashes. These periods were up to 3-5 minutes and would have impacted the users on that server. The servers with the higher
number in the queue seemed to be utilized less than the servers with the lower numbers but it is unclear if this is random or actual behavior. We plan to request a follow-up sample of five servers. Overall usage of the Citrix servers was up slightly in CPU as well as Memory usage.
-
8/13/2019 Network Performance Monitoring
37/37
Any Questions?