© 2013 IBM Corporation

SPN101 Leveraging the Power ofIBM Lotus Notes and Dominoat NationwideDavid O’Neal | Consultant, Infrastructure Engineering, NationwideWouter Aukema | CTO, Trust FactoryFlorian Vogler | CEO, panagenda

What we‘ll cover today

Introduction

IBM Notes and Domino @ Nationwide

What we did– Collect Data (what sources & some stats)

What we found– Confirmations & Opportunities– Configuration, Usage, Performance, Security

What it means– Short Term Quick Wins– Long Term Strategic Takeaways

Conclusions / What we learned

Q&A

2

Introduction – who‘s who?

About Nationwide– Nationwide Mutual Insurance Company, based in Columbus, Ohio, is one of the largest

and strongest diversified insurance and financial services organizations in the U.S. and is rated A+ by both A.M. Best and Standard & Poor’s. The company provides customers a full range of insurance and financial services, including auto insurance, motorcycle, boat, homeowners, pet, life insurance, farm, commercial insurance, annuities, mortgages, mutual funds, pensions, long-term savings plans and specialty health services.

About Trust Factory– Trust Factory‘s DNA provides true insight into server performance and scaling

opportunities. DNA is also used by IBM worldwide as Domino DoubleCheck.

About panagenda– With more than 5.5 million licenses of its products, panagenda helps customers in over

70 countries analyze and optimize their IBM environments.

3

IBM Notes and Domino @ Nationwide

The Nationwide Notes/Domino Environment

– Production use began in 1997 with version 3.3– Migrated/Consolidated to Notes from cc:Mail and a variety of different mainframe email

systems – Current environment

• 6 Domino Domains• 200+ Domino servers on Microsoft Windows® (8.x mixture – mostly 8.5.3 for mail)

(Mail, Management, Application, Blackberry and Good servers)

• Active / Passive clustering accross two data centers• 56,000+ Notes clients (mostly 8.5.2)• 15,000,000+ messages routed weekly• ~20 Sametime 8.5.2 IFR1 servers using Domino and WebSphere• ~1200-1400 Domino applications with 700 being active

4

IBM Notes and Domino @ Nationwide

What is Nationwide trying to accomplish by performing this in-depth analysis?

– Server:• Discover inconsistant configurations, and find gaps where Domino does not readily

report items that could potentially turn into problems.

– Client:• Discover and inventory client side settings, configurations and local databases to get

a better understanding of client health and functionality.

– Environmental:• Combine server and client findings to get a holostic view of our Notes/Domino

environment.

5

What we did

Collect data from Domino servers– statrep.nsf– log.nsf– catalog.nsf– directories (names.nsf,

DA)

Inventory Notes clients– notes.ini– desktop, bookmarks,

names– local databases– various OS and Notes

properties

6

Talk

What we did

Collect– log, statrep and catalog from 151 servers– 33,000 users used 35,000+ clients– 690,000+ documents with 315,000+ attachments collected = 3,5 GB of raw data– 1.6 million desktop icons, 1.5 million local databases, 5.4 million notes.ini entries,

8.5 million client and OS details

Analyze– DNA: Compared this engagement against 2+ million other users– 100+ views created consuming 30+ GB of disk space

Interpret and Correlate

… and now for the meat …

7

12 April 2023

8

Domino Environment Overview

1 Domino Directory

39,725 Users Registered

153 Servers Registered

39,369 Groups Registered

34,057 Users Active 47,178 Databases Touched

151 Servers Analyzed 133,540 Databases Deployed

4 Domino Releases

4 Operating Systems

82,131 Db Storage (GB)

1,477,390 Views Defined

381 View Storage (GB)

494,006 Views Indexed

1,361,855 ACL Entries

1,370,468 Group Members

12 April 2023

9

DNA BenchmarkActive versus Registered Users

0 %

20 %

40 %

60 %

80 %

100 %

Nationwide Lowest Customer DNA Average Highest Customer

Unused Licenses, Web Users, Regular Absense

34,057 active users

12 April 2023

10

DNA BechmarkTime Online

-

5

10

15

20

25

Ses

sion D

ura

tion

(min

s per

ses

sion)

-

10

20

30

40

50

60

Online

Tim

e

(hours

per

use

r)

Session Duration 3 1 4 22

Online Time 24 2 23 77

Nationwide Lowest Customer DNA Average Highest Customer

On average with DNA

12 April 2023

11

User Demand Profiling(Nationwide, 34,057 active accounts)

0%

5%

10%

15%

20%

25%

2 4 6 8 10 12 14 16 18 20 22 24

Distinct Hours Online per Day

System AccountsOffice WorkersRemote Workers

12 April 2023

12

End User Demand Characteristics

Nationwide

0%

25%

50%

75%

100%

check new mail 6% 0% 0% 1% 0% 0%

system dbs 6% 0% 0% 1% 0% 0%

mail files 80% 85% 79% 85% 86% 99%

directories 3% 1% 5% 4% 0% 0%

applications 5% 14% 16% 9% 14% 1%

Notes Sessions Document Reads Document Writes Db Transactions Network Traffic Session Duration

Only mail servers in Scope for DNA

End User Demand Characteristics

Other IBM Customer

0%

25%

50%

75%

100%

check new mail 19% 0% 0% 3% 0% 1%

system dbs 13% 4% 0% 5% 1% 1%

mail files 33% 24% 76% 54% 55% 72%

directories 16% 41% 3% 11% 11% 4%

applications 19% 31% 22% 27% 33% 23%

Notes SessionsDocument ReadsDocument WritesDb TransactionsNetwork TrafficSession Duration

12 April 2023

13

Extreme high docreads on Directory databases

12 April 2023

14

0

1

10

100

1.000

10.000

100.000

1.000.000

10.000.000

100.000.000

1 10 100 1.000 10.000 100.000 1.000.000 10.000.000 100.000.000

Kilo

Byte

s Se

nt t

o Se

rver

.

KiloBytes Read from Server

User Demand on 16739, DatabasesNationwide

369, Application Dbs 560, Domino Directory Dbs 15209, Mailfiles 55, Mailin databases 143, Server Mail Boxes 403, System databases

Showing only databases touched by >1 users.(47,175 databases touched by all users)

Majority of apps are MC

12 April 2023

15

End User Demand at NationwideClassified by Demand Level

0% 25% 50% 75% 100%

User Sessions

Network Traffic(server to client)

Network Traffic(client to server)

DatabaseTransactions

DocumentReads

DocumentWrites

Extreme (1) Intensive (16) Moderate (804) Light (33,236)

1 user does 15% of total network demand

12 April 2023

16

Domino Servers at NationwideClassified by Maximum Session Concurrency

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

80

85

90

95

87 23 23 17 1

Very Low< 50

Low50 - 249

Average250 - 749

Normal750 - 1499

High>= 1500Level

Servers

Redistributing the load can reduce nr. of servers with up to 87

12 April 2023

17

End User DemandSession Concurrency

0

5,000

10,000

15,000

20,000

25,000

30,00020

12-1

2-04

00 04

08

12

16

20

2012

-12-

05 0

0 04

08

12

16

20

2012

-12-

06 0

0 04

08

12

16

20

2012

-12-

07 0

0 04

08

12

16

20

2012

-12-

08 0

0 04

08

12

16

20

2012

-12-

09 0

0 04

08

12

16

20

2012

-12-

10 0

0 04

08

12

16

20

Concu

rren

t U

ser Ses

sions

Max Observed Maximum: 27,179

12 April 2023

18

How Much is Notes Network Compression Used?

Enabled75%

Disabled25%

Includes Traffic from Users and Servers

Network Compression

# Users making use of

Notes Network Compression

0%

25%

50%

75%

100%

Persons Servers

% A

ctive

Use

rs

Enabled

Disabled

Very few customers have this properly

implemented

12 April 2023

19

Deployment Integrity

11 Group Cycles Detected

Integrity check # Databases

Duplicate Replica On Same Server 380

Duplicate Template On Same Server 341

Replicas Acting As Different Template 610

Same Replica But Different Inheritance -

Grand Total 1,331

Document Type Item Type Nr of Documents

group docs listname 3

mail-in docs fullname 22

person docs fullname 2

Grand Total 27

Entries appearing in multiple documents

PubNames, DirCat & DA at

risk (!)

12 April 2023

20

Basic Security Checks

Access Level Databases Templates

Author 84 102

Editor 11 -

Manager 3 302

Reader 2,507 222

Grand Total 2,605 626

Databases with Anonymous Access

1st Customer with NO issues :)

Variations found Accounts

'password' 0

'secret' 0

firstname 0

lastname 0

shortname 0

companyname 0

Grand Total 0

Internet Password Strength

And now for the client-side analysis …

21

Diving right into client-side analysis

The following slides dive intovarious client-side details

In many cases, the Nationwideenvironment is surprisingly clean

– Your environment willmost probably look very different

22

Notes 8.0.2 & 8.5.2

Although there are 1,817 clients with 8.0.2,only 26 have Create_R8_Databases enabled =they do not leverage the benefits of ODS 48

23

Operating system details

Various different client-side operating systems in use:

24

Local replicas of public addressbook

Local replicas of the public addressbook beyond cutoff

Risk of replicating deleted documentsback into server-side replica

Enable PIRC:

25

Local addressbooks:Version mismatch

Checked rows show configurationswhere names design matchesclient version

– (might still have wrong ODS, though)

In general, design mismatch ofsystem databases

– slows down client startup and beyond– causes unexpected behaviour or

non-functioning of Policies

can be fixed by– making sure clients have correct templates– removing TemplateSetup= from notes.ini

26

Local bookmark.nsfs:Version mismatch

Checked rows show configurationswhere bookmark design matchesclient version

– (might still have wrong ODS, though)

27

Local cache.ndk:Version mismatch

Checked rows show configurationswhere cache design does NOT matchclient version

– (might still have wrong ODS, though)

Cache.ndk must be deleted andre-created together withCREATE_R85_DATABASES=1in notes.ini - for it to have properdesign and ODS(make sure client has correct cache.ntf)

28

Local desktop.ndk:ODS issues

Checked rows show configurationswhere desktop ODS is NOT ideal

29

Local log.nsfs:Version mismatch

Checked rows show configurationswhere log design does NOT matchclient version

– (might still have wrong ODS, though)

30

Notes.ini:Log=

A couple of users have multiple log= lines in notes.ini

Since only the first entry is actually read in such a case,logging does not work as expected for those users

31

More on ODS levels

Various databases and templates do not havean ideal ODS …:

AddingCREATE_R85_DATABASES=1andNSF_UpdateODS=1to notes.ini can help!

32

More notes.ini entries …

Less than 1% of all users have port compressiondisabled, but 25% of all traffic is uncompressed must be enabled on BOTH servers and clients identify servers that are used by users but haveport compression disabled

33

EXTMGR_ADDINS= …

Various users haveEXTMGR_ADDINSentries in notes.ini which areseperated by a blank

surprisingly DOES work(side-effects unknown)

34

Mail

Who encrypts email when saving emails?

Who encrypts sent email?

Who signs sent emails?

35

Calendar

Users with wrong(?) calendar settings?

1=Sunday2=Monday3=Tuesday5=Thursday

36

Cache.ndk

Users where Cache= in notes.ini points to– A dedicated file/path– Partly filepaths in which users

might not have write permissions(e.g. Notes program files directory)

37

Clustering / Loadbalancing

FailoverSilent (defaults to 0) is disabled for almost all users:

38

Client Configuration Health

Basic vs. Standard clients

Clients with wrong InstallMode= setting in notes.ini

39

… same for ini:InstallType=

Identifying Client/Admin/Designer configurations:

40

Hardware/OS details: disk space

Users with too little free disk space– might soon call helpdesk– may experience stability issues– have high disk fragmentation = slooooow

41

Hardware/OS details: memory

Users with too little memory– again, sloooow(er) client experience

42

Locations: do not use IP addresses as mailserver names …

A couple of users have an IP address configured as their mailserverbreaks Policies

DNS names as mailservers could become a problem if the DNS domain were ever to be renamed …

43

Mailfile replicas?

98 users work off a local replica+330 managed replica users

BUT: 3,407 users have a local replicaand: 149 users have more than one mail replica …

– 39 of these local replicas are beyond cutoff:

44

ECL

Open doors

45

Analyzing desktop icons (special vs. local vs. server)

46

Analyzing desktop icons (details)

196,930 local databases with an icon (e.g. bookmark.nsf)

380,243 local databases without an icon (e.g. help files, cache.ndk, …)

1,266 templates on desktops

37,108 templates not on desktops (think shared data directory)

36,865 replicas without any icon

267 replicas without a local icon

2,686 replicas without a server icon

862,395 template replicas without any icon

14 template replicas without a local icon

765 template replicas without a server icon47

Summing it up …

48

Conclusions

Mission accomplished– Provided a holistic view across servers and clients

Mission not accomplished (yet)– This is work in process

Nationwide is the 1st customer out of many that leverages most of the features/potentials of the N/D 8.5 platform

– Optimization potential almost exclusively in areas “without features”– Implemented Domino password security the way it should be

49

What we learned

Detailed data helps to leverage IBM Notes and Domino to its fullest potential

… and helps shifting from reactive to proactive

Assumptions vs. Evidence– Eliminate best guess/hope based working

Find out and focus on what really matters

50

Q&A

Thank you for listening!

51

52 © 2013 IBM Corporation

Legal disclaimer© IBM Corporation 2013. All Rights Reserved.

The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.