migration from ibm domino to ibm verse
TRANSCRIPT
1216 - Smooth Migration from IBM
Domino to IBM Verse : The Vital
Metrics You Do Need
Jérôme Deniau, move4ideas / September, 16th 2016
About Jérôme Deniau
• Co-founder & CTO at
• Member of Penumbra – an international consortium of selected Business Partners
pooling their talents and resources
• Designer of for Domino in 2006
• IBM Design Partner member
• Certified IBM/Lotus Instructor since 1993
• Certified Advanced Domino Administrator and Developer since Lotus-Notes R3
Abstract• IBM Domino server is having an exceptional functionality and rich
features which fit perfectly for customer and their very own business
needs.
• IBM Verse defines a #NewWayToWork that comes with Social
powerful features you can use and activate without compromising your
IBM Domino investment.
• How to migrate your IBM Domino server mail system to IBM Verse
simply without no hassle downtime and make your end-users happy
with the migration process.
Agenda
• Why Are We Here
• Pre-requisites
• W4
• Metrics
Why Are We Here
• Audience
Main target: IT Department and C-level but it is not limited
• Architecture change
• Migration headaches
• Where is my starting point
• #NewWayToWork
• Keep It Simple (but NOT) Stupid
• Keep in mind the 4 V’s (Value/Variety/Velocity/Volume)
Prerequisites
• Before Rush
Knowledge
Hard work
Attitude
Prerequisites - Knowledge
• This is the very first step
Can’t do without!
• Source
Have a good knowledge of your Domino environment and
architecture
• Target
Know your target and Procedures
On-premise
Hybrid
Pure-cloud
Prerequisites – Hard Work
• Prepare your team and your system
Complete checklist of your system
No rush
The simpler the better
Prerequisites - Attitude
• Last but not least
Be positive
Be invested
Be inventive
Be an evangelist
• Make it a success
W4
• The W power
The “Why We Should Move” is really far behind us by now
• The four vital points:
Who
What
Where
When
W4
W4 – What For?
• You will be able to define a set of population to migrate them altogether
When
WhatWho
Where
W4 – The Who Dilemma• Active vs inactive users
What defines active people vs inactive people
One can be active without answering mails
Database Activity
Mail rules
• Relations: people who collaborate, work together
• Domino applications and mail agents
• External usersWho
W4 – The What Dilemma
• Smartphones vs IBM Notes addicts
• Attachments: Enterprise data
• Huge databases
• Sensitive data
What
W4 – The Where Dilemma• Limited with IBM Verse for the cloud version
Asia Pacific
Europe
United States
• For this session metrics will take in account these targets instead:
Hybrid
On-premise
Pure cloud (private or public) Where
W4 – The When Dilemma• Last step at last
Once populations are defined
Deployment plan to run the migration at the right time for the population to
migrate
• Off-hours
• Off-peak
• Avoid Monday morning burstWhen
W4 – The Big Picture
W4 – The Big Picture
WhenWhatWho
Where
W4 – The Big Picture
WhenWhatWho
Where
Metrics For The Who Dilemma
Use Case
• We want to migrate only active users!
How to get this information from Domino ?
Active vs Inactive Users
Active vs Inactive Users• You do need to distinguish between false active and false inactive
Comparing databases sizes
Domino built-in tools for accurate information
Database properties: user activity detail
Domino log.nsf ‘Database Usage’ view
Domino activity logging (activity.nsf)
License Tracking (userlicenses.nsf)
Active vs Inactive Users• Quick and dirty metrics based on database files sizes
Export the mail files view from IBM Administrator on regular basis
Too simple and quite confusing
CTRL-C + Paste into a spreadsheet or better use IBM Watson Analytics
Active vs Inactive Users• Less dirty: log.nsf using the Usage views
Statlog should run regularly
Active vs Inactive Users• Quick and dirty (cont.)
Database sizes metrics are insufficients
View indexes sizes are missing (you can get rid of them with
purge or compact –D)
Take care of:
o Updall –c (in clustered or non clustered environment)
Need a tool to analyze correctly the information
Active vs Inactive Users
• Database user detail
No_Force_Activity_Logging=0 and Statlog task should run
Pros:
• simple to use
• C api calls can be used to grab the information
Cons:
• Results are on a database basis
• Erratic problems
• Limited use due to the 32kb limit (Wait for the RNext 16MB)
Solution: use scanEZ from Ytria
Active vs Inactive Users• Database user detail samples
• IBM Notes
• Ytria DatabaseEZ
Active vs Inactive Users
• Domino log.nsf
Information is set by the Statlog task
Add Log_Sessions=1 or 2 in notes.ini
Remove Log_Disable_Session_Info=1 if exists in notes.ini
Active vs Inactive Users• Domino activity logging
Simple configuration
Domino.Notes.Session
Domino.Notes.Database
Validation: show server
IBM Technote for retention period
#1230016 (from 2010) default is 7 days so:
o Log=logfiles/log.nsf,1,0,7,40000,[1-6]
Active vs Inactive Users
• Domino activity logging (continued)
Run the Activity analysis tool
Active vs Inactive Users• License Tracking
• Simple configuration
• Adminp is required to aggregate (so take care of that)
• All views under ‘Active Users’ can be a good starting point
Active vs Inactive Users
• Conclusions
A user who does not send mail can be active
Message Tracking Collector?
Automatic or manual reports in reports.nsf:
o Top 25 xxx
o Messages size
o Message volume
Tips• Mail Tracking Collector retention period
Default is 30 days
Accommodate this value according to your needs
Use Case
• We want to migrate people involved in ‘communities’.
Relationship
Relationship• How to define groups of people working together
Do we have metrics out of the box?
Technically:
Complicated if we talk about Communities detection for Domino
Complicated if talking about Dynamic Graphs
Complicated to implement Louvain method
• So what is available and simple to use?
Relationship• Message Tracking Collector
Yes but what for? Better compare manually the information
Inefficient for relationship but fields can be useful
Relationship• Increasing Domino logging for mail:
Log_MailRouting=30
Result:
Once again limited: but
Data is stored in log.nsf in the “Mail Routing” view
o If LOG_MailRoutingToMiscEvent=1 is not specified
Data is stored in text file(s) console log(s) in <Domino data Folder>/IBM_TECHNICAL_SUPPORT
Relationship• Domino Mail Journaling
High volume on some sites
Needs replication between servers
Caution due to high volumes and data retention delays
Needs development to build a map for user relationship
Do not encrypt $TUA nor $MessageID
$TUA field can help you group mail exchanges on one subject amongst a group of user
o Beware: you can have multiple $TUA fields
Relationship• Activity Logging
Domino.Mail entry type
As-is needs a bit of design to make it work the way you need it
Use the activity logging analysis tool from IBM Administrator
Deposited view
Processed view
Relationship• What the analysis tool does is:
Get the documents from log.nsf
Form=“Mail Routing”
Relationship
• Organizational Units can help
Group people according to their OU’s
• Domino mail server
Group people according to their home server
Relationship• Conclusions
Not really an out of the box functionality
Need a specific development
Or a third party tool
Use Case
• We want to distinguish between ‘Human’ and ‘non Human’
users
Domino Applications & Mail Agents
Domino Applications & Mail Agents• In a wonderful world
Designers are using a ‘fake’ or non user mail account when the application or the agent sends a memo
Designers can use two users information
Return-path: in case of error (Field Principal or From)
User reply address: for a normal reply (field ReplyTo)
(optionally) define a fake mail address in agent security property field “Run on behalf of”
Designers have used NotesLog or Log classes and an agentLog database
Domino Applications & Mail Agents
• Activity logging
Insufficient, lacks of information on agent type and
Does not generate information for the following agents types:
o Web agent
o Manually run agent from the IBM Notes client
o Scheduled agents that run locally on a client
Domino Applications & Mail Agents
• ICAA application supported
• Keep the agents and applications on premise or Hybrid Verse with SoftLayer
• Give a distinguished email address to agents that send mail
With Designer
With correct fields (for MailSend or agents that creates document directly into mailboxes)
This way you can use Domino logs to distinguish between agents and end-users.
Domino Applications & Mail Agents• Conclusion
Ask the developpers
To give mail agents a mail account that can be tracked easily
To use From/Principal and ReplyTo fields correctly into the code
To user agentlog or any log facility when sending mail
o NOT in log.nsf nor Domino Server Console please!
Manage Domino scheduled agents and log the information
Use Case
• We want to take care of external users
External Users
External Users
• Concern is: external users to your company that connect to the
Domino server(s) and use Domino mail functionality
• How many external users use the Domino mail service internally?
• Domino Passthru
Activity Logging option
Other considerations
• Are my HTTP passwords strong enough to protect current and future
information?
Tips• Check HTTP password weakness
Use Domino HTTP stack
Use DIIOP (Domino Corba program)
Develop a JAVA agent with NotesFactory
Loop with Domino users
Use default passwords
Use a dictionary
Store results in an IBM Notes database
Metrics For The What Dilemma
Use Case
• How many mails are sent from my mobile users?
Smartphones vs IBM Notes Addicts
Smartphones vs IBM Notes Addicts
• Concern is: Get the Traveler users addicts
• Metrics:
Number of active users
Number of mails from the handhelds
Smartphones vs IBM Notes Addicts
• Traveler Users
Tell traveler users enabled
Tell traveler users disabled
• Deeper statistics
What is activated
Increase the Log level first
Smartphones vs IBM Notes Addicts• What is activated
Tell traveler log show
Tell traveler log handler
What is my level
Which handler can I check?
Handlers
Tell traveler log handler logNsf level finest
Tell traveler log handler audit level ALL
Smartphones vs IBM Notes Addicts• How to activate user info trace:
Rename the NTSLoggingTemplate.properties in NTSLogging.properties
Located in <DominoDataDir>/traveler/cfg
Define LogLevel=ALL
Define Privacy according to your needs, default is 0x001E (shows everything except BODY and Attachments) everything is 0x8000
OR Tell traveler log fields *
And without any parameter to stop -> “Tell Traveler log fields”
Restart task Traveler for activating or stopping the logging
Smartphones vs IBM Notes Addicts• info trace results are located:
<DominoDataDirectory>/IBM_TECHNICAL_SUPPORT/traveler/logs/xml
mmdd-hhmm-XXXXX_Sync_<CollectionId>_<SyncKey>_out_200.xml
mmdd-hhmm-XXXXX_SmartReply_<NOTEUNID>_in.xml
mmdd-hhmm-XXXXX_SmartForward_<NOTEUNID>_in.xml
mmdd-hh-mm_Ping_in.xml
mmdd-hh-mm_Ping_out_200.xml
Smartphones vs IBM Notes Addicts
Smartphones vs IBM Notes Addicts• The simpler the better
How to track Traveler users
The $Mailer field contains the “Traveler” string
Mail journaling:
define the correct rule!
o Internal users and all documents
o Beware of the size of the resulting database!
Do not encrypt the fields you do need for analysis
o $Mailer
Smartphones vs IBM Notes Addicts• Create a view in mail journaling template
From an existing view (Person)
Add a column for $Mailer field
Add totals
Smartphones vs IBM Notes Addicts• Conclusions
Number of mails sent from a device
Nothing out of the box with Traveler logs
o Need to analyze the log files in xml directory for IBM Lotus Traveler
Mail Journaling
o Need really few design enhancements
Need third party software
o Access to the database (Non-HA and HA)
o Live and simple information: audit4mail from move4ideas
Use Case
• We want to get rid of non enterprise data, mainly based on
attachments not on content.
Attachments
Attachments• Concern: an excellent starting point to start using IBM Files
Remove files from mail files to the repository they deserve
Company information has to be at the company level
• DAOS is not a concern for metrics
DAOS obfuscate file location (in or out NSF files)
Quota apply to logical size NOT the physical size
After migration, DAOS will need to be pruned
Attachments
• A basic and efficient approach for a global picture
Reset: set stat <stat name>
Show stat mail.* (or IBM Administrator interface)
TotalKBDelivered
TotalKBTransferred
TotalKBTransferred.SMTP
DeliveredSize.*
Attachments• Mail journaling
Define the rule for internal users only
Already available in IBM Notes mail journaling databases
Beware of mail journaling databases size
You can customize the view for more information
Extensions of attachments
Attachments
• Use Event task
Create an event generator for:
Message transferred to
Message delivered to
Attachments• (Mis)Use DAOSEST
Download version from IBM web site
Currently 1,7
Use –v option
Beware: on really huge architecture it can never end.
Solution DAOSEST_BUCKETS in notes.ini
o DAOSEST_BUCKETS=16,64,128,256,512,768,1024,2048,3087,4096
o Any value of 10k can be set
Use Case
• We want to know how many huge databases we do have and who
owns such databases.
Huge Databases
Huge Databases• Concern: not a target concern but the source concern
One mail file of 51k documents (31Gb)
16 hours to migrate to SmartCloud in the past (non DAOS)
• At which size do we consider a database is huge?
Time to replicate
Time to compact (even with option –#)
Views indexes size (Welcome Domino 9.2!)
Daosified or non daosified NSF databases
Huge Databases• The main problems are
Internal network bandwith
Slow disks (yes it still exists!)
Overloaded Domino servers
No or bad scheduled FUC tasks
o Fixup
o Updall
o Compact
The notesenv folder for the Onboarding Manager requires space
Huge Databases• Main problems (continued)
Calendar
No ‘Calendar cleanup’ execution on a daily/weekly basis
Orphan calendar entries with attachments in iNotes
o Cancelled reservations are kept within mail files in iNotes
Track attachments
Enterprise or personal
Track view indexes
Track databases size
Huge Databases• Create replica databases with cluster_admin_on
Faster than any other solution (DAOS or NOT)
cl copy srv!!database.nsf TransfertServer!!database.nsf REPLICA
• Archive or not Archive?
A real question
Archive Pro
Get rid of inefficient data
Archive Con
Aggregate them back
Huge Databases
• How to track the huge database
IBM Administrator: copy and paste the Files view into a spreadheet Software
Log.nsf: View Usage/By Size
Use Case
• We want to check for sensitive data and take a decision to migrate that
information or not.
Sensitive Data
Sensitive Data• Attachments vs content
Attachments extensions can help BUT can be obfuscated
Unions vs VIPs vs C-Levels
• Forwards of sensitive data
• Blind copy of sensitive data
• Encryption
Are my users do use encryption for sensitive data when mailing the information to internal or external users?
Sensitive Data
• If forwards matter
Use the Event task
Message forwarded to xxx
Message forwarded to xxx from yyy
• Mail Journaling
Activate the mail journaling function
Other Considerations
• I want more information on Traveler data than the current one I do
have
• What about user data
Archives
User mail rules
Number of contacts
Tips• Traveler (for the non HA server)
Just try: te traveler sql select * from sys.systables
• User mail file content (LotusScript or Java agent or …)
Loop through all mailfiles
Get number of archives and their names
Get number of mail rules
Get number of contacts (private or not)
Export information to a notes database
Export information to a CSV file
Metrics For The Where Dilemma
Use Case
• We want to migrate people from one region or ‘site’ together.
The Where Dilemma• Concern:
Think of collaboration teams
Think of really bad networks
New-Caledonia is an excellent example
• What to catch:
IP addresses from the sender
Cross check with relationships
IP Address• IP Address can help define people location
Log_Sessions=2 in Domino notes.ini
23/11/2015 10:01:09 Opened session for 'CN=Jérôme Deniau/O=Move4Ideas' (Release 9.0) SessId 520221FB Ip '109.8.206.35' 'Auth' 'C'
Inbound SMTP only: Mail Journaling (again)
Do not encrypt the “Received” field
Create a view that displays the “Received” field(s)
Analyze the field content
IP Address and Cross Checking• Cross check log sessions and mail statistics
Mail statistics: Transferred vs Delivered
TotalRouted: means number of recipients
Transferred: means number of physical message transferred
Delivered: means number of physical messages delivered
Use Event task
Architecture
• Use of Domino Server task
MAPS
Can help in discovering the mail routing between servers
No interest if only one Domino cluster manages all users
Metrics For The When Dilemma
Use Case
• We want to define the best time range to migrate a population.
When
When
• Concern:
Avoid user dissatisfaction
• Need to define the best time for switching users
• Communication is essential
• Metrics
Time when user is not sending mail
Problem with smartphone users
o Always on line
Europeans policy on mail during the week-end
When• Metrics can help but
Solution is more enterprise policy defined
Once population are defined
You can force the switch time and date
You can inform the end user for the correct procedures.
You can start replication and onboarding process during off-peak hours
• So what about metrics then
When
• Determine number of mails exchanged during your off-hours
Problematic with worldwide companies
Track internal sender and define the best time range for switching
Depends on replication
Depends on mail availability on the other side.
Use Domino mail journaling features and to an analyze on off hours/off-
days for internal senders
Session Conclusion
• Metrics are the starting point
• Metrics help you understand how people work within your company
• This session gives you the main axis to start
• Do not forget to compare the metrics from the source platform to the
targeted system after the migration.
Thank you for listening!