Download - JPMC Sec Lending
Project Code: MXC-W061
DML to SLX Integration at JPMorgan Chase
A Major Qualifying Project Report
Submitted to the faculty of Worcester Polytechnic Institute
In partial fulfillment of the requirements for the Degree of Bachelor of Science
Submitted By: _______________________ _______________________
Michael J. Kristan Megan R. Slonski
Project Center: Wall St. New York, NY B Term 2006 Sponsoring Agency:
JPMorgan Chase & Co.
Submitted To: Project Advisors: ________________________ _________________________
Michael Ciaraldi Arthur Gerstenfeld
On-Site Liaisons: Marisa Giliberti Danielle Rumore Brian Kenney
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
i
Abstract
This MQP aimed to assist JPMorgan Chase with the successful completion of the
high priority DML to SLX Integration project. In the pre-design phase, work was carried
out to map data fields between the two systems and to analyze the anticipated effect of
this project on server performance. Concurrently, a thorough project plan was created to
be used in a critical path analysis. Struggles in creating this led to the recommendation of
a process for better managing projects.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
ii
Authorship
This Major Qualifying Project is a collaborative effort of both team members. In
compiling research and writing this paper each of us dedicated a significant amount of
time and effort. There was an overall equal contribution to the executive summary,
introduction, and background section made by both team members. Megan Slonski was
mainly responsible to the sections pertaining to the critical path analysis and project
management process improvements. Michael Kristan was primarily responsible for the
sections regarding capacity planning and data mapping.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
iii
Executive Summary
As the 17th largest corporation in the United States, JPMorgan Chase & Co.
provides a wide range of banking and financial services to customers worldwide. One
such service is known as securities lending. Securities lending is the market practice
whereby securities are temporarily transferred from a lender to a borrower. For the
period of the loan, the lender is protected against default through receipt of collateral
from the borrower. JPMorgan acts as the facilitator in this transaction by matching
lenders to qualified borrowers. JPMorgan also invests the collateral received on the loan
and splits the return on investment between all three parties.
In order to maintain its outstanding reputation, JPMorgan Chase & Co. must
utilize new technology, maximize efficiency of its internal processes, and continually
improve the services provided to its clients. Understanding the importance of these key
business strategies, JPMorgan executives decided that the software Securities Lending
Xpress (SLX) should be the sole front-end trading platform used to initiate all securities
lending loans globally. The reason for this decision is that SLX has checks built in that
ensure compliance to audit and government regulations. Also SLX is built on the most
up-to-date, flexible platform when compared to the other systems. This will allow future
changes to be made more quickly and at a lower cost. In this vision the other systems,
including International Securities Lending System / Dan McLune (DML), would only be
used as loan maintenance systems and would cease to perform any loan initiation activity.
The current state of the business requires that several incremental projects be completed
in order to turn this vision into a reality. One such project is to integrate DML and SLX
so that information pertaining to loan maintenance activity in DML is passed in the form
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
iv
of a message to SLX throughout the day and overnight using a feedback loop. Only then
can SLX use this information to accurately update lender limit and borrower credit line
utilizations.
The work done on this Major Qualifying Project (MQP) aimed to assist JPMorgan
with the successful completion of this important project. Specifically, work was done to
accomplish four main goals. The first two relate to pre-design activities that needed be
completed prior to the design and build phases of the project. Success of the pre-design
activities would contribute to the successful implementation of the feedback loop. The
first goal of this MQP was to assist in mapping and matching the data fields between the
two systems to ensure that SLX receives all of the information it needs from DML to
properly update its database. The second goal was to analyze the anticipated effects of
the feedback loop on servers and mainframes so that additional resources could be
introduced and performance would not be compromised. The third goal of this MQP was
to gather the information needed to conduct a critical path analysis that would examine
ways to expedite the delivery of the project. From this a fourth goal was established,
which was to recommend possible improvements to project management practices that
could result in shorter project durations, fewer delays, and more accurate estimates of
project cost and delivery dates for this and future projects.
The objective of the data mapping exercise was to examine the fields that DML
has defined in a loan snapshot and compare that to the data fields SLX needs to receive to
properly update borrower credit and lender limit utilizations. One of the challenges was
that the fields in DML do not directly map to the fields in SLX. Therefore, an in-depth
analysis needed to be conducted to examine how the data fields from DML should be
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
v
manipulated so that they are in a useable format for SLX. As part of the data mapping
exercises, sample loans were observed from the production environment. Unfortunately
a problem occurred when taking the raw loan data from DML and inputting it into a
Microsoft Access database that would allow it to be analyzed. This raw data is known as
a loan master file. There were more loan records in the loan master file than Microsoft
Access allowed to be imported at once. To get around this, a program was developed that
would split the file into multiple, smaller files that could be imported into Microsoft
Access one at a time. With this accomplished, the JPMorgan team could begin to analyze
how DML fields would map to SLX fields.
The second objective, capacity planning, was to carry out a comprehensive
analysis on the SLX servers and to gather enough information to make infrastructure
decisions moving forward. The primary deliverable was a document that gives the reader
an understanding of the resources required to run SLX in a production environment and
to analyze the impact that the DML feedback loop will have on the systems. Information
was gathered from many sources and compiled together to form a complete picture of the
resources that SLX depends on. One of the primary data sources was a feedback loop
that was formerly created as part of a project to integrate SLX with the domestic
counterpart to DML known as SLMU. The plan was to investigate the current
performance of the SLX servers with the SLMU feedback loop in use. The next step was
to compare the number of SLMU messages to the anticipated number of DML messages
to predict the effect that the DML feedback loop would have on SLX server performance.
The analysis showed that the anticipated volume of DML messages between
11AM and 1PM is so high that it will diminish the SLX server performance. The
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
vi
performance will be diminished to such an extent that SLX will not be able to process
messages from DML in the time deemed acceptable by business executives. This will
hinder the booking of new loans and the updating of lender and borrower utilizations
which will result in lost opportunity for JPMorgan.
The third objective of this MQP was to gather the information needed to conduct
a critical path analysis that would examine ways to expedite the delivery of the DML to
SLX Integration Project. This project was important because it was deemed a high
priority within the firm. A number of steps needed to be completed to gather the
information that was required to utilize the critical path method in the most effective way.
The basis for this analysis would be a thorough and cohesive project plan, which was
created as a product of this objective. Meetings were held with each member of the
project team to determine all project tasks, time estimates for the completion of each task,
and the network of task dependencies. Work was also done with the project delivery
manager to compile a list of all resources that would contribute to the project and to
determine the percentage of each resource’s time that would be allocated to this initiative.
All of this information was then incorporated into the project plan to be used in the future
management of the project.
The difficulty experienced in compiling the information needed for a thorough
project plan exposed the need for the fourth goal of this MQP, which was to recommend
improvements to project management processes. The improvements were aimed at
making the processes more efficient, streamlined, simple and/or error-proof. In order to
make these improvements, a thorough investigation of project management practices was
conducted surrounding the following activities: providing project cost and duration
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
vii
estimates to the Investment Council for project funding, creating an accurate project plan
and using it keep the project on schedule, and tracking action items that arise during team
meetings. This evaluation showed that there is a lot of room to make improvements to
how projects are managed. The biggest observation is that there is not a defined process
based on best practices in place in the Securities Lending Technology group. A different
process is used for each new project. This results in confusion, poor practices being used,
and time wasted by all team members learning a new system for each project. Therefore,
we have developed a process for managing projects that encompasses the aforementioned
aspects of project management to be used by all project managers. It is based on the
inefficiencies observed with current practices and on what was learned about the project
needs that project managers are responsible to meet. The implementation of this process
is expected to result in shorter project duration, fewer delays, and a more accurate sense
of project cost and end dates due to increased communication across team roles, greater
accountability and awareness of due dates, and simpler, repeatable processes that are less
prone to error.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
viii
Acknowledgements
We would like to thank Worcester Polytechnic Institute and our sponsor
JPMorgan Chase for giving us the opportunity to complete this project with the Securities
Lending Technology group at JPMorgan. Particularly we would like to thank Danielle
Rumore, Brian Kenney, Vitoriana Morais, Mike Salamina, Reynaldo Generoso, Besley
Nelson, Peter Ennis, and our liaison Marisa Giliberti for their continuous support and
commitment to helping us make this a successful project and a meaningful learning
experience.
Furthermore, we would like to thank our advisors Professor Arthur Gerstenfeld
and Professor Mike Ciaraldi for their guidance and assistance in helping us develop our
project and write this report.
.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
ix
Table of Contents
Abstract............................................................................................................................... i Authorship ......................................................................................................................... ii Executive Summary......................................................................................................... iii Acknowledgements ........................................................................................................ viii Table of Contents ............................................................................................................. ix Table of Figures................................................................................................................ xi Table of Tables ................................................................................................................ xii 1. Introduction................................................................................................................... 1 2. Background ................................................................................................................... 5
2.1. JPMorgan Chase & Co............................................................................................. 5 2.2. JPMorgan Worldwide Securities Services............................................................... 6 2.3. Securities Lending ................................................................................................... 7 2.4. JPMorgan’s Role in Securities Lending ................................................................ 10 2.5. The Technology Behind JPMorgan Securities Lending ........................................ 13
2.5.1. Custody Management Systems – COSMIC & TITAN................................... 13 2.5.2. Front-end Trading - SLX ................................................................................ 14
2.5.2.1. SLX Hardware ......................................................................................... 14 2.5.2.2. SLX Trade Views .................................................................................... 14
2.5.3. Legacy Trading Application - DML............................................................... 15 2.5.4. Legacy Trading Application - SLMU............................................................. 15
2.6. Project Delivery Framework for JPMorgan Worldwide Securities Services ........ 16 2.7. SMLU to SLX Integration Project......................................................................... 17
3. Methodology ................................................................................................................ 20 3.1. Critical Path Analysis ............................................................................................ 20 3.2. Project Management Process Improvements ......................................................... 23 3.3. Data Mapping Exercises ........................................................................................ 23
3.3.1. Parsing the DML Loan Master ....................................................................... 24 3.3.1.1. Design ...................................................................................................... 25 3.3.1.2. Implementation ........................................................................................ 25
3.3.2. Aggregation and Summarization of Data Fields............................................. 26 3.4. Capacity Planning Analysis ................................................................................... 27
3.4.1. Server Specifications ...................................................................................... 27 3.4.2. Used Server Resources ................................................................................... 28 3.4.3. SLMU Message Statistics ............................................................................... 28
4. Results and Analysis ................................................................................................... 30 4.1. Critical Path Analysis ............................................................................................ 30 4.2. Project Management Process Improvements ......................................................... 37
4.2.1. Investment Council Estimates and Project Plan Creation............................... 38 4.2.1.1. Current Process ........................................................................................ 38 4.2.1.2. Problems .................................................................................................. 39
4.2.2. Use of Microsoft Project................................................................................. 43 4.2.3. Action Item Tracking...................................................................................... 44
4.3. Data Mapping Exercises ........................................................................................ 45 4.3.1. Parsing the DML Loan Master ....................................................................... 45
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
x
4.3.2. Aggregation and Summarization of Data Fields............................................. 48 4.4. Capacity Planning Analysis ................................................................................... 48
4.4.1. Processor Analysis .......................................................................................... 48 4.4.2. Memory Analysis............................................................................................ 49 4.4.3. Storage Capacity ............................................................................................. 51 4.4.4. SLMU Message Statistics ............................................................................... 52 4.4.5. DML Projections............................................................................................. 54
5. Recommendations ....................................................................................................... 56 5.1. Project Management Process Improvements ......................................................... 56
5.1.1. The Estimation Workbook.............................................................................. 58 5.1.2. Microsoft Project and the Critical Path Method ............................................. 64 5.1.3. Action Item Tracking...................................................................................... 67 5.1.4. Weekly Status Meetings ................................................................................. 70
5.2. IT Equipment Upgrades for SLX........................................................................... 71 Works Cited..................................................................................................................... 73 Appendix A : Securities Lending Technology Systems Relationships ....................... 75 Appendix B : High Level Task Document for Project Plan Creation........................ 76 Appendix C : Snapshots of the STMate Tool ............................................................... 80 Appendix D : Screenshots of the same loan in DML and SLX................................... 82 Appendix E : DML Loan Master Parser Source Code ............................................... 84 Appendix F : DML Loan Master Parser JUnit Test Case .......................................... 88 Appendix G : Microsoft Access SQL Queries .............................................................. 89
Join DML/SLX Brokers................................................................................................ 89 DML Brokers without matching SLX Records ............................................................ 89 SLX Brokers without matching DML Records ............................................................ 89 Unique LOAN-TYPE Values ....................................................................................... 89 Unique SEC-LOC Values ............................................................................................. 89 Unique SEC-TYPE Values ........................................................................................... 89
Appendix H : Invitation to Final Presentation............................................................. 90 Appendix I : Review of SLX Server Performance....................................................... 91
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
xi
Table of Figures
Figure 1: PDF Project Phases (Project Delivery Framework, 2006) ................................ 16 Figure 2: Current Flow of Securities Lending Information (Kenney, 2006).................... 19 Figure 3: Design of DML Loan Master Parser (Java Trademark, 2006).......................... 25 Figure 4: Flow Chart for Business Analyst....................................................................... 31 Figure 5: Flow Chart for Quality Assurance Team .......................................................... 32 Figure 6: Flow Chart for DML Team ............................................................................... 33 Figure 7: Flow Chart for SLX Team................................................................................. 34 Figure 8: Flow Chart for SLX Mainframe Team.............................................................. 35 Figure 9: STMate Snapshot Showing Unnecessary Task List.......................................... 41 Figure 10: Class Dependency Diagram for DMLLoanMasterParser ............................... 46 Figure 11: Instability Metric for DMLLoanMasterParser (green dot).............................. 46 Figure 12: Efferent Couplings for DMLLoanMasterParser.............................................. 47 Figure 13: Class and Package Statistics for DMLLoanMasterParser............................... 47 Figure 14: Production Server CPU Usage ........................................................................ 49 Figure 15: Memory Usage on APP06 (GTI Hobbit, 2006) .............................................. 50 Figure 16: Memory Usage on APP10 (GTI Hobbit, 2006) .............................................. 50 Figure 17: Memory Usage on DBP01 (GTI Hobbit, 2006) .............................................. 50 Figure 18: /local/app/a_slx on APP06 frequently uses more than 90% of quota ............ 51 Figure 19: /app/a_slxftp on the JIP NAS is near capacity ................................................ 52 Figure 20: SLMU Messages in SLX................................................................................. 53 Figure 21: Comparing SLMU Performance Between Oct and Nov ................................. 54 Figure 22: SLMU and Anticipated DML Messages over a Trading Day......................... 55 Figure 23: Project Management Process........................................................................... 57 Figure 24: Estimation Workbook- Instructions ................................................................ 59 Figure 25: Estimation Workbook- Team Member Worksheet ......................................... 60 Figure 26: Estimation Workbook- Summary Sheet by Phase .......................................... 62 Figure 27: Estimation Workbook- Summary Sheet by Role ............................................ 63 Figure 28: Diagram of Two Layer Project........................................................................ 65 Figure 29: Action Item Tracking Sheet Instructions......................................................... 68 Figure 30: Action Item Tracking Sheet for Action Items ................................................. 69 Figure 31: Action Item Tracking Sheet for Risk Items..................................................... 69
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
xii
Table of Tables
Table 1: Acceptable Non-cash Collateral (Devasthali, 2006) ............................................ 7 Table 2: Key Team Members and Roles........................................................................... 22 Table 3: SLX Production Server Hardware (Ghelani, 2006)............................................ 28 Table 4: STMate Entry Based on Complexity of Repeated Tasks ................................... 42 Table 5: Code Metrics for DMLLoanMasterParser.java .................................................. 45
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
1
1. Introduction
As the 17th largest corporation in the United States, JPMorgan Chase & Co.
provides a wide range of banking and financial services to customers worldwide. In
order to maintain its outstanding reputation, JPMorgan Chase & Co. must utilize new
technology, maximize efficiency of its internal processes, and continually improve the
services provided to its clients. Understanding the importance of these key business
strategies, JPMorgan executives decided in 2004 that the software Securities Lending
Xpress (SLX) should be the sole front-end trading platform used to initiate all securities
lending trades globally. The reason for this decision is that SLX has checks built in that
ensure compliance to regulations and lender rules and limits. Also SLX is built on the
most up-to-date, flexible platform when compared to the other systems. This will allow
future changes to be made more quickly and at a lower cost. In this vision, the other
systems, International Securities Lending System / Dan McLune (DML), Securities
Lending Menu (SLMU), and GlobalOne, would only be used as loan maintenance
systems and would cease to perform any loan initiation activity. The current state of the
business requires that several incremental projects be completed in order to turn this
vision into a reality. One such project is to integrate DML and SLX so that information
from DML is passed to SLX throughout the day and overnight.
To understand the need for this project, one must understand the current process
and the implications it had on successful securities lending transactions. At the time,
DML was the system responsible for international loan maintenance activities. It tracked
actions such as marks, recalls and paybacks and used this information to immediately
update borrower credit line utilization and lender limit utilization. JPMorgan holds its
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
2
borrowers in the securities lending program to various restrictions, based on their
contract, dictating how much a particular borrower can have in outstanding loans at a
time. This is known as their credit limit. As loans are repaid and new ones are taken out,
the utilization of a borrower’s credit limit changes. DML tracks these activities and
immediately updates borrower credit line utilization so that new loans are not initiated
that will exceed the borrower’s credit limit. In a similar manner, JPMorgan custody
clients work with JPMorgan agents to determine lender limit guidelines. These limits and
restrictions are based on how much of their securities a lender wishes to have out on loan
at any time and to whom the securities are lent. As international loans are modified
throughout the day, DML tracks the activity and updates lender limit utilization so that
JPMorgan does not lend out more securities than the lender feels is too risky to have out
at one time.
The problem lies in the fact that throughout the day DML does not feed this
information to SLX, which is the most frequently used platform for initiating new loans.
Without knowledge of DML activities, and therefore without accurate intraday credit line
utilization figures in SLX, two different unacceptable situations may result. Utilization
may be understated in SLX, which can result in trades being booked that should not have
been because there is actually insufficient available credit line. If such a trade is
instructed to DML, it is rejected since DML does not have sufficient credit line available.
On the other hand, utilization could be overstated in SLX, which results in lost
opportunity since trade initiation is not permitted if there is insufficient available credit
line reflected.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
3
In both instances, the Database team must be contacted to manually adjust the
available credit line in SLX. This is time-consuming for both the Database staff and the
traders and in time critical situations can still result in loss of opportunity.
Therefore, the main objectives of the DML to SLX Integration Project are:
1. SLX to have accurate international utilization at start of day
2. SLX to have accurate intraday international utilization
3. traders able to view a current picture of each open loan in SLX rather than having
to toggle between SLX and DML
4. users able to define at will which value should be used to impact utilization: either
the required collateral or the DML Calculated Value (Kenney, 2006)
At a very high level this will be accomplished by creating a feedback loop for
intraday updates and an overnight batch file. The feedback loop will create intraday
messages in response to loan activity in DML. When SLX processes one of these
messages, lender/borrower utilizations will be updated and a snapshot of the loan will be
created and visible to SLX users. A key aspect in the success of this project is to make
sure that SLX calculates risk factors and therefore fractional limits in exactly the same
way that DML does so that both systems display the same utilizations and limits.
The work done on this Major Qualifying Project (MQP) aimed to assist JPMorgan
with the successful completion of this important project. Specifically, work was done to
accomplish four main goals. The first two relate to the pre-design activities that needed
be completed prior to the design and build phases of the project. Success of the pre-
design activities would contribute to the successful implementation of the feedback loop.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
4
The first goal of this MQP was to assist in mapping and matching the data fields between
the two systems to ensure that SLX receives all of the information it needs to properly
update its database. The second goal was to analyze the anticipated effects of the
feedback loop on servers and mainframes so that additional resources could be introduced
and performance would not be compromised. The third goal of this MQP was to gather
the information needed to conduct a critical path analysis that would examine ways to
expedite the delivery of the project. From this a fourth goal was established, which was
to recommend possible improvements to project management practices that could result
in shorter project durations, fewer delays, and more accurate estimates of project cost and
delivery dates for this and future projects.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
5
2. Background
2.1. JPMorgan Chase & Co.
As the 17th largest U.S. corporation, JPMorgan Chase & Co. provides banking
and financial services to customers worldwide. Its annual revenues for 2005 were 79.9
billion dollars, which is an impressive 10.6 percent increase from the previous year.
According to Fortune Magazine, JPMorgan Chase trails slightly behind Citigroup and
Bank of America, who rank 8th and 12th respectively on the Fortune 500 (Fortune, 2006).
With origins dating back to the 1700’s, JPMorgan has a rich legacy of more than
200 years of banking and financial services experience. J. P. Morgan Chase & Co., is the
result of a merger between JPMorgan and Chase Manhattan in 2000, which then acquired
BankOne more recently in 2004 (History, 2006). JPMorgan Chase & Co. uses two
brands to provide financial services to its customers. According to its 2005 Annual
Report, the Chase brand provides a full range of banking and asset management services
to small businesses, corporations, and governments. Services include but are not limited
to Consumer Banking, Home Finance, Credit Cards, Auto Finance, Education Finance,
Business Credit, Equipment Leasing, and Real Estate (2006). The JPMorgan brand has
eight lines of business. Those businesses are Investment Banking, Research, Asset
Management, Private Banking, Worldwide Securities Services, Treasury Services,
Private Equity, and Private Client Services (Home, 2006). Through continued excellence
in providing these services, JPMorgan has remained a prominent member of the financial
industry and plays a major role on the international stage. JPMorgan Chase & Co is a
component of the 30 member Dow Jones Industrial Average and has over 1.2 trillion
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
6
dollars in assets. Both of these qualities are a reflection of the company’s long term
stability. (Annual Report 2005, 2006)
2.2. JPMorgan Worldwide Securities Services
With 12.9 trillion dollars of securities under custody, the Worldwide Securities
Services (WSS) division of JPMorgan is the leader in the security services industry.
Through its consultative approach and proven experience, JPMorgan partners with clients
to assess and attend to their individual needs. JPMorgan Worldwide Securities Services
help clients to optimize efficiency, diminish risk, and increase revenue by providing a
range of services which include: American Depositary Receipts, Broker Dealer Services,
Clearance and Collateral Management, Domestic & Global Custody, Fund Services,
Global Debt Services, Investment and Liquidity Solutions, Outsourcing Solutions,
Performance Measurement, Securities Lending, Securities Settlement and Clearing, and
Specialty Trust Services. With a presence in over 90 markets, JPMorgan Worldwide
Securities Services leverages its size and capabilities to help its global client base realize
their financial goals. The business’ client base includes asset managers, pension funds,
mutual funds, broker dealers, financial institutions and hedge funds (WSS, 2006).
JPMorgan Worldwide Securities Services is led by Executive Vice President
Michael Clark. After joining JPMorgan in 1994, Mr. Clark assumed leadership roles in
the Institutional Trust Services business in 2000 and the Investor Services business in
2005, which evolved into the Worldwide Securities Services business. Under his
leadership JPMorgan Worldwide Securities Services has continuously received top
ranking in industry surveys. ISF Magazine named it “#1 Overall Cash Provider” in 2006
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
7
and JPMorgan Worldwide Securities Services received the award for “Best Investor
Services” in the Fourth Annual Waters Ranking by Waters Magazine (WSS, 2006)
2.3. Securities Lending
Securities lending, a service provided by JPMorgan Worldwide Securities
Services, is the market practice whereby securities are temporarily transferred from a
lender to a borrower. For the period of the loan, the lender is protected against default
through receipt of collateral from the borrower, usually in the form of cash, although non-
cash collateral is sometimes accepted. At the end of the loan term the securities are
returned to the lender. Table 1 below displays the acceptable forms of non-cash
collateral for securities lending.
Acceptable Collateral Cash
Government Securities Equities REPO
Commercial Paper C/D
(Yankee/Domestic/Eurodollar) Time Deposits
Medium Term Notes U.S. Agencies Master Notes Bank Notes
Funding Agreement/GIC Asset Backed Securities
Money Market Funds
Table 1: Acceptable Non-cash Collateral (Devasthali, 2006)
During the period of the loan, both the title of the lent securities and the title of
the collateral securities transfer ownership between parties. Along with the titles come
the rights associated with owning the security. The borrower now has the right to sell or
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
8
lend the security to a third party, has voting privileges, and receives the economic
benefits of owning the security, such as dividends and coupons. Commonly, the contract
between the lender and borrower stipulates that the borrower make equivalent payments
back to the lender for money received in this manner (Devasthali, 2006). Similarly,
through contractual stipulations lenders may reserve the right to recall the lent securities
in order to resume voting privileges. Lending firms such a JPMorgan act as
intermediaries within securities lending and facilitate the lender/borrower relationship.
More information on JPMorgan’s role in securities lending is given in Section 2.4 below.
Lenders are typically JPMorgan custody clients such as insurance companies,
mutual funds, endowments, and pension funds. The term custody client refers to a
situation where the client owns a particular security, but the security is in the care of
JPMorgan. In this case JPMorgan is also responsible for maintaining the securities under
custody, performing activities such as tracking stock splits, dividend payouts, stock price
fluctuations, and mergers and acquisitions. Generally, a good lender has the following
characteristics:
• a diversified portfolio with a variety of lendable securities
• a large enough securities portfolio to make lending worthwhile because there is a
small margin of return associated with securities lending
• passive trading strategies resulting in few loan recalls when compared to actively
managed portfolios.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
9
When the lender wants to sell a security in their portfolio that happens to be on
loan to a borrower, the lender must recall the loan to make the sale. Therefore, passively
managed portfolios make for better lenders (Devasthali, 2006).
Lenders benefit from securities lending because they receive incremental returns
on their securities that would otherwise lie dormant in their investment accounts. It
provides an opportunity for lenders to make money with very little risk. Lenders make
money through securities lending in a few ways. First, if the collateral received on the
loan is in the form of cash, JPMorgan invests that cash in a short-term investment. The
profit made on the investment is then split between JPMorgan, the lender, and the
borrower. If the collateral is in a non-cash form, the borrower pays a fee which is split
between JPMorgan and the lender. A lender also makes money in instances when the
lender is subject to high withholding taxes on dividends or interest, that the borrow is not
subject to. Here the security would be transferred to the borrower, who would receive the
dividend at a lower tax rate. The benefit from the difference in the tax rates is split
between the lender and the borrower. Lastly, lenders can earn money through securities
lending in the case when stock issuers offer stockholders the choice of receiving a cash
dividend or reinvesting it in additional securities at a discount price. If the lender cannot
take advantage of the discounted price because ownership of additional shares would
make their holdings larger than permitted under investment guidelines, the securities can
be sourced to a borrower who shares the economic benefit of the discounted prices with
the lender (Devasthali, 2006).
In addition to the financial benefits described above, borrowers make money
through securities lending because it allows them to cover the short position. A lot of
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
10
money can be made from an activity called selling short. In this case, the person borrows
shares of a stock through a securities lending transaction and then immediately sells
them. If the price of the stock falls the borrower re-buys the shares at the lower price and
repays the original lender to settle the loan. The difference in the selling price and the re-
purchase price is the profit for the borrower minus fees associated with the securities
lending transaction (Morris & Siegel, 1993). Similarly, Brian Kenney, a business analyst
for JPMorgan, describes that a borrower can make money in a situation where a broker
sells shares of a particular stock not yet owned, to a client to meet client demand. During
the float period (after the sale but before the shares are delivered to the client) the broker
borrows the shares through a securities lending transaction, hopefully at a lower price, to
give to the original client. Again the price difference less lending fees is the profit for the
broker (personal communication, October 24, 2006).
2.4. JPMorgan’s Role in Securities Lending
JPMorgan acts as a middleman between the lender and the borrower in securities
lending transactions. It facilitates the lender/borrower relationship and maintains all
transaction throughout the duration of the loan. As one of the top ranked custodians for
securities worldwide, JPMorgan primarily performs discretionary lending where it is the
custodian for the lendable assets. JPMorgan’s achievements in this area of business are
reflected in its receipt of several prestigious rankings, such as custodian of the year from
ICFA Magazine in 2005 and number one securities lender and global custodian by
Finance Asia (WSS, 2006). Globally, there are only eight major players in the securities
lending business. It takes a huge amount of capital to enter into the business and the
amount of money made on each loan is marginal. Therefore, to be a successful securities
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
11
lending agent one needs numerous clients each with a large portfolio of lendable
securities.
JPMorgan credits its success in the securities lending industry to its highly refined
securities lending program. JPMorgan has 390 lenders and 121 borrowers currently
involved in its program. The resulting, vast inventory of lendable securities provides
borrowers with a wide selection of securities from which to borrow. Likewise, with such
a large number of borrowers, lenders’ securities are utilized more frequently. This
maximizes lenders’ financial gain from participation in the program.
As an integral part of its securities lending program, JPMorgan works closely
with the lender to define lending limits and rules that govern how their securities can be
lent. An example is when a lender states that no more than 100 million dollars of
securities can be lent out at a time. Another example is when a lender declares that they
wish to hold back 25 percent of a certain holding from lending. These lending limits can
be broad and apply to an entire portfolio, or they can be very detailed, with specific rules
for each type of security. Furthermore, lenders have the option of restricting to whom
their securities can be lent. However, JPMorgan thoroughly screens each borrower
before allowing them to participate in the lending program. All of these lending limits
are then used by JPMorgan to match lenders to appropriate borrower loan requests
(Devasthali, 2006).
Another aspect of the JPMorgan program is that it protects its lenders and ensures
the risk associated with securities lending is marginal in three ways. First, JPMorgan
offers a broad indemnification plan that protects lenders in the case that the borrower
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
12
defaults on the loan or goes bankrupt. Second, JPMorgan requires borrowers to provide
at least 102 percent of the value of the lent security in collateral, in order to account for
fluctuations in the market values of the lent securities and the collateral. For example, in
lower risk loans, JPMorgan will require that the borrower furnish JPMorgan with
collateral that is equal to 102 percent of the value of the securities being borrowed.
According to Brian Kenney, an example of a lower risk loan is an established domestic
brokerage such as Merrill Lynch borrowing a sum of US stock and providing US dollars
as collateral. For higher risk loans, a rate of 105 percent may be charged. An example of
this would be a trader in Australia acting on behalf of an Israeli brokerage borrowing a
Russian stock and providing euros and a letter of credit from a bank in Spain as collateral
(personal communication, October 24, 2006). Lastly, the risk associated with the
investment of the cash collateral is the only risk that JPMorgan securities lending clients
are exposed to. To minimize this risk, JPMorgan utilizes conservative investment
strategies and develops strict investment guideline and restrictions with each of their
lending clients. This allows lenders to dictate how much risk they are exposed to.
The JPMorgan securities lending program offers unique options to its lenders and
borrowers that help to distinguish it from its competitors. JPMorgan custody clients who
agree to participate in the securities lending program are not charged a fee for their
participation. Bank of New York and JPMorgan are the only two securities lending
service providers that allow borrowers the option of using non-cash forms of collateral
against their loans. JP Morgan accepts U.S. Treasuries and Letters of Credit.
Additionally, JPMorgan recently introduced a second type of lending to its program
called non-custody lending, where JPMorgan is the lending agent but does not have
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
13
custody of the lendable assets. This is attractive because it opens the door for numerous
new clients to participate in securities lending with JPMorgan.
2.5. The Technology Behind JPMorgan Securities Lending
JPMorgan relies heavily on versatile and reliable technology in order to be one of
the major players in the securities lending business. To successfully facilitate securities
lending transactions, JPMorgan uses a variety of programs to manage its inventory of
securities and to initiate and maintain loans. These systems allow traders to obtain real
time data regarding all transactions and holdings in the securities lending service. This
project, along with other initiatives within JPMorgan, aims to integrate these systems in
order to optimize the uses of each and streamline securities lending processes. A better
understanding of these systems is vital in appreciating the relationships between them and
the processes currently being used by the securities lending group. A diagram detailing
the relationships between systems can be found in Appendix A.
2.5.1. Custody Management Systems – COSMIC & TITAN
JPMorgan has two systems in production that manage its inventory of lendable
securities. The inventory of international securities resides in a system called COSMIC
whereas the domestic securities in custody reside in TITAN. These repositories are
responsible for managing who owns a particular security and if it is out on loan. The
systems do not however know or care about the details of a loan. Access to COSMIC
and TITAN are indirect and are handled through front-end trading platforms.
(Devasthali, 2006)
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
14
2.5.2. Front-end Trading - SLX
The front-end trading platform used by the majority of traders at JPMorgan is
SLX. Currently, ninety-three percent of all US loans originated in SLX. SLX is one of
the newest applications that is currently in production and offers traders easier access to
data when compared to the legacy systems. SLX ensures compliance with auditing and
federal trade guidelines. It also is designed to ensure that all lenders have an equal
opportunity to lend their securities, which is required by government regulations. SLX is
based on an Oracle/UNIX platform and was written in a proprietary language called
Tenfold.
2.5.2.1. SLX Hardware
SLX production systems are hosted on multiprocessor UNIX based IBM AIX
nodes. The Global IT Services Group within JPMorgan offers tools and utilities that
monitor CPU, memory, disk utilization, and network traffic for each node. Real time
data is taken in five minute snapshots and is posted at midnight. Raw data is also
archived for future access. (GTI Hobbit, 2006)
2.5.2.2. SLX Trade Views
On the omni level (also known as the trade level) the borrower sees the entire loan
as one large package, regardless of the number of lenders associated with the loan. On
the component level (also known as the position or sequence level) traders can look at a
loan in more detail. If multiple lenders have securities lent in a particular loan the
sequence level view gives details about each lender and the value of its securities
associated with that loan. According to Brian Kenney, business analyst at JPMorgan
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
15
Worldwide Securities Services, the reason that the second level exists is because it allows
for better accounting and tracking of each loan (personal communication, October 25,
2006).
2.5.3. Legacy Trading Application - DML
At the present time, there is a legacy application that coexists with SLX and is
situated between SLX and COSMIC. This application is called DML and is used
primarily by international traders that are not based in the US or in the UK. Although
DML can perform loan activity, upon completion of this project DML will only be
responsible for loan maintenance. DML exists on a mainframe and can be accessed
through mainframe session thin clients such as RUMBA. DML is a Sunguard application
written in COBOL. DML uses DB2 and VSAM as a database platform. (Devasthali,
2006)
2.5.4. Legacy Trading Application - SLMU
SLMU is the counterpart to DML and is used for US-domestic securities lending
loan maintenance. SLMU has recently been phased out as a trading platform in favor of
SLX. A feedback loop was created between SLX and SLMU to feed SLX with real time
data related to updates in SLMU. SLMU still exists as an interface to TITAN but is no
longer used to process trades. Like DML, SLMU is a COBOL/DB2 platform application.
(Devasthali, 2006)
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
16
2.6. Project Delivery Framework for JPMorgan Worldwide Securities Services
The Project Delivery Framework (PDF) is a common and integrated project life
cycle framework using the best practices from JPMorgan Chase. The framework consists
of three tiers of 8 phases each, with required and recommended deliverables and
processes per phase. The tier classification of a project, as Tier 1, Tier 2 or Tier 3,
is based on project cost, risk and complexity. It is expected that all projects sponsored
and led by teams within JPMorgan Worldwide Securities Services will adhere to the
PDF. The figure below shows the breakdown of project phases as required for all
JPMorgan Worldwide Securities Services projects.
Figure 1: PDF Project Phases (Project Delivery Framework, 2006)
As part of the PDF process the project manager and delivery manager needs to
meet with the JPMorgan Chase Investment Council (IC) to obtain approval and funding
three times for each project. One section of these presentations to the Investment Council
is an estimation of the cost and completion date of the project. At the first meeting with
the IC, the estimate, known as the E1 Estimate, needs to have a 50% confidence interval.
At the second meeting, the E2 Estimate is due and should have a confidence interval of
30%. At the final IC meeting, the E3 Estimates are presented which require a 10%
confidence interval. The accuracy of each subsequent estimate is tighter because it is
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
17
gathered at a later date when more information about the tasks required to complete the
project and resource constraints have been identified (Project Delivery Framework,
2006).
In the Securities Lending Technology Division a tool called STMate is used to
gather each estimate. Each key team member on the project enters anticipated tasks and
durations of each task into the worksheet. All entries are compiled in a summary sheet
and a cost and project duration estimate is then generated. Sample snapshots of the tool
with information from the E1 Estimate of this project can be viewed in Appendix B.
2.7. SMLU to SLX Integration Project
Prior to the DML to SLX Integration Project, JPMorgan undertook a very similar
project to integrate SMLU and SLX. In order for SLX to become the solitary front-end
trading platform, it must reflect accurate borrower credit line utilization. SLMU is used
for domestic loan maintenance activities such as re-pricing and returns. The problem
arises from the fact that these activities impact credit line utilization, but since SLMU did
not relay information to SLX throughout the day, these impacts were not reflected in
SLX utilization figures. Therefore, the objectives of this project became:
• The reflection of accurate borrower credit line utilization in SLX related to loans
maintained on SLMU. Utilization is to be reflected accurately in SLX both
intraday and end of day and is therefore to be based on SLX loan initiation and
cancellation as well as on information provided intraday and end of day by the
SLMU to SLX feedback loop.
• The continued reflection of approximated borrower credit line utilization related
to loans maintained on DML (all international markets except the UK) based on
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
18
SLX loan initiation and cancellation as well as on information provided by the
end of day DML MIS file
In order to accomplish these objectives, the project team created a feedback loop
between SLX and SLMU. The result is that each time a loan is updated in SLMU a
message is sent to SLX with this information. Borrower credit utilization and lender
limit utilization are then updated in SLX within two minutes. This ensures that only
loans that do not exceed lender/borrower limits are initiated and that clients’ participation
in loan activity is maximized. Upon completion of the SLX to SLMU Integration
Project, the flow of information between the key systems in securities lending could be
represented in Figure 2 below. Since the SLX/DML integration project occurred after
this project, the diagram also represents the status quo of information flow prior to the
completion of the DML to SLX Integration Project.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
19
INT’LBORROWERCREDIT LINEUTILIZATION
Intraday:
New SLX Loans (+)SLX Cancels(-)
Nightly:
Recalc based on MISfiles
USBORROWERCREDIT LINEUTILIZATION
Intraday:
New SLX Loans (+)SLX Cancels(-)New SLMU / TitanLoans (+)SLMU Returns (-)SLMU / Titan COACAdj (+/-)
Nightly:
Recalc based on SODSnapshots from SLMU
BORROWERCREDIT LINEUTILIZATION
Intraday:
New Loans (+)Marks (+/-)Returns (-)COAC Adj (+/-)Cash Adj (+/-)
BORROWERCREDIT LINEUTILIZATION
Intraday:
New Loans (+)Marks (+/-)Returns (-)COAC Adj (+/-)Cash Adj (+/-)Cancels (-)
SLX SLMU DML
US FEEDBACK LOOPIntraday and Nightly
Input into SLX Intraday:New SLX Loans
SLX Cancels
Input into SLMU or DML Intraday:New Loans
MarksReturns
COAC AdjCancels
MIS File(used for nominees
other than DTC, FEDand Crest)
Nightly
Figure 2: Current Flow of Securities Lending Information (Kenney, 2006)
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
20
3. Methodology
3.1. Critical Path Analysis
As mentioned previously, this project was very important to the JPMorgan
business because it laid the foundation for several new products and services to be
developed that would bring significant amounts of new revenue to the company. Without
this project, these new products and services could not be developed. With this in mind,
there was a serious push from management to expedite the project. The critical path
method was determined to be the superior tool for analyzing ways to expedite the project.
A critical path analysis, based on a comprehensive project plan, allows managers to
prioritize activities for the effective management of project completion. Often projects
don’t finish on time because of multi-tasking and lack of prioritization. The critical path
method works to identify which tasks are critical to the on-time completion of a project.
Priority and special attention are subsequently given to these tasks because if the
completion of a critical task is delayed then the whole project is delayed. If feasible,
additional resources should be allocated to critical tasks to expedite their completion
(Internet Center for Management and Business Administration, Inc.).
A number of steps needed to be completed to gather the information that was
required to utilize the critical path method in the most effective way. The basis for this
analysis was an accurate and thorough project plan. Three key pieces of information are
required to begin constructing a project plan. The first is a listing of all tasks and
activities required to complete the project. The second is estimates regarding the time
needed to complete each task. The third is a list of predecessors for each task. A
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
21
preliminary project plan had been constructed prior to the start of this project. However,
as the project changed and developed, the original project plan needed to be modified
substantially.
A big challenge in obtaining these three types of information was that the project
was still far from completion, barely entering the design phase. At this point in the
project there were still many more questions than answers about how the project goals
would be accomplished. Each member in the project team, each with their own area of
expertise, had unique thoughts as to which tasks were necessary to realize the objectives
of the project. This led to disconnects in the overall picture of how the project would be
completed. These disconnects prohibited the creation of a cohesive project plan.
To obtain the information needed to create a cohesive project plan, the first step
was to create a document with the high level tasks and key deliverables of the project,
using information from the original project plan. The document also contained a list of
the four key objectives of the project and a note reminding resources that all tasks should
add value to fulfilling either a direct objective of the project or one of the documents
required by the Project Delivery Framework (PDF) process. This document was then
distributed to the project team for review. A meeting was held to discuss the document
where the agenda was to 1) brainstorm if all high level tasks were accounted for and 2)
collectively come to an agreement on a single strategy for how the project would be
completed. The document was then updated with information collected at the meeting.
The final version of the document can be seen in Appendix B.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
22
At that point all team members had the same vision of the strategy that would be
used to complete the project. Subsequently, the data needed to update the project plan
could be gathered. The major team members were determined and can be seen in the
table below.
Team Member Specialty Area
Brian Kenney Business Analyst
Mike Salamina SLX Development
Peter Ennis SLX Mainframe Development
Rey Generoso DML Development
Besley Nelson Quality Assurance
Table 2: Key Team Members and Roles
Individual interviews were set up with each key team member to gather information
needed for creating the project plan and performing the critical path method. The agenda
for each interview contained the following points:
1. Document all activities that a particular individual needs to complete for the
extent of the project.
2. Estimate the number of days each activity will take to complete if the owner is
exclusively devoted to the tasks completion.
3. Identify the predecessors of each activity and the earliest possible start of each.
4. Receive suggestions for improving the STMate Tool used for making estimates
regarding project cost and duration.
The High Level Task Document and each person’s STMate spreadsheet from the E1
estimates were used to guide the interviews.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
23
3.2. Project Management Process Improvements
Another objective of this project was to recommend ways in which project
management processes could be improved. The improvements were aimed at making the
processes more efficient, streamlined, simple and/or error-proof. To complete this
objective the current process for managing projects needed to be understood and
evaluated. This was partially accomplished by going through the experience of executing
many steps of the current system to develop a project plan and gather information for
estimates to present to the Investment Council. Throughout the execution of these
activities, a list of notes about inefficiencies and possible sources of error inherent in the
process was maintained. Information about current processes and ways to improve them
was also gathered in two other key ways. Informal discussions were held with the project
manager and the project team regarding their struggles and experiences with current
practices. Lastly, research was conducted using the Microsoft Project help function to
investigate proper usage of the tool and additional capabilities that were not previously
being leveraged when managing JPMorgan projects.
3.3. Data Mapping Exercises
The goal of the data mapping exercise was to examine the fields that DML has
defined in a loan snapshot and compare that to what data fields SLX needs to receive to
properly update borrower credit and lender limit utilizations. One of the challenges was
that the fields in DML do not directly map to the fields in SLX. Therefore, an in-depth
analysis needed to be conducted to examine how the data fields from DML should be
manipulated so that they are in a useable format for SLX. Once this was accomplished, to
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
24
verify that the manipulations are correct, the team will create a proof of concept. This
will entail creating a mock intraday message using the DML data fields as inputs that
outputs the same information in the SLX format.
To the surprise of the development team, the data was too complex to create a
one-to-one mapping such as saying the broker field in DML matched the broker field in
SLX. According to Danielle Rumore, calculations proved to be extremely difficult to
trace because of the extensive amount of COBOL logic that was built around each field
in DML over the past decade. (Personal Communication, November 15, 2006). Danielle
also explained that up-to-date documentation does not exist for DML, so very few people
know exactly what each field means. As part of the data mapping exercises, sample
loans were observed from the production environment. Screenshots of one such loan can
be found in Appendix D.
3.3.1. Parsing the DML Loan Master
The first encountered problem was taking the raw loan data from DML, known as
a Loan Master File, and putting it into a database that allowed it to be analyzed. The loan
master file contains over 220,000 records stored in a comma delimited file called a CSV
(Comma Separated Value) file. Due to software limitations in Microsoft Excel XP, only
the first 65,536 records could be displayed because row numbers are stored in a 16 bit
integer. Importing data into Microsoft Access using the Microsoft .txt Jet Engine was
also problematic because opening the text file for parsing caused a virtual memory
overflow on the workstation.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
25
3.3.1.1. Design
To get around the file size limitation, a program was developed that would split
the CSV into multiple files that had less than 65,000 records. A parser was designed that
would read in the CSV file one line at a time and immediately output it to a target CSV
file. A counter would be in place to keep track of how many records were written to the
target file. Once the user-defined limit was reached, the parser would send the data out to
another file. A high level design is shown in Figure 3.
Figure 3: Design of DML Loan Master Parser (Java Trademark, 2006)
3.3.1.2. Implementation
To implement this parser, the Java programming language was used.
Development occurred in an Eclipse 3.2 development environment using the Sun Java
JS2E SDK v1.5. A third party CSV parser package called com.csvreader.CsvReader was
used in the implementation because of its high benchmark ratings. One of the benefits of
using this package is that none of the data is buffered in memory, so it can process a CSV
file of any size.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
26
The software uses nested for loops to scan the records. The outer loop reads one
line at a time. The inner loop reads each column and writes it to the target output text
file. If however, the number of columns in a row did not equal 125 (a hard coded
constant equal to the expected number of columns a DML loan), the parser would output
that entire row to a different file to be analyzed later.
Command line arguments were implemented to allow the user to specify the input
file name, the output file name, and the maximum number of records that each output file
is allowed to contain. The only known bugs associated with the program are the
following:
• Using an integer value greater than Integer.MAX_VALUE for integers will cause
problems when counting the number however that value in Java 1.5 is 2^31 – 1
which is larger than the number of integers allowed in Excel anyways.
• Using long filenames or name with spaces causes problems with the file name
string. We suggest using the MS-DOS 8.3 file naming convention when
referencing file names in the command line parameters.
The compiled .class file was then placed on a shared network drive so that any
user that has the 5.0 release of the Java Runtime Environment (JRE) could parse a loan
master. The source code for the program can be found in Appendix I.
3.3.2. Aggregation and Summarization of Data Fields
Once the data was imported into Microsoft Access, there was a need to run
various queries on the data in order to find patterns or unique values. According to
requirements and needs, the following queries were developed:
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
27
• Queries on the SLX/DML broker tables
• Join DML/SLX Brokers • DML Brokers without matching SLX Records • SLX Brokers without matching DML Records
• Queries on the DML Loan Master
• Unique LOAN-TYPE values • Unique SEC-LOC values • Unique SEC-TYPE values • Filter out closed loans
These queries were written in Microsoft Access SQL Query format. The queries
can been found in Appendix J. These queries along with the Microsoft Access database
were then posted to a shared folder so that any person that wanted to run these queries
could do so.
3.4. Capacity Planning Analysis
The goal of the capacity planning analysis was to carry out a comprehensive
analysis on the SLX servers and to gather enough information to make infrastructure
decisions moving forward. The primary deliverable was a document that gives the reader
an understanding of the resources required to run SLX in a production environment and
to analyze the impact that the DML feedback loop will have on the systems. Information
was gathered from many sources and compiled together to form a complete picture of the
resources that SLX depends on. The complete document can be found Appendix I.
3.4.1. Server Specifications
The first step in the capacity planning analysis was trying to understand what the
current servers had for hardware. Through documentation and utilities provided by
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
28
JPMorgan’s Global Technology Infrastructure (GTI Hobbit, 2006). The primary focus is
on the production system. General information about production server hardware can be
found in Table 3. All SLX servers are IBM AIX servers with PowerPC processors
running a UNIX environment. The servers can also allocate a percentage of the total
number of physical processors to a specific application (in this case SLX).
APP06 APP10 DBP01 Total Number of Processors 4 2 8
Processors assigned to SLX 4 2 3.5
Processor Speed 1.7 GHz 1.7 GHz 1.7 GHz
Processor Type 64 bit 64 bit 64 bit
L2 Cache Size 1.5 MB 1.5 MB 1.5 MB
Physical Memory 7.75 GB 7.75 GB 62 GB
Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/paging00 : 6GB
SLX Disk Drives 9 11 6
Table 3: SLX Production Server Hardware (Ghelani, 2006)
3.4.2. Used Server Resources
The next step was to look at the current performance of the servers. There were
two areas of particular interest, current processor usage and allocated memory. Both of
these pieces of information could be obtained from a JPMorgan website that provided
real-time server statistics. (GTI Hobbit, 2006) For the analysis, data was gathered for the
same week (October 15th – October 21st) to maintain consistency. The three areas of
focus are processor usage, memory allocation, and storage quotas.
3.4.3. SLMU Message Statistics
The final step was to merge in statistics from SLMU and projected DML activity
onto the plot of production environment. SLMU data was gathered from the SLX
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
29
production environment query on the SLMU feedback loop. DML data was exported by
the DML system administrators and emailed to requestors.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
30
4. Results and Analysis
4.1. Critical Path Analysis
The majority of the information gathered for the critical path analysis was
obtained during the interviews with key members of the project team. Each interview
focused on identifying the tasks that a particular resource would need to complete as part
of the project and what the task dependencies are. Some team members were able to
provide task durations during these meetings. Timing was such that E2 Estimates, which
are the second round estimates that should have a 30 percent confidence interval, were
due a short time after these interviews. Therefore, these meetings served a dual purpose:
to aid in the creation of a thorough project plan and to aid in filling out the STMate tool
for the E2 Estimates. Task durations for those team members that did not provide them
in the interviews were gathered from the E2 Estimate spreadsheets once they were
completed. From these meetings, flow charts were created with the tasks for each team
member to visually highlight predecessors and task dependencies. The flow charts for
each resource can be viewed in the following figures.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
31
Figure 4: Flow Chart for Business Analyst
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
32
Figure 5: Flow Chart for Quality Assurance Team
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
33
Define DML fieldsanalysis for SLX
resources(Data Mapping)
DML Unit TestPlan/Strategy
Backout RecoveryPlan
DML Calculations15 Days
E2 Estimates3 Days
Brainstorm HighLevel Design w/
DML Team2 Days
Detailed Design(DD)
SOD File
DD Logical Unit ofWork DD Snapshot
DD Process LogicDesign for Loan
Master
DD Prototyping
Start Document 2
UpdateDocuments
1,3, 5
DD Document
Documents1. Traceability Matrix2. ECM3. Backout Recovery Plan4. Turnover Document5. Implementation & Rollout Document
DD Walkthrough
Intraday (I/O Module)-Create Module for
Snapshot(Build, Test, Peer Code
Review)
Intraday (I/O Module)-Create Module for Logic of
Unit Work(Build, Test, Peer Code
Review)
SOD-Programming &
Copybook(Build, Test, Peer
Code Review)
SOD- JCL Proc(Build, Test, Peer
Code Review)
SOD- CA7 SetupScheduling
(Build, Test, PeerCode Review)
Intraday- Modify ExistingModules for DML onlines(Build, Test, Peer Code
Review)
Intraday- Modify ExistingModules for Mark Process
for Japanese Assets(Build, Test, Peer Code
Review)
SIT Tests in DML
Start Document 4
Revise Documents1,2,3,5
Support QA andUAT Testing
Finalize AllDocuments
DML - ReyGeneroso
Figure 6: Flow Chart for DML Team
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
34
Figure 7: Flow Chart for SLX Team
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
35
Brainstorm HighLevel Design
SLX Mainframe-Peter Ennis
Data MappingCompletion (Informationregarding outputs from
DML and necessaryinputs to SLX)
Detailed DesignSOD Batch
Program Specs
Detailed DesignIntraday BatchProgram Specs
Detailed DesignIntraday OnlineProgram Specs
Build CopybookFile Layouts
Build CopybookCommon Logic
Module
Build JobsInfrastructure
CICS, MQ, VSAMor DB2
Build SODProgram
Build IntradayOnline Program
Enter Build Phase
Build End-of-intraday Batch
Program
Enter Testing
Write Unit TestStrategyWrite SIT Strategy
Execute SIT
Execute Unit Tests
Support QA
Figure 8: Flow Chart for SLX Mainframe Team
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
36
All of the information gathered from the interviews, the flow charts, and the E2
Estimates was then entered into Microsoft Project to create an initial project plan. The
next step was to identify all of the resources that would be working on the project. All
team members’ vacation schedules and bank holidays were then integrated into the
project plan to accurately portray the project schedule. The next step that we were able to
complete was to work with the delivery managers in charge of the team members to
investigate how much of each resource’s time could be allocated to working on this
particular project. The delivery manager is the supervisor to most team members and is
responsible for the thorough, timely delivery of the project. The project plan was again
updated with this information so that it would accurately represent task durations.
That was all of the information that we were able to gather during our time at
JPMorgan. At that point the project plan was handed back to the project manager to
complete the analysis. The final step in creating an accurate and thorough project plan to
use in the critical path analysis is to assign resources to each project task. After that the
project manager can use Microsoft Project’s Detailed Gantt Chart view to see the critical
path. The last step is to analyze the critical path to determine ways to accelerate the
project. The following options should be considered and analyzed:
1. Allocating extra resources to critical tasks
2. Revising task dependencies to allow for more scheduling flexibility
3. Breaking the new functionality developed by this project into individual features
that could be delivered to production separately and at different times
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
37
4.2. Project Management Process Improvements
This section aims to document and evaluate current project management
processes in place in the Securities Lending Technology division of JPMorgan. The three
areas of project management practices that were evaluated in this project are: 1) the
process for gathering information for Investment Council estimates and project plan
creation, 2) the use of Microsoft Project plans to help manage the project and 3) the
tracking of action items identified in team meetings. In addition to a discussion
surrounding each of these three areas, there is also a section about general observations
regarding the culture of project teams and how projects are executed. These observations
are important in understanding how to improve the way projects are managed to result in
faster project delivery without compromising quality.
Overall, project management within JPMorgan Securities Lending Technology
does not have a defined set of best practices. The process used to manage projects differs
between project managers and even between projects by the same project manager. This
results in confusion, poor practices being used, and time wasted by team members
learning a new system for each project. For the most part deadlines are very fluid and
team members are not held accountable when deadlines are missed. Another observation
is that there is a poor sense of team within the group. Many people seem to have tunnel
vision around their specific role and seldom offer input or suggestions regarding aspects
of the project that they are not directly responsible for. This poor communication across
team roles contributes to the lack of a clear, big picture throughout the team of how the
project will be executed and the objectives satisfied.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
38
4.2.1. Investment Council Estimates and Project Plan Creation
4.2.1.1. Current Process
Two important functions performed by JPMorgan Project Managers are to
provide project cost and duration estimates to the Investment Council for project funding
and to create a project plan. As stated in the methodology section of this paper, three
main types of information need to be collected from each project team member in order
to create a project plan. The first is a listing of all tasks and activities required to
complete the project. The second is estimates regarding the duration of each task.
Duration specifically means the amount of actual working time necessary to complete the
task. It does not mean the elapsed time between the start of the task and the completion
of the task. The third is a list of predecessors for each task. The first two types of
information are also necessary for providing estimates to the Investment Council.
The current process for gathering this information is to first send out the STMate
tool template to each team member. The team members fill in the STMate tool
worksheet for their job type (e.g. business analyst) with tasks and estimated durations for
each task. They also compile a list of assumptions they used when making the estimates.
Each person sends his/her STMate worksheet and list of assumptions to the project
manager. The project manager then takes the total estimated man hours and associated
cost from each worksheet and creates a separate overall project summary in Excel. The
STMate tool does have the capacity to compile a summary of all of the estimates if all
worksheets are in the same workbook, however, this function does not get used. Next the
team meets to discuss the assumptions each resource used when making their estimates.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
39
The purpose is to compile an overall assumptions list to be used in the presentation to the
Investment Council.
At this point, all of the necessary information for Investment Council funding
request presentations has been obtained. However, a project plan cannot be created
without a thorough understanding of task predecessors. The project manager gathers this
information in a piecemeal fashion in a few different ways. For tasks that are assigned to
a single resource, the project manager meets with that person to discuss the order in
which the tasks need to be completed. For tasks that require a group effort, the project
manager both meets with the delivery manager to discuss possible task dependencies and
also attends brainstorming meetings where the team develops a strategy for completing a
certain group of tasks. In all instances, the project manager spends large amounts of time
gathering task predecessors. This is a very inefficient process. Due to the piece meal
fashion in which task predecessors are identified and relayed to the project manager to
enter into the project plan, several weeks can pass before a thorough, accurate project
plan can be analyzed and used to drive the project.
4.2.1.2. Problems
While the current system does accomplish the task at hand, there are many
inefficiencies and potential sources for error associated with it. The majority of the
problems lie in the STMate tool. As a brief overview, as stated in the Background
Section of this paper, the tool is an Excel workbook template that has worksheets within
it for each type of team member. The types of team members are Project Manager,
Business Analyst, Quality Assurance, Technical Manager, and Hardward Systems. Each
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
40
team member fills in the worksheet designed for their role with all anticipated tasks and
effort estimates for each. The tool then calculates the total man-days each resource
requires to complete their share of the project and the associated cost. This information is
displayed in a summary sheet. Some example snapshots of the tool can be seen in
Appendix C. The main problem is that the tool is unnecessarily complex with no clear
set of use instructions. There is a very comprehensive list of all possible project tasks
that could potentially be completed in each resource spreadsheet. Many team members
feel that the list is too cumbersome to read through because projects in this division vary
so greatly that the majority of the tasks on the list do not pertain to a given project. The
goal is to remind people of any tasks they may not have remembered without the list.
People should just leave the estimated time for non-applicable tasks at zero. However,
there is not a set of instructions that explains this explicitly, which leads confusion. The
figure below illustrates this point. It is a snapshot of a technical manager’s E1 estimate.
The majority of the tasks do not apply to the DML to SLX Integration Project and
therefore, have a duration of zero days entered into the columns in blue.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
41
Figure 9: STMate Snapshot Showing Unnecessary Task List
Another way in which the tool is unnecessarily complex is that it is designed
around the functionality that allows a person to enter estimates based on the complexity
of a task. This function is applicable if a task involves repeating an activity several times,
where the level of difficulty is not the same for each iteration. A very basic example of
how this functionality works is that a chef is making an elaborate dinner and wants to
make a project schedule. As part of the dinner he will make six pies: 1 pumpkin, 3 apple,
and 2 strawberry rhubarb. He estimates that the pumpkin pie is easy to make and will
take 1.5 hours, the strawberry rhubarb pies are medium difficulty and will take 2 hours
each, and the apple pies are of a high difficulty and will take 4 hours each to make. The
tool has three sets of columns to calculate an overall estimate for making pies. In the
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
42
activity units by complexity columns he would enter the number of pies for each
complexity level. In the effort static data columns he would enter the amount of time it
takes to make a pie for each complexity level. The effort calculation columns are the
product of the two other sets of columns. The table below shows a simplistic version of
how the chef would enter this data into the STMate tool. The total estimated time to
make pies is 17.5 hours.
Activity Units By Complexity
Effort Calculation (Man Hours) Effort Static Data
Task High Medium Low High Medium Low Total High Medium LowMake Pies 3 2 1 12 4 1.5 17.5 4 2 1.5
Table 4: STMate Entry Based on Complexity of Repeated Tasks
The problem with this functionality is that in Securities Lending Technology
projects most tasks are unique and not repeated and therefore, do not require the use of
this functionality. However, the template for the tool is designed around this capability,
leaving no place to enter simple estimates for a one time task. Many times the project
manager is forced to meet with team members to discuss how they entered estimates into
the tool to make sure the way the project manager is interpreting the estimates is the way
the team member intended. The complexity of this aspect of the tool is not justifiable
when considering the rare instances in which it is useful, and the confusion and extra time
it introduces to the estimation process.
There are some other smaller issues with the tool and the way it is used. Many
cells have built-in formulas that should not be changed, but these cells are not locked. A
user can easily erase or modify these formulas without realizing it. Additionally, the
STMate tool and the way it is currently used, does nothing to ensure that the tasks one
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
43
team member believes are necessary to complete the project align with the tasks another
team member envisions. As mentioned previously, due to the wide range of peoples’
expertise, it is a challenge with many of these projects to have all project resources
collectively come to an agreement on a single strategy for how the project should be
completed. The process for gathering the data needed for estimates and project plan
creation should be designed to encourage all team members to have a common ideology
and subsequent high-level tasks in mind when they provide their list of tasks, estimated
durations, and predecessors. Lastly, the STMate tool is not designed to capture task
dependencies and predecessors. Likewise, there is no formal process for gathering this
information and passing it to the project manager, who can then incorporate it into the
project plan. The project managers receive this information informally little by little
through attending project meetings and holding discussions with team members about
project deliverables as they become due. However, a thorough and timely understanding
of task dependencies is vital to the creation of a useful project plan.
4.2.2. Use of Microsoft Project
Using all of the data discussed in the previous section, the project manager creates
a project plan using Microsoft Project. When new tasks arise through team meetings and
progress on the project, team members notify the project manager who updates the
project plan. The main use of the project plan for JPMorgan Securities Lending Project
Managers is to determine a completion date for the overall project and to keep track of all
deliverables and key tasks associated with the project. However, the tool can be
leveraged much more to effectively manage the project and schedule tasks to ensure a
speedy delivery. In the current system, rather than using the project plan to dictate when
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
44
team members should be working on specific tasks to give priority to critical tasks and
ensure proper allocation of resources, team members notify the project manager of when
they will be able to complete tasks and then the project plan is updated based on this
information. In this case it is used more as a tracking tool than a planning tool. With this
method people are not held as strictly to due dates and the project can easily be delayed.
Also in the current system the project plan is mainly used by the project manager. The
other team members do not follow the project plan and its schedule to dictate when they
work on which tasks. Therefore, team members are not aware of which activities are
critical tasks and as such do not know how to best prioritize the tasks assigned to them.
4.2.3. Action Item Tracking
Because Securities Lending projects often require a project team with widely
varied roles and backgrounds, many brainstorming sessions, working meetings, and
update meetings are held to discuss the tasks required to complete the project. New
action items are often identified at these meetings. These action items are too granular to
put into the project plan but are essential to the successful completion of key deliverables
and the overall project. Since they are not captured in the project plan, another system is
required to track action items. The current system is that after most meetings, the
meeting coordinator sends out a recap of the meeting via email. Part of this is a list of
action items and the person responsible for the completion of each. There is no
consistent method in place for checking back on the status of these action items. Also
there is no one place that houses a compiled list of action items. The project team
members are responsible for searching through their email to find action items. Since
there are so many meetings each with a separate meeting recap email and also so many
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
45
other emails received daily, it can become very time consuming for a team member to
keep track of all of the action items assigned to them. Likewise it is even more difficult
for the project manager to track the progress on all of the action items associated with a
project. This system is very inefficient and can cause team members to lose sight of
important tasks that are vital to a timely and successful project completion.
4.3. Data Mapping Exercises
4.3.1. Parsing the DML Loan Master
The loan master parser successfully parsed the comma separated file and did so
without causing strain on the workstation. Running time for this program was under 10
minutes. The program was also created with good object-oriented software design
practices in mind. Table 5 shows important code metrics for this program. These metrics
were generated by the Eclipse Metrics plug-in.
Cyclomatic Complexity
Lines of Code
Num of Levels
Number of Parameters
Number of Statements Method
8 126 5 1 58 main(java.lang.String[])
2 19 2 2 6 setOutputFileName(java
.lang.String, int)
2 18 2 1 6 setErrorFileName(java.l
ang.String)
Table 5: Code Metrics for DMLLoanMasterParser.java
Because this program is contained all within one class, there are very few object
dependencies. An analysis was performed on the class and package level with the use of
the CAP (Code Analysis Plugin) plugin for Eclipse. The results for the package are
shown in Figure 10, Figure 11, Figure 12, & Figure 13.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
46
Figure 10: Class Dependency Diagram for DMLLoanMasterParser
Figure 11: Instability Metric for DMLLoanMasterParser (green dot)
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
47
Figure 12: Efferent Couplings for DMLLoanMasterParser
Figure 13: Class and Package Statistics for DMLLoanMasterParser
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
48
4.3.2. Aggregation and Summarization of Data Fields
Due to inefficiencies of Microsoft Access, the queries written in Appendix G took
a very long time to run. It was decided to run the queries once and then export the results
to a Microsoft Excel workbook, as the data in the referenced tables did not change after
the initial import.
4.4. Capacity Planning Analysis
The findings of the capacity planning analysis on the servers can be broken down
into three categories: processor, memory, and storage. Each category looks at current
bottlenecks and anticipated bottlenecks. The complete analysis can be found in the
Capacity Planning document at Appendix I.
4.4.1. Processor Analysis
Data was extracted from the servers showing how busy the processors were at a
given point in time during the day. Data points were recorded every five minutes. To
paint a picture of what a typical day looked like, the median value was taken (out of the
seven days) for each of the five minute intervals. The median was used instead of the
mean because weekend downtime would have skewed the plot for certain time intervals.
Each of the three production servers was then plotted over a 24 hour horizontal axis.
This plot can be found in Figure 14.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
49
Median production server CPU usage (10/15/06 - 10/21/06)
0
0.2
0.4
0.6
0.8
1
0:001:00
2:003:00
4:005:00
6:007:00
8:009:00
10:0011:00
12:0013:00
14:0015:00
16:0017:00
18:0019:00
20:0021:00
22:0023:00
0:00
Time of day (Eastern Time)
CPU
usa
ge (a
s a
ratio
of 1
00)
% APP06 CPUUtilization
% DBP01 CPUUtilization
% APP10 CPUUtilization
6 per. Mov. Avg. (%APP06 CPUUtilization)
6 per. Mov. Avg. (%DBP01 CPUUtilization)
6 per. Mov. Avg. (%APP10 CPUUtilization)
Figure 14: Production Server CPU Usage
As illustrated in Figure 14, CPU usage has two areas of peaks. The first peak
starts at 9:00 PM and runs until 3:00 AM. The second peak starts at 3:00 PM and ends
around 7:00 PM. These peaks can be attributed to SLX batch processes that are running
during those periods. Surprisingly the server load is under 50% for most of the trading
day. As a result, processing time for messages is under the 2 minute service level
agreement.
4.4.2. Memory Analysis
Data was extracted from the servers showing how much physical and virtual
memory was used during the day. In the case of memory, a bigger time interval was
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
50
observed because memory changes less rapidly. Memory allocations for servers APP06,
APP10, and DBP01 can be seen in Figure 15, Figure 16, & Figure 17 respectively.
Figure 15: Memory Usage on APP06 (GTI Hobbit, 2006)
Figure 16: Memory Usage on APP10 (GTI Hobbit, 2006)
Figure 17: Memory Usage on DBP01 (GTI Hobbit, 2006)
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
51
Figure 15, Figure 16, & Figure 17 show memory usage on each of the servers
over a 48 day span. The plot in blue represents physical memory usage. The plot in red
represents swap space via hard disk paging files. One interesting observation is that
APP06 in Figure 6 appears to run out of physical memory by the 4th day of uptime. The
server then falls over to allocating and using swap space.
4.4.3. Storage Capacity
Data was extracted from the servers that showed how full each hard drive volume
was on all production servers. The goal was to find volumes that were at risk of
exceeding their quota after the implementation of the DML feedback loop. Two hard
drive volumes were identified of being at risk (volumes that are more than 80% full).
These volumes are shown in Figure 18 & Figure 19.
Figure 18: /local/app/a_slx on APP06 frequently uses more than 90% of quota
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
52
Figure 19: /app/a_slxftp on the JIP NAS is near capacity
4.4.4. SLMU Message Statistics
The next step was to observe intraday processing of SLMU messages as they hit
the SLX systems. Information was extracted from SLX queries which gave an hourly
breakdown of the number of SLMU messages that were processed during that hour. In
addition to the number of messages, the average time it took to process a single message
during that hour period was returned from the query. This information was plotted on top
of the CPU graph found in Figure 14. This allowed for correlations to be made between
how busy the processors were and how long it took to process an SLMU message. This
graph can be found in Figure 20. According to the graph, the processing time for an
SLMU message is more than two minutes between 3:30PM and 6:00PM daily. This
means that SLX is not meeting the service level agreement (SLA) that guarantees
processing time within two minutes.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
53
Average SLMU messages & processing time (from 10/15/06 - 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)
6 per. Mov. Avg.(% DBP01 CPUUtilization)
Figure 20: SLMU Messages in SLX
The infrastructure team has commented on this and remarked that there are no
memory limitations on TSJIPUAPP06 server. Piyush Ghelani, system administrator of
JPMorgan’s servers said, “the illusion of being out of memory comes from the fact that
users do not count file cache usage feature of the Unix kernel. Actual memory usage is
4.125 GB.” (Personal Communication, November 22, 2006) Starting after week 44 the
system administrators also limited the size of the file cache so that it would resize itself to
prevent the swap space from being used. Even with this change in place, the processing
times for SLMU messages remain unchanged as shown in Figure 21. This graph was
created using the same methods that were used in creating the October performance
assessment. The only difference is that for November statistics, the week of November
12th was used.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
54
October Performance vs November Performance
0
3000
6000
9000
12000
15000
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
Projected DMLMessages
Oct. SLMUMessages
Nov. SLMUMessages
Processing SLA of2 minutes
Oct. SLMUprocessing time
Nov. SLMUprocessing time
Figure 21: Comparing SLMU Performance Between Oct and Nov
4.4.5. DML Projections
The final step was to take intraday trading information from DML and to project
when and how many messages would be sent to SLX during the day. The method of
plotting the information was to look at the sum of the loan and component updates during
the hour period and to treat each as a single message. This closely matches what SLMU
does when sending messages to SLX. The plot was then shown on the same graph as the
SLMU messages and CPU usage. In doing this, one could see if the SLMU message
peaks matched up with DML peaks. This combined graph can be seen below in Figure
22.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
55
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Figure 22: SLMU and Anticipated DML Messages over a Trading Day
In looking at the graph, DML is anticipated to have a spike in messages between
11:00AM and 1:00PM. As a result, it is predicted that the processing time will be greater
than two minutes during this time frame and thus, the two minute SLA will not be met
between 11:00AM and 1:00PM.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
56
5. Recommendations
5.1. Project Management Process Improvements
Based on our evaluation of current project management practices and our analysis
of the areas in need of improvement, we have developed a project management process
that encompasses many of the responsibilities of JPMorgan project managers. In
particular, the process involves the three areas that have been focused on throughout this
paper: gathering project cost and duration estimates for the Investment Council, creating
a project plan and using it to manage the project, and tracking action items. The
flowchart below illustrates the process that we have developed. We recommend that
JPMorgan adopts this process. The main recommendations that summarize this process
are:
1. Use the Estimation Workbook in place of STMate to gather information for
estimates and project plan creation.
2. Take full advantage of Microsoft Project and the critical path method to schedule
tasks and better manage projects.
3. Use the Action Item Tracking Sheet to track the status of action items and project
risks.
4. Hold weekly status meetings to discuss progress and encourage communication
across project roles.
The following sections explain this process in greater detail. We also recommend that
the firm conduct a more in-depth study to identify project management best practices and
develop a common process to be used by all project managers for all projects in
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
57
JPMorgan Securities Lending. This effort could possibly be spearheaded by another
team on WPI students.
Figure 23: Project Management Process
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
58
5.1.1. The Estimation Workbook
We have developed a new tool called the Estimation Workbook that we
recommend be used in place of STMate to gather estimates for Investment Council
project funding requests. The Estimation Workbook is a simple Excel workbook that will
capture all of the necessary information from project team members, while eliminating
the unnecessary complexity and confusion associated with the STMate tool. To ensure
success in using this tool, the project team should follow the process that we have
developed for its use.
The project manager should first hold a meeting with the entire project team to
make a list of the project milestones and deliverables and their dependencies. Each team
member should then work offline to develop a list of all foreseen tasks that will be
required to accomplish the project milestones and deliverables. The team should then
come together again to compile a comprehensive list of all project tasks and task
dependencies. The team should reference the list of tasks in STMate to make sure they
account for all tasks that will need to be completed. For clarification purposes, a
milestone is a significant accomplishment or intermediate project goal. A deliverable is
any tangible outcome that is produced by the project. These can be documents, plans,
computer systems, buildings etc. Lastly, a task is any piece of work that is undertaken
that adds value to the creation of a project deliverable or fulfillment of a project
objective.
Once this list is compiled and the team agrees on it, the project manager should
set up the Estimation Workbook. A new Estimation Workbook will need to be created
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
59
for each project and should be based on the example that we have created for this project.
In every Estimation Workbook there will be tabs named Instructions, Assumptions,
Summary Sheet by Phase, and Summary Sheet by Role. There will also be a tab for each
key project team member labeled with their role in the project. Each team member tab
and the summary sheet by role tab should have the comprehensive list of project tasks
along the left edge. The milestones and deliverables should be in bold with the respective
subtasks listed directly beneath each. After the project manager sets up the Estimation
Workbook with all worksheets, columns and formulas, he/she should post it on a shared
drive and notify the project team that it is ready for their inputs. The figures below are
snapshots of the sample Estimation Workbook that we have developed.
Figure 24: Estimation Workbook- Instructions
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
60
This instruction sheet is a summary of the process that we are recommending. It
gives instructions for the project manager on how to create the Estimation Workbook and
also for team members on how to fill it in with their estimates. This instruction worksheet
should be copied into every new Estimation Workbook.
Figure 25: Estimation Workbook- Team Member Worksheet
Each member of the project team that is required to provide estimates should have
their own tab in the Estimation Workbook. The tab should be labeled with the team
member’s role such as BA for Business Analyst or QA for Quality Assurance. The tabs
should all be identical and should look like the figure above. They will each have the
comprehensive list of all project tasks in the left hand column and a list of the project
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
61
phases along the top edge of the worksheet. Once the project manager sends out
notification that the workbook is posted on the shared drive, each team member should
open the workbook and find the tab labeled with their role. They should then fill in their
estimates in man-days for how long they think each task will take them in each phase of
the project. These are estimates of pure effort not of elapsed duration. If a task does not
apply to a particular team member, then that team member should leave that task blank or
enter a zero. Each team member should also click on the assumption tab and review the
list of assumptions started by the project manager. If a team member wants to modify or
add to the list of assumptions, he/she should do so, making sure to note their name and
the date next to any changes. Team members should not enter any information into the
summary sheets. These sheets are linked to the team member estimate tabs and will
aggregate automatically. Once a team member finishes entering their information into
the Estimation Workbook, he/she should save the changes on the shared drive and notify
the project manager.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
62
Figure 26: Estimation Workbook- Summary Sheet by Phase
The Summary Sheet by Phase provides the project manager with total time and
total cost broken down for each phase of the project. The time totals are shown in days,
hours, and months. Once all team members have filled in their worksheet, the project
manager should pull the file off the shared drive. Since this summary sheet is linked to
the data entered in the team member estimate sheets, the project manager just needs to fill
in the cost table with the cost per day of each team member. Once that is complete, the
Estimation Workbook will show the total time and cost for each phase of the project.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
63
Figure 27: Estimation Workbook- Summary Sheet by Role
The summary sheet by role is also linked to the team member estimation sheets
and contains the same comprehensive list of project tasks. When the project manager
sets up the Estimation Workbook, he/she should enter the task predecessors in the right
hand column of this tab. After all team members enter their estimates, this summary
sheet shows the total time and cost for each team member along the bottom edge of the
worksheet. The time totals are again given in days, hours, and months. Also, along the
right side of the sheet, next to the task predecessors are total man-days for each task
across all team members. Between the two summary sheets, the project manager should
have total duration and cost estimates broken down in a variety of ways that will be
useful in presenting to the Investment Council.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
64
We realize that this process requires a substantial amount of time upfront to
compile the comprehensive list of project tasks and dependencies; however, we feel the
benefits of gathering estimates using the Estimation Workbook outweigh this extra
upfront time. Some of the most important benefits are:
• Clearer understanding of project tasks by all team members because of the
comprehensive task list on all estimate tabs
• Cost and duration totals broken down in several ways
• Eliminates the need for the project manager to manually merge the estimates from
each team member into one estimate
• Simpler design and linked worksheets reduces the time all team members have to
spend providing estimates
5.1.2. Microsoft Project and the Critical Path Method
The next step in our proposed project management process is to create a project
plan in Microsoft Project using the comprehensive task list, task predecessors, and task
effort estimates from the Estimation Workbook. The project plan should be set up in
such a way that it has two layers. The high level layer should include the network of
project milestones and deliverables and their dependencies. Below this, there should be a
more granular layer with the network of tasks needed to complete each deliverable and
reach each milestone. A diagram illustrating this point is shown below. The pink boxes
show the network of project deliverables. The blue circles represent the network of tasks
required to accomplish each deliverable.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
65
Figure 28: Diagram of Two Layer Project
The project manager and the delivery manager should also work together to
compile a list of all of the resources that will be working on the project. Then they
should look up each person’s vacation schedule and determine what percentage of each
person’s time will be allocated to that particular project. Once all of this information is
entered into Microsoft Project, it will automatically create a project schedule. The
project manager should use the Detail Gantt Chart view to see the critical path for the
project. All critical tasks are highlighted in red and should be given priority. To ensure
that resources are not over-allocated and that the schedule depicts realistic dates, the
project manager should re-schedule concurrent tasks assigned to each resource to give
priority to critical tasks and subsequently fill in non-critical tasks in the open areas of the
schedule.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
66
We recommend that project managers take a Microsoft Project training class to
ensure they are leveraging as many functions of the program as possible. The main goal
is to have the most accurate project plan as possible by accounting for such things as
vacation time and resource allocation to other projects. With a solid project plan as the
foundation, the project manager can ensure on-time delivery by giving priority to critical
tasks. It may even be possible to expedite the completion of a project by analyzing
options such as allocating extra resources to critical tasks or breaking the project into key
deliverables that could be delivered separately and at different times.
Once the project manager develops a sound project plan, the project team should
use it to prioritize and manage their work loads and due dates. To help with this, we
recommend that the project manager print and post relevant sections of the project plan
on a highly visible wall. Using the project plan in this way ensures that it is used as a
planning and tracking tool, rather than simply a tracking tool as it was with the current
system. The foreseen benefits of this process are as follows:
• Substantially less time required by a project manager to create the project plan
because information is taken directly from the Estimation Workbook
• More accurate sense of project end date
• More likely on-time delivery because all team member recognize and give
priority to critical tasks
• Fewer project delays due to increased accountability and awareness of due dates
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
67
5.1.3. Action Item Tracking
We recommend that the JPMorgan Securities Lending Technology division begin
using the following process and tool to track the status of action items that arise through
formal and informal team meetings. As team members identify new action items, they
should send an email to the project manager with the following information:
• A description of the action item
• The milestone or deliverable from the project plan that the action item will help to
fulfill (the team should refer to the project plan for list of tasks and deliverables)
• The open date and due date of the action item
• The team member responsible for the action item
This should also be extended to tracking risks associated with the project. When
new risks are identified, the same information from above should be sent in an email to
the project manager. Additionally, the email should contain a brief risk mitigation
strategy.
The project manager should then use the Action Item Tracking Sheet that we have
developed to input this information for each action item and risk. The tool is an excel
workbook with three worksheets. The first is an instruction sheet for the project manager
and can be seen below.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
68
Figure 29: Action Item Tracking Sheet Instructions
The other two sheets are where the actual action items and risks are input and
tracked. The project manager should input all information from the team emails in the
appropriate sheet. These two sheets can be seen in the figures below. The initial status
of all action items and risk items should be open. As progress and status changes are
made on action items, notification should be sent to the project manager through email.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
69
Figure 30: Action Item Tracking Sheet for Action Items
Figure 31: Action Item Tracking Sheet for Risk Items
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
70
Using the Action Item Tracking Sheet in the manner stated above will result in
many benefits for Securities Lending Technology project teams at JPMorgan. Action
items that are essential to completing project deliverables will all be housed in one
location. This will eliminate the need to search through numerous emails and the risk
that the team loses sight of important action items. The new capability to track the status
and outcome of action items and risks ensures a more thorough completion of action
items and mitigation of risk. This will result in higher quality project deliverables.
Lastly, the due dates and updates at weekly status meetings will increase accountability
of team members and help to keep the project on schedule.
5.1.4. Weekly Status Meetings
The other key part of our process is that the delivery manager holds weekly status
meetings. At these meetings the team should have a discussion, led by the delivery
manager, to bring the team up to date on the status of project milestones, to determine a
strategy for how to reach the next project milestones, and to identify next steps.
Additionally, we recommend that the team discuss any open action items with pressing
due dates or that are pending closure. During this time, each team member should give a
brief report regarding the status of the items they own. If a team member believes an
action item or risk status should be changed to closed, the team should come to a
consensus that sufficient work has been done to close the item. After these meetings the
project manager should update the Action Item Tracking Sheet, which should be housed
in PKS for all team members to review frequently. The project manager should also
update the project plan, if applicable, after the weekly status meetings. He/she should
then print and post a new version of the project schedule on the wall.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
71
These weekly status meetings will ensure that all team members are aware of the
current status of the project, even regarding areas where they are not directly involved.
We believe this will boost communication and the flow of ideas between team members
and also bring any problems or roadblocks to the forefront sooner. We also feel that the
weekly status meetings will increase accountability and adherence to deadlines because
team members will be required to report on their progress each week.
5.2. IT Equipment Upgrades for SLX
Based on the capacity planning research and analysis, we have developed a list of
recommendations to aid JPMorgan with the successful implementation of the DML to
SLX feedback loop into production, such that server performance is not compromised.
Our recommendations are as follows:
• JPMorgan should use the capacity planning document, found in Appendix I, to
estimate server loads and to upgrade hardware on the servers in order to meet
future demand.
• A member of the infrastructure team should be contacted and brought on board
for the rest of this project. This individual should be responsible for maintaining
and updating the capacity planning document.
• The infrastructure team should perform further analysis if necessary on quality
assurance, disaster recovery, and development environments to ensure that all
SLX environments have adequate resources.
• Storage space on two volumes should be increased to meet future demands from
the DML feedback loop. These storage volumes are:
o /dev/slxp1-slxhom1lv
o jipnas1:/vol/jipnas1_app0/tss_slx_prd1
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
72
• Add more processors in order to meet the two minute service level agreement for
intraday message processing time. Without these upgrades the service level
agreement will not be met between 11:00AM and 1:00PM and 3:30PM and
6:00PM.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
73
Works Cited
[Annual Report 2005] (2006). Annual Report 2005. Retrieved November 30, 2006 from http://files.shareholder.com/downloads/ONE/62656512x0x34666/D25C186E-7D76-4BA3-AC27-445C45622401/ar05_complete.pdf
[Fortune] (2006). Fortune500 2006 Our Annual Ranking of America’s Largest Corporations. Retrieved October 23, 2006. from http://money.cnn.com/magazines/fortune/fortune500/full_list/
Ghelani, Piyush (2006). WSS Unix webdoc 1.1 for JPMChase. Retrieved November 30 2006 from http://dcfdoc.gis.chase.com
[GTI Hobbit] (2006). Hobbit Monitor 4.1.2: WSS Unix Health Status. Retrieved November 30, 2006 from http://wssunixmonitor.gis.chase.com/
[History] (2006). The History of Our Firm. Retrieved November 30, 2006 from http://www.jpmorganchase.com/cm/cs?pagename=Chase/Href&urlname=jpmc/about/history
[Home] (2006). JPMorgan Home Page. Retrieved November 20, 2006 from http://www.jpmorgan.com
Internet Center for Management and Business Administration, Inc. (2006). CPM- Critical Path Method. Retreived November 16, 2006 from http://www.netmba.com/operations/project/cpm/.
[Java Trademark] Java and the Java logo are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. or other countries.
JPMorgan Chase & Co. (2006). Securities Lending Overview. New York, NY: Amar Devasthali.
JPMorgan Chase & Co. (2006). Comprehensive Business Requirements Document: DML to SLX Integration. New York, NY: Brian Kenney.
Morris, K.M. & Siegel, A.M. (1993). The Wall Street Journal Guide to Understanding Money & Investing. New York: Lightbulb Press, Inc.
[Project Delivery Framwork] (2006). WSS Project Delivery Framework (PDF). Retrieved November 29, 2006, from http://tss-cb-cto-tech.jpmorganchase.com/dept/WSS/permits.asp
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
74
[WSS] (2006). About Worldwide Securities Services. Retrieved October 23, 2006, from http://www.jpmorgan.com/cm/ContentServer?c=TS_Content&pagename=jpmorgan%2Fts%2FTS_Content%2FGeneral&cid=1114735359705.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
75
Appendix A: Securities Lending Technology Systems Relationships
Diagram courtesy of JPMorgan (Devasthali, 2006)
Mar
ket
Inte
rfac
es
Cle
arin
g &
Settle
ment
CO
SM
IC•
Int’l
Cus
tody
Infr
ast
ruct
ure
Dom
estic
Wo
rld
Len
d•
Non-
cust
ody
[Sun
Gar
d]
Inte
rnat
iona
lR
epo
rtin
g
WSS and Other Corp
User Interfaces Core Processing
SL
X•
Tra
de I
nitia
tion
EQ
UIL
EN
DD
alla
s M
essa
ge
Hu
b(W
MQ
I)
DM
L•
Inte
rnat
ion
al
Glo
bal
On
e•
UK
Dom
est
ic
[SunG
ard
]
Vie
ws
Po
rtfo
lio
Rep
ort
ing
TIT
AN
•U
.S.
Cus
tody
SL
X
Mai
nfr
ame
•M
essa
ge
Rou
ting
& T
ran
sfo
rma
tion
Inte
rnet
Bro
wse
r
3rd
Par
ty
Cu
sto
dia
ns
Cre
stU
DT
’s
UD
T’s
Rec
onci
liatio
ns,
Acc
rual
s, a
nd o
the
r bu
sine
ss f
unc
tions
SE
I•
B1
U.S
. C
usto
dy
Ass
et
Man
age
men
t
SL
AM
/ C
TE
•C
ash
Colla
tera
l M
gm
t
Blo
ckad
e•
LC
Col
late
ral
Allo
c/R
epo
rtin
g
SL
MU
•D
ome
stic
Lo
anet
Pir
um
DT
CF
ED
Eu
roC
lear
FT
ID
Glo
bal
-V
iew
s
MIS
Tra
de
In
itia
tion
Loa
n M
ain
ten
ance
E-m
ail
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
76
Appendix B: High Level Task Document for Project Plan Creation
We would like to brainstorm if these are all of the major activities and begin to determine the predecessors of each. This will facilitate the E2 estimate process and subsequently a critical path analysis to examine ways to expedite the project. Also we would like to map out and agree upon the high level design for the completion of the project. Keep in mind: **Every high level task should add value to fulfilling either a required document for the
PDF process or one of the direct objectives for the project. ** The objectives are as follows: Objective 1 – SLX to have accurate international utilization at start of day Objective 2 – SLX to have accurate intraday international utilization Objective 3 – Enable traders to view a current picture of each open loan in SLX rather than having to toggle between SLX and DML. Objective 4 – The ability to define at will which value should be used to impact utilization: either the required collateral or the DML Calculated Value. Major Milestones by Phase Requirements Phase
Tasks Documentation SLX/DML High Level Pre-design Work
• DML Calculation of limits* • Verify that DML does not
calculate limits differently for different loan types
• DML Functions that affect utilization and date sensitivity around those functions*
Detailed Requirements Document
Data Mapping Exercise • Spreadsheet mapping from DML loan
master to the loan snapshot file layout • Analysis of borrower relationships in DML vs. SLX, valid values, valid codes, etc.
• Proof of Concept Prototype in SLX to
Initiate Requirements Traceability Matrix
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
77
validate data mapping (To include prototyping the End User Transactions Summary and Detailed Screens)
Initiate Test Strategy/Plan E2 Estimates Performance Capacity Planning
• Examine CPU and RAM usage (currently executing this task)
• Analyze Current Performance pertaining to SLMU feedback loop to identify inaccurate/inefficient processes and determine whether we can change existing processes due to poor performance that will ameliorate the use of the future SLX/DML Feedback Loop (If changes arise because of this task, there will be some significant Testing Effort required to test changes to the current process)
• Work with GTI to determine next steps for solving anticipated problems
• Track existing issues facing production and infrastructure teams on all SLX environments
Architecture Permit to Build
Business Case Design Phase
Tasks Documentation
Update Test Strategy
High Level Design document (author Brian with input from developers) [This doc will build upon the requirements document to
show the users a picture of our High Level Design Concept]
**Implementation of a common I/O
(input/output) Module in DML [Dependent upon previous tasks listed in asterisks: DML calculation of limits and DML
functions that affect…]
Detailed Design/System Specifications Doc
Messaging Task Finalize Release Schedule End User Transactions Architecture Step 3
Data Transactions Perimeter Infrastructure Risk Assessment Form
Design in SLX Update Requirements Traceability Matrix Design in DML E3 Estimate
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
78
Design in SLX MF Detailed Design Walkthrough
Design SOD File Build/Unit Test
Tasks Documentation Source Code Build/Unit Test DML Update Requirements Traceability Matrix Source Code Build/Unit Test SLX Revise Requirements
Source Code Build/Unit Test SLX MF Update End User Documentation Peer Code Review Initiate Application/System Documentation
SIT Tests Technology Recovery Action Plan (TRAP) Deployment to QA Update SQA Checklist
Update Tollgate Checklist Testing
Documentation Update End User Documentation
Finalize Requirements Traceability Matrix Revise Requirements
Initiate Production Readiness Review Update SQA Checklist
Update Tollgate Checklist
QA Testing Tasks SIT Volume Performance Test
Test Setup (getting access to programs, etc.) Write Test Scripts Test Script Review
Execute Error Handling Testing Request GTI Resources
Execute Performance Testing Assisted by GTI Resources Execute Functional Requirements Testing
Analyze Test Results Regression Testing
UAT Testing Test Setup Plan UAT
Execute UAT for DML
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
79
Execute UAT for SLX Execute UAT for SLX MF
Analyze Test Results Training We need to brainstorm ideas for a training strategy. Implementation
Tasks Documentation
Code Deployment DML Finalize Operations Turnover Documentation
Code Deployment SLX Update Tollgate Checklist Code Deployment SLX MF Finalize Ender User Documentation
Implement Changes to Batch Schedule Finalize Application/System Documentation
Finalize ECM Calendar Entry Finalize SLA Finalize Implementation Plan
• Create Implementation Tasks • Create ECMS Record (Allow 10-15
days for Review as part of Risk for the project)
Finalize Production Readiness Review
Update SQA Checklist Project Closure
Tasks Documentation Post Production Support (2 week warranty
period) Project Closure Document
Post Implementation Survey Customer Satisfaction Survey Finalize SQA Checklist Finalize Tollgate Checklist
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
80
Appendix C: Snapshots of the STMate Tool
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
81
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
82
Appendix D: Screenshots of the same loan in DML and SLX
Omni view of a loan in DML
Component view of a loan in DML
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
83
Trade level view of a loan in SLX Production
Position level view of a loan in SLX Production
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
84
Appendix E: DML Loan Master Parser Source Code
/** * DML Loan Master Parser * * Written by: Michael Kristan ([email protected]) * Securities Lending Technology * JPMorgan Worldwide Securities Services * * Copyright 2006 JPMorgan Chase & Co. All Rights Reserved. * Internal use only. Not for distribution or use outside * of the firm without permission from the author or the firm. * * * About this program: * This program parses a DML Loan Master file (in .csv format) * and splits it up into smaller parts to allow for easier * processing by the Microsoft Office suite of applications. * * Command line arguments (in order): * Maximum number of records that can appear in an output file * Input .csv file name * Output .csv file name * * Example usage: * java.exe DMLLoanMasterParser 10000 C:\input.csv C:\output.csv * * In the example above, the file input.csv is read and split up * into groups of 10,000 records. Each group is output into a csv file * containing the name C:\output * * Most java virtual machines will require the full package name * of the class when being executed. In that event, use * com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser after java.exe * * Notes: * The output file is modified by the software by appending * a number to the file name. Example, the first 10000 records * are stored in output1.csv, the second 10000 records are in * output2.csv. * * There is an interesting situation where some of the DML * loan records have extra commas in them which causes errors when * trying to import into Microsoft Access. Records that do not match * the expected number of columns (defined as 125 in constant * numberOfCSVColumns) are instead written into a file with an ERR * suffix (in this example it would be outputERR.csv). * * It has been found that using MS-DOS 8.3 file naming in the * arguments helps when trying to load files that have long file names * or spaces. Example, instead of passing H:\SLX Integration\Loan Master.csv * use H:\SLXINT~1\LOANMA~1.CSV * * This program has been tested on the Sun Java 5.0 SDK & JRE * platforms. The runtime environment for this version of java can * be found at http://java.sun.com This software makes use of a * non-standard java package called csvreader. The class files for * this package can be freely obtained at http://www.csvreader.com at * the time of this writing. * * To compile this file, you need the Java SDK. The command to * compile is javac.exe DMLLoanMasterParser.java. To run this program, * you only need DMLLoanMasterParser.class and do not need the SDK. * */ package com.jpmorgan.wss.seclend.dmlslx;
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
85
// External classes that DMLLoanMasterParser depends on import com.csvreader.CsvReader; // Used to read CSV files import com.csvreader.CsvWriter; // Used to create CSV files /** * DMLLoanMasterParser * * This program only uses one class, DMLLoanMasterParser. All * functions in this program are static and are accessed via the * void main function. * * @author Michael Kristan */ public class DMLLoanMasterParser { /** * Main function * * This function does the parsing of the CSV file as described above * in the program description. There is no error checking to ensure * that the first parameter is an integer or that the paths in the other * parameters are valid. * * @param args A string array that contains the expected arguments as described above * @throws Exception in the event that the wrong number of command arguments are passed */ public static void main(String[] args) throws Exception { // Validate that we have enough command line arguments if(args.length != 3) throw new Exception("Not enough command line arguments"); // Extract argument parameters into integers and character strings final int maxRecordsPerFile = Integer.parseInt(args[0]); String paramInputFileName = args[1].trim(); String paramOutputFileName = args[2].trim(); // Output general information to the console screen System.out.println("DML Loan Master Parser\nJPMorgan Securities Lending\n"); System.out.println("Number of records per file : " + maxRecordsPerFile); System.out.println("Input file name : " + paramInputFileName); System.out.println("Writing contents of input file to .CSV files, please be patient..."); // Declare the CSV Reader and the CSV Writer for errors CsvReader myReader = new CsvReader(paramInputFileName); CsvWriter errWriter = new CsvWriter(setErrorFileName(paramOutputFileName)); // Setup parameters for size of documents and output file name int fileNumberCounter = 0; // Used for output file names String outputFileName = setOutputFileName(paramOutputFileName, fileNumberCounter); int errNumberCounter = 0; // Used for looking at statistics // Setup parameters for the definition of the CSV file final int numberOfCSVColumns = 125; // Boolean based on the myReader.readRecord() output boolean recordsLeftToRead = true; // Create a loop that will run until the end of the input CSV file is reached while(recordsLeftToRead == true) { // Declare new output file stream and print name to console screen CsvWriter myWriter = new CsvWriter(outputFileName);
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
86
System.out.println("Output file name : " + outputFileName); // Loop populate an output CSV file until the maxRecordsPerFile // upper bound is reached. Then break out and have the parent // loop create a new CSV file for(int index = 0; index < maxRecordsPerFile; index++) { // Read a record and implement early exit recordsLeftToRead = myReader.readRecord(); if(recordsLeftToRead == false) break; // If we run into a situation where the the number of columns are not // equal to the defined constant (numberOfCSVColumns), output that row // to a different file so that the regular CSV file doesn't break when // importing into Microsoft Access if(myReader.getColumnCount() != numberOfCSVColumns) { for(int colCounter = 0; colCounter < myReader.getColumnCount(); colCounter++) { errWriter.write(myReader.get(colCounter)); //System.out.print(myReader.get(colCounter)); } System.out.println("\tOutput Record number : " + index + ", COL_NUM_ERROR: Columns = " + myReader.getColumnCount() + " Skipping..."); errWriter.endRecord(); // Adds a CR and LF to the text file // Because we're not sending our output in this if statement to myWriter, // there is no need to update the counter that measure the numbers of // records being written to that file. Therefore as a hack, we need to decrement // the index counter by one because it will get incremented automatically by the // for loop. index--; // Increment the error number counter. errNumberCounter++; } else { // Iterate through the number of columns and output line for(int colCounter = 0; colCounter < myReader.getColumnCount(); colCounter++) { myWriter.write(myReader.get(colCounter)); //System.out.print(myReader.get(colCounter)); } System.out.println("\tOutput Record number : " + index); myWriter.endRecord(); // Adds a CR and LF to the text file } } // Close the output file stream myWriter.flush(); myWriter.close(); // Increment the file number counter and set the new output file name fileNumberCounter++; outputFileName = setOutputFileName(paramOutputFileName, fileNumberCounter); } // Close the file reader and error writer myReader.close(); errWriter.flush(); errWriter.close(); System.out.println("\nFile creation complete!\n\tNumber of errors : " + errNumberCounter); }
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
87
/** * setOutputFileName * * This method parses the output filename and addes a numeric counter to the end of the * file name but before the .csv file suffix. * * @param fileName The name of the file that needs to be parsed * @param fileNumberCounter The integer that needs to be concatentated * @return A file name with the fileNumberCounter appended to the file prior to the .csv suffix */ public static String setOutputFileName(String fileName, int fileNumberCounter) { // Parse the string so that we only have the portion before the .csv suffix // then add the filenumber and then add the .csv suffix if(fileName.indexOf(".") > 0) fileName = fileName.substring(0, fileName.indexOf(".")); //System.out.println(fileName); return fileName + fileNumberCounter + ".csv"; } /** * setErrorFileName * * This method parses the output filename and addes a numeric counter to the end of the * file name but before the .csv file suffix. * * @param fileName The name of the file that needs to be parsed * @return A file name with the characters 'ERR' (without quotes) appended to the file * prior to the .csv suffix */ public static String setErrorFileName(String fileName) { // Parse the string fileName so that we only have the portion before the .csv // suffix. Then add the letters ERR and then add the .csv suffix if(fileName.indexOf(".") > 0) fileName = fileName.substring(0, fileName.indexOf(".")); return fileName + "ERR.csv"; } }
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
88
Appendix F: DML Loan Master Parser JUnit Test Case
package com.jpmorgan.wss.seclend.dmlslx; import junit.framework.TestCase; /** * Test for DMLLoanMasterParser helper methods * * @author Michael Kristan */ public class TestDMLLoanMasterParser extends TestCase { /** * Test method for {@link com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser#setOutputFileName(java.lang.String, int)}. */ public void testSetOutputFileName() { String inputStringA = "H:\\output.csv"; String outputStringA1 = "H:\\output1.csv"; String outputStringA2 = "H:\\output2.csv"; String outputStringA_99 = "H:\\output-99.csv"; assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 1), outputStringA1); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 2), outputStringA2); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, -99), outputStringA_99); inputStringA = "H:\\output"; assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 1), outputStringA1); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, 2), outputStringA2); assertEquals(DMLLoanMasterParser.setOutputFileName(inputStringA, -99), outputStringA_99); } /** * Test method for {@link com.jpmorgan.wss.seclend.dmlslx.DMLLoanMasterParser#setErrorFileName(java.lang.String)}. */ public void testSetErrorFileName() { String inputStringA = "H:\\output.csv"; String outputStringA1 = "H:\\outputERR.csv"; assertEquals(DMLLoanMasterParser.setErrorFileName(inputStringA), outputStringA1); inputStringA = "H:\\output"; assertEquals(DMLLoanMasterParser.setErrorFileName(inputStringA), outputStringA1); } }
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
89
Appendix G: Microsoft Access SQL Queries
JOIN DML/SLX BROKERS SELECT DISTINCTROW [_internal_Remove Quotes from SLX Borrowers].[MR Short Name], [_internal_Remove Quotes from SLX Borrowers].[Relationship Short Name], [_internal_Remove Quotes from SLX Borrowers].DBRELID, [_internal_Remove Quotes from SLX Borrowers].[BG Short Name], [_internal_DMLBrokers Query].[ACCT-LINE], [_internal_DMLBrokers Query].[ACCT-PREFIX], [_internal_DMLBrokers Query].[ACCT-NO], [_internal_DMLBrokers Query].[SETTL-CODE], [_internal_DMLBrokers Query].[COLAT-CODE], [_internal_DMLBrokers Query].[ACCT-TYPE], [_internal_DMLBrokers Query].[_ACCT-NO] AS DMLID FROM [_internal_DMLBrokers Query] INNER JOIN [_internal_Remove Quotes from SLX Borrowers] ON [_internal_DMLBrokers Query].[_ACCT-NO]=[_internal_Remove Quotes from SLX Borrowers].DMLID;
DML BROKERS WITHOUT MATCHING SLX RECORDS SELECT [_internal_DMLBrokers Query].[ACCT-LINE], [_internal_DMLBrokers Query].[ACCT-PREFIX], [_internal_DMLBrokers Query].[ACCT-NO], [_internal_DMLBrokers Query].[_ACCT-NO], [_internal_DMLBrokers Query].[SHORT-NAME], [_internal_DMLBrokers Query].[SETTL-CODE], [_internal_DMLBrokers Query].[COLAT-CODE], [_internal_DMLBrokers Query].[ACCT-TYPE] FROM [_internal_DMLBrokers Query] LEFT JOIN [_internal_Remove Quotes from SLX Borrowers] ON [_internal_DMLBrokers Query].[ACCT-NO] = [_internal_Remove Quotes from SLX Borrowers].DMLID WHERE ((([_internal_Remove Quotes from SLX Borrowers].DMLID) Is Null));
SLX BROKERS WITHOUT MATCHING DML RECORDS SELECT [_internal_Remove Quotes from SLX Borrowers].DMLID, [_internal_Remove Quotes from SLX Borrowers].[MR Short Name], [_internal_Remove Quotes from SLX Borrowers].[Relationship Short Name], [_internal_Remove Quotes from SLX Borrowers].DBRELID, [_internal_Remove Quotes from SLX Borrowers].[BG Short Name] FROM [_internal_Remove Quotes from SLX Borrowers] LEFT JOIN [_internal_DMLBrokers Query] ON [_internal_Remove Quotes from SLX Borrowers].DMLID = [_internal_DMLBrokers Query].[ACCT-NO] WHERE ((([_internal_DMLBrokers Query].[ACCT-NO]) Is Null));
UNIQUE LOAN-TYPE VALUES SELECT DISTINCT [Loan Master].[LOAN-TYPE] FROM [Loan Master];
UNIQUE SEC-LOC VALUES SELECT DISTINCT [Loan Master].[SEC-LOC] FROM [Loan Master];
UNIQUE SEC-TYPE VALUES SELECT DISTINCT [Loan Master].[SEC-TYPE] FROM [Loan Master];
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
90
Appendix H: Invitation to Final Presentation
You are cordially invited to attend a presentation hosted by Megan Slonski and Michael Kristan of Worcester Polytechnic Institute where they will discuss their contributions to the SLX-DML Integration initiative within WSS Securities Lending Technology. Date: December 14th, 2006 Time: 3:30PM – 4:30PM Location: 28th Floor Conference Center, 1 Chase Manhattan Plaza, New York, NY Dial-In: 1.866.870.8212 Participant Passcode: 66370645 Details As part of our college graduation requirements we must complete a project that ties together our four years of study. This project is referred to as a Major Qualifying Project (MQP) at the university. For our MQP, we spent time at JPMorgan Chase assisting with the SLX-DML Integration project, which was initiated by the Securities Lending Technology team within JPMorgan Worldwide Securities Services. Our contributions include a critical path analysis, recommendations for improvements to project management processes, data field mapping, and server capacity planning. This project was advised by Professors Arthur Gerstenfeld and Michael Ciaraldi of WPI. Individuals who can not physically attend are welcome to connect via the phone number listed above. Time will be reserved for questions and answers. About Worcester Polytechnic Institute (WPI) Founded in 1865 in Worcester, Mass., WPI was one of the nation’s first engineering and technology universities. WPI's 18 academic departments offer more than 50 undergraduate and graduate degree programs in science, engineering, technology, management, the social sciences, and the humanities and arts. WPI's world-class faculty work with students in a number of cutting-edge research areas, leading to breakthroughs and innovations in such fields as biotechnology, fuel cells, information security, materials processing, and nanotechnology. Students also have the opportunity to make a difference in communities and organizations around the world through the university's innovative Global Perspective Program. There are more than 20 WPI project centers throughout North America and Central America, Africa, Australia, Asia, and Europe. High level project description In 2004, a decision was made to adopt SLX as a strategic front end single trading platform, replacing the trading activity performed on the three lending platforms (SLMU, DML, Global One), leaving them in place only for loan maintenance activities. During 2005, SLX development primarily focused on enabling a viable front end for US domestic trading with the delivery of an SLMU to SLX feedback loop. The "feedback loop" design approach adopted for the US Front-End will be leveraged now for the creation of a DML to SLX loop as the first in a series of efforts that will be undertaken to enable SLX as a viable International trading front end.
This document is not intended for public distribution. Permission must be obtained from JPMorgan Chase & Co. to disclose this document.
91
Appendix I: Review of SLX Server Performance
The document in Appendix I provides JPMorgan Chase with a thorough analysis
of all server systems related to the DML to SLX Feedback Loop. This document is
currently in use by members of the project team as well as by the members of JPMorgan
Global Technology & Infrastructure (GTI) group.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Copyright © 2006 J. P. Morgan Chase Bank, N.A. All rights reserved.
This document contains proprietary and confidential information of JPMorgan. Reproduction, disclosure, or use of any portion of this document without specific written authorization from JPMorgan is strictly prohibited. This restriction applies to the information on every page of the document. Contents may be disclosed only to authorized JPMorgan employees and consultants for the purpose of performing their job responsibilities.
Worldwide Securities Services
Securities Lending DML to SLX Integration
Review of SLX Server Performance
Version 0.5
Project Code: 71707
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 2 of 31
Table of Contents Introduction ................................................................................................................................................. 4
Purpose of Document............................................................................................................................. 4 Definitions ............................................................................................................................................... 4
Glossary of Terms & Acronyms ....................................................................................................4 SLX Environments ........................................................................................................................6
Hardware Specifications ......................................................................................................................... 6 Production server specifications ...................................................................................................6 Development server specifications ...............................................................................................7 Disaster Recovery server specifications.......................................................................................7 Quality Assurance server specifications.......................................................................................7 Cross reference of SLX hard disk volumes to UNIX mounted file systems..................................8
Analysis...................................................................................................................................................... 11 Current Database Usage...................................................................................................................... 11
CPU load on production servers.................................................................................................11 Possible zombie processes ........................................................................................................12 Memory allocation on production servers...................................................................................12 Disk space on production servers...............................................................................................14
SLX New Loan Volumes....................................................................................................................... 15 SLMU Volumes..................................................................................................................................... 17
Number of SLMU messages.......................................................................................................17 Processing times for messages..................................................................................................19 Table sizes..................................................................................................................................19 Anticipated volume increases .....................................................................................................20
DML Volumes ....................................................................................................................................... 20 Current DML intra-day load ........................................................................................................20 Table Sizes .................................................................................................................................22 Anticipated volume increases .....................................................................................................22 Requirements around timing of DML batch processes...............................................................23
Recommendations and Next Steps ......................................................................................................... 23 Recommendations................................................................................................................................ 23 Next Steps ............................................................................................................................................ 24
Reference Documents .............................................................................................................................. 25 Document Control..................................................................................................................................... 25
Document History ................................................................................................................................. 25
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 3 of 31
Document Storage................................................................................................................................ 25 Appendices................................................................................................................................................ 26
PowerPoint Slides For Capacity Planning............................................................................................ 26
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 4 of 31
INTRODUCTION PURPOSE OF DOCUMENT The purpose of this document is to give the reader a comprehensive understanding of the resources required to run SLX in a production environment and to analyze the impact that the DML feedback loop will have on the systems. Information has been gathered from many sources and compiled together to form a complete picture of the resources that SLX depends on. The goal is to provide enough information in order to make infrastructure decisions moving forward.
DEFINITIONS
Glossary of Terms & Acronyms
Term Description APD06 APD06 is short for the server name TSJIPUAPD06. APD06 is the development server for
all SLX environments (including production, DR, & QA).
APP06 APP06 is short for the server name TSJIPUAPP06. APP06 is the main SLX processing server for SLXP.
APP10 APP10 is short for the server name TSJIPUAPP10. APP10 is the user SLX server for SLXP. JPMorgan traders interface through this server.
APR05 APR05 is short for the server name TSMCCUAPR05. APR05 is a disaster recovery SLX server that is not located in the same area as the SLX production servers.
APR06 APR06 is short for the server name TSMCCUAPR06. APR06 is a disaster recovery SLX server that is not located in the same area as the SLX production servers.
APQ05 APQ05 is short for the server name TSMCCUAPQ05. APQ05 is a quality assurance SLX server.
APQ06 APQ06 is short for the server name TSMCCUAPQ06. APQ06 is a quality assurance SLX server.
Autoborrow Autoborrow is an intelligent, fully automated way for borrowers and lenders to transact US and non-US equity and fixed income securities. Customizable schedules and rules automate the process between the borrower and lender, making the trade execution simple. Autoborrow is part of the Equilend trading system.
Cache Cache is a collection of data duplicating original values stored elsewhere or computed earlier, where the original data is expensive (usually in terms of access time) to fetch or compute relative to reading the cache. Once the data is stored in the cache, future use can be made by accessing the cached copy rather than refetching or recomputing the original data, so that the average access time is lower.
CPU The CPU (also known as the Central Processing Unit or the processor) is the main processor on the server. The current CPU load measures how busy the server is.
DBP01 DBP01 is short for the server name TSJIPUDBP01. DBP01 is the production server that hosts the SLX Oracle Database.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 5 of 31
Term Description DML DML (also known as SLS) is the Securities Lending platform currently used to support
trading and loan maintenance activities for international assets.
DR DR stands for disaster recovery.
Equilend Formed in 2001 by its ten original investors, EquiLend delivers standardized, centralized and global securities lending solutions to the equity and fixed income markets, while providing operational efficiencies across organizations. The EquiLend platform is designed to process equity and fixed income securities lending transactions on a global basis.
GB GB stands for gigabyte. It is a measure of size on computers. A gigabyte is equal to 1,024 MB (megabytes). Gigabytes are frequently used when measuring sizes of hard disks and sometimes used to measure memory.
GHz GHz stands for gigahertz. It is a measure of frequency and is used with computer when discussing speed. The higher the frequency, the faster the processing speed of a computer. 1 GHz is equal to 10-9 seconds.
MB MB stands for megabyte. It is a measure of size on computers. A megabyte is equal to 1,024 KB (kilobytes). Megabytes are frequently used to measure size of memory.
NAS Network-attached storage (NAS) is hard disk storage that is set up with its own network address rather than being attached to the department computer that is serving applications to a network's workstation users. By removing storage access and its management from the department server, both application programming and files can be served faster because they are not competing for the same processor resources. The network-attached storage device is attached to a local area network (typically, an Ethernet network) and assigned an IP address. File requests are mapped by the main server to the NAS file server.
Network-attached storage consists of hard disk storage, including multi-disk RAID systems, and software for configuring and mapping file locations to the network-attached device. Network-attached storage can be a step toward and included as part of a more sophisticated storage system known as a storage area network (SAN).
SAN In computing, a storage area network (SAN) is a network designed to attach computer storage devices such as disk array controllers and tape libraries to servers. As of 2005, SANs are common in enterprise storage.
SLA SLA stands for Service Level Agreement. An SLA is a contract or agreement that guarantees a certain level of performance or turnaround time usually between two applications or systems.
SLMU SLMU (also known as Securities Lending MenU) was the Securities Lending platform previously used by US domestic traders. SLMU is only used for loan maintenance now and has been integrated into SLX by a feedback loop.
SLX SLX (also known as Securities Lending eXpress) is a front-end trading platform that is strategically envisioned to replace all trading activity currently performed on any of the lending platforms with those platforms then being retained for loan maintenance type activities only.
SLXP SLXP stands for the production environment of SLX.
Swap/Page file A swap file (or swap space or, in Windows NT, a pagefile) is a space on a hard disk used as the virtual memory extension of a computer's real memory (RAM). Having a swap file
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 6 of 31
Term Description allows your computer's operating system to pretend that you have more RAM than you actually do. The least recently used files in RAM can be "swapped out" to your hard disk until they are needed later so that new files can be "swapped in" to RAM. In larger operating systems (such as IBM's OS/390), the units that are moved are called pages and the swapping is called paging.
One advantage of a swap file is that it can be organized as a single contiguous space so that fewer I/O operations are required to read or write a complete file. In general, Windows and Unix-based operating systems provide a default swap file of a certain size that the user or a system administrator can usually change.
TB TB stands for terabyte. It is a measure of size on computers. A terabyte is equal to 1,024 GB (gigabytes). Terabytes are frequently used when measuring sizes of large hard disk arrays that store log files, databases, or other data-intensive files.
QA QA stands for quality assurance. Development efforts on SLX are first tested on the QA servers and environments before being phased into production.
SLX Environments SLX Type Environment Names Related Server(s) Production SLXP TSJIPUAPP06, TSJIPUAPP10, TSJIPUDBP01
Development SLXG, SLXD, SLXI, SLXT TSJIPUAPD06
Disaster Recovery (DR) SLXP TSMCCUAPR05, TSMCCUAPR06
Quality Assurance (QA) SLXQ, SLXR TSMCCUAPQ05, TSMCCUAPQ06
HARDWARE SPECIFICATIONS Production server specifications APP06 APP10 DBP01
Total Number of Processors 4 2 8
Processors assigned to SLX 4 2 3.5
Processor Speed 1.7 GHz 1.7 GHz 1.7 GHz
Processor Type 64 bit 64 bit 64 bit
L2 Cache Size 1.5 MB 1.5 MB 1.5 MB
Physical Memory 7.75 GB 7.75 GB 62 GB
Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB /dev/hd6 : 4 GB
/dev/paging00 : 6GB
SLX Disk Drives 9 11 6
Network Interfaces en8 : 10.75.107.54 en8 : 10.75.107.62 en25 : 10.75.107.22
% Allocation to SLX 100% 100% 14.29%
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 7 of 31
APP06 APP10 DBP01
Monthly Operating Cost $1,218.00 $1,218.00 $2,493.52
Development server specifications APD06
Total Number of Processors 2
Processor Speed 563.3 MHz
Processor Type 64 bit
L2 Cache Size 1.5 MB
Physical Memory 7.75 GB
Known Paging Files /dev/hd6 : 6GB
SLX Disk Drives 10
Network Interfaces en2 : 10.75.85.30
% Allocation to SLX 100%
Monthly Operating Cost $1,447.77
Disaster Recovery server specifications APR05 APR06
Total Number of Processors 2 1
Processor Speed 1.7 GHz 1.7 GHz
Processor Type 64 bit 64 bit
L2 Cache Size 1.5 MB 1.5 MB
Physical Memory 7.5 GB 7.5 GB
Known Paging Files /dev/hd6 : 4GB /dev/hd6 : 4GB
SLX Disk Drives 11 9
Network Interfaces en11 : 10.31.77.67 en11 : 10.31.77.75
en12 : 10.31.87.98
% Allocation to SLX 100% 100%
Monthly Operating Cost $1,218.00 $1,218.00
Quality Assurance server specifications APQ05 APQ06
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 8 of 31
APQ05 APQ06 Total Number of Processors 2 3
Processor Speed 1.7 GHz 1.7 GHz
Processor Type 64 bit 64 bit
L2 Cache Size 1.5 MB 1.5 MB
Physical Memory 7.5 GB 7.5 GB
Known Paging Files /dev/hd6 : 4 GB /dev/hd6 : 4 GB
SLX Disk Drives 11 10
Network Interfaces en3 : 10.31.77.65
en4 : 10.31.87.73
en3 : 10.31.77.73
en4 : 10.31.87.93
% Allocation to SLX 100% 100%
Monthly Operating Cost $1,218.00 $1,218.00
Cross reference of SLX hard disk volumes to UNIX mounted file systems Device path Mounted File System QuotaAPP06 /dev/slxp1-sftwrelv /software 640 MB
/dev/slxp1-local1lv /local 192 MB
/dev/slxp1-slxhomlv /local/app/a_slx 8.06 GB
/dev/slxp1-mqmhomlv /local/app/mqm 64 MB
/dev/slxp1-mqdatalv /local/data/mqm 64 MB
/dev/slxp1-mqmloglv /local/data/mqm/log 320 MB
/dev/slxp1-mqmerrlv /local/data/mqm/errors 64 MB
/dev/slxp1-gtamsglv /local/app/gtamsg 64 MB
/dev/slxp1-oraclelv /local/SLX/oracle 2.97 GB
APP10 /dev/slxp2-local1lv /local 192 MB
/dev/slxp2-slxhomlv /local/app/a_slx 8.06 GB
/dev/slxp2-mqmhomlv /local/app/mqm 64 MB
/dev/slxp2-mqdatalv /local/data/mqm 64 MB
/dev/slxp2-mqmloglv /local/data/mqm/log 320 MB
/dev/slxp2-mqmerrlv /local/data/mqm/errors 64 MB
/dev/slxp2-gramsglv /local/app/gtamsg 64 MB
/dev/slxp2-oraclelv /local/SLX/oracle 2.97 GB
/dev/slxp2-homedirlv /local/home 320 MB
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 9 of 31
Device path Mounted File System Quota/dev/slxp2-ibmhttlv /local/IBMHTTPD.8 512 MB
/dev/slxp2-slxloglv /local/app/a_slx/logs 64 MB
DBP01 /dev/dbp01-slxora1 /local/SLX/oracle 8 GB
/dev/dbp01-slxolog /local/SLX/logs/oracle 2.92 GB
/dev/dbp01-slxoaud /local/SLX/oraaudit01 1 GB
/dev/dbp01-slxwora /local/SLX/work/oradba 1 GB
/dev/dbp01-SLXobck /local/SLX/orabackup01 54.5 GB
/dev/dbp01-SLXoftp /local/SLX/oraftp 32 GB
APR05 /dev/slxp1-sftwrelv /software 2.5 GB
/dev/slxp1-local1lv /local 1.5 GB
/dev/slxp1-slxhom1lv /local/app/a_slx 8 GB
/dev/slxp1-mqmhomlv /local/app/mqm 512 MB
/dev/slxp1-mqdatalv /local/data/mqm 512 MB
/dev/slxp1-mqmerrlv /local/data/mqm/errors 512 MB
/dev/slxp1-gtamsglv /local/app/gtamsg 512 MB
/dev/slxp1-oraclelv /local/SLX/oracle 16 GB
/dev/slxp1-mqmloglv /local/data/mqm/log 2.5 GB
/dev/slxp1-ibmhttlv /local/IBMHTTPD.8 512 MB
/dev/slxp1-slxloglv /local/app/a_slx/logs 512 MB
APR06 /dev/slxp2-local1lv /local 1.5 GB
/dev/slxp2-slxhomlv /local/app/a_slx 8 GB
/dev/slxp2-mqmhomlv /local/app/mqm 512 MB
/dev/slxp2-mqdatalv /local/data/mqm 512 MB
/dev/slxp2-gtamsglv /local/app/gtamsg 512 MB
/dev/slxp2-mqmerrlv /local/data/mqm/errors 512 MB
/dev/slxp2-oraclelv /local/SLX/oracle 16 GB
/dev/slxp2-mqmloglv /local/data/mqm/log 2.5 GB
/dev/slxp2-ibmhttlv /local/IBMHTTPD.8 512 MB
APQ05 /dev/slxq1-locallv1 /local 128 MB
/dev/slxq1-slxhomlv /local/app/a_slx 12.4 GB
/dev/slxq1-slxftplv /local/app/a_slxftp 4 GB
/dev/slxq1-usrhomlv /local/home 320 MB
/dev/slxq1-oraclelv /local/SLX/oracle 2 GB
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 10 of 31
Device path Mounted File System Quota/dev/slxq1-mqdatalv /local/data/mqm 64 MB
/dev/slxq1-mqmerrlv /local/data/mqm/errors 64 MB
/dev/slxq1-mqmhomlv /local/app/mqm 64 MB
/dev/slxq1-gtamsglv /local/app/gtamsg 64 MB
/dev/slxq1-sftwrelv /software 640 MB
/dev/slxq1-ibmhttlv /local/IBMHTTPD.8 512 MB
APQ06 /dev/slxq2-slxhomlv /local 160 MB
/dev/slxq2-usrhomlv /local/app/a_slx 12.4 GB
/dev/slxq2-usrhomlv /local/home 320 MB
/dev/slxq2-oraclelv /local/SLX/oracle 2 GB
/dev/slxq2-mqdatalv /local/data/mqm 64 MB
/dev/slxq2-mqmerrlv /local/data/mqm/errors 64 MB
/dev/slxq2-mqmhomlv /local/app/mqm 64 MB
/dev/slxq2-gtamsglv /local/app/gtamsg 64 MB
/dev/slxq2-sftwrelv /software 640 MB
/dev/slxq2-ibmhttlv /local/IBMHTTPD.8 512 MB
APD06 /dev/slxd1-slxora /local/slx/oracle 2 GB
/dev/slxd1-gtamsg /local/app/gtamsg 32 MB
/dev/slxd1-mqdatalv /local/data/mqm 256 MB
/dev/slxd1-mqlogslv /local/data/mqm/logs 256 MB
/dev/slxd1-mqhomelv /local/app/mqm 256 MB
/dev/slxd1-slxhomlv /local/app/a_slx 27.7 GB
/dev/slxd1-slxoralv /local/SLX/oracle 3 GB
/dev/slxd1-slxftplv /local/app/a_slxftp 4 GB
/dev/slxd1-appesmlv /local/app/esm 256 MB
Shared Network Attached Storage (NAS) jipnas1:/vol/jipnas1_app0/tss_slx_prd1 /app/a_slxftp 4 GB
jipnas1:/vol/jipnas1_app0/tss_slx_prd2 /app/a_slx/slxp/SLXP/export 4 GB
jipnas1:/vol/jipnas1_app0/tss_slx_prd3 /app/a_slx/slxp/SLXP/out 4 GB
mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX /app/a_slxftp 4 GB
mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX2 /app/a_slx/slxq/SLXQ/export 2 GB
mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX3 /app/a_slx/slxq/SLXQ/out 2 GB
mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX4 /app/a_slx/slxr/SLXR/export 2 GB
mccfiler10-infra:/vol/mccfiler10_shared/TSS_QA_SLX5 /app/a_slx/slxr/SLXR/out 2 GB
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 11 of 31
ANALYSIS CURRENT DATABASE USAGE
In order to do a successful capacity study on SLX, attention needs to be paid to the current state of server resources. In the following sections, an analysis was made on CPU, memory, and disk space on SLX production servers. DR, QA, and development were not included in this study because performance isn’t a priority on those servers. They also share resources with other systems that affect performance on SLX.
CPU load on production servers In Figure 1, the production server’s CPU usage was plotted on a 24 hour horizontal axis. The
values for each point were calculated by observing all 7 days of CPU values for the specified time interval and taking the median value. The median was used instead of the mean because weekend downtime would have skewed the plot for certain time intervals. A six term moving average was plotted on top of the data points to emphasize trends.
Median production server CPU usage (10/15/06 - 10/21/06)
0
0.2
0.4
0.6
0.8
1
0:001:00
2:003:00
4:005:00
6:007:00
8:009:00
10:0011:00
12:0013:00
14:0015:00
16:0017:00
18:0019:00
20:0021:00
22:0023:00
0:00
Time of day (Eastern Time)
CPU
usa
ge (a
s a
ratio
of 1
00)
% APP06 CPUUtilization
% DBP01 CPUUtilization
% APP10 CPUUtilization
6 per. Mov. Avg. (%APP06 CPUUtilization)
6 per. Mov. Avg. (%DBP01 CPUUtilization)
6 per. Mov. Avg. (%APP10 CPUUtilization)
Figure 1 - Production server CPU usage
There are two areas of peaks. The first peak period starts around 9:00 PM and ends at 3:00 AM. The second peak period starts 3:00 PM and ends around 7:00 PM. Surprisingly, the server load is under 50% for most of the US trading day.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 12 of 31
Possible zombie processes On APP06 there were 4 instances of the wait process that were allocating 16% of total CPU
usage (16% per process split over 4 processors). These processes were initiated on February 11th, 2006. Further investigation is needed to determine why these processes are consuming so many resources while the other wait processes are not. Table 1 shows the top processes on the server APP06.
[ps]USER PID %CPU %MEM SZ RSS TTY STAT STIME TIME COMMANDroot 12294 16.1 0 48 44 - A 11-Feb-06 261490:39 waitroot 20490 16.1 0 48 44 - A 11-Feb-06 260903:48 waitroot 16392 16 0 48 44 - A 11-Feb-06 260090:25 waitroot 8196 16 0 48 44 - A 11-Feb-06 259121:55 waita_slx 2150436 3.7 4 233760 233680 - A 12:49:15 170:55:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 2224178 3.1 2 123380 123328 - A 12:49:15 146:00:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1511664 3 5 296064 295984 - A 12:49:16 137:38:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1990868 2.4 4 257536 257456 - A 12:49:15 110:45:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 282740 2.3 3 163272 163192 - A 12:49:16 104:31:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 2093268 2.1 2 115688 115608 - A 12:49:15 99:22:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 532526 2 2 114376 114300 - A 12:49:16 94:39:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 815148 2 2 140768 140688 - A 12:49:16 90:49:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1810524 1.5 2 135460 135380 - A 12:49:16 68:59:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 528512 0.7 2 95060 94952 - A 12:49:15 31:59:00 /app/a_slx/slxp/SLXP_I/bin/tftskmst fcenginea_slx 1228806 0 0 9128 9144 - A 12:49:14 1:33 /app/a_slx/slxp/SLXP_I/bin/tfsrvmgr SLXP_I_TSJIPUAPP06
Table 1 - Top processes on APP06
Memory allocation on production servers Figure 2, Figure 3, & Figure 4 show memory usage on each of the servers over a 48 day span.
The plot in blue represents physical memory usage. The plot in red represents swap space via hard disk paging files. One interesting observation is that APP06 in Figure 6 appears to run out of physical memory by the 4th day of uptime. The server then falls over to allocating and using swap space.
Figure 2 - Memory usage on APP06
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 13 of 31
Figure 3 - Memory usage on APP10
Figure 4 - Memory usage on DBP01
The infrastructure team has commented on this and remarked that there are no memory limitation or constrains on TSJIPUAPP06 server. The illusion of being out of memory comes from the fact that users do not count file cache usage feature of the Unix kernel. Actual memory usage is 4.125 GB. Starting after week 44, the system administrators also limited the size of the file cache so that it would resize itself to prevent the swap space from being used. A performance assessment was taken after week 44 to reassess the processing times. Data was collected during the week of November 12th in the same way that it was done in Figure 9. Figure 5 shows that processing times have not significantly changed after the adjustment was implemented.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 14 of 31
October Performance vs November Performance
0
3000
6000
9000
12000
15000
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
Projected DMLMessages
Oct. SLMUMessages
Nov. SLMUMessages
Processing SLA of2 minutes
Oct. SLMUprocessing time
Nov. SLMUprocessing time
Figure 5 - Comparing October performance to November
Disk space on production servers Given the number of mounted file systems on each server, only graphs showing disk utilization greater than 80% for an SLX drive are shown. Despite the fact that these drives are mostly full, there is no evidence that short term growth will cause these volumes to become completely full.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 15 of 31
Figure 6 - /local/app/a_slx on APP06 frequently uses more than 90% of quota
Figure 7 - /app/a_slxftp on the JIP NAS is near capacity
SLX NEW LOAN VOLUMES Table 2 shows intraday statistics on how many new loans are booked during a trading day. This data was compiled from loan statistics captured between January 2006 and June 2006. There are two different types of loans. The first are manual loans which are hand initiated by JPMorgan securities lending traders via the SLXP client. Autoborrow loans are automatically initiated loans that are processed through Equilend. A majority of loans are autoborrow loans which are handled automatically by SLX servers and do not require user intervention.
Based on the gathered data, the average manual loan is worth $8,305,877.38 and the value of an autoborrow loan $927,666.75. These numbers are strictly averages and are used for estimating financial loss due to SLX downtime.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 16 of 31
Loans Booked - Average Hourly Breakdown From: 1-Jan-06 To: 4-Jun-06
Manual Loans Autoborrow loans Total
Time - ET # of Loans # of Loans # of Loans
Hourly Avg. 32 69 100 Daily Avg. 757 1,653 2,410
0:00 3 0 3 1:00 5 1 6 2:00 17 1 18 3:00 16 1 16 4:00 14 25 40 5:00 43 57 100 6:00 87 107 194 7:00 72 337 410 8:00 66 316 382 9:00 90 201 291
10:00 78 244 322 11:00 66 147 214 12:00 84 130 215 13:00 75 78 152 14:00 9 0 9 15:00 3 0 3 16:00 1 0 1 17:00 2 2 4 18:00 3 1 4 19:00 3 1 5 20:00 3 1 4 21:00 5 0 5 22:00 5 1 6 23:00 7 0 7
Table 2 - New loans booked in SLX
This data was then plotted against the recorded CPU usage. This plot can be found in Figure 8. An analysis of processor usage during the day shows that new loan bookings correlates very closely to CPU load. Servers APP06 and DBP01 closely follow new loan booking between the hours of 3:00AM and 2:00PM. Fortunately, because the servers have excess CPU capacity between 3:00AM and 3:00PM (nearly 60% of CPU time is idle time) there is plenty of room for growth without risking a performance hit. While it has not been proven that high processor activity on APP06 and DPB01 impact the SLX client response time that traders encounter (they communicate via the less stressed APP10), loan bookings may take slightly longer to process in the background.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 17 of 31
New loans compared to CPU usage
0
0.2
0.4
0.6
0.8
1
0:001:00
2:003:00
4:005:00
6:007:00
8:009:00
10:0011:00
12:0013:00
14:0015:00
16:0017:00
18:0019:00
20:0021:00
22:0023:00
0:00
Time of day (Eastern Time)
CPU
usa
ge (a
s a
ratio
of 1
00)
1
10
100
1,000
10,000
100,000
Num
ber o
f new
loan
s
% APP06 CPUUtilization
% DBP01 CPUUtilization
% APP10 CPUUtilization
New loan bookings
6 per. Mov. Avg. (%APP06 CPUUtilization)
6 per. Mov. Avg. (%DBP01 CPUUtilization)
6 per. Mov. Avg. (%APP10 CPUUtilization)
Figure 8 - New intraday loans in SLX
SLMU VOLUMES
Number of SLMU messages The total number of SLMU messages per day is 485,098*. That number is broken down into four categories:
• Message In Credit Snapshot (MICS)
• File In Credit Snapshot (FICS)
• Message In Loan Snapshot (MILS)
• File In Loan Snapshot (FILS)
*according to SLXP at 7:43PM on 11/13/2006
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 18 of 31
Credit Snapshot Loan Snapshot
Instance Type: File In 69,534 232,059
Instance Type: Message In 47,339 136,166
Table 3 - Breakdown of SLMU Messages by type
From the data in, further derivations can be made for SLMU messages as it passes through the buffer tables. These are expressed in Table 4.
Formula Value
SLXTradeSnapshotBuffer2 FICS x 2 139,068
SLXPositionSnapshotBuffer2 FILS – FICS 162,525
SLXPositionSnapshot FILS – FICS 162,525
SLXTradeSnapshot FICS 69,534
Table 4 - Calculations based on SLMU messages
In Figure 9 the CPU data from the production servers has been overlaid with SLMU messages and processing times over the course of the same sampled week. The main SLMU message peak occurs between 3:00 PM and 5:00 PM. There is a secondary peak of messages that occurs at 10:00 AM.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 19 of 31
Average SLMU messages & processing time (from 10/15/06 - 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)
6 per. Mov. Avg.(% DBP01 CPUUtilization)
Figure 9 - SLMU Messages in SLXP
Processing times for messages Processing of SLMU messages should take under 2 minutes (120 seconds) according to the service level agreement (SLA). According to Figure 9, the SLA is currently not being met between the hours of 3:00 PM and 2:00 AM. Processing time is directly related to server CPU load. During this period, both APP06 and DBP01 use more than 50% of the available processing capacity. Although the high processing time occurs outside of the US trading day, this is unacceptable from an international standpoint.
Table sizes SLX uses a series of tables to store and process SLMU intraday messages. Table 6 shows each of the tables the SLMU feedback loop uses to process along with their sizes. Most of this data is retained for one day and the tables are purged nightly. The SLMU tables are part of the main SLX table space which has a set quota. Table Q shows how much space is being used by SLX tables.
TABLESPACE TOTAL_MB FREE_MB USED_MB PCT_FREE SLX 24528 8267 16258 33 UA 10731 1804 8926 16
Table 5 - Table space statistics on SLX and UA
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 20 of 31
OWNER SEGMENT NAME SIZE IN MB SLX_DBA SLX_POSITION_SNAPSHOT 184 SLX_DBA SLX_PSTN_SNPSHT_BFFR2 184 SLX_DBA SLX_SNAPSHOT_BUFFER1 352 SLX_DBA SLX_TRADE_SNAPSHOT 72 SLX_DBA SLX_TRADE_SNAPSHOT_BUFFER2 96 SLX_DBA SLX_UPDATE_LIMIT_USED 30 SLX_DBA SLX_UPDT_BRRWR_CRDT_LN_USD 192
Table 6 - SLMU Feedback loop table sizes in SLX
SLX also gets a start of day file from SLMU that contains all snapshot information. This file is 148MB in size.
Anticipated volume increases Given that SLMU should only be used for loan maintenance activities, the volume increase should be very small.
DML VOLUMES
Current DML intra-day load Hourly observations were taken based on DML activity. This plot was added to the previous chart and shown below in Figure 10.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 21 of 31
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Figure 10 - SLMU and anticipated DML messages over a trading day
The alarming part of the DML message projection is the 11AM spike which occurs during the US trading day. A spike will potentially cause the processors to hit their 100% utilization ceiling and then have an impact on regular SLX activity (as highlighted in Figure 8). Using a loan data from SLX, if a 1 minute downtime occurs each hour between 10 and 2 due to the DML feedback loop spike, the potential loss for JPMorgan is $20million in new loans. This prediction is very conservative but does highlight the need for further investigation. A breakdown in loss can be found in Table 7.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 22 of 31
From: To: 4-Jun-06
# of Loan
Total ValueUSD
# of Loan
Total Loan ValueUSD
Hrly Av. 32 $ 262,011,497.64 69 $ 63,876,620.12
Daily Av. 757 $ 6,288,275,943.25 1,653 $ 1,533,038,882.82 23:00 - 00:00 3 9,776,454.73$ 0 55,334.76$ 00:00 - 01:00 5 21,394,272.78$ 1 1,714,372.40$ 01:00 - 02:00 17 113,351,879.75$ 1 5,441,366.77$ 02:00 - 03:00 16 129,951,300.08$ 1 3,253,248.69$ 03:00 - 04:00 14 83,025,547.10$ 25 37,447,879.91$ 04:00 - 05:00 43 884,015,442.57$ 57 354,717,281.86$ 05:00 - 06:00 87 1,016,299,575.04$ 107 143,778,231.86$ 06:00 - 07:00 72 539,109,513.86$ 337 295,179,792.66$ 07:00 - 08:00 66 536,976,007.01$ 316 229,954,281.45$ 08:00 - 09:00 90 838,829,903.57$ 201 117,932,077.76$ 09:00 - 10:00 78 560,998,062.46$ 244 93,985,724.65$ 10:00 - 11:00 66 388,989,641.01$ 147 53,361,969.91$ 11:00 - 12:00 84 386,406,884.98$ 1 1 4,580,315$ 130 107,176,712.33$ 1 2 1,643,972$ 6,224,287$ 12:00 - 13:00 75 430,055,601.56$ 1 1 5,765,786$ 78 79,039,907.30$ 1 1 1,016,427$ 6,782,213$ 13:00 - 14:00 9 171,728,279.02$ 1 0 -$ 0 1,849,706.81$ 1 0 -$ -$ 14:00 - 15:00 3 75,867,084.33$ 0 2,719,934.33$ 15:00 - 16:00 1 5,162,395.33$ 0 69,085.51$ 16:00 - 17:00 2 4,235,037.57$ 2 1,526,454.28$ 17:00 - 18:00 3 6,633,586.62$ 1 605,583.31$ 18:00 - 19:00 3 10,135,257.59$ 1 926,060.65$ 19:00 - 20:00 3 13,884,355.94$ 1 1,421,584.56$ 20:00 - 21:00 5 14,946,578.59$ 0 234,267.84$ 21:00 - 22:00 5 23,914,216.79$ 1 596,853.58$ 22:00 - 23:00 7 22,589,064.98$ 0 51,169.65$
Total Loss : 13,006,500$
Total potential value of lost
loans
Time - ET
Loans Booked - Average Hourly BreakdownEquiLend Autoborrow LoansManual Loans
Outage Time -
Minutes
Potential # of Lost Loans
Potential Value of Lost Loans
USD
Outage Time -
Minutes
1-Jan-06
Potential # of Lost Loans
Potential Value of Lost Loans
USD
# of Requests Expired or
Not Processed
Table 7 - Financial loss due to SLX downtime
Table Sizes Because DML will use a feedback loop system similar to that of SLMU, the expected sizes of the tables should be equal to sizes in Table 5 & Table 6. Increasing the SLX table space for the new DML feedback loop should only add about 1GB of data at most and so it will not push the SLX table space over its quota. The start of day loan file from DML is roughly 220MB in size which again will not have a significant impact on free storage.
Anticipated volume increases Over a 6 month observed period, DML activity has grown at a linear rate. As shown in Figure 11, the number of loans grows by 1,990 every 90 days. The total number of loan components grows by 4,774 every 90 days. Following this trend, by July 31, 2007, DML will have 43,223 loans and 82,510 components. That is a 23% increase and a 61% increase from the previous year respectively. A table showing the observed 6 month period can be found in Table 8.
Loans Components3/31/2006 33087 472156/30/2006 35242 512609/11/2006 36703 55973
Table 8 - DML loan and component statistics
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 23 of 31
Growth of DML Activity
y = 22.115x - 825078R2 = 0.9978
y = 53.04x - 2E+06R2 = 0.9885
0
10000
20000
30000
40000
50000
60000
3/14/2006 4/28/2006 6/12/2006 7/27/2006 9/10/2006 10/25/2006
Date
Num
ber o
f loa
ns &
com
pone
nts
LoansComponentsLinear (Loans)Linear (Components)
Figure 11 - Number of DML loans and components
In Figure 11, the formula that is next to each line is a best fit linear regression generated by Microsoft Excel. The value for X is equal to the number of days after 1/0/1900 (Excel’s interpretation of dates as integers)
Requirements around timing of DML batch processes DML batch processes will generate more messages for SLX to process. Examples of batch processes include nightly recalculations and the early morning Japanese price batch process. Starting times of these batches should be observed. If they happen when the SLX server load is high, the SLA of 2 minutes will not be met.
RECOMMENDATIONS AND NEXT STEPS RECOMMENDATIONS The recommendations can be summarized into the following points:
• Review the service level agreements. The proposed service level agreement (guarantee) of a two minute maximum processing time of a transaction is not being met between the hours of 1AM and 3AM and between 3:30PM and 6:00PM. While the 1AM – 3AM time is not applicable because it is in the middle of batch processing, the SLA should be modified for 3:30PM – 6PM.
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 24 of 31
• Batch processes should be scheduled if possible to not coincide with SLMU peaks.
• There is no immediate need to increase file storage capacity to accommodate the DML feedback
loop
NEXT STEPS The next steps going forward can be summarized into the following points:
• Establish contact with a member of the infrastructure team and maintain regular contact for the
rest of the project’s duration.
• Identify times of all SLX and DML batch processes
• Determine if there is a need to increase file storage capacity on the volumes /dev/slxp1-slxhom1lv & jipnas1:/vol/jipnas1_app0/tss_slx_prd1 to allow for future growth
• Determine which volume the SLX tables are stored on (we think it’s /dev/dbp01-slxora1)
• Come up with a way to translate DML messages into storage space
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 25 of 31
REFERENCE DOCUMENTS
Reference Document Location AIX Server description webdoc http://dcfdoc.gis.chase.com
GTI Hobbit, real-time server monitoring utility http://wssunixmonitor.gis.chase.com
GTI Memory and CPU plotting macro CPU&RAM SLXv5.xls
SLMU / SLX feedback Loop documentation (multiple docs) Worldwide Securities Services PKS /
SLEP - TSS_WSS-INV-SLX/DML Integra - TSS_WSS-INV-SLX/DML Integration /
Research / Information
SLX Production Incident Impact Calculator S:\SECLEND\Production\Production Support\SLX\SLX Production Incident Impact Calculator.xls
DOCUMENT CONTROL DOCUMENT HISTORY
Version Name Date Description 0.1 Michael Kristan 11/14/2006 Initial Draft 0.2 Michael Kristan 11/20/2006 Revised based on comments from Brian Kenney
and from Michael Salamina. 0.3 Michael Kristan 11/22/2006 Revised after Scope/Requirements Review on
11/21. Expanded section on SAN storage and updated recommendation list.
0.4 Michael Kristan 11/30/2006 Revised to include performance assessment from November (post-swapfile fix), cost analysis from Danielle Rumore.
0.5 Michael Kristan 12/3/2006 Added SLMU feedback loop table sizes and financial impact from downtime.
DOCUMENT STORAGE System Name: Worldwide Securities Services PKS Project ID: 71707 Project Name: SLEP - TSS_WSS-INV-SLX/DML Integra - TSS_WSS-INV-SLX/DML Integration
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 26 of 31
APPENDICES POWERPOINT SLIDES FOR CAPACITY PLANNING
Capacity PlanningAn analysis of performance on SLX servers
Michael J KristanSecurities Lending Technology
JPMorgan Worldwide Securities Services
Observed data (SLMU)
• An analysis was made on the effect of SLMU messages on SLX servers during the week of October 15th.
• A report was run which took an hourly snapshot of the total number of SLMU messages processed during that hour
• SLMU hourly data was then aggregated over 7 days (including weekends)
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 27 of 31
Observed data (CPU)• CPU usage data was gathered off of three
servers for that week period (10/15-10/21)– TSJIPUDBP01– TSJIPUAPP06– TSJIPUAPP10 (not shown in these graphs)
• In order to remove any severe outliers, the median value of the 7 day samples was used for each time interval– Median was used instead of average because the
servers went offline for part of the weekend (resulting in an observed CPU usage of 0%) and it would have skewed the average
Observed data (DML)
• DML data was projected onto the graph in order to help determine when processing spikes may occur
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 28 of 31
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000Pr
oces
sing
Tim
e (s
econ
ds)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Response time is directly related to processor usage, not messages
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 29 of 31
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Number of messages however has an influence on processor usage
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000Pr
oces
sing
Tim
e (s
econ
ds)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Response time is quick when both CPUs are under 50%
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 30 of 31
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
SLA is not being met right now
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000Pr
oces
sing
Tim
e (s
econ
ds)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Risk: DML messages are high from 11AM – 1PM
Worldwide Securities Services – DML to SLX Integration - Project Code: <71707> Review of SLX Server Performance
Date Created: 14-Nov-06 Date Revised: 19-Dec-06 Last Saved By: Michael Kristan
© 2006 J. P. Morgan Chase Bank, N.A.
All Rights Reserved. Confidential And Proprietary
For Internal Use Only - Do Not Duplicate
Owner: Securities Lending Version: 0.5 Page: 31 of 31
Performance Data on APP06 & DBP01 (based on data from 10/15/06 to 10/21/06)
0
0.2
0.4
0.6
0.8
1
1:00 2:00 3:00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 17:00 18:00
Time of day (Eastern Time)
# of
mes
sage
s an
d C
PU u
sage
(as
a ra
tio o
f 100
)
1
10
100
1,000
10,000
100,000
Proc
essi
ng T
ime
(sec
onds
)
% APP06 CPUUtilization
% DBP01 CPUUtilization
SLMUMessages/25000
DMLMessages/25000
SLMU messageprocessing time
Processing SLA of2 minutes
6 per. Mov. Avg.(% APP06 CPUUtilization)6 per. Mov. Avg.(% DBP01 CPUUtilization)
Good news: SLMU peak occurs when DML messages are low
Next steps
• Take a closer look at memory usage and disk usage on servers
• Take a larger sample (2-3 weeks instead of 1)
• Identify times of all SLX batch processes• Come up with a formula to model CPU
usage based on messages