Management Report and KPIs

Report to the Management Board on Q4 2014 Produced January and February 2015

KEY PERFORMANCE INDICATORS

Content – Request Submissions

Target  –  completed  (including  not  accepted)  –  70%  2014  Q4  –  66.3%  

    August  Septemb

er   October   November   December  %  on  hold  (excluding  not  accepted)   21.11%   21.45%   21.07%   20.92%   21.43%  

Content – Projects 2014

0%  

10%  

20%  

30%  

40%  

50%  

60%  

70%  

80%  

90%  

100%  

Content   Mapping  

Red  

Green  

CONTENT  PROJECTS   Content   Content   Mapping   Mapping  Green   12   70.59%   5   100.00%  Red   5   29.41%   0   0.00%  

Red projects •  Pre-coordination roadmap and

expression library •  Context value sets for situation content •  Development of a Diagnostic Imaging

Reference Set •  Alignment of SCT content with

procedure classifications •  Pharmaceutical content international

value proposition

Content – Release Details (concepts/descriptions)

0  

5000  

10000  

15000  

20000  

25000  

30000  

35000  

40000  

45000  

Jan  2010  

July  2010  

Jan  2011  

July  2011  

Jan  2012  

July  2012  

Jan  2013  

July  2013  

Jan  2014  

July  2014  

Jan  2015  

New  descripIons  added  

New  Concepts  added  

•  July  2014  –  descripIons  added/changed  represents  the  changes  to  the  medical  devices  representaIon  

0  

5000  

10000  

15000  

20000  

25000  

30000  

35000  

DescripIon  changes  

Concept  changes  

Customer Service - Freshdesk

Freshdesk went into use gradually beginning in April 2014, so Q2 numbers are not entirely accurate. New procedures were put into place in Feb. 2015 to improve response time. SLA levels vary (see bar chart) so they cannot be put on the line charts.

Education

80 39

4 21 56

15

121

80

33 18

31

4

E-Learning - Completed Unique Views Q3 Q4

0

5

10

15

20

Q3 Q4

Online Tutorials/Assessments

# Online Tutorials/Assessments Available

Human Resources

Q1 Q2 Q3 Q4

# Members 25 27 27 27

# Large Vendor Agreements

0 0 0 0

# New Salsa Licenses

51 25 30 39

# New Collab Agreements

0 0 2 0

Appropriate Representation at External Events

100% 100% 100% 100%

% Survey Results Showing Strong Product/Service Levels

71% (Member survey)*

n/a n/a 98% (Showcase survey)**

Stakeholder Management

*Member survey completed in Nov. 2013 but presented here as a baseline. “IHTSDO meets my overall service requirements” 13% strongly agree + 58% agree; “To what extent has our service delivery met your expectations?” average response score: 2.25, where 1-exceeds expectations, 2-meets expectations, 3-slightly below expectations, 4-does not meet expectations **Showcase survey: 56 respondents; 98% agreed or strongly agreed that it was a valuable use of their time, 84% said the Showcase met or exceeded their expectations.

Concepts  per  release 2014 2015 Added403,836                                         417,209                               13,373                    

Release  on  time?Defects  in  release Opportunities 75,132,477                    

Defects* 40,364                                  Defect  percentage 0.054%

January  2015  SNOMED  CT  International  Edition  

Yes,  on  31  Dec.  2014

Technical - Delivery

RF2  Compliance All  file  columns  present,  in  the  correct  order PASSAll  columns  contain  the  correct  data  types

Release  type  integrity Delta  files  are  subsets  of  the  Snapshot  files PASSSnapshot  component  states  are  subsets  of  Fullcomponent  statesNew  Full  release  is  same  as  previous  Full  release  +new  Delta

Concept/component  integrity Each  component  possesses  required  components,  and PASSnone  added  in  error

Quality  Assurance

*Language reference set member were erroneous; subsequently corrected

Technical - Applications & Architecture Customer Service

Tooling Issues logged in JIRA against Mapping, MLDS, Browser and Refset Mgmt Tools

Freshdesk Customer Support (internal and external stakeholders) Tickets Resolved On Time

Resolution First Contact Resolution

Average First Reponse Time

Average Response Time

Average Resolution Time

Q4 2014 67 67% 73% 5 hours 49 mins 4 hours 13 mins 2 days 16 hours Target 85% 75% 5 hours 4 hours 2 days

Technical - Applications & Architecture Infrastructure Uptime (%)

Oct 2014 Nov 2014 Dec 2014 IHTSDO Collabnet 100.00 99.99 99.99 IHTSDO Daily Build Browser 100.00 100.00 100.00 IHTSDO Freshdesk 99.97 IHTSDO JIRA 99.99 99.98 99.98 IHTSDO Lookup Service 100.00 IHTSDO Mapping Service 97.03 95.83 95.56 IHTSDO Release Service API 100.00 100.00 100.00 IHTSDO SALSA 99.99 99.95 99.84 IHTSDO SNOMED CT Browser 100.00 100.00 100.00 IHTSDO Web Site 100.00 99.94 100.00 SIRS 97.07 99.92 99.31 Average Uptime (%) 99.41 99.51 99.47

Target Uptime (%) 99.9

Technical - Applications & Architecture Online Service Usage Statistics

Sessions Avg. Session Duration

Bounce Rate

IHTSDO SNOMED CT Browser 14,273 00:01:13 85.87% IHTSDO Web Site (from Nov 1, 2014) 13,876 00:03:02 42.89% IHTSDO Mapping Tool 1,479 00:36:31 11.70% SNOMED In Action 1,296 00:00:52 49.38% SNOMED CT Documentation 942 00:00:48 84.08% IHTSOD Freshdesk Support Portal 602 00:02:00 47.18%

Technical - Applications & Architecture Project Success

Target Live Date

Actual Live Date

Variance Notes

IHTSDO SNOMED CT Browser

2014-10-01 2014-09-18 -13

IHTSDO Web Site 2014-10-27 2014-10-27 0 IHTSDO Mapping Tool 2014-07-25 2014-08-05 11 Member Licensing & Distribution Service (MLDS)

2014-12-01 2015-01-12 42 Delayed to align with new affiliate license

IHTSDO Refset Mgmt Tool 2014-12-19 2015-03-31 102 Multiple delivery points, rolled over into 2015 for directory functionality

        Average Variance

28.4

Target Variance

0

Project Status

6 14

13

57 58

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Q3 Q4

Project Status

Green

Amber

Red

Number of projects declined in Q4 because some projects were combined. Red projects in Q4 include taking a step back to develop a strategy for refsets/subsets, mapping activities put on hold pending the completion of the ICD-10 maps, work on vaccines that require precursor activities, delays in getting a group of consultant editors, need to finish the SCT languages work, deferment of work on DITA, delays in the vendor guide, and supplier delays for the Refset Management tool.

UPDATES BY LINE OF BUSINESS

▪  Successful completion of content authoring for the Jan 2015 release ▪  Detailed planning for content development activities for 2015 ▪  Consultant Terminologist Program is progressing, with a successful

face-to-face group meeting and allocation of a group project ▪  Substance redesign - progressing on the alignment of content with

ChEBI and the development of molecular entity hierarchy ▪  LOINC – technology preview published ▪  Publication of medical devices release and GMDN map – planning

for required work in 2015 ▪  Publication of updated GP/FP subset and ICP-2 map ▪  Reviews of ICD-11 chapters, and subsequent content

enhancements – update on delivery of ICD-11 chapters ▪  Finalized the completion of the ICD-10 project to provide a complete

map

Content – 2014 Q4 Achievements

▪  Management and authoring of SIRS requests ▪  Management of feedback on LOINC technology preview ▪  Functioning work - initiate project ▪  Publication of updated GP/FP subsets and ICPC-2 map ▪  Preparation for publication of molecular entity hierarchy tech preview ▪  Use case review for pharmaceutical requirements for review by MF ▪  Development of E-learning syllabus (basic and advanced authoring) ▪  Consultant Terminologist program ▪  ICD-11 alignment – continuation of review and authoring new content

arising from review work ▪  Publishing candidate baseline of completed SCT to ICD-10 map ▪  Commencement of work with UKTC to review the SCT to ICD-10 map ▪  Complete requirements specification for key features of replacement

authoring tool

Content – 2015 Q1 Key Objectives

▪  Technology previews ▪ Molecular entity technology preview ▪  LOINC technology preview ▪  Substances technology preview ▪  Functioning technology preview

▪  LOINC content - first release of joint content ▪  Finalizing report on anatomy alignment with FMA, as part of the

contract with University of Washington ▪  Planning for future development of anatomy content based on the

report from University of Washington (continued next slide)

Content – 2015 Q2-Q4 Key Objectives

Content – 2015 Q2-Q4 Key Objectives (continued from previous slide)

▪ Management and authoring of SIRS requests ▪ Microbiology value set(s) ▪  Allergy content - enhancing

existing content ▪ Molecular entity ▪  Substances ▪  Pharmaceutical content ▪ Dentistry content ▪ Nursing diagnoses/procedures

content as agreed with ICN as part of collaboration agreement

▪ Development of editorial guidance for functioning content, and provision of new content ▪ Delivery and authoring of CMT

content as per agreed schedule with Kaiser Permanente ▪ Continuation of ICD-11 alignment

work ▪ Delivery of baseline release of

completed SNOMED CT to ICD-10 map

Items of content delivery for July 2015, also including Jan 2016 items:

▪  Potential disruption of authoring productivity due to development and delivery of new authoring tool ▪  Lack of clarity for the delivery of a clear process for escalation of

editorial issues available internally and externally will lead to uncertainty ▪  Potential lack of required skills within the authoring resource to

ensure the team can meet the challenging times ahead ▪  Lack of feedback on the alpha releases and technology previews

will lead to uncertainty of the ability to implement the solution

Content – 2015 Key Risks

▪ Customer Relations Roadmaps completed for all three regions ▪ Member Satisfaction survey issued ▪  Participation at national and regional conferences with positive

outcomes: ▪ National SCT event in Estonia ▪ HIMSS Middle East conference ▪  INFOLAC conference in Uruguay ▪ National SCT stakeholder engagement event in Malaysia ▪ Asia eHealth Information Network UHCICT conference in the

Philippines

Customer Relations – 2014 Q4 Achievements

▪ Member Satisfaction survey results received and summary results published ▪  Prospective Member pipeline updated and Impact Statements

developed ▪  “SCT in Action” redesign plan developed ▪  Finalize regional CRM Action Plans based on the regional

roadmaps

Customer Relations – 2015 Q1 Key Objectives

▪ Delivery against agreed regional CRM Action Plans ▪ Regional Staffing Models, Scope and Costing completed ▪ Regional Calendar of Meetings & Events established ▪  KPIs refined ▪ Work with LOB’s to simplify documentation for Members ▪ Development of Regional Marketing Plans

Customer Relations – 2015 Q2-Q4 Key Objectives

▪  The existing core delivery resources that are mainly centralized in Europe would not be scalable to support Members in Asia Pacific and Americas regions ▪  Insufficient resource capacity to scale up the engagement as

outlined in the vendor engagement strategy

Customer Relations – 2015 Key Risks

▪ Collaboration status and activities with all organizations as identified in Operation Plan 2014-15 ▪  Presentation at WHO-FIC annual meeting ▪  Started communicating to SDOs plans regarding antecedent

versions of SNOMED CT

Collaboration – 2014 Q4 Achievements

▪ Confirm 2015 work plan and communication activities with WHO and HL7 ▪  Formalize affiliate licensing arrangements with SDOs using

SNOMED CT in their products/materials e.g. DICOM, IEEE, HL7 ▪ Draft collaboration agreement with AMA with initial joint work

areas and seek approvals where required ▪  Sign off agreement and work plan with INSERM (Orphanet) ▪ Move forward the IHTSDO role in taking over the chair role with

JIC

Collaboration – 2015 Q1 Key Objectives

▪ Monitor delivery of agreed collaborative work plan within

individual LoB and also with partners ▪  KPI’s re-defined ▪  Service Level Targets Established. ▪ Define Regional requirements for collaboration support

Collaboration – 2015 Q2-Q4 Key Objectives

▪  Lack of access to clinical specialists for SNOMED CT/ICD-11 may delay progress of the work ▪ Dependence on potential and actual collaborating partners on

both establishing updated/new agreements and undertaking work

Collaboration – 2015 Key Risks

▪ Completion of main elements of Foundation course for E-Learning Reference Group ▪ Delivery of draft guidance document on terminology binding ▪ Delivery of a specification document updating SNOMED CT

compositional grammar ▪ Delivery of a draft specification document on expression

constraint language ▪ Delivery of a draft guidance document on analytics with

SNOMED CT ▪ Delivery of a draft document on options for exchanging

constrained post-coordinated expression ▪ Delivery of a draft report of a survey of requirements for wider

use of SNOMED CT in research and innovation activities

Education – 2014 Q4 Achievements

▪  Finalize preparation of course material for Foundation course ▪  Start first (small) and second (full) intakes to course ▪ Revise Foundation course material based on feedback from first

intake ▪  Plan and commence development of Implementation course ▪ Delivery of draft document on SNOMED CT query language

specification

Education – 2015 Q1 Key Objectives

▪  Assess completion of four intakes to Foundation course ▪ Complete preparation of course material for Implementation

course ▪  Start and complete first intake to Implementation course ▪ Manage content course delivery ▪ Draft document on template extensions to SNOMED CT query

language ▪ Delivery of updates to various SNOMED CT guidance

documents

Education – 2015 Q2-Q4 Key Objectives

▪  Disruption of key staff time by overhead of running courses for large numbers of students impacting other work

▪  Delay in delivery of presentation and assessment materials for implementation course ▪ Dependency of staff time on impact of feedback/issues from

Foundation course ▪ Dependency on external contractors with limited time

▪  Uncertainty about future demand for courses with cost of meeting potential demand or reputational risk of failing to meet demand

▪  Negative reactions to unsuccessful course completion – i.e. disputing assessment result

▪  Lack of available time from other teams to contribute to content course (and potential mapping course and refset course) development

▪  Implementation guidance documentation developments and updates delayed by priority focus on education

Education – 2015 Key Risks

▪  Successful running and completion of the Terminology Server RFP ▪ MVP development on Members Licensing & Distribution Service

(MLDS) completed in time for go live in early Jan 2015 ▪  Successful recruitment of three new team members to support

the tooling effort in 2015 ▪  Feedback & input from both Malaysia and Canada

Technical – App & Arch – 2014 Q4 Achievements

▪ Roll out of MLDS for IHTSDO in Jan. and Members in Feb. ▪  Approval and purchase of Terminology Server (TS) ▪  Integration of the IHTSDO SCT Browser to the TS ▪  Integration of WB exports to the TS so that current authoring is

being stored in the TS ▪ Minimum Viable Product (MVP) Translation Service beta for

members ▪  Batch Import integrated to the TS ▪ Refset Management Tool available for members, and integrated

with TS ▪  Publishing IT policies covering

▪ Data Protection ▪ Software Development Licensing, and ▪ Data Retention, Backup and Deletion.

Technical – App & Arch – 2015 Q1 Key Objectives

▪  IHTSDO content team will no longer need to rely on using the IHTSDO Workbench by end of Q4. This incorporates MVP’s of the following: ▪  template authoring ▪ batch authoring

▪ Migration away from Collabnet for both infrastructure and TeamForge well before the end of Q4. This includes: ▪ house cleaning of existing Collabnet content and users ▪ moving to JIRA to replace the current use of trackers ▪ moving to Confluence (current product of choice) for

technical documentation and project collaboration

Technical – App & Arch – 2015 Q2-Q4 Key Objectives

▪ On further investigation the Terminology Server is found to be less compliant than expected

▪  Tight development and delivery timelines without much room to maneuver

▪  Although improved since 2013, there is a continuing dependency on suppliers as fallback positions. There is a risk that this reliance can lead to the suppliers holding the organization to "ransom" in regards to costs and timings of deliverables.

▪  In the event of accessibility being denied to tools and applications for whatever reason this can lead to major reductions in productivity and impact the timing of releases.

Technical – App & Arch – 2015 Key Risks

▪ Release Management ▪  SNOMED CT Spanish Edition released to Members Oct 31 ▪  SNOMED CT International Edition released to Members Dec 31 ▪  LOINC Technology Preview co-released with Regenstrief in Oct

▪  SNOMED Factory/Process Development ▪  SNOMED CT release service used in the production of the Jan

2015 International Edition ▪  SNOMED CT release validation framework minimum viable

product completed and used in the validation of the Jan 2015 International Edition

▪  SNOMED Factory/End to End Service Development ▪  Initial planning for the Workbench Replacement Project

completed, resulting in funding approval

Technical – Delivery – 2014 Q4 Achievements

▪ Release Management ▪ Develop a product lifecycle management process, align the release

management portfolio with the content roadmap, and publish product maintenance schedules for SNOMED CT products in development, maintenance, and deprecation phases ▪  The related deliverable: the portfolio, along with current and planned

schedules and statuses, published on the Web by Mar 31 ▪ Workbench Replacement

▪ Demonstrate the end to end throughput of content, and deliver a minimum viable product (MVP) batch import service into production

▪  SNOMED CT Extended Logic Profile ▪ Understand and document the impact on Members. This will feed

into a roll-out plan, which will enable the delivery of pending content projects

Technical – Delivery – 2015 Q1 Key Objectives

▪ Release Management ▪  Achieve routine product lifecycle management ▪ Obtain buy-in for and introduce the use of the new product

development process, which will front-load decisions about content and software development requirements, and quality validation ▪  Leverage the new quality validation framework to improve quality

validation coverage across the entire product portfolio ▪  Begin partnering with NRCs to understand how extensions may

be managed and released using IHTSDO processes and tools

(continued next slide)

Technical – Delivery – 2015 Q2-Q4 Key Objectives

(continued from previous slide)

▪ Workbench Replacement ▪ Deliver a set of authoring services that are integrated from

request through publication. The aim is firstly to provide functionality to release content based on the current content roadmap, and secondly to improve the throughput of content by eliminating the inefficiencies inherent in the Workbench system. The focus in 2015 is on a minimum feature set (MVP), to be stabilized and enhanced during 2016.

Technical – Delivery – 2015 Q2-Q4 Key Objectives

▪ Risk to the quality of SNOMED CT releases: ▪ New staff are inexperienced in our domain ▪ Newly developed software tends to be unstable

▪ Risks to the Workbench Replacement Project ▪  Project timelines are ambitious ▪  There have already been delays due to the need for an

independent evaluation of the terminology server RFP ▪  Project personnel are 100% allocated. Additional projects or

tasks that would put timelines at risk ▪  An ambitious content release schedule for 2015 risks starving

the project team of SME input and testing resources

Technical – Delivery – 2015 Key Risks