Standard Script All-Hands meeting

September 29, 2014

1

• Collaboration is the Key – Mat • PhUSE CSS Background – Mary• Standard Script Group – Hanming• Projects

– P01: Adrienne Bonwick– P02: Dante Di Tommaso– P03: Michael Carniello– P04: Sally Cassells– P05: Jean-Marc Ferran/Eric Sun– P07: Dirk Spruck– P08: Mary E Nilsson

• Upcoming Events - Dirk

Agenda

In a Silo World

3

IndustryAnalytics * Code/Tools/Scripts

* Software

)(xf

AcademicAnalytics * Code/Tools/Scripts

* Software

)(xf

FDAAnalytics * Code/Tools/Scripts

* Software

)(xf

Literature/ResearchTrial Design

Data Standardization

Double programming

Literature/ResearchTrial Design

Data Standardization

Literature/ResearchTrial Design/Actual Trial

Data Standardization

?

??

In a Integrated World

In a Collaborative World

Note: This Graph is from Mat Soukup, FDA

• Create a collaborative environment• Develop analytics repository

Collaboration is the Key to Success in the Future

• Determining Factors:– Entrepreneurial Spirit– Creativity and Innovation– Cross-cultural

Communications– Interpersonal Skills– Information Technology

6

http://2.bp.blogspot.com/-YXUyPEtc5EA/Ur5yCdUdvaI/AAAAAAAAIJE/vT2daLp0b10/s1600/collaboration3.jpg

• Computational Science Symposium (CSS) Working Groups– 5th year in a Series of Annual Computational Science

Meetings established to support the work of FDA’s Center for Drug Evaluation and Research (CDER), Office of Computational Science (OSC)

– 3rd Year working with PhUSE (first two were with the DIA)– FDA’s motivation: To work collaboratively with industry,

vendors and regulators in a “non-competitive space” to find solutions and advance the computational sciences/technology associated with the development and regulation of new medical products (drugs, biologics, etc.)

– Perceived benefits of collaboration and crowd sourcing

PhUSE CSS Background

7

PhUSE CSS Working Groups

Optimizing Use of Data Standards

Standard Scripts for Analysis and

Reporting

White Papers on Analysis and Displays

Script Repository

Communications

Emerging Technologies

Semantic Technologies Non-Clinical

8

PhUse CSS Working Groups

Summary of Standardization

Data Collection Systems

Observed Datasets

Analysis Datasets

Tables, Figures

and Listings

Clinical Data Flow

Trial Design

PRM SDTM ADaM No TFL Stds Exist

Industry

Standards

Alignment

CDASH

?TransCelerate (TAs)

Script Repository In the Cloud

11

• Development of Standard Scripts for Analysis and Programming Group was formed during 2012 PhUSE CSS March 2012 meeting

• Current Leadership– Mary Nilsson (Industry co-lead, Steering Committee

Liaison)– Hanming Tu (Project Management)– Steve Wilson (FDA Liaison)– Mat Soukup (FDA Liaison)

Working Group: Formed in 2012

Working Group: Description

• Goal: Establish a platform for the collaborative development of specialized program code to be used as analytical tools for the clinical trial research, reporting, and analysis. This includes:– Identification of areas that can benefit from a standard

set of analyses• Within a topic area, includes recommendations for analyses,

tables and figures

– Developing a process and guidelines for documentation and management of scripts

– Harness data standards whenever feasible

Working Group: Two Focus Areas

• Initially three sub-groups– Sub Group 1: works on developing and prioritizing lists of

analysis scripts to be developed – Sub Group 2: works on process for developing, validating and

documenting scripts– Sub Group 3: works on technology platform to support script

development, testing and documentation.

• Two Focus Areas:– White Paper Development: standards, formats, etc.– Repository Platform: platform, process, development, access,

validation, etc.

Working Group Repository: Google Code

14

• P01: Look for existing scripts and store them in the repository – Adrienne Bonwick and Aiyu Li

• P02: Define qualification steps for scripts in the repository – Dante Di Tommaso

• P03: Maintain and enhance platform (repository) for sharing scripts – Mike Carniello/Hanming Tu

• P04: Legal ownership and issues in open source repository – Sally Cassells

• P05: Create templates and metadata for documenting scripts and coding practices

– Jean-Marc Ferran/Eric Sun

• P06: Refine process for creating and editing scripts in Google Code – Closed, any remaining tasks moved to Project 03

• P07: Implement and further develop communication plan for standard scripts – Dirk Spruck

• P08: Create white papers providing recommended display and analysis including Table, List and Figure shells

– Mary Nilsson

Working Group: Projects

Working Group: Accomplishments

• Script Repository has been created– Google Code chosen as the repository– Some scripts exist in the repository– MIT license chosen– Process guidelines developed– First FDA Script set added to the repository– Continue calling for additional scripts

• White Papers Describing Recommended Analyses, Tables, Figures, and Listings– For common measurements (e.g., safety)– One white paper is complete; 5 others in progress

Remit:• Look for existing scripts and store them in the repository

Progress• Received the “Demographics” package from the FDA• Reviewed and updated to work on any PC SAS package using FDA Pilot

Data• Provided working instructions• Added to Code Repository

• Starting to investigate MAED

P01:Adrienne Bonwick and Aiyu Li

Standard Scripts Project 2

2014 - Proposal for Qualification of Standard Scripts

Main Sections

Summary of prior proposal, 2013Updated proposal, July 2014

Main Sections

Summary of prior proposal– Concepts, definitions & metadata– Test data considerations– Heavy vs. Light qualification

Updated proposal

Proposal from 2013

• Anyone can submit a script, according to a check list• Categorize scripts by complexity

– Complexity:

low, medium, high, software

– Output:

tabulated data, analysis data, table, figure, listing

• Metadata for script should indicate– Type of output:

tabulated data, analysis data, table, figure, listing

– Study design:

parallel, crossover, etc

– State of qualification:

?

Proposal from 2013

• Test data– Overall project should have minimum test data (SDTM & ADaM)– Scripts can propose new test data, must pass (Data fit? Open CDISC?)– Share program to produce test data, never binary test data

• 2 levels of qualification– Light vs. Heavy: according to complexity of script & output– Common elements include

• header• adhere to good programming practices• clear scope of script (e.g., study design(s))• test data matche scope & pass "FDA Data Fit" assessment (Open CDISC?)• documentation available for: usage, inputs, outputs, dependencies

Proposal from 2013

• Heavy qualification– Beta package includes:

minimal elements for contribution• Specification & Documentation (could be in pgm header)• Test data (Data Fit? or Open CDISC?)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests reproduced

– Draft• Write qualification plan, Review tests for completeness/suitability (e.g.,

Branch testing – are all conditional blocks/combos tested?)– Test

• Peer Review: Write qualification report, incl. log/output from tests– Final

Proposal from 2013

• Light qualification– Beta package includes

skip if >1 yr production use without ERROR– Draft

minimal elements for contribution• Specification & Documentation (could be in pgm header)• Test data (Data Fit? or Open CDISC or other, as appropriate)• Tests & Expected results defined• Peer Review: GPP, Specs & Docn reviewed, Tests reproduced• Write qualification plan, Review tests for completeness/suitability (e.g.,

Branch testing – are all conditional blocks/combos tested?)– Test

• Peer Review: Write qualification report, incl. log/output from tests– Final

Proposal through CSS 2104

Peer Review Checklist Heavy Light

Requirement specification X ?

Documented or perhaps only documented in header X

User Guide X X

SDTM/ADaM used in input/output X X

Open CDISC validator or Data Fit used to check input/output X X

GPP in source X X

Run according to Requirement specification X ?

Tested by qualification plan, tests & results all Peer reviewed X ?

Tested by End users X ?

Robust without red errors in contributor's production environment X X

Robust and used in FDA (other) scripts repository, ranked ****** X

Main Sections

Summary of prior proposalUpdated proposal

– Objectives– Definitions: Qualification, States, Roles– Metadata and Test data– State Transitions

Proposal 2014 - Objectives

• For End-users– Clear overview of resources available, incl. purpose & state of each– Inspire confidence from first user experience– Ease-of-use:

• clear messaging from scripts• Reproducible results in user's own environment• Consistency of scripts, learning first one makes remaining familiar

– Ease of converting users to contributors

• For Contributors & Standard Scripts Team– Standardize workflows and checklists– Modularize routine components (e.g., FUTS for dependency checking?)– Automate testing, issue identification (e.g., concept similar to Spotfire/R compatibility)– Centralize & consolidate information & results

Qualification Proposalmeaningful terms in bluehttp://www.phusewiki.org/wiki/index.php?title=File:WG5_P02_Proposal_-_2014.pptx

• Qualification Instructions (see embedded template ð) – Certification phase of Qualification applies to new scripts and tests– Confirmation phase applies to updates of existing scripts

• States:

Contributed, Development, Testing, Qualified• Roles

– Contributor: Anyone with appropriate skills & interests– Developer: CSS Working Group 5 volunteer familiar with objectives**– Tester: CSS WG 05 volunteer familiar with objectives**– Environment Tester: Anyone in industry community able to set up automatic

test replication in their work environment– Reviewer: Author of white papers, designers of script targets**

** suggests a quick-start onboarding page in CSS Phusewiki

WG05-P02 Qualification plan (template).do

Qualification Proposal

• Metadata for scripts should indicate:

YML proposal– Whitepaper & output ID uniquely identify the target

<Script: Source, Title, Target>

– Programming language & version

<Language: name, version>

– Type of output:

<Script: Type>

– Study design:

<Script: StudyDesign>

– State of qualification <Stage>:

<Qualification: Stage>

– OS is not included, since should be indicated in OS compatibility report• Test Data requirements

– available as a program or script (text, not binary)– based on expected standards (SDTM, ADaM)– data requirements clearly & accurately specified for each script– include expected results from these data for defined tests/checks

Qualification Proposal

• Transitions

"Contributed" is the original State of all scripts– to Development, checklist includes

by Developer & Reviewer• R & D confer on suitability of contribution. Suitable starting point?

[ may require analysis details, specs, from contributor ]• D reviews components (checklist to be completed)• D works with Contributor to complete minimum components

[ including Test Data and Coverage of defined tests ]• D adds standard parameter, dependency checking• D writes Qualification instructions .docx (see template, above)

– to Testing, checklist includes

by Tester• Review Qualification instructions, consider coverage of tests• Execute Qualification instructions• Work with Developer to complete execution successfully

Qualification Proposal

• Transitions continued– to Qualified

by Tester & Environment Tester & Reviewer• T updates reference test outputs from certification/confirmation• E updates & executes local tests (posting PASS/FAIL results)• R confirms script output matches intention• R confirms qualification process covers important elements and

considerations. • R also provides user (rather than technical) feedback?• Achieve "Qualified" state when all tests in all test environments PASS

(i.e., match outputs that T has certified and/or confirmed) and that R agrees that target is achieved

Qualification Proposal

• Efforts Required– Top priority

• Finalize Qualification states, roles, workflow, checklists, and templates – Next priorities

• Design test structure in google code• Develop scripts that will allow Environment Testing• Develop general components (e.g. parameter, dependency checking)• Identify Environment Testers based on

– Host environment– SAS or R version

• Identify opportunities to automate qualification. E.g.,– Environment Testers to post results back as machine readable– Script green-light/red-light qualification matrix on Phusewiki

Remit:• Maintain and enhance script repositoryProgress• CSS 2014 Scriptathon work catalogued and displayed (see next slide)• PharmaSUG Scriptathon work to be added to repository• PhUSE 2014 Scriptathon plannedWhat’s next?• Do we use Scriptathons as vehicle for adding code?• Should we spend effort in updating/perfecting what we have?• Scriptathon targets are White Paper displays … should we add SDTM and

ADaM scripts as well?

P03:Hanming Tu and Mike Carniello

• https://code.google.com/p/phuse-scripts/wiki/yml_idx

P04: Legal ownership and issues in the Open Source Repository

Summary

16 Sep 2014

Background

• Members of WG5, the PhUSE Standard Script Development group, wanted to ensure that users of posted scripts are protected from any potential claims of ownership from organizations employing individuals who contribute scripts.– The group recommended that a ‘click through’

agreement be obtained from a lawyer and posted on the GoogleCode site. This became P04 for our group.

Assignment March 2013

• Get more scripts in the Script Repository– Networking– Broad Communications

• Address any remaining validation, legal, process issues

• Finalize first white paper – Goal to finalize within a couple of months

• Begin work on the remaining white papers– Looking for more active participants– Goal to be final or near-final by next year

Summary of Next Steps (2013)

• The work has now been divided into 8 projects– P01: Look for existing scripts and store them in the

repository (Kevin Kane)– P02: Define validation steps for scripts in the

repository (Lina Jørgensen)– P03: Maintain and enhance platform (repository)

for sharing scripts (Hanming Tu)– P04: Legal ownership and issues in open source

repository (Kevin Kane/Sally Cassells)

Summary of Next Steps (2013)

• 8 projects (cont’d)– P05: Create templates for documenting scripts and

coding practices (Jean-Marc Ferran)– P06: Refine process for creating and editing scripts

in Google Code (Kevin Kane)– P07: Implement and further develop

communication plan for standard scripts (Dirk Spruck)

– P08: Create white papers providing recommended display and analysis including Table, List and Figure shells (Mary Nilsson)

Actions for P04

• Sally contacted Chris Decker to request that PhUSE provide a legal contact who could develop the agreement.

• Chris provided draft agreement in May 2013 for review.

• The agreement is currently posted on the GoogleCode site and in the PhUSE Wiki . The document name is SS 15591982 1 UKGROUPS Harmony Individual Contributor License AgreementDOC.

Issues, Discussions, Decisions

• UK vs. US– The Draft Agreement cites UK law rather than US

(Section 8). Adding similar clauses for the US would require the services of a US lawyer.

• Individually owned tools– Contributor grants full ’worldwide, non exclusive, royalty

free, irrevocable license to the IP’. – If tool was developed collaboratively, contributor should

get agreement from collaborators before posting.• Click through

– Not yet technically click through but contributors can read and agree to the terms.

Next Steps, Recommendations

• Investigate possibility of modifying the language in Section 8 to indicate that the agreement applies fully within the US.

• Consider a leadership team vote to finalize the draft agreement.

• Link to agreement PhUSE Standard Scripts Contributor Agreement

Remit:• Create templates and metadata for documenting scripts and coding

practices

Progress• Templates for program documentation and user guides created• Next steps:

– Document 2 existing scripts from the repository using these and send them for review in order to get more detailed feedback on the structure based on examples

– Define metadata for TFL analysis. Link up with Emerging Technology / Analysis metadata WG to see where we can align and re-use each other’s findings

Standard Scripts WG - P05:Jean-Marc Ferran (Qualiance)& Eric Sun (Sanofi)

Project Description• Create white papers providing recommended displays and

analyses including Table, Figure, and Listing shells for clinical trial study reports and summary documents.

• The intent is to begin the process of developing industry standards with respect to analysis and reporting for measurements that are common across clinical trials and across therapeutic areas.

• Script developers could then create scripts consistent with the recommendations for all to use, improving efficiency and safety signal detection.

P08: Mary Nilsson

Standard Script All-Hands meeting

P07 - Communications

45

• Maintain Website with Group Updates• Maintain Conference and Paper Log (update needed)

• Distribute News (e.g. PhUSE Newsletter)• Help Preparation of Conference Material• Technical Assistance of White Paper Review• Part of New CSS Communications Group

(New CSS Letter Planned)• Backup and Help is Needed

Current Activities

Progress - The following white papers have been finalized:• Analyses and Displays Associated with Measures of Central Tendency –

With a Focus on Vitals, ECGs, and Labs in Phase 2-4 Clinical Trials and Integrated Summary Documents (finalized in October 2013)

• Analyses and Displays Associated with Non-Compartmental Pharmacokinetics – With a Focus on Clinical Trials (finalized in March 2014)

Note: Final white papers can be found on the PhUSE website, Publications section. (www.phuse.eu)

P08: Mary Nilsson

Progress - The following white papers are in progress:• Analyses and Displays Associated with Outliers or Shifts from Normal to Abnormal – With a

Focus on Vitals, ECGs, and Labs in Phase 2-4 Clinical Trials and Integrated Summary Documents (third draft targeted for public comment in September 2014)

• Analyses and Displays Associated with Adverse Events – With a Focus on Phase 2-4 Clinical Trials and Integrated Summary Documents (first draft targeted for public comment in September 2014)

• Analyses and Displays Associated with Demographics, Disposition, and Medications – With a Focus on Phase 2-4 Clinical Trials and Integrated Summary Documents (final white paper targeted for October 2014)

• Analyses and Displays Associated with Hepatotoxicity – With a Focus on Phase 2-4 Clinical Trials and Integrated Summary Documents (first draft targeted for public comment by the end of 2014)

• Analyses and Displays Associated with QT Studies (first draft targeted for public comment by the end of 2014)

Note: Draft white papers (where applicable) can be found on the PhUSE Wiki, Development of Standard Scripts Project 08 wiki page (www.phusewiki.org)

P08: Mary Nilsson

• How you can help– When a white paper has been finalized, or when a

white paper is ready for broad review, please help spread the word within your organization and across your contacts. Please include your medical colleagues. Thanks!

P08: Mary Nilsson

• Continue to build Script Repository– Continue creating code for tables and figures as outlined in white

papers– Look for other existing code sharing libraries

• Continue white paper development– Consider updating the first white paper– Finalize PK white paper this month– Finalize remaining 4 white papers in 2014– Consider additional white papers to develop

• Communications– Presentations at various conferences– Implement other ideas to increase awareness

Plans for Upcoming Year

Interested In Learning More?

• Sign up for the PhUSE working group mailings– Go to www.listbox.com– Join this group: “CSS WG5: Developing Standard Scripts”

(CSS-WG-Standard-Scripts@listbox.com)• Sign up to be a contributor or committer in the

script repository (Google account required)– Send an email to one of the repository maintainers to

request to join the Google Code PhUSE/FDA script repository

– Go to repository at https://code.google.com/p/phuse-scripts/ and start interacting

Reference URLs

• PhUSE home: http://phuse.eu/• PhUSE Wiki: www.phusewiki.org• PhUSE Standard Script Group: http://

www.phusewiki.org/wiki/index.php?title=Standard_Scripts

• PhUSE Repository: – http://code.phuse.com– https://code.google.com/p/phuse-scripts/

Q&A