using web service technologies for incremental, real-time data transfers from edc to sas
DESCRIPTION
Using web service technologies for incremental, real-time data transfers from EDC to SAS. Andrew Newbigging Vice President, Integrations Development 19 th October 2010. Medidata Solutions, Inc. Proprietary - Medidata and Authorized Clients Only. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/1.jpg)
Using web service technologies for incremental, real-time data transfers from EDC to SAS
Andrew NewbiggingVice President, Integrations Development19th October 2010
Medidata Solutions, Inc. Proprietary - Medidata and Authorized Clients Only. This document contains proprietary information that shall be distributed, routed or made available only within Medidata, except with written permission of Medidata.
![Page 2: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/2.jpg)
Introduction
General Considerations
Web Services
Conclusion
Agenda
![Page 3: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/3.jpg)
© 2010 Medidata Solutions Worldwide
Introduction
• Data transfers from EDC to SAS are typically cumulative (all data) transferred in batch mode (infrequently) in files formatted in a SAS proprietary format:
• How can we utilize modern technologies and standards to improve efficiency, reliability and information density?
3
EDC SAS
![Page 4: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/4.jpg)
Introduction
General Considerations
Web Services
Conclusion
Agenda
![Page 5: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/5.jpg)
© 2010 Medidata Solutions Worldwide
Cumulative data volumes in a clinical study
5
![Page 6: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/6.jpg)
© 2010 Medidata Solutions Worldwide
Average daily change
6
![Page 7: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/7.jpg)
© 2010 Medidata Solutions Worldwide
Cuumulative vs. incremental
7
Cumulative Incremental
Repeated re-transfer of unchanged data: inefficient and time-consuming
Only data changes transferred: maximum efficiency
Difficult to achieve real-time data transfer
Near real-time transfer possible
Entire data set always sent – no data lost if one transfer fails
How to recover from a transfer error: checksum/resend protocol required
![Page 8: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/8.jpg)
© 2010 Medidata Solutions Worldwide
Data transfer formats
Desirable features:• Support any clinical study design and data• Human-readable• Self-describing (metadata)• Support for incremental or cumulative transfers• Open, not proprietary, format
8
![Page 9: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/9.jpg)
© 2010 Medidata Solutions Worldwide
Data transfer formats and standards
9
Text SAS CDISC SDTM
CDISC ODM
All studies Y Y N Y
Human readable
Y N Y Y
Metadata N (Y) N Y
Incremental / cumulative
(Y) N N Y
Open standard
N N Y Y
![Page 10: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/10.jpg)
© 2010 Medidata Solutions Worldwide
CDISC ODM – Clinical data structure
10
![Page 11: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/11.jpg)
© 2010 Medidata Solutions Worldwide
CDISC ODM - Example
11
![Page 12: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/12.jpg)
Introduction
General Considerations
Web Services
Conclusion
Agenda
![Page 13: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/13.jpg)
© 2010 Medidata Solutions Worldwide
Web services
• Web services are application programming interfaces (API) that are accessed via the Hypertext Transfer Protocol (HTTP)
– Simple Object Access Protocol (SOAP) is one style– Representational State Transfer (REST) is our preferred
approach
13
![Page 14: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/14.jpg)
© 2010 Medidata Solutions Worldwide
REST
• REST uses HTTP methods or verbs:– GET– PUT– POST– DELETE
• To access objects via Uniform Resource Identifiers (URI)– https://innovate.mdsol.com/RaveWebServices/
studies/Mediflex/datasets/regular/AE
• Returning HTTP status codes:– 200 OK– 401 Unauthorized– 404 Not Found
14
![Page 15: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/15.jpg)
© 2010 Medidata Solutions Worldwide
REST in the browser - 1
15
![Page 16: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/16.jpg)
© 2010 Medidata Solutions Worldwide
REST in the browser - 2
16
![Page 17: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/17.jpg)
© 2010 Medidata Solutions Worldwide
REST in the browser - 3
17
![Page 18: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/18.jpg)
© 2010 Medidata Solutions Worldwide
REST from the command line - 1• curl -u username:password -H 'Content-Type:text/xml' -v
https://innovate.mdsol.com/RaveWebServices/studies/Mediflex/datasets/regular/AE
• * About to connect() to innovate.mdsol.com port 443 (#0)• * Trying 70.42.99.224... connected• * Connected to innovate.mdsol.com (70.42.99.224) port 443 (#0)• * SSLv3, TLS handshake, Client hello (1):• * SSLv3, TLS handshake, Server hello (2):• * SSLv3, TLS handshake, CERT (11):• * SSLv3, TLS handshake, Server finished (14):• * SSLv3, TLS handshake, Client key exchange (16):• * SSLv3, TLS change cipher, Client hello (1):• * SSLv3, TLS handshake, Finished (20):• * SSLv3, TLS change cipher, Client hello (1):• * SSLv3, TLS handshake, Finished (20):• * SSL connection using RC4-MD5• * Server certificate:• * subject: O=*.mdsol.com; OU=Domain Control Validated; CN=*.mdsol.com• * start date: 2007-03-28 17:49:39 GMT• * expire date: 2017-04-03 14:34:46 GMT• * subjectAltName: innovate.mdsol.com matched• * issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.;
OU=http://certificates.godaddy.com/repository; CN=Go Daddy Secure Certification Authority; serialNumber=07969287
• * SSL certificate verify ok.
18
![Page 19: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/19.jpg)
© 2010 Medidata Solutions Worldwide
REST from the command line - 2• * Server auth using Basic with user 'username'• > GET /RaveWebServices/studies/Mediflex/datasets/regular/AE HTTP/1.1• > Authorization: Basic ********************************• > User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8l
zlib/1.2.3• > Host: innovate.mdsol.com• > Accept: */*• > Content-Type:text/xml• > • < HTTP/1.1 200 OK• < Date: Tue, 14 Sep 2010 01:18:05 GMT• < Content-Type: text/xml• < • <?xml version="1.0" encoding="utf-8"?>• <ODM • FileType="Snapshot" • FileOID="96741552-97f4-4035-aad3-e9f12459ca20" • CreationDateTime="2010-09-14T01:18:05.255-00:00" • ODMVersion="1.3" • xmlns:mdsol="http://www.mdsol.com/ns/odm/metadata"• xmlns:xlink="http://www.w3.org/1999/xlink"• xmlns="http://www.cdisc.org/ns/odm/v1.3"• >• …• …• …
19
![Page 20: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/20.jpg)
© 2010 Medidata Solutions Worldwide
Incremental requests
https://innovate.mdsol.com/RaveWebServices/ studies/Mediflex/datasets/regular/AE? start=2010-09-01T15:00:00
ODM TransactionType:InsertUpdateRemove
20
![Page 21: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/21.jpg)
© 2010 Medidata Solutions Worldwide
REST and SAS Clinical Data Integration(CDI)
21
![Page 22: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/22.jpg)
© 2010 Medidata Solutions Worldwide
REST and SAS CDI
• GET list of studieshttps://innovate.mdsol.com/RaveWebServices/
studies/
• GET study metadatahttps://innovate.mdsol.com/RaveWebServices/
/studies/Mediflex/metadata/
• GET clinical datahttps://innovate.mdsol.com/RaveWebServices/
/studies/Mediflex/datasets/
22
![Page 23: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/23.jpg)
© 2010 Medidata Solutions Worldwide
Challenges
• Consistency– Incremental transfers are more efficient, but how can
the overall integrity of transferred data be assessed?– Hash functions (md5, sha1, etc) being investigated
• Metadata versions– To accommodate changes during a study (for example
a protocol amendment) CDISC ODM may have multiple metadata versions
– There are no constraints on changes between versions– Extra care needed to ensure that the correct metadata
version is applied to each data point
23
![Page 24: Using web service technologies for incremental, real-time data transfers from EDC to SAS](https://reader036.vdocuments.site/reader036/viewer/2022062422/56813af7550346895da37ea5/html5/thumbnails/24.jpg)
Introduction
General Considerations
Web Services
Conclusion
Agenda