data transport standard nathan chitty software architect nelnet april 24 th, 2007

Data Transport StandardNathan Chitty

Software ArchitectNelnet

April 24th, 2007

Trading Partner BDTS Service

Trading Partner BProcessing Application

Trading Partner AData Application

Trading Partner ADTS Client

Internet(https)

37

1 8

5 6

1. Data application creates data to be sent and hands off to DTS Client to send.2. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository.3. DTS Client makes request to DTS Service sending the data.4. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.5. DTS Service extracts data from DTS Request and hands off to a processing application.6. Processing application returns processed results to the DTS Service.7. DTS Service responds to the DTS Request with the processed results.8. DTS Client uses public key (step 2) to verify signature and then hands processed results back to the data application.

Repository Repository

2 4

• Topics– Data Transport Standard (DTS) Defined– History of DTS Project– Current state of DTS– Business Flow Implementations– Technical Architecture discussion

• DTS – Data Transport Standard – is a specification for an adjunct to or a replacement for existing data transport mechanisms:– FTP, “Secure FTP”, and/or Email– PGP / GnuPG encryption– SecretAgent

• What is DTS?– DTS is a specification not a product– DTS was developed by the Postsecondary

Education Standards Council (PESC) for exchanging data for:

• Inquiries• Reports• Transactions

• What is DTS?– DTS uses Internet technologies to facilitate

real time data exchange and transaction processing

– DTS builds on stable technologies, not specific products

– DTS, once implemented, reduces programming and per-transaction costs through standardization

• DTS Technologies utilized– WSDL (Web Services Description Language)– SOAP (Simple Object Access Protocol)– WS-I (Web Services Interoperability)– HTTP (Hyper Text Transfer Protocol)– X.509 Certificates– zLib compression– Other Web Services specs (WS-*)

• History of DTS - Topics– Original Business need/Justification– Business needs/requirements

• DTS SOAP:Header elements

– Version 1• Interoperability issues• Security and Public key management

• History – Original Need/Justification– Overcome inefficiencies of existing transports

• Commonrecord:Commonline (CR:C) would reach current “standard” limits

• Separate encryption/authentication steps from transport

• Delivery confirmations

– Industry desire to attain “real-time” processing

• Business Requirements– Guarantee Delivery– Support immediate and deferred processing– Accommodate various technical platforms– Highly secure– Support larger payloads– No distribution royalties– Based on open standards– Useful for internal and external communications– Single transport for all business needs without

evaluating payload

• DTS addresses delivery issues– Web service implementation

• Requires synchronous response– Status message– Processed result payload– SOAP:Fault

• No response from service = client error

– Order of delivery handled by client• guaranteed confirmation enables confirmed hand-

offs

• DTS is Highly Secure– Use of Secure Sockets Layer (SSL)

• HTTPS – same as online banking• Data encryption in the same step as transport

– No longer a separate step or application

– Use of Digital Signatures• X.509 certificate and certificate authorities• Strong authentication• Non-repudiation

• DTS can accommodate various technical platforms– Operating Systems

• Use of broader IT industry standards, not specific products• Actually proven to work with Java and .Net

– Size/Type of organizations• Client only implementations (smaller organizations)

– Push-Pull Deferred – Immediate

• Service and Client implementations– Push-Push Deferred– Immediate

• Additional Data Elements defined– Elements defined but the contents are left to

line of business

• Experience proved the necessity of certain information– DTS SOAP:Header elements

• Request Headers– DTSRequestRouting– DTSRequestServiceExpectation – DTSRequestPayloadType– DTSRequestPayloadBytes– DTSRequestSignature

• Response Headers– DTSResponseRouting– DTSResponseAcknowledge – DTSResponsePayloadType– DTSResponsePayloadBytes– DTSResponseSignature

• DTSRequestRouting and DTSResponseRouting– Routing Header elements

• sourceId• sourceIdSubCode • recipientId• recipientIdSubCode• UUID• transmissionDateTime

• Why SOAP:Headers?– Immediate/deferred processing

(ServiceExpectation / Acknowledge)– Larger payloads (PayloadBytes)– Single transport for all process without

evaluating payload (PayloadType)– Useful for internal and external transport

(Source/Recipient)

• Why SOAP:Headers? (cont.)– Map current EEAT filename components

• Filename– Format: A-B-X_Y_Z-M– Example: CRC01_REQUEST-02-DOE_004444_01-123

• DTS Header Elements– DTSRequestPayloadType = A– DTSRequestRouting

» sourceIdSubCode = X» sourceId = Y_Z» UUID = M

• DTS Version 1– Approved by PESC May, 2006– How did we do it?– Problems encountered

• Overcame

– Almost there!• Security and Key Management

• How did we do it?– Identified what and the proposed how– Created a working proof of concept for how

• Simple HelloWorld Service• Added simple type SOAP:Headers

– Not interoperable

• Added complex type SOAP:Headers– Not interoperable

• Interop problems with SOAP:Header– xsi:type attribute must be in header tag

• Java includes and requires this attribute by default– Null reference exception during deserialization

• .Net does not need the attribute

• Overcoming the Interop problem– SOAP transmitted across the wire of primary

importance• Element Name• Type Attribute

– .Net class declarations were augmented to force xsi:type attribute to be created during serialization

• Details are in the Reference Implementation Guide at http://www.pesc.org/workgroups/datatransport/

• DTSv1 Security and Key Management– Highly Secure

• Using SSL (https) for transport• X.509 Signatures

– Strong Authentication– Non-repudiation– Protects against man-in-the-middle and modification of

data

• DTSv1 Security and Key Management – However,– DTS Signature proprietary header element– Out-of-band certificate exchange required

• Key management and exchange practices not improved

• Current State of DTS (version 2)– New tools– No changes to interfaces– Reorganization of SOAP Headers– Integration of WS-Security (WS-S)

• New Tools– Recent versions of Axis Java toolkit removed

requirement of xsi:type• Reduced complexity for .Net implemenations• Easier inclusion of other WS-* specifications

– WSS4J and MS-Web Service Enhancements (WSE)

• WS-Security integration

• Reorganization of Headers (more like other WS-* specifications)

<DTSRequestRouting xsi:type="DTSRequestRouting" xmlns="http://www.datatransportstandard.com">

<UUID> <transmissionDateTime> <sourceID> <sourceIDCode> <recipientID> <recipientIDSubCode></DTSRequestRouting><DTSRequestPayloadType

xsi:type="DTSRequestPayloadType" xmlns="http://www.datatransportstandard.com">

<value>CRC01Request</value></DTSRequestPayloadType><DTSRequestPayloadBytes

xsi:type="DTSRequestPayloadBytes" xmlns="http://www.datatransportstandard.com">

<value>53</value></DTSRequestPayloadBytes><DTSRequestServiceExpectation

xsi:type="DTSRequestServiceExpectation" xmlns="http://www.datatransportstandard.com">

<value>Immediate</value></DTSRequestServiceExpectation><DTSRequestSignature

xsi:type="DTSRequestSignature" xmlns="http://www.datatransportstandard.com">

<value>someting</value></DTSRequestSignature>

<DTSRequestHeader xmlns:dts="urn:org:pesc:datatransport" soap:mustUnderstand="1" soap:actor="urn:org:pesc:datatransport/dts" xmlns="urn:org:pesc:datatransport">

<dts:DTSRequestRouting> <dts:DTSUUID> <dts:DTSSourceID> <dts:DTSSourceIDSubCode> <dts:DTSRecipientID> <dts:DTSRecipientIDSubCode> </dts:DTSRequestRouting> <dts:DTSRequestPayloadType> <dts:DTSRequestPayloadBytes> <dts:DTSRequestServiceExpectation> </DTSRequestHeader>

<wsse:Security soap:mustUnderstand="1"> <wsu:Timestamp> <wsu:Created> <wsu:Expires> </wsu:Timestamp>... <SignatureValue>…</wsse:Security>

Version 1 Version 2

• WS-Security– Removed proprietary Signature element– Replaced with WS-Security <Security>

complex element• Including <BinarySecurityToken> element

• Binary Security Token Element– WS-S specification allows for the X.509

certificate to be included in the transmission– X.509 Certificate must be signed by

Certificate Authority (CA)• No need to exchange public key out-of-band• Verification by chain of trust including the CA• Single point of contact for updates and revocation

• More Interoperability problems– .Net implementing WS-S via WSE

• Adds WS-Addressing elements to SOAP• Creates interop problem with java• Solution:

– Output handler in ASP Pipeline to remove WS-Addressing elements

• Current State wrap up– Version 1 approved (5/2006)– Version 2 with Change Control Board– Electronic Exchange Advisory Team has

published Technical Manual with DTSv1 implementation defining specifics

• very little will be changed for DTSv2

• Business Flow Implementations– Expected transactional flows





Internet(https)

37

1 8

5 6

1. Data application creates data to be sent and hands off to DTS Client to send.2. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository.3. DTS Client makes request to DTS Service sending the data.4. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.5. DTS Service extracts data from DTS Request and hands off to a processing application.6. Processing application returns processed results to the DTS Service.7. DTS Service responds to the DTS Request with the processed results.8. DTS Client uses public key (step 2) to verify signature and then hands processed results back to the data application.


2 4

Immediate




Internet(https)

1C1F

1A 1E1A. Data application creates data to be sent and hands off to DTS Client to send.1B. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository.1C. DTS Client makes request to DTS Service sending the data.1D. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.1E. DTS Service extracts data from DTS Request and stores it for guaranteed processing later.1F. DTS Service responds to the DTS Request acknowledging the data was received.


1B 1D

Trading Partner BDeferred Storage

“Push/Push” (part 1)


Trading Partner ADTS Service

Trading Partner BDTS Client

Internet(https)

2C2F

2A. Trading Partner B finishes processing the dat and hands off to DTS Client to send.2B. DTS Client looks up Endpoint URL (and public key) for Trading Partner A from their repository.2C. DTS Client makes request to DTS Service sending the data.2D. DTS Service looks up public key of Trading Partner B from their repository and verifies signature of payload.2E. DTS Service extracts data from DTS Request and stores it to process later.2F. DTS Service responds to the DTS Request acknowledging the data was received.

Repository

Repository

2D

2B

2A

2E“Push/Push” (part 2)




Internet(https)

1C1F

1A 1E1A. Data application creates data to be sent and hands off to DTS Client to send.1B. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository1C. DTS Client makes request to DTS Service sending the data.1D. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.1E. DTS Service extracts data from DTS Request and stores it for guaranteed processing later.1F. DTS Service responds to the DTS Request acknowledging the data was received.


1B 1D

Trading Partner BDeferred Storage

“Push/Pull” (part 1)




Internet(https)

2B2E

2D

2A. DTS Client looks up Endpoint URL (and public key) for Trading Partner B from their repository2B. DTS Client makes request to DTS Service asking for data.2C. DTS Service looks up public key of Trading Partner A from their repository and verifies signature of payload.2D. DTS Service looks for processed data for Trading Partner A.2E. DTS Service responds to the DTS Request with any data available, if no data is available the returns “Nothing to Provide”.2F. DTS Client verifies signature from Trading Partner B, and extracts payload for processing.

Repository

Repository

2A

2C

2F

“Push/Pull” (part 2)

• Reference Implementation Architecture– Client Application– Client Core– Service Core– Service Application

• Client Application– Knows nothing of SOAP or Web Services– Implements Client Core Interface

• “Setters” and “Getters” of DTS specific elements

– Houses specific business logic

• Client Core– Knows nothing of business logic– Uses properties set to construct the SOAP– Interface for “setting send” and “getting

returned” elements– Handles the communication to Service Core-

DTS Specification

• Service Core– Accepts transmissions from Client Core– Implements Service Application Interface

• “Setters” and “Getters” of DTS specific elements

– Creates return SOAP • Format return acknowledgement or data from

Service Application• Construct SOAP faults

• Service Core (cont)– Isolated business logic

• Examples– Invoke Service Application based on payload– Place payload in “queue”

• Service Application– Interface for “setting sent” and “getting to be

returned” elements– Houses specific business logic– Knows nothing of SOAP or Web Services

Internet

Client Application

Client Core Service Core

Service Application

DTS SpecDTS Spec

Entity A Entity BPlatform specific communication

Platform specific communication

Connecting the Layers

Internet

Client Application


Service Application

DTS SpecDTS Spec



Client Application


Internet

Client Application


Service Application

DTS SpecDTS Spec



Client Application

Client Application


Internet

Client Application


Service Application

DTS SpecDTS Spec


Client Application

Client Application

Service Application



ESB

Internet

Client Application


Service Application

DTS SpecDTS Spec

Point A Point B

Client Application

Client Application

Service Application

Platform, but DTS interface Platform, but DTS interface


ESB

Internet

Client Application


Service Application

DTS SpecDTS Spec

Point A Point B

Client Application

Client Application

Service Application

Platform, but DTS interface Platform, but DTS interface

ExternalClient Application


Questions?

Nathan ChittyNelnet

Software [email protected]

(904) 631-1308

Thank You!

data transport standard nathan chitty software architect nelnet april 24 th, 2007

Documents