EDI data in webMethods Trading Networks

Author:Narendra Thota

Date: 27-Oct-04

Introduction:

This BOK is based on the experiences of EDI Support & Maintenance Project. The project involves supporting & maintaining 8 EDI applications i.e. Invoice (Supplier 810), Advance Ship Notice (Supplier 856), Receiving Advice (Customer 861), Shipping Schedule (Customer 862), Planning Schedule with Release Capability (Customer 830), Purchase Order (Customer 850), PO Acknowledgment (Customer 855), Advance Ship Notice (Customer 856). The process includes communication with Trading Partners to get the Data, analyzing the data against ANSI/EDIFACT standards, testing the data in existing systems (Mainframes, Harbinger, webMethods), coming up with new webMethods mappings depending on requirements.

The content is organized in to three sections: 1) EDI data format/Envelope structure 2) Setting up the Trading Partner profiles in webMethods Trading Networks 3) Running sendToTN service and validating the results.

EDI data format/Envelope Structure

ANSI X12 and EDIFACT are most widely used EDI standards. This section talks about the structure of ANSI X12 transaction sets. ANSI is an abbreviation for the American National Standards Institute that has been coordinating standards in the United States since 1918. ANSI offers an open forum for all concerned to identify their needs, plans to meet those needs, and agreement on the proposed standards. The Institute has a number of committees including the ANSI Accredited Standards Committee X12 (ANSI ASC X12). This committee is a voluntary standards committee that consists of subcommittees representing both private and public sectors in many industries. The subcommittees use a consensus process to propose a new standard or changes to existing standards. These standards enable the electronic exchange of business transactions. The standard that has been recommended by this committee is known as the ANSI ASC X12 standards. It is sometimes called the ANSI X12 standard or simply the X12 standard.

WHAT IS A TRANSACTION SET?

A transaction set is a single business document such as a Purchase Order, Invoice, orShipment Notice. There are hundreds of Transaction Sets available in the ANSI ASC X12 standards. Each set of transaction data is identified by a three digit code number.

Many transaction sets have three parts. a) Header Area, b) Detail Area, c) Summary Area. Segments that may be used in each of these parts, within a specific document (i.e., invoice), are specified in associated tables defined in the X12 Standards document.

Transaction Sets Begin with the Transaction Set Header (ST) segment and End with the Transaction Set Trailer (SE) segment. These two segments are the innermost level of the three levels of envelopes within an ANSI transaction.

ENVELOPE STRUCTURE

For every transaction there are three levels of enveloping: Transaction Set Functional Group Interchange

The following diagram identifies the three levels and how they relate to each other:

ISA-IEA: Interchange EnvelopeGS-GE: Functional Group EnvelopeST-SE: Transaction Set Envelope

TRANSACTION SET ENVELOPE

The innermost level is the Transaction Set identified by the ST/SE segments. The ST segment always has two data elements. A third data element is optional for version/releases 004020 and later. They are:

Transaction Set ID (e.g., 850) Control Number (e.g., 1001) Implementation Convention Reference (e.g., X099) optional and available for

Version/releases 004020 and later

The SE segment contains the Number of Included Segments in the transaction set and the same Control Number as the ST segment. The Transaction Set ID identifies the transaction set being enveloped using the three-digit Transaction Set ID code.

Examples of these codes are: 850 Purchase Order 810 Invoice 997 Functional Acknowledgement

The Number of Included Segments and Transaction Control Number provide data integrity for transaction control via segment counts and control numbers. ST02 should always be equal to SE02 in a valid transaction data.

ST~850~0001SE~14~0001FUNCTIONAL GROUP ENVELOPE

The second (middle) level of enveloping is the FUNCTIONAL GROUP Envelope. Its purpose is to group similar types of Transaction Sets within a transmission. The definition of ‘similar’ varies by the version/release of ANSI ASC X12 being used. For example the November 1987 (and prior) release for the PO group may contain the Planning Schedule (830), the Purchase Order (850), the Purchase Order Acknowledgment (855), the Purchase Order Change (865), the Order Status Inquiry (869) or the Order Status Report (870).

For later versions/releases and for new Transaction Sets that are being developed, most

ISA

IEA

GS

GS

GE

GE

STSE

STSE

STSE

STSE

Transaction Sets are being assigned to a unique Functional Group. The Functional Group Envelope is defined by the GS/GE segments. The GS segment has a number of data elements. An example of some is:

Functional Group Set ID Format and Version of the document, and Date/Time Stamp Numbers

The Functional Group Set ID identifies the function group set being enveloped using a two character Functional Group Set ID code. Refer to the Standards manual for a complete list of Functional ID codes. Examples of these codes are:

PO Purchase Order SH Shipment Notice/Manifest FA Functional Acknowledgment

The Transaction Sets Count and Functional Group Control number provides data integrity for transaction control via the transaction. The Format and Version of the document are used to identify the ANSI ASC X 12 releases being used. Example: X for the format and 004010 for the version/release. GS06 should always be equal to GE02 in a valid Functional Group data.

GS~PO~0247921940100~ARROW2~20041025~1816~344~X~004010GE~1~344

INTERCHANGE ENVELOPE

The outermost level is the INTERCHANGE envelope that is defined by ISA and IEA segments. The ISA/IEA data elements start with the letter I and are found at the end of the data element dictionary in the Standards manual. The Interchange envelope encloses the data from one sender to one receiver. The ISA is a fixed length segment. Some items contained in the ISA/IEA segments are:

Structured mailbox addresses of the sender and receiver Interchange control numbers Counts of the functional groups within the interchange Time/Date stamp (similar to functional group, but does not include the century) Version of the interchange envelope Characters in the ISA segment used for data element separators, sub-element separators, and

segment terminators

ISA05 is Interchange Sender Qualifier; ISA06 is Interchange Sender Id, ISA07 is Interchange Receiver Qualifier and ISA08 is Interchange Receiver Id. ISA13 should always be equal to IEA02 in a valid Interchange data.

ISA~00~ ~00~ ~01~024792194 ~ZZ~ARROW2 ~041025~1816~U~00401~000000772~0~T~:IEA~2~000000772

Setting up the Trading Partner profiles in webMethods Trading Networks

Partner Profiles

In Trading Networks, partner profiles are set up to contain all the descriptive information about a trading partner, including the partner's name, address, contact information, External ID numbers (Duns, GSID, etc), and classification code (Customer/Supplier).

Document Types

Document Types are used to recognize the type of document that is being submitted to Trading Networks, whether it is an inbound EDI or RN document, or an outbound EDI or RN Document. Inbound and Outbound document types are set up using samples of the data being recognized. For inbound EDI documents, Document Types are set up to recognize the EDI Envelope; for inbound RosettaNet PIPs, Document Types are set up to recognize the root tag and version. For outbound documents, Doc Types are set up to recognize the application layout of the outbound request queue.

If a document comes into Trading Networks and it is not recognized by a Document Type, Trading Networks will ignore the document and it will not be processed.

Processing Rules

Processing Rules contain the directions for how the document will be processed. Separate rules are set up for each different version of EDI transactions and RN pips. If a Processing Rule is not found to match the criteria, the default processing rule will be executed.

Transaction Analysis

Transaction Analysis is display of all the transactions that have been sent or received by Trading Networks. Transactions can be queried by using sender/receiver information, document type, date range, and processing status.

Activity Log

The activity log is a listing of each action taken within Trading Networks. It also lists the date, time, and user who took the action.

Agreements

In our case, the Agreement is set up specifically for Arrow. For our purposes, it is only used for EDI inbound transactions to determine the Routing Mode (GS Only, which means we recognize a trading partner by the GSID) and the rejection level of each different functional group. How GS Routing and rejection is used will be shown later in this document.

Inbound Document Processing Through Trading Networks

Inbound document processing begins with the document arriving into Trading Networks and ends when the document is delivered to the backend system in the appropriate application format. A number of steps are taken to complete this process, which can be encompassed by the following three general areas:

1. The document is recognized by trading networks and associated with a Document Type2. The Sender and Receiver ID's are matched against the valid trading partner setups3. The Sender ID, Receiver ID, and Document Type are used as criteria to match to a Processing Rule that

will kick off the service that will process the document and send it to the backend system.

Default Agreement:Ÿ GS Routing OffŸ Sender Qualifier: *Ÿ Receiver Qualifier: *

Trading Partner ProfileŸ GSID Pairs matched in

TNŸ Gets Sender &

Receiver Internal TNId's

Inbound GenericService Invoked

Processing RuleŸ Processing Rule is

matched on:Ÿ TN Internal Sender &

Receiver ID'sŸ Docment TypeŸ EDI Functional Group

(GS01)Ÿ Transaction Version

(GS08)

TN uses Sender &Receiver ISA's to

match anAgreement.

Ÿ ISA Sender IDŸ ISA Receiver IDŸ Production Mode (ISA15)Ÿ GS Only Routing ModeŸ Sender Qualifier: GSIDŸ Receiver Qualifier: GSID

Envelope Table:Ÿ Checks Production ModeŸ Matches ISA SID/RID PairŸ Uses qualifiers from Agreement to get GSID

Pair

Inbound EDI Transaction Flow

EDI DocumentReceived in

Trading Networks

Default AgreementSelected

Document TypeDocument is

recognized byDocument Type as

EDI

Mainframe

EDI Inbound Processing

When EDI documents are received into TN they follow the path outlined below.

1. EDI Inbound document is received and submitted to Trading Networks.

2. The X12 Envelope Document Type and the X12 Group Document Type both recognize the data as EDI.

3. Since the document is now recognized as EDI data, the sender and receiver ISA ID's are read and TN tries to match them to an Agreement. Arrow currently uses only one default agreement. When the TN cannot match to a trading partner specific Agreement, the default Agreement is selected.

4. The default Trading Partner Agreement (TPA) contains some of the instructions the document needs to process:

The GS Routing Mode is set to "OFF" and the senderQualifier and receiverQualifier are both set to *. This means that the document will not follow the routing mode set in the TPA, it will rely on what is set up in the interchange table (see step X below).

persistMultipleDocEnvelope is set to false. Arrow does not use this setting. In a case where a single Van file is submitted to TN, shis setting specifies whether to save the original EDI document to the Trading Networks database (in it's entirety). Since we submit separate ISA enveloped documents into TN, this setting is not needed.

processingMode is set to "Interchange Defined." We have set it up this way to use the processing mode sent on an inbound document in the ISA15 field to match against the processing mode set up in the Interchange table.

splitOption is set to "Group" so that only X12 group documents are sent to TN for processing. If this setting were set to "Document," we would process everthing at a document level (instead of a group level).

InRejectLevel - Reject levels of all inbound (and outbound) documents are set up in the TPA.

5. Since Trading Networks is set up on a GSID level for EDI, the default TPA is set up to route the document to the interchange table for routing instructions. Using the ISA envelope information, the document is matched to an Arrow-Trading Partner pair in the interchange table. Once a match is made, it takes the "Qualifier" to use to match on a group pair in the Group table.

6. The GSID pair from the Group table is sent back into Trading Networks to match against the Trading Partner Profiles, which are also set up with "GSID" qualifiers. Each Trading Partner Profile has a webMethods generated internal id.

7. When the GSID pairs are matched, these internal id's, the Document Type, the functional group code (GS01) and the version (GS08) are used to find a match with the criteria set up in the Processing Rules. If a processing rule fitting the criteria is not found, the document will be processed according to the

processing indicated in the Default Processing Rule. The Default Processing Rule does not kick off a service, it simply changes the document status to "Ignored" and the document does not get processed.

è It is important that the processing rules be ordered in an efficient way because they are processed in a top-down approach. They need to be ordered by specific criteria to generic criteria by transaction type. Note that the Default Rule is at the bottom of the list. This is because if the Default Rule is hit first, the document will not be processed.

8. Once the Processing Rule is matched, it invokes the services that process the document and send it to the back end systems. For EDI, Processing Rules are set up based on the Sender ID, Receiver ID, and Document Type on a Group Level:

This processing rule will accept match an Inbound EDI 810 from any sender, but with only Arrow Electronics, "Arrow C" and "Arrow G" as receivers, and only for the Document Type X12 Group.

If a document meets these criteria, additional matching criteria must be met:

This Processing Rule will only be invoked if the EDI Group Type (GS01) is equal to "IN," and the EDI Version (GS08) is equal to "4010."

All EDI Processing Rules must be set up with Extended Criteria including the EDI Version and the EDI Group Type.

Once the Processing Rule is selected based on the above criteria, the appropriate services will be invoked:

In this case, the driver called ibEDIProcess is a generic service which invokes the map based on the inputs for the specific business group, map driver, doc type, application schema name, pip schema name, and out file format.

RosettaNet Inbound Process

RosettaNet inbound PIPS process differently from EDI in the following ways:

- RN Trading Partners are set up with Duns numbers (as opposed to GSID)

- Because they are not GSID driven, the generic inbound process does not access the Trading Partner Agreement or the Envelope Table.

- When a RN document is received, it is submitted to TN from Viacore through the receiveInner service, where it is recognized by a Document Type and matched with a Processing Rule.

For RosettaNet, the document is recognized by the Root Tag, along with the Payload Version of the document. In the following document type setup, Trading Networks will recognize a document that has a Root Tag of Pip3A4PurchaseOrderConfirmation, and a Payload Version of V02.00, as being a PIP3A4vV02.00 Purchase Order Confirmation:

Once the inbound document is recognized, Trading Networks searches for a Processing Rule to see how the document should be processed.

The selected Sender ID, Receiver ID, and Selected Document Type are set up to match the criteria on inbound documents. RosettaNet has no additional match criteria.

Once the processing rule recognizes the sender, receiver, and document ID, a generic inbound service is kicked off based on the inputs for the specific business group, map driver, doc type, application schema name, pip schema name, and out file format:

These are the inputs to the services that process the document and send it to the mainframe.

Outbound Document Processing Through Trading Networks

Outbound document processing begins with the document arriving into Trading Networks in an Application format and ends when the document is delivered to the Trading Partner in the appropriate format (EDI Wrapped Data; RN XML). Outbound document processing follows the same basic procedure through Trading Networks as it follows in inbound document processing:

1. The document is recognized by trading networks and associated with a Document Type

2. The Sender and Receiver ID's are matched against the valid trading partner setups

3. The Sender ID, Receiver ID, and Document Type are used as criteria to match to a Processing Rule that will kick off the service that will process the document and send it to the Trading Partner.

RN Outbound Processing

While EDI uses the GSID as the External ID Type, RosettaNet Trading Partners are set up with the Duns #'s as the External ID.

When RN documents are received into TN they follow the path outlined below:

1. Application data is picked up from the Queue and submitted to Trading Networks, along the Sender ID, Receiver ID (Duns #) and a "Doc Type" value:

2. These values are compared to the Document Types set up for outbound documents. When a match is found on the Doc Type value, the Sender and Receiver ID's are extracted from the data and used to match on a Processing Rule.

3. The Sender ID, Receiver ID, and Doc Type are matched against each processing rule in a top-to-bottom order. When these criteria are matched with a Processing Rule, the rule will invoke the RN generic service which will in turn kick off the map and convert the data to XML:

4. Once the data has been converted to a RosettaNet PIP, it is sent to Viacore.

EDI Outbound Process

EDI Trading Partners are set up on a GSID level. Our outbound applications use internal and external id's to indicate the internal division from which Arrow is sending, and to indicate to which division of the trading partner we are sending.

Application data contains a code for both Arrow's and the trading partners internal divisions. For the trading partner, we are calling it the Application ID.

When EDI transactions are received into TN, the follow the path outlined below:

1. When Application data is submitted to the request queue, it contains the Sender ID (Arrow's internal ID), the Receiver ID (the trading partner's Application ID), and a "Doc Type" Value. The service that picks up the data from the request queue uses the Application ID to match to a trading partner in Trading Networks, and gets the GSID that is cross-referenced in the trading partner External ID table:

In this example, the Application ID 987654321 is submitted to Trading Networks. It matches with the APP External ID for the trading partner with GSID 123456789. The service returns the GSID 123456789 to the pipeline to continue processing the document.

This service performs the same function as the External Cross-Reference table in Harbinger.

2. The document is then submitted to TN with the Trading Partner GSID, the Arrow GSID, and the Doc Type. The Doc Type value is compared to the Document Types set up for outbound documents, and when a match is found, the Sender and Receiver ID's are extracted from the data and put into the pipeline.

3. The Sender ID, Receiver ID, and Document Type are matched against each Processing Rule in a top-down order. When the criteria are matched with a Processing Rule, the rule will invoke the EDI generic service which will in turn kick off the map and convert the data.

4. Since Trading Networks is set up on a GSID level, ISA enveloping information is stored in a separate table that is accessible from the webMethods Integration Server:

The Interchange Info table contains an ISA pair. Within the ISA pair is a GSID pair.

5. When the document is ready to be enveloped, a service runs that will submit the Sender and Receiver GSID's to the Interchange table to get the ISA information needed to wrap the data:

After matching on the GSID pair, the service picks up the delimiters and enveloping information,

… as well as control number information ….

Running sendToTN service and validating the results

sendToTN service takes the EDI data (EBCDIC format or ASCII format) as input and submits the control to Trading Networks. The information in the EDI data i.e. sender ISA/GS id, Functional Group code, and version number is compared against the list of processing rules in the Trading Networks. Accordingly a suitable map driver will be chosen to process the data . Generated application data will be sent to MQ. The JCL programs extract the data from the queue to process a series of screens.

The status of the sendToTN run gets recorded in Ecommerce Toolkit (EC Toolkit). If the input EDI data complies with standards and mappings are done correctly, status will be “IB Doc sent to MQ”. Non-compliance of the data results in error and segment/data element details are shown along with error code. If Trading Networks is not set up for a Trading Partner (TP), Transaction then the sendToTN run results in Invalid/Sender Receiver Id.