Download - Application Engine ETL
Application Engine ETL – Direct access to external databases from PeopleSoft made easy
Session #08714Ocotber 6, 2011
Your Presenters David Sexton PeopleSoft ver. 5.01 Senior Systems Analyst “Application Engine ETL – Direct access to external databases from
PeopleSoft made easy”, presentation at HEUG Alliance ’11, March, 2011.
“Justifying a Data Governance Initiative” presentation at HEUG Alliance ’11, with Lumen Data, March, 2011.
“Justifying a Data Governance Initiative” presentation at Oracle Open World ’10, with Lumen Data, September, 2010.
“Putting the Service into SOA – Consuming WSDLs in PeopleSoft”, presentation at HEUG Alliance ‘09, Anaheim, March, 2009.
“Web Services for PSQuery”, presentation at Oracle Open World, September, 2008.
“Web Services for PSQuery”, presented at HEUG Alliance ‘08, March, 2008.
“Extending Applications with Java”, presented at HEUG Alliance ‘07, March, 2007.
“Real World Web Services Integration from PeopleSoft”, presented at HEUG Alliance ‘07, March, 2007.
University of Utah
28,000 students 18,000 employees Joining PAC 12 2002 Olympic Opening and
Closing Ceremonies Tier 1 Research Instition Medical School and Hospital 30 minutes from….
+ =
PeopleSoft Timeline Sep 97 : SA Campus Community / Admissions / Recruitment
Jul 98 : FS Purchasing / Accounts Payable / General Ledger
Oct 98 : Human Resources
Apr 99 : Go live HE Upgrade 7.0 to 7.5 / Time & Attendance (KRONOS)
May 99 : HR Base Benefits / Payroll
Apr 00 : HE HR/SA Upgrade 7.5 to 7.6 / SA Financial Aid
May 00 : SA Student Financials
Feb 01 : FS Upgrade 6.0 to 7.02
Oct 02 : HE HR/SA Upgrade 7.6 to 8.0 / Enterprise Portal 8.4
Dec 03 : Upgrade 8.4 to 8.8 Enterprise Portal
July 04 : FS Upgrade 7.02 to 8.4
Mar 06 : HE HR/SA Upgrade 8.9
Feb 10 : FS Upgrade 8.4 to 9.0
Oct 10 : HR/SA Upgrade 9.0
PeopleTools 8.50.12
We have been working with PeopleSoft for awhile
!!!!!! Disclaimer !!!!!! This presentation is based on our
experience This presentation contains techniques
only endorsed by us, and not necessarily by our employers
Hopefully this presentation encourages you to share your experience at next year’s Openworld conference.
If you have questions, please ask!
Overview
What is App Eng ETL ? Why we developed it
Why not SOA/ Web Services Advanced Use Case What is required DEMO Where you can get it
App Eng ETL – what is it
We create an application engine program that leverages the PeopleSoft record object to dynamically create insert statements. These statements are executed against the external database through a JDBC connection which is instantiated in PeopleCode.
App Eng ETL – what is it
Transform the data in the PeopleSoft record/view
Extract the data from PeopleSoft using the SQL and Record objects
Load the data using a JDBC connection to the external databse
App Eng ETL Why ?
Started MDM/Data governance/ Integration project Nov 2009
Session Justifying a Data Governance Initiative: Get Approval the First Time #29312 Monday Mar 28 4:30 – 5:30
People want Data ! The more the better
We wanted something that was Easy to maintain/reusable Easy to consume
App Eng ETL Why not Web Services?
SOA requires public services, data sets your organization has declared public
https://wiki.doit.wisc.edu/confluence/display/EBOS/Repository+Home
We’re not there yet Each data request must be approved individually
Each integration is point to point
Consuming web services (wsdl) is not a common skill set (on our campus)
People are excited about it, but can they use it today?
Thank You to Jim Phelps at Wisconsin
App Eng ETL Why not Web Services?
Web Services are best suited where the data request is high frequency
And the response data is low bandwidth Good use of web services
App Eng ETL Why ?
• Build an SQR• Creates a file• Get permissions on the destination server• SFTP the File
Destination does
We do
Custom process to load the file into their database
We do
• Create View• Configure – 1 page• Run a process
Destination doesNothing – data shows up in their data base
App Eng ETL Configure
App Eng ETL Run
App Eng ETL Positives
All integration objects are managed/show on compare reports SQL is in a view – provides a layer or abstration, should
not be impacted by upgrades All integrations are easily visible Low cost per additional integration
No additional coding, configuration only Less moving parts
Customers LOVE it !
App Eng ETL what is required ?
1) Server name where external DB is, DB sid
Jeff.Robbins.awesome testschema
2) Username/Password for the external DB
3) /PSHOME/Appserv/Classes directory contains
driver manager (“java.sql.DriverManager”) The MySql driver (“com.mysql.jdbc.Driver”)
App Eng ETL what is required ?
4) Record/View in Peoplesoft containing the data to be integrated, all fields
will be sent
5) A record on the external database with the same structure as the PeopleSoft record/view
field names, field types, field lengths of the source and target table must match EXACTLY Generate the DDL from the PeopleSoft
record/view, send to external database
Demo configure and run
WHAT ABOUT SECURITY?Gee Dave, this sounds great, but is the
transmission of the data secure? Muliple options to secure the tranmission
Oracle Wallet Talk to your Sales Rep Encyrption options at the JDBC level Source and destination must have Oracle Wallet
Other Vendors supply JDBC drivers that have encrytpion capabilities
Customs options we have investigated Custom Java using ports and SSL SSH tunneling with JDBC*
http://www.ibm.com/developerworks/data/library/techarticle/dm-0312lurie/index.html
SSH Tunnel
Key Code PiecesSSH script
# Referenced in UU_DATA_PUSH application engine. Used for # integrating PeopleSoft data to remote systems. This script # sets up a secure tunnel to a remote host to connect to a # remote database securely. local_port=$1 remote_host=$2 remote_database_host=$3 remote_database_port=$4 ssh -f -g -L $local_port:$remote_database_host:
$remote_database_port $remote_host sleep 60
Key Code Pieces Call SSH script from PeopleCode
&script_path = &path | "ssh_tunneling.sh"; &local_port = &properties.get(UU_DP_RQST_AET.URL | ".local_port"); &remote_host = &properties.get(UU_DP_RQST_AET.URL | ".remote_host"); &remote_database_host = &properties.get(UU_DP_RQST_AET.URL |
".remote_database_host"); &remote_database_port = &properties.get(UU_DP_RQST_AET.URL |
".remote_database_port"); &shell_command = "/usr/bin/ksh -c """ | &script_path | " " | &local_port | " "
| &remote_host | " " | &remote_database_host | " " | &remote_database_port | """";
&processId = Exec(&shell_command, %Exec_Asynchronous + %FilePath_Absolute);
Questions?
Contacts David Sexton
Senior Systems Analyst
University Information Technology
University of Utah