© 2010 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property of their respective owners.

Best Practices at AT&TOracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite

Bryon Rickey

Ravi Kayarthodi

September 20th, 2010

Agenda Welcome and Introductions

Supply Chain Management (SCM) environment

Overview/highlights

Architecture

Partnership with Oracle to transform our Business

Key project highlights

Future initiatives

Solution Support Center (SSC) and on-site support

Q&A

About AT&T

AT&T is one of the world’s largest Telecommunications companies

AT&T Global Headquarters is based in Dallas, TX

AT&T has approximately 290,000 employees

AT&T has more than 90 million wireless customers

AT&T has more than 16 million high speed internet subscribers

AT&T has more than 2.5 million uverse subscribers

Exclusive U.S. provider of iPhone 3G / 3GS / 4

AT&T SCM: Architecture highlights

Oracle E-Business Suite Centric Scale-up with cluster database on very large

infrastructure High availability with Oracle Data Guard, Oracle

GoldenGate, and site level redundancy Central inventory/financial management; utilizes

product hub, planning/forecast with ASCP/Demantra Order orchestration and brokering with in-house

developed Oracle based application system Internet deployment for interaction with external

suppliers Management reporting/profitability analysis;

procurement and spend analytics Industry leading warehouse/distribution

management systems

AT&T SCM Platform: Financial Integration Both Network and Retail SCM - financial

counterparts are integrated in the same platform: Shared AP, AR, PA, FA, and GL

Supply Chain transactions leverage same referential data and accounting rules

Financial integration of supply chain transactions are seamless and are recorded real-time in the general ledger

No need for reconciliation between sub ledgers Ability to forecast financial implications based on

SCM forecasts Online financial budgeting validation on

purchasing and requisitions Analytical GL Cube and Inventory Cube

Results

AT&T has saved more than $500 million and has improved its purchasing power by combining accounts with its vendors.

Estimated to save $1B in inventory costs, and $300M per year in reduced operating costs.

Warehouses have been able to keep inventories low without missing SLA, resulting reduced overall inventory costs.

Ability to identify sold merchandise, resulting in fewer fraudulent returns

Industry Recognition

2010 IDG’s CIO AwardRecognizes organizations around the world that

exemplify the highest level of operational and strategic excellence in information technology

Systems Architecture

AT&T SCM: E-Business Suite Centric

Oracle E-Business Suite 11i

Other Billing Systems

HR - Employee

EDI

Warehouse/Distribution System

Asset Tracking

Direct Fulfillment

Spreadsheet

Lease Management

Retail Planning

Construction

General Ledger Receivables

Accounts Payable Purchasing

Inventory

Order Management Assets

AT&T Corporate Systems

Tax

Projects

Archive Returns

Procurement partners

COR Stores

Planning/ Forecasting -

Network

Order Broker/Orchestrator

Legacy Procurement

Provisioning

Other Channels

External Facing; Suppliers and

Auctions

Technician Ordering

SO

AM

idd

lew

are

Procurement

3PL portal

TMS

Demand Planning

Profitability Analysis

Procurement & Spend Analytics

AT&T SCM: Key Systems Architecture

Oracle E-Business Suite 11i

Inventory Planning Purchasing Financials

Corporate Asset TrackingDistribution Center

Warehouse Mgmt Inventory Tracking

ReceiptsReceipts

Shipments

EDI EDI

Audits

Repairs

APPOODP eAuction GL

FAPAASCPOM RNF

Moves

OpusOpus

Oracle 11i E-Business Environment

Dealers/iStore

Dealers/iStore

Store Replenishment

Store Replenishment Direct FulfilmentDirect Fulfilment

National Retailers

National Retailers

DSL and uVerse HSIA

DSL and uVerse HSIA

StoreStore XBM/InsuranceXBM/Insurance Dealer/NRDealer/NRService DepotsService Depots SalesSales ReturnsReturns

Order Management

Order Management

Forward Orders

UpfrontUpfront

Reverse Company Owned Retail

416 CPU Cores & 1.7TB of Memory in DB Cluster

14TB of storage for transaction DB

3TB of storage for Archived DB

PlanningPlanning PurchasingPurchasing InventoryInventory ReceivablesReceivables FinancialsFinancials

Oracle E-Business Suite 11i Apps

11i DB

EPLEPL

RU30RU30

AT&T SCM Platform: Oracle10g RAC Infrastructure: VCS+ CRS

AT&T SCM Platform: Transaction Volume

Approximately 3.35 million logons to the production areas per day

2.06 billion SQL executions per day Approximately 136.5 Billion logical reads and 41TB

of physical reads performed per day Approximately 7 TB of writes performed per day Approximately 1.05 billion user calls and 52

million transactions completed per day 200,000 concurrent request per day

AT&T SCM Platform: Operational architecture

Decoupled month-end book closing preparation Statistics gathered on a clone

Separate reporting environment – refreshed nightly; Inventory and GL Cubes for reporting

Revenue share and allocation on a separate system

Direct supplier interaction with Internet-deployed application modules

Utilizes market making mechanisms (sourcing)

AT&T SCM: Oracle 11i Dev/Test Configurations

PRODL-TRA

RPT

R-TRA

DEV

QA

DEV

DEVQA

DEV

DEVQA

DEVDEV

QA

DEV

Data Guard

Data Guard

Flex Clone

FTP

HDS – USP-V HDS – USP-VBothell, WA

MET1

Shadow Image

NetAppGOLD

Data Guard/

Custom Shipping

NetAppMaster

Flex Clone

MET2

Shadow Image

NetAppGOLD

NetApp

Master

Data Guard/Custom Shipping

Flex Clone

Flex Clone

Shadow Image

SnapMirror

Alpharetta, GA

FTP

Storage Efficiency with NetApp de-duplication Tools Performance test environment on tier-1 storage

AT&T SCM: Oracle Automatic Storage Management Local and Remote DR databases and also

Reporting databases have been migrated to ASM storage to meet storage and capacity needs Avoided SFRAC costs Performance

SCMP DR/BC database converted to RAC database Saved SFRAC license cost on E25K frames

Comply with AT&T standard for cluster database (ASM is required)

Expected to save cost in the future when we upgrade other Corporate Systems’ databases

AT&T SCM: Archive and Purge

Archive Stats 16 custom and core modules

including (GL, INV, OM, AR, WSH, PO, FA)

Executed weekly, monthly, quarterly, and yearly

History instance Informatica Data Archive

application used Schedule is monitored real-time

to ensure adherence

Purge Stats 70 jobs via Oracle concurrent

programs Executed daily and weekly Schedule is monitored real-time

to ensure adherence

AT&T Supply ChainGeographic Distribution – DB Flow to Test and Dev

XX

XX

Prod

Dev/Test

DR Site

Test Site 1

Development Site 1

Primary Site

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XX

XXXX

XX

XXXX XX

Development Site 2

XX

XXXX

XX

Test Site 2

XX

XXXX

XX

AT&T SCM: Spend Analytics

Top 5 Categories

Top 5 Suppliers

Line Of Business

Spend

Contract Leakage Off Contract Spend Payables Leakage

Oracle Business Intelligence Enterprise Edition (Oracle OBIEE) Powered

Sourcing from Oracle E-Business Suite and Legacy Applications

AT&T SCM: Database Maintenance/Performance Management

Monthly month-end close of books stats validation – database statistics preparation on production clone

Table Re-organization – routine and ad-hoc Periodic index rebuild Daily performance call Daily/weekly/monthly/yearly purge/archive

schedules – vigilance in adhering to them Extensive performance monitoring on all levels

of the system

Partnering with Oracle Advanced Customer Service

Oracle Solution Service Center (SSC)

SSC: Project Spotlight: Patch Recommendations

23

Project Activities Description

o Identification of available latest required patches for AT&T environment

An initiative to incorporate E-Business Suite recommended patches, such as, Rollup (RUP) in AT&T’s release schedules for 2010o Participation in meetings to review

recommendations

Value Addition SSC and AT&T worked together to create a new process to efficiently identify required

patches with the goal to minimize the impact of rolling them out in our production environment and to current/future projects: Quarterly document for AT&T environments with the latest recommended patches. Participation on AT&T IT meetings to clarify recommendations and questions Provide detailed patch analysis on a ad-hoc basis

Successful roll out of AP.O patchset along with other Financials patches in February 2010: stabilized production environment and brought environment to the latest code base.

Successful completion of the ASCP Rup 37, Collections Rollup (RUP) 32 and Application Technology Group (ATG) RUP-7 for the April 2010 production rollout.

SSC: Project Spotlight: Performance Improvement

Project Activities Description

o Understand AT&T batch window requirements and transaction volumes

An initiative to address various performance issues related to Month-End-Close processes and other batch processes after Oracle 10g database upgrade.

o Identification and root cause analysis of problem areas

o Implementation of resolutions

Value Addition Active participation on Conference bridge calls and OWC sessions. Engage Oracle development groups (Financials and Apps Performance) to provide SQL

tuning improvements. Improve month end close activities like AR close, depreciation and asset retirement,

optimizer, Invoice Validation processes. Maximize performance during month end, preventing delay; AT&T was able to successfully

meet the SLAs for the Month End Close Processes month after month. Provide recommendations for major upgrades (DB, Applications and Tools)

SSC: Critical SR Spotlight: Other Key areas of focusProject Activities Description

o Workflow background process

WF_DEFERRED queue growing exponentially; AT&T engages SSC to evaluate and address the issue.upgrade.

o RAC – CRS Configuration

Collect diagnostic and configuration dataTroubleshoot frequent RAC node reboot/performance issue post upgradeDiscover unforeseen problemsValidate AT&T configuration with RAC Best Practices

o Device Life Cycle Database UpgradeProvided upgrade expertise and support (pre and post deployment for 11g R2 upgrade.

Value Addition Active participation on conference bridge calls and OWC sessions Review current setups and schedules of workflow Background Process Engage on Severity-1 SR and work collaboratively with Oracle development to provide workaround (eg.

skip script) Work with at&t and Oracle development organization to provide scripts to clear workflow DEFERRED

queue Monitor the implementation of recommended suggestions Engage Oracle Development and RAC COE team Engage on Architectural design discussions Stand-by support for DB/RAC upgrade deployment activities Ensure successful implementation of RAC upgrade Rollout

SSC: AT&T Targeted Services: Operations The Services Yield to the Customer targeted for the next 6-month planning period is comprised of key Support activities, aligned with key customer objectives/activities towards achievement of AT&T’s Account Goals

Supply Chain Transformation (Streamline Operations for newly deployed modules)

Work with ACS sales to determine appropriate support for upcoming SC rollout. Determine/secure funding for new modules (Demantra and Transportation Management Suite)

Develop, Enhance and Secure development sponsors Participate in scheduled project “Deep Dive” calls with PMO and project team Site visit to Memphis to understand warehouse and logistic operations – understand

business functions. Supply Chain Business – develop key relationships with Retail and Network business users

11i Financials and Supply Chain Production Supportability (System) Next generation iPhone launch and 2010 Holiday Retail Season readiness “Top Ten” Prioritization

SSC: AT&T Targeted Services: Transportation Manager OneATT- New Horizon to transform Network and Asset Management: Logistics TMS

Key Secondary Activities Participate in scheduled project “Deep Dive” calls with PMO and project

team Work with PMO/SSC to determine appropriate support model for New Horizon’s Work closely with OCS /PMO on project milestones – develop relationships Continue to maintain Dev OTM Network Supply Chain Business – develop key relationships Site visit to Warehouses SSC walk-thru’s

Key Deliverables Enhance Supply Chain SSC presence and understand usage

and architecture for OTM Site visit to understand implementation, document in

configuration guide. Performance testing as requested. Amend SSC contract to include OTM products

SSC: Oracle E-Business Suite Release 12/Database 11g R2 Upgrade Upgrade existing Oracle 11i.10 SCM and Planning environments to E-

Business Suite R12. This will enhances supportability for our complex environment and provide additional needed functionalities

Three stage upgrade (2 – 3 year journey) Supply Chain planning to release 12 RDBMS for 10g to 11G Supply chain platform to release 12

Objective: System Availability, Supportability, and Software Currency

Key Secondary Activities SSC “Top Ten” Prioritization: Service Request review meetings with Supply Chain,

Financials, and DBA support teams Patching/Release Management strategy for Release 12 interoperability; Leverage Tools and Templates Daily SSC Performance Calls

Key Deliverables Proactive Diagnostics (Configuration/Performance) Contracted deliverables Semi-Annual Account Review

Service Highlights Focused Engineering Team of Oracle Subject Matter Experts Fast track solutions based on intimate Knowledge of customer

Environment and business needs Proactive Software Advice and Knowledge Transfers Performance and Patch Assessments Single point of contact for all technical issues - Technical Lead Single point of contact for all escalations – SDM 24/7 coverage

Solution Support Center

Value Mitigate Upgrade Risks substantially All project milestones are met Ensured optimal usage of Oracle features and functionality Reduced overall Maintenance/upgrade/ training costs

Focused Engineering Team Single Point Accountability –

SDM Exclusive 1 800 number

24x7x365 Review of Critical Issues as

Needed Patch Analysis

Stand-by Support for Critical Go-lives

Proactive Alerts / Patch Notifications

Project Planning / Migration Assistance

Periodic Review Sessions Applied Research

Key Delivered Features

Solution Support Center

Questions

For More Information

On this presentation, e-mail Bryon Rickey (BR3681@att.com) or

Ravi Kayarthodi (RK8409@att.com).

For general information on Advanced Customer Services, go to

oracle.com/acs

or e-mail

acsdirect_us@oracle.com