optimizing oracle deployments in distributed data …faculty.ccc.edu/mmoizuddin/cisco live...
TRANSCRIPT
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
1
© 2008 Cisco Systems, Inc. All rights reserved. Cisco PublicSession_IDPresentation_ID 2
Optimizing Oracle Deployments in Distributed Data Centers
BRKAPP-2018
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
2
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3BRKAPP-201814438_04_2008_c2
Cisco CVD and DCAP
Cisco Validated Design Program
Data Center Assurance Program (DCAP)
Validated Design Guides
System Assurance Testing
App Deployment Guides
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4BRKAPP-201814438_04_2008_c2
DCAP Topology
1 2 3 4 5 6 7 8 9
42
3
8
1 5
6
7
9
DataBase Cluster
Branch Client Application Tier
ISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
3
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5BRKAPP-201814438_04_2008_c2
Other Cisco Live Breakout Sessions that You May Want to Attend
BRKAPP-1016 Running Applications on the Branch Router
BRKAPP-2014 Deploying AXG
BRKAPP-2013 Best Practices for Application Optimization illustrated with SAP, Seibel and Exchange
BRKAPP-2011 Scaling Applications in a Clustered Environment
BRKAPP-1009 Introduction to Web Application Security
BRKAPP-3006 Troubleshooting WAASBRKAPP-2005 Deploying WAAS
BRKAPP-3003 Introduction WAAS
BRKAPP-2002 Troubleshooting ACEBRKAPP-1001 Server Load Balancing Design
ApplicationTierISRGSS WAE 512 ACE WAE 7326
Relevancy
Data BaseCluster
Products we will discuss in our Packet Walk
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6BRKAPP-201814438_04_2008_c2
Cisco Solutions for Oracle
Oracle Apps lack built in site selection and replication capabilities. GSS
Disaster Recovery Limitations
Oracle Apps lacks built-in functionality of providing key HA features
Scalability/High Availability
Oracle Apps lacks built-in functionality of providing LB capability
Load Balance
WAN characteristics hinder performance of Oracle Applications
Latency
DescriptionProblem
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
4
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7BRKAPP-201814438_04_2008_c2
DCAP: Geographical Map
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 8BRKAPP-201814438_04_2008_c2
DCAP Packet Walk Overview(L3 is “So Yesterday”)
DataBase Cluster
Application
Presentation
Session
Transport
Network
Data Link
Physical
Application
Presentation
Session
Transport
Network
Data Link
Physical
Application
Presentation
Session
Transport
Network
Data Link
Physical
Network
Data Link
Physical
Session
Transport
Network
Data Link
Physical
Application
Presentation
Session
Transport
Network
Data Link
Physical
Session
Transport
Network
Data Link
Physical
Application
Presentation
Session
Transport
Network
Data Link
Physical
Application
Presentation
Session
Transport
Network
Data Link
Physical
IE and Firefox on Win XP
Named/bind 9.2.4 MS DNS
Server2003 Enterprise
Version 2.0.2
Version 12.4(12)
Version4.0.11b34
VersionA1.6.3
Version4.0.11b34
eBusiness Oracle 11i
Oracle 10g R2 RAC
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
5
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 9BRKAPP-201814438_04_2008_c2
Begin Packet Walk
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10BRKAPP-201814438_04_2008_c2
User Datagram Protocol, Src Port: 4112 (4112), Dst Port: domain (53)Domain Name System (query)
Transaction ID: 0x000fFlags: 0x0100 (Standard query)Questions: 1Answer RRs: 0Authority RRs: 0Additional RRs: 0Queries
wwwin-oefin.gslb.dcap.com: type A, class INName: wwwin-oefin.gslb.dcap.comType: A (Host address)Class: IN (0x0001)
Client DNS Query to Branch NS
Simulate Client BrowserLoadrunner (testing)
Important timers, TTL’s, MS Resolver
Browser CachingIE, Mozilla, Safari, etc
User Datagram Protocol, Src Port: domain (53), Dst Port: 4112 (4112)Domain Name System (response)
Transaction ID: 0x000fFlags: 0x8400 (Standard query response, No error)Questions: 1Answer RRs: 1Authority RRs: 0Additional RRs: 0Queries
wwwin-oefin.gslb.dcap.com: type A, class INName: wwwin-oefin.gslb.dcap.comType: A (Host address)Class: IN (0x0001)
Answerswwwin-oefin.gslb.dcap.com: type A, class IN, addr 101.1.33.50
Name: wwwin-oefin.gslb.dcap.comType: A (Host address)Class: IN (0x0001)Time to live: 5 secondsData length: 4Addr: 101.1.33.50
HKEY_CURRENT_USER\SOFTWARE\Microsoft\Windows\CurrentVersion\Internet Settings
Value Name: DnsCacheTimeoutData Type: REG_DWORDRadix: Decimal
Value: (time in seconds)
Trace Data (Client dns query)
Trace Data (NS dns response)
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
6
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 11BRKAPP-201814438_04_2008_c2
Branch NS Receives Client A Query
DCAP 4.0 DNS (BIND/MS DNS
Gotchas:
Be aware of TTL145C PACKET UDP Rcv 10.0.20.3 0008 Q [0001 D NOERROR] (11)wwwin-oefin(4)gslb(4)dcap(3)com(0)145C PACKET UDP Snd 201.1.33.11 1024 Q [0000 NOERROR] (11)wwwin-oefin(4)gslb(4)dcap(3)com(0)A58 PACKET UDP Rcv 201.1.33.11 1024 R Q [0084 A NOERROR] (11)wwwin-oefin(4)gslb(4)dcap(3)com(0)A58 PACKET UDP Snd 10.0.20.3 0008 R Q [0084 A NOERROR] (11)wwwin-oefin(4)gslb(4)dcap(3)com(0)
Delegated sub-zone: gslb.dcap.com.
gslb 600 NS dca-gss-1.dcap.com.gslb 600 NS dca-gss-2.dcap.com.gslb 600 NS dcb-gss-1.dcap.com.gslb 600 NS dcb-gss-2.dcap.com.
Delegated sub-zone: wwwin-oefin.gslb.dcap.com.;wwwin-oefin NS dca-gss-1.gslb.dcap.com.wwwin-oefin NS dca-gss-2.gslb.dcap.com.wwwin-oefin NS dcb-gss-1.gslb.dcap.com.wwwin-oefin NS dcb-gss-2.gslb.dcap.com.
Configuration (Branch NS Zone file)
Log file on Name Server showing dns delegation
1010001110000001101010101010111110101010111
dns
*.gslb.dcap.com is delegated to all 4 GSS’s
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 12BRKAPP-201814438_04_2008_c2
dcb-gss-1.gslb.dcap.com#show statistics keepalive101.1.33.12 OFFLINE 101.1.33.22 ONLINE 101.1.33.31 ONLINE 101.1.33.32 ONLINE101.1.33.34 OFFLINE 101.1.33.35 OFFLINE 101.1.33.36 ONLINE 101.1.33.50 ONLINE201.1.33.34 OFFLINE 201.1.33.35 OFFLINE 201.1.33.53 OFFLINE 201.1.33.59 ONLINE
GSS Receives A Querydca-gss-1.gslb.dcap.com#show statistics keepalive101.1.33.12 OFFLINE 101.1.33.22 ONLINE 101.1.33.31 ONLINE 101.1.33.32 ONLINE101.1.33.34 OFFLINE 101.1.33.35 OFFLINE 101.1.33.36 ONLINE 201.1.33.32 ONLINE201.1.33.34 OFFLINE 201.1.33.35 OFFLINE 201.1.33.53 OFFLINE 201.1.33.59 ONLINE
ACE VIPS
Show screen (GSS) Show screen (GSS)
1010001110000001101010101010111110101010111
dca-gss-1.gslb.dcap.com
ACE VIPS
dca-gss-2.gslb.dcap.com
dcb-gss-1.gslb.dcap.com
dcb-gss-2.gslb.dcap.com
dca-agg-1-ace-1
dca-agg-2-ace-1
dcb-ss-1-ace-1
dca-ss-2-ace-1
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
DCA
DCA
DCB
DCB
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
7
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13BRKAPP-201814438_04_2008_c2
GSS Decision Process
dca-gss-1.gslb.dcap.com#show statistics keepalive http-head all
IP: 101.1.33.50 Keepalive => 101.1.33.50Termination Method: ResetStatus: OFFLINEKeepalive Type: FastDestination Port: 8000Http Path: "/"Host Tag: ""Packets Sent: 1144808Packets Received: 1199599Positive Probe: 0Negative Probe: 54822Transitions: 0VIP GID: 151 LID: 32Keepalive GID: 836
Show screen (GSS Rule) Show screen (GSS Rule)
Show screen (GSS keepalives)
3 Balance Clauses
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
1010001110000001101010101010111110101010111
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14BRKAPP-201814438_04_2008_c2
No. Time Source Destination Protocol Info1 0.000000 10.0.20.3 10.0.20.2 DNS Standard query A wwwin-oefin.gslb.dcap.com
Frame 1 (85 bytes on wire, 85 bytes captured)Ethernet II, Src: Intel_53:0e:72 (00:07:e9:53:0e:72), Dst: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc)Internet Protocol, Src: 10.0.20.3 (10.0.20.3), Dst: 10.0.20.2 (10.0.20.2)User Datagram Protocol, Src Port: 1073 (1073), Dst Port: domain (53)Domain Name System (query)
No. Time Source Destination Protocol Info2 0.000149 10.0.20.2 201.1.33.11 DNS Standard query A wwwin-oefin.gslb.dcap.com
Frame 2 (85 bytes on wire, 85 bytes captured)Ethernet II, Src: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc), Dst: 00:19:aa:c1:96:d1 (00:19:aa:c1:96:d1)Internet Protocol, Src: 10.0.20.2 (10.0.20.2), Dst: 201.1.33.11 (201.1.33.11)User Datagram Protocol, Src Port: 29310 (29310), Dst Port: domain (53)Domain Name System (query)
No. Time Source Destination Protocol Info3 0.018354 201.1.33.11 10.0.20.2 DNS Standard query response A 101.1.33.50
Frame 3 (101 bytes on wire, 101 bytes captured)Ethernet II, Src: 00:19:aa:c1:96:d1 (00:19:aa:c1:96:d1), Dst: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc)Internet Protocol, Src: 201.1.33.11 (201.1.33.11), Dst: 10.0.20.2 (10.0.20.2)User Datagram Protocol, Src Port: domain (53), Dst Port: 29310 (29310)Domain Name System (response)
No. Time Source Destination Protocol Info4 0.018482 10.0.20.2 10.0.20.3 DNS Standard query response A 101.1.33.50
Frame 4 (101 bytes on wire, 101 bytes captured)Ethernet II, Src: Inventec_e5:c8:cc (00:a0:d1:e5:c8:cc), Dst: Intel_53:0e:72 (00:07:e9:53:0e:72)Internet Protocol, Src: 10.0.20.2 (10.0.20.2), Dst: 10.0.20.3 (10.0.20.3)User Datagram Protocol, Src Port: domain (53), Dst Port: 1073 (1073)Domain Name System (response)
dcb-gss-2 DNS-7-SELREQNAME[15925] Request from 10.0.20.2:32370 for wwwin-oefin.gslb.dcap.com, type is T_A, id is 9472
dcb-gss-2 DNS-7-SELREPPASS[15925] Reply is A, 1 addresses 101.1.33.50, NOERROR, TTL 5, AAfor wwwin-oefin.gslb.dcap.com, Request id is 9472
DNS DataFlow Client10.0.20.3
NS10.0.20.2
GSS201.1.33.11
GSS201.1.33.11
Client10.0.20.3
NS10.0.20.2
NS10.0.20.2
NS10.0.20.2
Trace taken on the Name Server (Client/NS/GSS)
GSS syslog
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
1010001110000001101010101010111110101010111
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
8
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15BRKAPP-201814438_04_2008_c2
Queuing the Packet!—Timeout to Talk About If a GSS Were to Fail
DCAP topology
NS Backoff timers
TTL issues
Can configure NS Probes on GSS to NS
No DNS responseis issued from the GSS
Name Server waiting for a response
?
wwwin-oefin.gslb.dcap.com
### NS Delegation ###.
NS dca-gss-1.dcap.com.NS dca-gss-2.dcap.com.NS dcb-gss-1.dcap.com.NS dcb-gss-2.dcap.com.
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16BRKAPP-201814438_04_2008_c2
Branch Router Intercepts TCP Traffic via WCCP and Forwards to Branch WAE
ip wccp 61ip wccp 62
interface GigabitEthernet0/1.10description to "Clients"encapsulation dot1Q 10ip address 10.0.10.1 255.255.255.0no ip unreachablesip wccp 61 redirect inip pim sparse-mode!interface GigabitEthernet0/1.11description "to Cisco WAE Appliances"encapsulation dot1Q 11ip address 10.0.11.1 255.255.255.0no ip unreachablesip wccp redirect exclude in!interface GigabitEthernet0/1.12description "To Wide Area Network"encapsulation dot1Q 12ip address 10.0.12.1 255.255.255.248no ip unreachablesip wccp 62 redirect inip pim sparse-mode
Configuration (ISR 3845)
1010001110000001101010101010111110101010111
Trace Data (ISR 3845)
No. Time Source Destination Protocol Info1 0.000000 10.0.10.2 101.1.33.50 TCP 58372
> 8000 [SYN] Seq=0 Len=0 MSS=1460
Frame 1 (62 bytes on wire, 62 bytes captured)Ethernet II, Src: Inventec_e5:e0:60 (00:a0:d1:e5:e0:60), Dst: Cisco_32:ab:81 (00:19:56:32:ab:81)Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)Transmission Control Protocol, Src Port: 58372 (58372), Dst Port: 8000 (8000), Seq: 0, Len: 0
Source port: 58372 (58372)Destination port: 8000 (8000)Sequence number: 0 (relative sequence number)Header length: 28 bytesFlags: 0x02 (SYN)Window size: 65535Checksum: 0xc79f [correct]Options: (8 bytes)
Maximum segment size: 1460 bytesNOPNOPSACK permitted
TCP Option 0x21 not present
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
9
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17BRKAPP-201814438_04_2008_c2
Branch WAE Receives TCP Packet from Branch Router via WCCP
classifier HTTPmatch dst port eq 80match dst port eq 8080match dst port eq 8000match dst port eq 8001match dst port eq 3128
map basicname File-System classifier AFS action optimize fullname Instant-Messaging classifier AOL action pass-throughname Remote-Desktop classifier Altiris-CarbonCopy action
pass-throughname Printing classifier AppSocket action optimize fullname File-System classifier Apple-AFP action optimize fullname Remote-Desktop classifier Apple-NetAssistant action
pass-throughname Instant-Messaging classifier Apple-iChat action pass-
throughname File-Transfer classifier BFTP action optimize fullname Systems-Management classifier BMC-Patrol action pass-
throughname Other classifier Basic-TCP-services action pass-
throughname P2P classifier BitTorrent action pass-throughname SQL classifier Borland-Interbase action optimize full
Configuration (Branch WAE) Trace Data (Branch WAE)
1010001110000001101010101010111110101010111
No. Time Source Destination Protocol Info
1 0.000000 10.0.10.2 101.1.33.50 TCP 58372 > 8000 [SYN] Seq=0 Len=0 MSS=1460
Frame 1 (62 bytes on wire, 62 bytes captured)Ethernet II, Src: Inventec_e5:e0:60 (00:a0:d1:e5:e0:60), Dst: Cisco_32:ab:81 (00:19:56:32:ab:81)Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)Transmission Control Protocol, Src Port: 58372 (58372), Dst Port: 8000 (8000), Seq: 0, Len: 0
Source port: 58372 (58372)Destination port: 8000 (8000)Sequence number: 0 (relative sequence number)Header length: 28 bytesFlags: 0x02 (SYN)Window size: 65535Checksum: 0xc79f [correct]Options: (8 bytes)
Maximum segment size: 1460 bytesNOPNOPSACK permitted
TCP Option 0x21 not present
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 18BRKAPP-201814438_04_2008_c2
ISR Receives Packet from Branch WAE and Forwards Packet to Its Destination (ACE VIP)
No. Time Source Destination Protocol Info173 3.265868 10.0.10.2 101.1.33.50 TCP 22994 > 8000
[SYN] Seq=0 Len=0 MSS=1432
Frame 173 (78 bytes on wire, 78 bytes captured)Ethernet II, Src: Cisco_19:bf:80 (00:15:c7:19:bf:80), Dst: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)802.1Q Virtual LAN
000. .... .... .... = Priority: 0...0 .... .... .... = CFI: 0.... 1000 0101 0100 = ID: 2132Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0, Len: 0
Source port: 22994 (22994)Destination port: 8000 (8000)Sequence number: 0 (relative sequence number)Header length: 40 bytesFlags: 0x02 (SYN)Window size: 65535Checksum: 0x8f36 [correct]Options: (20 bytes)
Maximum segment size: 1432 bytesNOPNOPSACK permittedUnknown (0x21) (12 bytes)
Configuration (Branch WAE) Trace Data (Branch WAE)
TCP Option 0x21 now present
ip wccp 61ip wccp 62
interface GigabitEthernet0/1.10description to "Clients"encapsulation dot1Q 10ip address 10.0.10.1 255.255.255.0no ip unreachablesip wccp 61 redirect inip pim sparse-mode!interface GigabitEthernet0/1.11description "to Cisco WAE Appliances"encapsulation dot1Q 11ip address 10.0.11.1 255.255.255.0no ip unreachablesip wccp redirect exclude in!interface GigabitEthernet0/1.12description "To Wide Area Network"encapsulation dot1Q 12ip address 10.0.12.1 255.255.255.248no ip unreachablesip wccp 62 redirect inip pim sparse-mode
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
1010001110000001101010101010111110101010111
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
10
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19BRKAPP-201814438_04_2008_c2
ACE Interfaces Explained
Features
Interesting config
HA (Failure)
Gotchas
interface vlan 2132description CLIENT_VLANbridge-group 10ip options allowmtu 2000no normalizationfragment min-mtu 68no icmp-guardaccess-group input BPDU-ALLOWaccess-group input anyoneaccess-group output anyoneservice-policy input ORACLE_TCP_TRAFFIC
no shutdown
interface vlan 1135description WAE_VLANip address 101.1.35.9 255.255.255.0alias 101.1.35.10 255.255.255.0peer ip address 101.1.35.8 255.255.255.0mtu 2000no normalizationfragment min-mtu 68mac-sticky enableno icmp-guardaccess-group input anyoneservice-policy input REMOTE-MGNTservice-policy input OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSservice-policy input NAT_POLICYno shutdown
Ensure return traffic from WAE’s follows return traffic flow
Allow TCP Option 0x21
Configuration (ACE)
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20BRKAPP-201814438_04_2008_c2
interface vlan 1105description CLIENT_VLANbridge-group 10ip options allowno normalizationaccess-group input anyoneservice-policy input REMOTE-MGNTno shutdown
interface vlan 1133description SERVER_VLANbridge-group 10ip options allowno normalizationaccess-group input anyonenat-pool 1 101.1.33.150 101.1.33.150 netmask 255.255.255.0 patservice-policy input REMOTE-MGNTservice-policy input ORACLE_VIPSno shutdown
interface bvi 10ip address 101.1.33.252 255.255.255.0alias 101.1.33.254 255.255.255.0peer ip address 101.1.33.253 255.255.255.0description CLIENT_SIDE_L3no shutdown
Congestive Collapse!—Timeout to Talk About Bridged vs. Routed Mode
ACE acts acts as a router
Servers default gateway is the ACE
ACE acts as a bump in the wire
Servers default gateway is the upstream router
interface vlan 1105description CLIENT_VLANip address 201.1.5.252 255.255.255.0ip options allowalias 201.1.5.254 255.255.255.0peer ip address 201.1.5.253 255.255.255.0no normalizationaccess-group input anyoneservice-policy input REMOTE-MGNTservice-policy input ORACLE_VIPSno shutdown
interface vlan 1133description SERVER_VLANip address 201.1.33.252 255.255.255.0alias 201.1.33.254 255.255.255.0peer ip address 201.1.33.253 255.255.255.0access-group input anyonenat-pool 1 201.1.33.150 201.1.33.150 netmask 255.255.255.0 patservice-policy input REMOTE-MGNTno shutdown
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
11
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21BRKAPP-201814438_04_2008_c2
ft interface vlan 1120ip address 101.1.20.100 255.255.255.0peer ip address 101.1.20.101 255.255.255.0no shutdown
ft peer 1heartbeat interval 600heartbeat count 10ft-interface vlan 1120
ft group 1peer 1priority 200associate-context c2inservice
FT Group : 1No. of Contexts : 1Context Name : c2Context Id : 1Configured Status : in-serviceMaintenance mode : MAINT_MODE_OFFMy State : FSM_FT_STATE_STANDBY_COLDMy Config Priority : 200My Net Priority : 200My Preempt : EnabledPeer State : FSM_FT_STATE_ACTIVEPeer Config Priority : 100Peer Net Priority : 100Peer Preempt : EnabledPeer Id : 1Last State Change time : Tue Feb 26 13:32:46 2008Running cfg sync enabled : Enabledrom activeStartup cfg sync enabled : Enabled
TCP Time Wait!—Timeout to Talk about If an ACE Fails
Dedicated FT VLAN between ACE ModulesAll Redundancy traffic is sent over this dedicated VLANTRP Protocol packets, Heartbeats, Configuration Sync packets, and State replication packets
ft interface vlan 1120ip address 101.1.20.101 255.255.255.0peer ip address 101.1.20.100 255.255.255.0no shutdown
ft peer 1heartbeat interval 600heartbeat count 10ft-interface vlan 1120
ft group 1peer 1peer priority 200associate-context c2inservice
FT Group : 1Configured Status : in-serviceMaintenance mode : MAINT_MODE_OFFMy State : FSM_FT_STATE_ACTIVEMy Config Priority : 100My Net Priority : 100My Preempt : EnabledPeer State : FSM_FT_STATE_STANDBY_COLDPeer Config Priority : 200Peer Net Priority : 200Peer Preempt : EnabledPeer Id : 1Last State Change time : Tue Feb 26 18:25:42 2008
Running cfg sync enabled : EnabledRunning cfg sync status : Running configuration sync has Startup cfg sync enabled : EnabledStartup cfg sync status : Startup configuration sync has completed
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22BRKAPP-201814438_04_2008_c2
Packet Arrives at the ACE (Packet #1)
DataBas
Branch
Applica
ISR
384
GSS
449
WAE
512
ACE
Mod
WAE
732
Branch
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
policy-map multi-match ORACLE_TCP_TRAFFICclass ORACLE_L4loadbalance vip inserviceloadbalance policy GO_TO_WAE_FARMloadbalance vip icmp-reply
class-map match-any ORACLE_L42 match virtual-address 101.1.33.50 tcp eq 8000
policy-map type loadbalance first-match GO_TO_WAE_FARMclass class-defaultserverfarm WAE_FARM backup ORAAPP_ORACLE_FARM
serverfarm host WAE_FARMdescription WAAS SERVERFARM TRANSPARENT MODEtransparentpredictor leastconnsprobe WAE_ICMPrserver WAE_1conn-limit max 6500 min 5000inservice
rserver WAE_2conn-limit max 6500 min 5000inservice
rserver host WAE_1description WAE 1ip address 101.1.35.4inservice
rserver host WAE_2description WAE 2ip address 101.1.35.5inservice
No. Time Source Destination Protocol Info173 3.265868 10.0.10.2 101.1.33.50 TCP 22994 > 8000
[SYN] Seq=0 Len=0 MSS=1432
Frame 173 (78 bytes on wire, 78 bytes captured)Ethernet II, Src: Cisco_19:bf:80 (00:15:c7:19:bf:80), Dst: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)802.1Q Virtual LAN
000. .... .... .... = Priority: 0...0 .... .... .... = CFI: 0.... 1000 0101 0100 = ID: 2132Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0, Len: 0
Source port: 22994 (22994)Destination port: 8000 (8000)Sequence number: 0 (relative sequence number)Header length: 40 bytesFlags: 0x02 (SYN)Window size: 65535Checksum: 0x8f36 [correct]Options: (20 bytes)
Maximum segment size: 1432 bytesNOPNOPSACK permittedUnknown (0x21) (12 bytes)
Do not use NAT
WAE Threshold TCP Options Set via Branch WAE
Configuration (ACE) Trace Data (Client to WAE)
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
12
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23BRKAPP-201814438_04_2008_c2
ACE: Internal Mapping of TCP/UDP Flow from Client in Branch 2 (Show Conns)
640 1 in TCP 2132 10.0.20.2:46457 101.1.33.50:8000 ESTAB1346 1 out TCP 1135 101.1.33.50:8000 10.0.20.2:46457 ESTAB
ACE Connection ID
Client:SRC IP : SRC Port
SYN_SEENESTAB
CLOSED
INITSYN-ACK
ESTABCLOSED
VIP:PortNetwork
Proccessor
Non TCP is displayed as “- -”
TCP and UDP Flows = 2 X Internal Half Flows
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
1010001110000001101010101010111110101010111
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24BRKAPP-201814438_04_2008_c2
NAM—Where Did the Packet Go? DCAP Relies on the NAM for verification and troubleshootingService Modules
(ip.addr eq 101.1.33.50 and ip.addr eq 10.0.10.2) and (tcp.port eq 8000 and tcp.port eq 22994)Must look at both sides of the flow
Set to 1518 bytes in order to capture entire frames
dca-agg-1#show module Mod Ports Card Type Model --- ----- -------------------------------------- ---------------1 6 Firewall Module WS-SVC-FWM-12 1 Application Control Engine Module ACE10-6500-K93 1 Application Control Engine Module ACE10-6500-K94 8 Intrusion Detection System WS-SVC-IDSM-25 8 Network Analysis Module WS-SVC-NAM-27 2 Supervisor Engine 720 (Active) WS-SUP720-3B9 8 CEF720 8 port 10GE with DFC WS-X6708-10GE10 8 CEF720 8 port 10GE with DFC WS-X6708-10GE11 8 CEF720 8 port 10GE with DFC WS-X6708-10GE12 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX13 8 CEF720 8 port 10GE with DFC WS-X6708-10GE
analysis module 5 management-port access-vlan 200monitor session 1 destination analysis-module 5 data-port 1
Configuration (CAT6K)
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
13
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25BRKAPP-201814438_04_2008_c2
Delay the ACK!—Timeout to Talk WCCP vs. WAE in DC
DCAP 4.0 using WCCP interception in DCB and in DCA, using ACE to load balance to WAE’s.
Advantages to using WAE in the DC
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26BRKAPP-201814438_04_2008_c2
ACE Sending Packet to WAE (Packet #2)
interface vlan 1135description WAE_VLANip address 101.1.35.9 255.255.255.0alias 101.1.35.10 255.255.255.0peer ip address 101.1.35.8 255.255.255.0mtu 2000no normalizationfragment min-mtu 68mac-sticky enableno icmp-guardaccess-group input anyoneservice-policy input REMOTE-MGNTservice-policy input OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSservice-policy input NAT_POLICYno shutdown
policy-map multi-match OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSclass VIA_WAE_FARM_L4loadbalance vip inserviceloadbalance policy ORAAPP_ORIGIN_SERVERS
class-map match-any VIA_WAE_FARM_L42 match virtual-address 101.1.33.50 tcp any
policy-map type loadbalance first-match ORAAPP_ORIGIN_SERVERSclass class-defaultsticky-serverfarm sticky-ace-cookieinsert-http SRC_IP header-value "%is"
No. Time Source Destination Protocol Info174 3.268853 10.0.10.2 101.1.33.50 TCP 22994 > 8000
[SYN] Seq=0 Len=0 MSS=1432
Frame 174 (78 bytes on wire, 78 bytes captured)Ethernet II, Src: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01), Dst: Ibm_b4:37:2f (00:14:5e:b4:37:2f)
Destination: Ibm_b4:37:2f (00:14:5e:b4:37:2f)Source: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN000. .... .... .... = Priority: 0...0 .... .... .... = CFI: 0.... 0100 0110 1111 = ID: 1135Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0, Len: 0
Source port: 22994 (22994)Destination port: 8000 (8000)Sequence number: 0 (relative sequence number)Header length: 40 bytesFlags: 0x02 (SYN)Window size: 65535Checksum: 0x8f36 [correct]Options: (20 bytes)
Maximum segment size: 1432 bytesNOPNOPSACK permittedUnknown (0x21) (12 bytes)
SRC MAC of ACE
DST MAC WAE1
Configuration (ACE) Trace Data (ACE to WAE)
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
14
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27BRKAPP-201814438_04_2008_c2
Buffer the Packet!—Timeout to talk about what happens when a WAE Fails
ACE will take WAE out of rotation (what happens to current sessions?)
What happens when the WAE reaches the max connlimit?
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28BRKAPP-201814438_04_2008_c2
Packet Arrives at the DC WAE
policy-map multi-match ORACLE_TCP_TRAFFICclass ORACLE_L4loadbalance vip inserviceloadbalance policy GO_TO_WAE_FARMloadbalance vip icmp-reply
class-map match-any ORACLE_L42 match virtual-address 101.1.33.50 tcp eq 8000
policy-map type loadbalance first-match GO_TO_WAE_FARMclass class-defaultserverfarm WAE_FARM backup ORAAPP_ORACLE_FARM
serverfarm host WAE_FARMdescription WAAS SERVERFARM TRANSPARENT MODEtransparentpredictor leastconnsprobe WAE_ICMPrserver WAE_1conn-limit max 6500 min 5000inservice
rserver WAE_2conn-limit max 6500 min 5000inservice
rserver host WAE_1description WAE 1ip address 101.1.35.4inservice
rserver host WAE_2description WAE 2ip address 101.1.35.5inservice
No. Time Source Destination Protocol Info173 3.265868 10.0.10.2 101.1.33.50 TCP 22994 > 8000
[SYN] Seq=0 Len=0 MSS=1432
Frame 173 (78 bytes on wire, 78 bytes captured)Ethernet II, Src: Cisco_19:bf:80 (00:15:c7:19:bf:80), Dst: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)802.1Q Virtual LAN
000. .... .... .... = Priority: 0...0 .... .... .... = CFI: 0.... 1000 0101 0100 = ID: 2132Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0, Len: 0
Source port: 22994 (22994)Destination port: 8000 (8000)Sequence number: 0 (relative sequence number)Header length: 40 bytesFlags: 0x02 (SYN)Window size: 65535Checksum: 0x8f36 [correct]Options: (20 bytes)
Maximum segment size: 1432 bytesNOPNOPSACK permittedUnknown (0x21) (12 bytes)
Configuration (DC WAE) Trace Data (DC WAE)
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
15
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29BRKAPP-201814438_04_2008_c2
SYN (VLAN 1133) TCP OPTION 0x21 SEQ=4150100321SYN (VLAN 1135) TCP OPTION 0x21 SEQ=4150100321SYN (VLAN 1135) TCP OPTION 0x21 SEQ=4150100321SYN (VLAN 1133) TCP OPTION 0x21 SEQ=4150100321
SYN/ACK (VLAN 1133) NO TCP OPTION SEQ=1193771344SYN/ACK (VLAN 1135) NO TCP OPTION SEQ=1193771344SYN/ACK (VLAN 1135) TCP OPTION 0x21 SEQ=1193771344SYN/ACK (VLAN 1133) TCP OPTION 0x21 SEQ=1193771344
ACK (VLAN 1133) TCP OPTION 0x21 SEQ=2002616674ACK (VLAN 1135) TCP OPTION 0x21 SEQ=2002616674ACK (VLAN 1135) NO TCP OPTION SEQ=4150100322ACK (VLAN 1135) NO TCP OPTION SEQ=4150100322
Buffer the Packet!— Timeout to talk about WAE Auto Discovery Though ACE (Gotcha)
4 SYN’s
4 SYN/ACK’s
4 ACK’s
123456789101112
Change made to the sequence number during final acknowledgements of the TCP handshake (Packet #9)
SEQ Space Jump!
1234
5678
9101112
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30BRKAPP-201814438_04_2008_c2
Stateful inspection of WAAS optimized traffic requires that the inspecting device understand the sequence number shift on optimized TCP connectionsThe following software versions provide 100% interoperability with WAAS optimized connections:
IOSFW (Zone-based): 12.4(11)T2 or later
ASA/PIX: 7.2.x or later
FWSM: 3.2.1 or later
ACE: (all versions) TCP Norm, IP Options
Silly window syndrome!— Timeout to talk ACE/Firewall Integration
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
16
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31BRKAPP-201814438_04_2008_c2
WAE Sends Packet Back to ACE (Packet #3)
interface vlan 1135description WAE_VLANip address 101.1.35.9 255.255.255.0alias 101.1.35.10 255.255.255.0peer ip address 101.1.35.8 255.255.255.0mtu 2000no normalizationfragment min-mtu 68mac-sticky enableno icmp-guardaccess-group input anyoneservice-policy input REMOTE-MGNTservice-policy input OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSservice-policy input NAT_POLICYno shutdown
policy-map multi-match OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSclass VIA_WAE_FARM_L4loadbalance vip inserviceloadbalance policy ORAAPP_ORIGIN_SERVERS
class-map match-any VIA_WAE_FARM_L42 match virtual-address 101.1.33.50 tcp any
policy-map type loadbalance first-match ORAAPP_ORIGIN_SERVERSclass class-defaultsticky-serverfarm sticky-ace-cookieinsert-http SRC_IP header-value "%is"
No. Time Source Destination Protocol Info175 3.268869 10.0.10.2 101.1.33.50 TCP 22994 > 8000
[SYN] Seq=0 Len=0 MSS=1432
Frame 175 (78 bytes on wire, 78 bytes captured)Ethernet II, Src: Ibm_b4:37:2f (00:14:5e:b4:37:2f), Dst: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)
Destination: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01)Source: Ibm_b4:37:2f (00:14:5e:b4:37:2f)Type: 802.1Q Virtual LAN (0x8100)
802.1Q Virtual LAN000. .... .... .... = Priority: 0...0 .... .... .... = CFI: 0.... 0100 0110 1111 = ID: 1135Type: IP (0x0800)
Internet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.50 (101.1.33.50)Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 0, Len: 0
Source port: 22994 (22994)Destination port: 8000 (8000)Sequence number: 0 (relative sequence number)Header length: 40 bytesFlags: 0x02 (SYN)Window size: 65535Checksum: 0x8f36 [correct]Options: (20 bytes)
Maximum segment size: 1432 bytesNOPNOPSACK permittedUnknown (0x21) (12 bytes)
SRC MAC of WAE 1
TCP Options Set via Branch WAEWAE default route is ACE Alias IP
Configuration (ACE) Trace Data (WAE to ACE)
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32BRKAPP-201814438_04_2008_c2
ACE (Probes to DB Cluster via Application Tier)
probe http ORACLE_WEB_PAGE_CHECKport 8000interval 2faildetect 1passdetect interval 2credentials sysadmin sysadminrequest method get url /oa_servlets/AppsLoginexpect status 200 200
Hypertext Transfer ProtocolGET /oa_servlets/AppsLogin HTTP/1.1\r\n
Request Method: GETRequest URI: /oa_servlets/AppsLoginRequest Version: HTTP/1.1
Connection: Close\r\nAuthorization: Basic c3lzYWRtaW46c3lzYWRtaW4=\r\n
Credentials: sysadmin:sysadminHost: 101.1.33.47\r\n\r\n
Configuration (ACE)
Trace Data (ACE Probe to Server)
Trace Data (ACE Probe to Server)
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
17
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33BRKAPP-201814438_04_2008_c2
Delayed Binding!—Timeout to Talk about ACE Probes
dca-agg-1-ace-1/c2(config)# logging message <0-2147483647>dca-agg-1-ace-1/c2(config)# logging message 251006 level 7 Syslog message ID
dca-agg-1-ace-1/c2# show probe ORACLE_WEB_PAGE_CHECK detail
probe : ORACLE_WEB_PAGE_CHECKtype : HTTPstate : ACTIVEdescription : ----------------------------------------------
port : 8000 address : 0.0.0.0 addr type : -interval : 2 pass intvl : 2 pass count : 3 fail count: 1 recv timeout: 10 http method : GEThttp url : /oa_servlets/AppsLoginconn termination : GRACEFUL expect offset : 0 , open timeout : 10 expect regex : -send data : -
show probe detail
dca-agg-1-ace-1/c2# show logging message all Message logging:
message 100001: default-level 2 (enabled)message 101002: default-level 4 (enabled)message 101004: default-level 6 (enabled)message 101005: default-level 6 (enabled)message 101006: default-level 6 (enabled)message 101007: default-level 6 (enabled)
View the logging messages
dca-agg-1-ace-1: %ACE-3-251010 Health probe failed for server 101.1.33.47 on port 8000, connection refused by server
syslog output
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34BRKAPP-201814438_04_2008_c2
ACE Makes Decision and Forwards Packet to Application Tier (Packet #4)
policy-map multi-match OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSclass VIA_WAE_FARM_L4loadbalance vip inserviceloadbalance policy ORAAPP_ORIGIN_SERVERS
class-map match-any VIA_WAE_FARM_L42 match virtual-address 101.1.33.50 tcp any
policy-map type loadbalance first-match ORAAPP_ORIGIN_SERVERSclass class-defaultsticky-serverfarm sticky-ace-cookieinsert-http SRC_IP header-value "%is“
serverfarm host ORAAPP_ORACLE_FARM_WAAS_CONTENTprobe ORACLE_WEB_PAGE_CHECKrserver ORAAPP01inservice
rserver ORAAPP02inservice
rserver ORAAPP03inservice
No. Time Source Destination Protocol Info207 3.286888 10.0.10.2 101.1.33.5 HTTP GET / HTTP/1.1
Frame 207 (468 bytes on wire, 468 bytes captured)Ethernet II, Src: Cisco_fe:1b:01 (00:0b:fc:fe:1b:01), Dst: HewlettP_3e:5e:c0 (00:19:bb:3e:5e:c0)802.1Q Virtual LANInternet Protocol, Src: 10.0.10.2 (10.0.10.2), Dst: 101.1.33.5 (101.1.33.5)Transmission Control Protocol, Src Port: 22994 (22994), Dst Port: 8000 (8000), Seq: 1, Ack: 1, Len: 410Hypertext Transfer Protocol
GET / HTTP/1.1\r\nAccept: */*\r\nAccept-Language: en-us\r\nUA-CPU: x86\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1;.NET CLR50727)\r\nHost: wwwin-oefin.gslb.dcap.com:8000\r\nConnection: Keep-Alive\r\nCookie: oracle.uix=0^^GMT-5:00^p; CHOCO=r275366210\r\nIf-Modified-Since: Mon, 22 Oct 2007 16:17:38 GMT\r\nIf-None-Match: "534a8a-a2c-471ccd22"\r\n\r\n
Oracle Server Chosen
ACE inserting HTTP Headers
Configuration (ACE) Trace Data (ACE to Server)
1010001110000001101010101010111110101010111
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
18
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35BRKAPP-201814438_04_2008_c2
Oracle Login Page (Oracle E-Business Suite)
wwwin-oefin.gslb.dcap.com
Insert Trace Client to App server on port:8000
1010001110000001101010101010111110101010111
Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326 DataBase ClusterBranch Client Branch Name Server
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36BRKAPP-201814438_04_2008_c2
Packet to Application Tier (Packet #4)
grew wwwin-oefin OEFIN_dcap-dca-oraapp01.xml<externURL oa_var="s_external_url">http://wwwin-oefin.gslb.dcap.com:8000</externURL>
<webentryhost oa_var="s_webentryhost">wwwin-oefin</webentryhost><login_page oa_var="s_login_page">http://wwwin-
oefin.gslb.dcap.com:8000/oa_servlets/AppsLogin</login_page><externURL oa_var="s_external_url">http://wwwin-oefin.gslb.dcap.com:8000</externURL>
<webentrydomain oa_var="s_webentrydomain">gslb.dcap.com</webentrydomain>
Hypertext Transfer ProtocolGET / HTTP/1.1\r\nAccept: */*\r\nAccept-Language: en-us\r\nUA-CPU: x86\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2;
SV1;.NET CLR50727)\r\nHost: wwwin-oefin.gslb.dcap.com:8000\r\nConnection: Keep-Alive\r\nCookie: oracle.uix=0^^GMT-5:00^p; CHOCO=r275366210\r\nIf-Modified-Since: Mon, 22 Oct 2007 16:17:38 GMT\r\nIf-None-Match: "534a8a-a2c-471ccd22"\r\n
Trace Data (ACE to Server)
Configuration (Oracle)
Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326 DataBase Cluster
1010001110000001101010101010111110101010111
Branch Client Branch Name Server
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
19
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37BRKAPP-201814438_04_2008_c2
ACE Show ScreensInterface: vlan 1133 2132 service-policy: ORACLE_TCP_TRAFFICclass: ORACLE_L4VIP Address: Protocol: Port:101.1.33.50 tcp eq 8000 loadbalance:L7 loadbalance policy: GO_TO_WAE_FARMVIP Route Metric : 77VIP Route Advertise : DISABLEDVIP ICMP Reply : ENABLEDVIP State: INSERVICEcurr conns : 2 , hit count : 44 dropped conns : 0 client pkt count : 430 , client byte count: 26724 server pkt count : 322 , server byte count: 23295conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : GO_TO_WAE_FARMclass/match : class-defaultLB action :
primary serverfarm: WAE_FARMstate: UP
backup serverfarm : ORAAPP_ORACLE_FARMstate: UP
hit count : 44dropped conns : 0
Show screen (Client to WAE in DCA) Trace Data (ACE to Server)
Interface: vlan 1135 service-policy: OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSclass: VIA_WAE_FARM_L4VIP Address: Protocol: Port:101.1.33.50 tcp anyloadbalance:L7 loadbalance policy: ORAAPP_ORIGIN_SERVERSVIP Route Metric : 77VIP Route Advertise : DISABLEDVIP ICMP Reply : ENABLEDVIP State: INSERVICEcurr conns : 2 , hit count : 39 dropped conns : 6 client pkt count : 269 , client byte count: 14135 server pkt count : 106 , server byte count: 9465conn-rate-limit : 0 , drop-count : 0 bandwidth-rate-limit : 0 , drop-count : 0 L7 Loadbalance policy : ORAAPP_ORIGIN_SERVERSclass/match : class-defaultLB action :
sticky group: sticky-ace-cookieprimary serverfarm: ORAAPP_ORACLE_FARMstate: UP
backup serverfarm : -hit count : 39dropped conns : 0
DataBase Cluster
1010001110000001101010101010111110101010111
Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Client Branch Name Server
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38BRKAPP-201814438_04_2008_c2
Oracle and SSL
policy-map multi-match ORACLE_TCP_TRAFFICclass ORACLE_L4loadbalance vip inserviceloadbalance policy GO_TO_WAE_FARMloadbalance vip icmp-replyssl-proxy server PROXY_1
class-map match-any ORACLE_L42 match virtual-address 101.1.33.50 tcp eq 8000
policy-map type loadbalance first-match GO_TO_WAE_FARMclass class-defaultserverfarm WAE_FARM backup ORAAPP_ORACLE_FARM
parameter-map type ssl SSL_PARAM_1cipher RSA_WITH_RC4_128_MD5session-cache timeout 3600
ssl-proxy service PROXY_1key key1.pemcert cert1.pemssl advanced-options SSL_PARAM_1
# show crypto files Filename File File Expor Key/
Size Type table Cert-----------------------------------------------------------------------cert1.pem 1334 PEM Yes CERTkey1.pem 887 PEM Yes KEY
# show stats crypto server SSL Server Statistics:
SSL alert INTERNAL_ERROR sent: 0SSL alert USER_CANCELED sent: 0SSL alert NO_RENEGOTIATION sent: 0SSLv2 client hello received: 48SSLv3 client hello received: 1626TLSv1 client hello received: 108SSLv3 negotiated protocol: 1626TLSv1 negotiated protocol: 108SSLv3 full handshakes: 1456SSLv3 resumed handshakes: 90Cipher sslv3_rsa_rc4_128_md5: 1626Cipher sslv3_rsa_rc4_128_sha: 0Cipher sslv3_rsa_des_cbc_sha: 0Cipher sslv3_rsa_3des_ede_cbc_sha: 0Cipher sslv3_rsa_exp_rc4_40_md5: 0Cipher sslv3_rsa_exp_des40_cbc_sha: 0Cipher sslv3_rsa_exp1024_rc4_56_md5: 0Cipher sslv3_rsa_exp1024_des_cbc_sha: 0Cipher sslv3_rsa_exp1024_rc4_56_sha: 0Cipher sslv3_rsa_aes_128_cbc_sha: 0Cipher sslv3_rsa_aes_256_cbc_sha: 0TLSv1 full handshakes: 21TLSv1 resumed handshakes: 87Cipher tlsv1_rsa_rc4_128_md5: 108
SSL Parameter Map
SSL Proxy Service
Configuration (ACE) SSL on ACE
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
1010001110000001101010101010111110101010111
SSL Proxy Enabled
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
20
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39BRKAPP-201814438_04_2008_c2
Gotcha #1 (Incorrectly Formatted hrefs)
</script>width="100%"><a id="AppsNavLink" href="http://Insert this info for CLEAR TEXT Brakiong HREFS
SSL Connection Attempt
Leaving the SSL Domain
Incorrectly (Insecure) Formatted Protocol
RST sent to Client
ACE Module
1010001110000001101010101010111110101010111
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40BRKAPP-201814438_04_2008_c2
HREFS Are Now Formatted Correctly (https://)
</script>width="100%"><a id="AppsNavLink" href="https://wwwin-oefin.gslb.dcap.com:8000/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE&akRegionApplicationId=0&navRespId=54060&navRespAppId=272&navSecGrpId=0&transactionid=596166964&oapc=2&oas=qTiv-azim1E9ksbjtDKCTA.." class="xd">ABM Manager</a></td></tr><tr><td><imgsrc="/OA_HTML/cabo/images/t.gif" width="4"></td><td valign="top"><imgid="AppsNavLink" href="https://wwwin-oefin.gslb.dcap.com:8000/OA_HTML/OA.jsp?OAFunc=OAHOMEPAGE&akRegionApplicationId=0&navRespId=54061&navRespAppId=272&navSecGrpId=0&transactionid=596166964&oapc
SSL Connection Attempt
MaintainSSL Domain
Secure Formatted Protocol
ACE Module
1010001110000001101010101010111110101010111
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
21
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41BRKAPP-201814438_04_2008_c2
What Is the Problem with Redirects?
Since the web server is unaware that SSL offloading is occurring, the web server will send a “302 redirect”back to the client with a port and protocol in the location field for what it thinks the client is talking on.
The “302 redirect” back to the client will reference a non-secure port such as :8000 since that is what the Oracle server is actually listening on.
Ultimately, the client will follow this link, attempt to connect via a non-secure port, in this case :8000 and leave the SSL domain as we will see in the next slide.
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42BRKAPP-201814438_04_2008_c2
Gotcha #2 Insecure Redirect from
Frame 42 (60 bytes on wire, 60 bytes captured)Ethernet II, Src: Cisco_e8:1b:91 (00:19:aa:e8:1b:91), Dst: Intel_5d:8f:53 (00:07:e9:5d:8f:53)Internet Protocol, Src: 101.1.33.50 (101.1.33.50), Dst: 10.0.30.3 (10.0.30.3)Transmission Control Protocol, Src Port: 8000 (8000), Dst Port: 17769 (17769), Seq: 1437232247, Ack: 2713312565, Len: 0
Source port: 8000 (8000)Destination port: 17769 (17769)Sequence number: 1437232247Acknowledgement number: 2713312565Header length: 20 bytesFlags: 0x14 (RST, ACK)Window size: 32232
Insecure Redirect
RST sent to the Client
ACE Module
1010001110000001101010101010111110101010111
RST sent to Client
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
22
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43BRKAPP-201814438_04_2008_c2
Oracle SSL Java
Opening a form
ACE Module
1010001110000001101010101010111110101010111
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44BRKAPP-201814438_04_2008_c2
Java Console (SSL)
Java Console
Java Console
ACE Module
1010001110000001101010101010111110101010111
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
23
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45BRKAPP-201814438_04_2008_c2
SSL Sessions
Hypertext Transfer ProtocolGET / HTTP/1.1\r\nUser-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;
Hypertext Transfer ProtocolGET /OA_JAVA/oracle/apps/fnd/jar/fndforms.jar HTTP/1.1\r\nUser-Agent: Java1.3.1.21-internal\r\n
ACE Module
1010001110000001101010101010111110101010111
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 46BRKAPP-201814438_04_2008_c2
Oracle SSL Configuration
<webentryhost oa_var="s_webentryhost">wwwin-oefin</webentryhost><webentrydomain oa_var="s_webentrydomain">gslb.dcap.com</webentrydomain<webentryurlprotocol
oa_var="s_webentryurlprotocol">https</webentryurlprotocol><activewebport oa_var="s_active_webport"
oa_type="PORT">8000</activewebport><web_ssl_port oa_var="s_webssl_port" oa_type="PORT">443</web_ssl_port><login_page oa_var="s_login_page">https://wwwin-
oefin.gslb.dcap.com:8000/oa_servlets/AppsLogin</login_page>
All references must be HTTPS://
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
24
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47BRKAPP-201814438_04_2008_c2
Fix on ACE for Redirectspolicy-map multi-match OPTIMIZED_TRAFFIC_TO_ORIGIN_SERVERSclass FROM_WAE_VLAN
loadbalance vip inserviceloadbalance policy TO_ORIGIN_SERVERSloadbalance vip icmp-reply
policy-map multi-match ORACLE_VIPSclass ORACLE_VIP
loadbalance vip inserviceloadbalance policy ORACLE_LB_POLICYloadbalance vip icmp-replyssl-proxy server SSL_PROXY_1
policy-map type loadbalance first-match ORACLE_LB_POLICYclass class-default
serverfarm WAE_FARMpolicy-map type loadbalance first-match TO_ORIGIN_SERVERSclass class-default
serverfarm ORACLE_SERVERSaction URL_REWRITE
dca-agg-1-ace-1/c2# show stats http
+------------------------------------------++-------------- HTTP statistics -----------++------------------------------------------+LB parse result msgs sent : 29717 , TCP data msgs sent : 59395 Inspect parse result msgs : 0 , SSL data msgs sent : 84
sentTCP fin/rst msgs sent : 360 , Bounced fin/rst msgs sent: 9 SSL fin/rst msgs sent : 11 , Unproxy msgs sent : 29648 Drain msgs sent : 0 , Particles read : 29787 Reuse msgs sent : 0 , HTTP requests : 29733 Reproxied requests : 0 , Headers removed : 0 Headers inserted : 0 , HTTP redirects : 0 HTTP chunks : 0 , Pipelined requests : 0 HTTP unproxy conns : 29486 , Pipeline flushes : 0 Whitespace appends : 0 , Second pass parsing : 0 Response entries recycled : 0 , Analysis errors : 0 Header insert errors : 0 , Max parselen errors : 0 Static parse errors : 38 , Resource errors : 0 Invalid path errors : 0 , Bad HTTP version errors : 0 Headers rewritten : 3 , Header rewrite errors : 0
Branch Client Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326Branch Name Server
DataBase Cluster
1010001110000001101010101010111110101010111
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48BRKAPP-201814438_04_2008_c2
Component Close Up “Application Tier”—Configurations
/apps/oefin/appl_top/admin > df /apps/oefinFilesystem 1K-blocks Used Available Use% Mounted onnas-oefin.gslb.dcap.com:/vol/dca_oraapp_oefin
157286400 114323320 42963080 73% /apps/oefin
Oracle Application Tier is configured in Active/Active mode across dual datacenters using shared application top on Network Appliance NAS.
grep wwwin-oefin OEFIN_dcap-dca-oraapp01.xml<externURL oa_var="s_external_url">http://wwwin-
oefin.gslb.dcap.com:8000</externURL><webentryhost oa_var="s_webentryhost">wwwin-oefin</webentryhost>
<login_page oa_var="s_login_page">http://wwwin-oefin.gslb.dcap.com:8000/oa_servlets/AppsLogin</login_page>
<externURL oa_var="s_external_url">http://wwwin-oefin.gslb.dcap.com:8000</externURL>
<webentrydomain oa_var="s_webentrydomain">gslb.dcap.com</webentrydomain>
Oracle/ACE configuration
ApplicationServer
NME 502GSS Branch WAE ACE Module DC WAEBranch Client DNS Server DataBase Server
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
25
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49BRKAPP-201814438_04_2008_c2
Component Close Up “Application Tier”—Oracle Forms Configuration
Applet running in browser for brining up forms in
browser
<server_url oa_var="s_forms_servlet_serverurl">/forms/formservlet</server_url><servlet_comment oa_var="s_forms_servlet_comment"/><formservlet_session_cookie
oa_var="s_form_session_cookie">true</formservlet_session_cookie>>
Forms Servlet Configuration
•Client interface is provided via a java appliet in a web browser for Oracle forms based applications•Forms Listener servlet allows http/https transport of forms server traffic from client and supports standards •load balancing methods• Requires fewer ports to be open in firewall using servlet mode.
1010001110000001101010101010111110101010111
Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326 DataBase ClusterBranch Client Branch Name Server
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50BRKAPP-201814438_04_2008_c2
Component Close Up “Database Server”
<jdbc_url oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=dcap-rac-node1.gslb.dcap.com)(port=1531))(ADDRESS=(PROTOCOL=tcp)(HOST=dcap-rac-node2.gslb.dcap.com)(port=1531)))(CONNECT_DATA=(SERVICE_NAME=OEFIN.dcap.com))</jdbc_url
HA Configuration for APPS
DB tier relies on Application tier for communication from Clients as there is no direct communication b/w Client and DB tier
Oracle Database 10gR2 configured with Real Application Cluster (RAC) providing HA and scalability for the Database Tier
Shared SAN storage for RAC configured using Oracle Automatic Storage Management (ASM), various storage vendors (EMC, HP, Network Appliance), and MDS 9000.
Oracle Interconnect is configured using best practices recommended by Oracle corporation.
1010001110000001101010101010111110101010111
ApplicationServer
NME 502GSS Branch WAE ACE Module DC WAEBranch Client DNS Server DataBase Server
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
26
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51BRKAPP-201814438_04_2008_c2
App Server to DB Flow Through ACE
Packet Trace (App Server to DB Server)
640 1 in TCP 2132 10.0.20.2:46457 101.1.33.50:8000 ESTAB1346 1 out TCP 1135 101.1.33.50:8000 10.0.20.2:46457 ESTAB
ACE Connection ID
Client:SRC IP : SRC Port Non-Load Balanced Traffic
Network Processor
Non TCP is displayed as “- -” 101000111000
0001101010101010111110101010111
Application TierISR 3845GSS 4492 WAE 512 ACE Module WAE 7326 DataBase ClusterBranch Client Branch Name Server
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52BRKAPP-201814438_04_2008_c2
Disaster Recovery
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
27
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53BRKAPP-201814438_04_2008_c2
DR: DCA Down
DCA Down
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54BRKAPP-201814438_04_2008_c2
Time Line: DCB Oracle Offline
SSLdump With Decryption
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
28
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55BRKAPP-201814438_04_2008_c2
Time Line: DCB Oracle Application Servers Offline
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56BRKAPP-201814438_04_2008_c2
Time Line: DCB Oracle Online
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
29
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57BRKAPP-201814438_04_2008_c2
Time Line: DCB Oracle Offline
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58BRKAPP-201814438_04_2008_c2
In Summary: Cisco Solutions for Oracle
MDS,GSSDisaster Recovery
ACE,GSSScalability
ACE,GSSLoad Balance
WAASLatency
SolutionsProblem
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
30
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59BRKAPP-201814438_04_2008_c2
Q and A
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60BRKAPP-201814438_04_2008_c2
Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press
Check the Recommended Reading flyer for suggested books
Available Onsite at the Cisco Company Store
© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr
31
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61BRKAPP-201814438_04_2008_c2
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Passport points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008.
Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62BRKAPP-201814438_04_2008_c2