z/os communications server tcp/ip vipa (virtual ip … communications server tcp/ip vipa (virtual ip...

z/OS Communications Server

TCP/IP VIPA

(Virtual IP Address)

Linda Harrison

[email protected]

IBM Advanced Technical Skills

09/21/2011 www.ibm.com/support/techdocs Document PRS789

© IBM Corporation, 2011 --- Last Updated 09/21/2011

Trademarks

• The following are Registered Trademarks of the International Business Machines Corporation in the United States and/or other countries.– IBM

– z/OS

• The following are trademarks or registered trademarks of other companies.– Microsoft is a registered trademark of Microsoft Corporation in the United States and other countries.

• All other products may be trademarks or registered trademarks of their respective companies.

• Refer to www.ibm.com/legal/us for further legal information.



Agenda• Introduction

– Inbound Connections Without VIPA

– Two Types of VIPA

– OSA and Routing Background

– VIPA Requirements

– Which VIPA to Use

– DVIPA Recoveries and Failovers

– Source VIPA

• DVIPA Details– DVIPA Route and Alternate Distribution Options

– Sysplex Distributor Distribution Options

– Sysplex Distributor TN3270 LU Support

– Fast Response Cache Acclerator (FRCA)

– QDIO Acclerator

– Multi-tier Applications and non-z/OS Targets

– Sysplex Autonomics

– Sysplex-Wide Security Associations (SWSA)

• Syntax, Commands, and References– VIPA Syntax

– VIPA Commands

– Documents

• Appendices– Appendix A: Sysplex and Subplex Support

– Appendix B: Details of Static VIPA Backup and Load Balance

– Appendix C: Details of DVIPA Backup and Load Balance



Introduction:

Inbound Connections

Without VIPA



Original OSA with Inbound

Connections• Inbound Connection Request

– Remote Client and z/OS Server

• Remote client sends in a connection request.– Source in IP Header = Remote client IP address

– Destination in IP Header = z/OS server IP address

• Response is sent back to the client.– Source in IP Header = z/OS server IP address

– Destination in IP Header = Remote client IP address

• Destination OSA IP Address– Connection Request Source=10.1.2.109 and Destination=10.1.1.11

– Response Source=10.1.1.11 and Destination=10.1.2.109

– All Inbound packets are received over the destination OSA (unless OSA failure).

– An OSA outage may cause a connection drop.

• For a TCP connection, the same client/server IP addresses are used for the life of the connection.

• TCP/IP Routing Table is used to determine which OSA the outboundpackets are sent over.

– Static Routing and OSPF support multiple concurrent parallel routes if IPCONFIG MULTIPATH is configured.

– Policy Agent and NetAccess can effect the decision.

10.1.2.109

z/OS TCP/IP

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.1

Router1

10.1.2.1

OSA is

single point

of failure.



OSA QDIO Gratuitous ARP Fail-over

z/OS TCP/IP

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2Router210.1.2.2

10.1.1.1Router110.1.2.1

z/OS TCP/IP

OSA

10.1.1.11

OSA10.1.1.1110.1.1.12

10.1.1.2Router210.1.2.2

10.1.1.1Router110.1.2.1

• When devices are started by the TCP/IP stack the stack determines if there is already a parallel connection to the same network. If at a later time one of the OSA connections goes down the other OSA will "take-over" for it. The other OSA will send out a gratuitous ARP with the IP address of the failed OSA and "own" the IP address of the failed OSA until its recovery.

– Note that this does require that both OSAs did originally come up so that the stack marked them as parallel to the same network.

• If one of the OSAs fails (any failure that causes the LINK to go down), then OSA QDIO Gratuitous ARP Fail-over occurs.

– The failed OSA IP address is taken over by one of the working OSAs.

– A gratuitous ARP will be sent out to associate that IP address with the working OSA’s MAC (Media Access Card) address.

• OSA QDIO Gratuitous ARP Fail-over does not require any configuration.

• The same failover support exists for IPv6 where gratuitous neighbor advertisements are sent rather than gratuitous ARPs.

OSA backup

capability.

Connections

are not

dropped.



Introduction:

Two Types of VIPA



Static VIPA for Inbound

Connections• Remote Client and z/OS Server

• Remote client sends in a connection request.– Source in IP Header = Remote client IP address

– Destination in IP Header = z/OS server IP address

• Response is sent back to the client.– Source in IP Header = z/OS server IP address

– Destination in IP Header = Remote client IP address

• Destination VIPA IP Address– Connection Request Source=10.1.2.109 and

Destination=10.2.1.101


– Inbound packets are received over either OSA (depending on routing protocol).

– An OSA outage may NOT cause a connection drop.


• TCP/IP Routing Table is used to determine which OSA the outbound packets are sent over.

– Static Routing and OSPF support multiple concurrent parallel routes.


10.1.2.109

z/OS TCP/IP

VIPA 10.2.1.101

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.1

Router1

10.1.2.1



Dynamic VIPA (DVIPA)

• DVIPA dynamically moves to a different TCP/IP stack in the sysplex when the first stack fails.

• VIPA still addresses connection resilience but now also addresses application recovery. If an image is taken down, DVIPA backup policies can be used to define where the associated DVIPAs move to within the sysplex. DVIPA support also allows for manual movement of applications with associated DVIPA addresses.

• Only one stack at any point in time “owns” a specific DVIPA address.

• There is no maximum number of static VIPA interfaces, but the maximum number of dynamic VIPA interfaces is 1024.

z/OS TCP/IP

VIPA 10.2.1.101

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS TCP/IP

VIPA 10.4.1.104

OSA

10.1.1.13

OSA

10.1.1.14

z/OS TCP/IP

VIPA 10.2.1.101

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS TCP/IP

VIPA 10.2.1.101

VIPA 10.4.1.104

OSA

10.1.1.13

OSA

10.1.1.14



Sysplex Distributor (Distributed DVIPA)

z/OS TCP/IP

VIPA 10.2.1.101

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS TCP/IP

VIPA 10.4.1.104

OSA

10.1.1.13

OSA

10.1.1.14

Distributor Stack

z/OS TCP/IP

VIPA 10.2.1.101

z/OS TCP/IP

VIPA 10.2.1.101

z/OS TCP/IP

VIPA 10.2.1.101

Target StackTarget Stack

z/OS TCP/IP

VIPA 10.2.1.101

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS TCP/IP

VIPA 10.2.1.101

VIPA 10.4.1.104

OSA

10.1.1.13

OSA

10.1.1.14

Distributor Stack

z/OS TCP/IPVIPA 10.2.1.101

z/OS TCP/IPVIPA 10.2.1.101

z/OS TCP/IP

VIPA 10.2.1.101


• A Distributed DVIPA (Sysplex Distributor) is just a special kind of DVIPA.

• One Stack Performs Routing Functions (Distributor Stack)– Owns Sysplex-Wide VIPA And Advertises To Routers

– Routes Connection Requests To Application Hosts (Target Stacks)

• All inbound packets travel through the Distributing Stack.

• All outbound packets from the Target Stacks flow according to the Target Stack routing table so they do not necessarily flow through the Distributor Stack.

A distributed DVIPA exists on several

stacks, but is advertised outside the

sysplex by only one stack. This stack

receives all incoming connection requests

and routes them to all the stacks in the

distribution list for processing.



VIPA Benefits of Sysplex

• System z Administration Services– System Cloning

– Application Cloning

– Reduced Definitions via System Symbolics

• Sysplex Horizontal Growth.– Discovery of new sysplex members.

– Dynamic connectivity via XCF connectivity.

• Sysplex Single System Image (SSI).– Multiple images of the same application on different z/OS images may appear as a single

application to end users.

– Balance workload across Sysplex members.

– Minimize application failure impact.

– Transparent application workload movement between images.

– Automatic Recovery with Automatic Restart Manager (ARM).

• VIPA Independence from Physical Interfaces– Interface failure does not cause session failures.

• As long as one network interface is operational, sessions do not fail.

– VIPA addressing may be independent of the LAN addressing.• Changes to LAN addresses may be transparent to end users because the VIPA address does not

change.– No DNS and firewall changes required.



Introduction:

OSA and Routing

Background



OSA Modes

• OSA Mode is configured in the CHPID definition.– Some OSA cards have only one port. One OSA port and one OSA CHPID.

– Many OSA cards have two ports. Two OSA ports and two separate OSA CHPIDs.• Each OSA port on a two port card is configured independently in only one OSA Mode.

– New OSA-Express3 have four ports. Two pairs of OSA ports. Each pair has a separate OSA CHPID.• Each pair of OSA ports on a four port card is configured together in only one OSA Mode, independently from the other pair

of OSA ports on the card.

• Two OSA Modes for LAN attachment:– OSE Mode - non-QDIO LCS and LSA Protocols

� Only OSAs with copper LAN attachments

� z/OS supports SNA traffic using LSA (Link Station Architecture) protocol and TCP/IP traffic using LCS (LAN Channel Station) protocol

� Linux supports TCP/IP and CCL SNA traffic using LCS protocol

– OSD Mode - QDIO Protocol� OSAs with copper or fiber LAN attachments

� z/OS supports TCP/IP traffic

� Linux supports TCP/IP traffic

� Linux supports any protocol with QDIO Layer 2 configured

• OSA Non-QDIO versus QDIO– OSE Mode - non-QDIO LCS and LSA Protocols

• Requires OSA/SF (unless only used for TCP/IP and not shared between LPARs)� Every IP address, including VIPA, must be manually defined.

– OSD Mode - QDIO Protocol• Does not require OSA/SF

� All IP addresses are dynamically downloaded to the OSA from the TCP/IP stack.

� Any VIPA movement/changes are dynamically downloaded to the OSA from the TCP/IP stack.

See Techdocs item PRS3169 for details about

defining OSA to z/OS Communications Server.



IPCONFIG MULTIPATH

z/OS TCP/IP

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

BEGINROUTES

ROUTE 10.1.1.0/24 = OSALNK11 MTU 1492

ROUTE 10.1.1.0/24 = OSALNK12 MTU 1492

ROUTE DEFAULT 10.1.1.1 OSALNK11 MTU 1492




ENDROUTES

• IPCONFIG MULTIPATH "load balances" outbound packets– Static Routing and OMPROUTE OSPF support IPCONFIG MULTIPATH

– Default Multipath routing is per connection as opposed to per packet

• Failed first hop router– Static Routing Dead Gateway Detection

• TCP connection will eventually timeout (3 to 10 minutes) and TCP will redrive the route selection algorithm and hopefully get a successful connection

• UDP and RAW packets are lost

• With Static Routing HSRP/VRRP should be used between first hop routers– See http://cisco.com for HSRP and VRRP details.

– Dynamic Routing detects failures• OSPF defaults to 40 seconds and RIP takes up to 3 minutes



Multipath Consideration

• Outbound Load Balance– Multipath can be configured “per connection” or “per packet”. A connection is a

unique combination of source IP address, source port, destination IP address, and destination port.

– The “per connection” option is recommended because the “per packet” option may cause additional overhead (and network traffic) if packets are received out of sequence due to different paths through the network. However, if numerous connections appear to be coming from a single end point (ie. a firewall) then traffic will not be truly load balanced.

• The “per packet” option uses the OSAs alternating between them evenly for outbound packets. The “per connection” option uses the OSAs alternating been them evenly on a connection basis, therefore the OSAs are all utilized but the outbound traffic is not as evenly distributed as the “per packet” option. In this presentation it is still indicated that the “per connection” option provides outbound load balance, even though the real comparison of the outbound traffic may not appear equal between the OSAs.

– Additionally see “per packet” APAR PK42294.

• Inbound Load Balance– Inbound load balance is really determined by the first hop router.

– If the first hop router is capable of load balancing traffic across multiple OSAs when the destination is a VIPA address, then inbound traffic will be truly load balanced.

– Routers have static and OSPF load balancing capability similar to z/OS outbound Multipath. See http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a0080094820.shtml



PRIRouter, SECRouter, NONRouter

• All IP addresses in a TCP/IP HOME list are registered (dynamically downloaded) with the QDIO adapters.

– HOME changes are automatically sent to QDIO adapters.

• If the OSA receives any packets with its MAC as the destination and a destination IP address that is "unknown" (meaning not an IP address in the HOME list), then OSA does the following:

– If PRIRouter is defined (assuming the OSA is started to that stack) then all "unknown" packets are sent to the PRIRouter stack.

– If PRIRouter is not defined (the OSA is not started to any stack with PRIRouter)(could be that PRIRouter is coded but that OSA connection is down due to failure or other outage) then if SECRouter is coded all "unknown" packets are sent to the SECRouter stack. If multiple SECRouters then a random (unpredictable) stack with SECRouter coded will be sent the "unknown" packets.

• There is no way to set the order of precedence for the secondary routers.

• Multiple Secondary Routers are supported on Ethernet Only

– If only NONRouter is defined (or any PRIRouter and SECRouter connection are down due to failure or other outage) then all "unknown" packets are discarded by the OSA. NONROUTER is the default.

• Non-QDIO OSA (OSE mode) may define PRIROUTER and SECROUTER via OSA/SF.• IPCONFIG DATAGRAMFWD

– PRIRouter is used when traffic is routed through the stack to another stack. Keep in mind that if one stack is used to route to other stacks IPCONFIG DATAGRAMFWD is required.

• z/OS V1.6+ DATAGRAMFWD not required for Sysplex Distributor.

• If target TCP/IP stacks only have XCF connectivity, datagram forwarding still needs to be configured on the distributor as all packets originating from the target will be forwarded to the distributor.

• PRIRouter and VLAN ID– Packets with unknown IP addresses are passed to the “router” stack.

• If VLANID is specified then packets with unknown IP addresses are only passed to the “router” stack if the packets have a matching VLAN ID tag.

– Without VLANID• Across one hardware box (System z central processor complex (CPC)), PRIROUTER can only be specified in the profile of one TCP/IP

stack for the same OSA port.

– With VLANID• Across one hardware box (System z central processor complex (CPC)), PRIROUTER can only be specified in the profile of one TCP/IP

stack for the same OSA port for the same VLANID.

• PriRouter, SecRouter, NonRouter definition is ignored when VMAC parameter is defined.

– Recommendation: Use VMAC for shared OSA ports rather than PRIROUTER/SECROUTER.



Considerations• OSA VIPA Limitations

– OSA devices have a limit on the number of IP addresses (both IPv4 and IPv6 addresses) that can be registered to the device. The limit is dependent on the microcode level of the OSA-Express device. This limit applies across all TCP/IP stacks that share the OSA-Express device. When defining a large number of VIPAs, take care not to exceed this limit. If the limit is exceeded, IP addresses beyond the limit will not be registered with the OSA-Express devices, and incoming packets with those IP addresses will not be routed to the correct stack unless that stack is designated as the primary router.

• RIP Routing and VIPA in Same (Sub)Network as OSAs– If using the RIP routing protocol and host route broadcasting is not supported by adjacent routers (that is,

adjacent routers are unable to learn host routes), the following restrictions for VIPA addresses must be applied in order to benefit from fault tolerance support:

• If you use subnetting and VIPA addresses are in the same network as the physical IP addresses, the subnetwork portion of any VIPA addresses must not be the subnetwork portion of any physical IP addresses in the network. In this case, assign a new subnetwork for the VIPA address.

• If subnetting is not used on any physical interface, the network portion of any VIPA address must not be the network portion of any physical IP address in the network. In this case, assign a new network for the VIPA address, preferably a class C network address.

– If using the RIP routing protocol and host route broadcasting is supported by adjacent routers (that is, adjacent routers are able to learn host routes), the network or subnetwork portions of VIPA addresses can be the same across multiple z/OS TCP/IP stacks in the network.

• Spanning Tree Protocol– If using a DVIPA when connecting an OSA-Express Gigabit Ethernet QDIO device to a intelligent bridge or

switch, ensure that the Spanning Tree Protocol (STP) on the intelligent bridge or switch is configured properly for DVIPA giveback and takeover operations. See the IP Configuration Guide for more details on STP problems.

• Port Fast Mode– If using VIPA along with an intelligent bridge or switch, ensure that ’Port fast mode’ (Cisco) is enabled. This

helps to decrease the amount of time the VIPA is unreachable in scenarios where there is dynamic movement of VIPA (dynamic or static). For more information, see your bridge or switch manual.



Introduction:

VIPA Requirements



Base vs. Parallel Sysplex

• Coupling Facility is not required for Base Sysplex.

• Coupling Facility is required for Parallel Sysplex.

• Structures are not defined in a Base Sysplex.

• Structures are defined in COUPLExx and stored in Coupling Facility in a Parallel Sysplex.

• DVIPA requires Parallel Sysplex or Base Sysplex.– TCP/IP only requires Parallel Sysplex coupling facility structures for Sysplex-wide Security

Association (SWSA), SysplexPorts, and Explicit Bind Port Range structures (covered later in thispresentation).

• DVIPA requires DynamicXCF.– Each stack must have an IPv4 or IPv6 DYNAMICXCF address, or both. This address is used by

distributing stacks to determine which destination targets are in the sysplex. When using DVIPAs, do not define an IUTSAMEH link. All Dynamic XCF Interfaces are created automatically if the DYNAMICXCF statement is configured.

• VIPAROUTE may be defined in the VIPADYNAMIC block to send DVIPA data traffic over an alternate IP path.

– XCF communications about DVIPAs, target stack server availability, etc. still use XCF communications.

YesYesDVIPA

YesNoStructures defined

YesNoCoupling Facility installed

YesYesXCF signaling or messaging

Parallel

Sysplex

Base

Sysplex • In Base Sysplex XCF signaling or messaging

flows over ESCON CTCs, or ICP (Internal

Coupling Peer Channel) in CEC (COUPLExx).

• In Parallel Sysplex XCF signaling or messaging

flows over Coupling Facility (COUPLExx).



Static VIPA Summary

No

No

One of the following is required:

•PRIROUTER must be defined on all the OSAs.

•OSAs must not be shared with other LPARs.




Requires PRIROUTER?

YesYes, with

OSPF.No, with RIP.

NoNo

QDIO

OSAs with

dynamic

routing

Yes

Yes, if router

supports

multipath.

YesYes

QDIO

OSAs with

static

routing

Yes

Yes, with

OSPF.

No, with RIP.

NoNo

Non-QDIO

OSAs with

dynamic

routing

No

Yes, if router

supports

multipath.

YesNo

Non-QDIO

OSAs with

static

routing

Dynamic

OSA

backup?

Inbound load

balance of

OSAs?

HSRP/VRRP

required on

first hop

router?

VIPA in

same

subnet as

OSAs?



DVIPA Summary

No

No




OSAs must be shared between the

VIPA primary stack and VIPA backup

stack and PRIROUTER/SECROUTER

must be defined.

Requires PRIROUTER?

Yes

Yes

Yes

No

Dynamic

OSA

backup?

YesYes, with

OSPF.No, with RIP.

NoNo

QDIO

OSAs with

dynamic

routing

Yes

Yes, if router

supports

multipath.

YesYes

QDIO

OSAs with

static

routing

Yes

Yes, with

OSPF.

No, with RIP.

NoNo

Non-QDIO

OSAs with

dynamic

routing

No

Yes, if router

supports

multipath.

YesNo

Non-QDIO

OSAs with

static

routing

Dynamic

TCP/IP

Stack

Backup?

Inbound load

balance of

OSAs?

HSRP/VRRP

required on

first hop

router?

VIPA in

same

subnet as

OSAs?



OMPROUTE OSPF Coding

• OMPROUTE OSPF Routing is not required but is highly recommended for VIPA.• Define all the DVIPAs which the host might own.• If OSPF is being used VIPAs should be defined as OSPF_Interface rather than INTERFACE.• OMPROUTE supports a maximum of 255 physical IPv4 interfaces. There is no theoretical limit on

how many VIPAs can be configured.• Ranges of DVIPA interfaces can be defined using the subnet mask parameter on the

OSPF_Interface or Interface statement. The range defined will be all the IP addresses that fall within the subnet defined by the mask and the IP address. The IP address parameter must be the subnet number of the range being defined, not a host address within that range. The following example defines a range of DVIPA addresses from 10.138.165.81 to 10.138.165.94:

– OSPF_InterfaceIP_address = 10.138.165.80Name = dummy_nameSubnet_mask = 255.255.255.240;

– The Name parameter is not actually used for DVIPAs. The names of DVIPA interfaces are assigned dynamically by the stack when a DVIPA interface is created. Therefore, the Name field set on the Interface or OSPF_Interface statement for a DVIPA will be ignored.

• Optimize Performance– To minimize routing table size and advertisements that have to be processed, try to put z/OS and the

sysplex into a stub or totally stubby area or isolate areas with BGP or EIGRP.– To mimimize OSPF adjacencies, try to avoid OMPROUTE becoming the designated router.

• May be unavoidable on Hipersockets LAN since no routers attached.

– Only use debug tracing when necessary.• Use CTRACE tracing whenever possible.

– Consider taking XCF links out of the OSPF domain, if not used for data traffic.• Define them as “Interface” rather than “OSPF_Interface” - Cuts down significantly on adjacencies.

• OSPF uses multicast packets– Disable multicast snooping on switches with shared OSAs attached to them.



Introduction:

Which VIPA to Use



What type of VIPA to use?

• Static VIPA– When you only want the IP address to be used on one TCP/IP stack.

• DVIPA– When you only want the IP address to be used on one TCP/IP stack at any given

time but you want the dynamic backup capability of moving the IP address to a different stack at failure.

• If you only ever want the application to run on one system at a time, but you need the application to backup to another system when the primary stack or app goes down, then DVIPA without Sysplex Distributor is what you need.

– There is no maximum for static VIPA interfaces, but the maximum number of dynamic VIPA interfaces is 1024.

• Sysplex Distributor– When you have multiple Servers that are identical to your clients, you want all

clients to use the same Server identity, but one instance of a server cannot handle the total number of clients that you have.

• If you want the application to run "identically" on multiple systems concurrently and have connections distributed (or load balanced) to those target systems, then Sysplex Distributor is what you need.

• If an application running on only one system does not have the capacity to support all the clients you want to support, and the application may run "identically" on multiple different systems, then Sysplex Distributor is what you need.



Application Specific DVIPA

• A DVIPA may be defined such that the VIPA is not brought up by the TCP/IP stack until an application does a Bind(). This is known as an application specific DVIPA.

• If the application goes down, the DVIPA is deleted.– If using MODDVIPA to create the DVIPA, then MODDVIPA would be needed to delete the DVIPA from the stack.

• The application may be restarted on a different z/OS in the Sysplex via Automatic Restart Manager (ARM), etc. At that time the application may issue a Bind() and the DVIPA would be brought up by the second TCP/IP stack.

• Only one stack at any point in time “owns” a specific DVIPA address.

z/OS TCP/IP

VIPA 10.2.1.101

Application APPL1

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS TCP/IP

OSA

10.1.1.13

OSA

10.1.1.14

z/OS TCP/IP

VIPA 10.2.1.101

Application APPL1

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS TCP/IP

VIPA 10.2.1.101

Application APPL1

OSA

10.1.1.13

OSA

10.1.1.14



Multiple Application-instance vs.

Unique Application-instance• Multiple Application-instance Scenario

– Some applications will accept client requests on any IP addr by binding to INADDR_ANY or IN6ADDR_ANY (i.e.. TN3270 & FTP). If the appl is identical on multiple z/OS images in sysplex, a DVIPA may move between stacks.

– The stacks in the sysplex do all the work of activating a DVIPA in the event of a failure.

– A Multiple Application-instance DVIPA is defined with a profile statement:• VIPADYNAMIC VIPADEFINE… …ENDVIPADYNAMIC

• or VIPADYNAMIC VIPABACKUP… …ENDVIPADYNAMIC– Backup allows ranking to determine order. VIPABACKUP does not require VIPADEFINE.

• The DVIPA is active when the stack starts and processes the profile statement.

– Multiple Application-instance DVIPA may be distributed (Sysplex Distributor) with a profile statement:

• VIPADYNAMIC VIPADISTRIBUTE DEFINE... …ENDVIPADYNAMIC

• Unique Application-instance Scenario– For some applications, each application instance must have a unique IP address.

– Each application instance is assigned a unique IP address as its DVIPA. Before defining individual DVIPAs, one or more blocks of IP addresses must be defined for these DVIPAs, and the individual DVIPAs must be defined from within the blocks. Each block should be represented as a subnet, so that a VIPARANGE statement can be defined for it.

– A Unique Application-instance DVIPA is defined with a profile statement:• VIPADYNAMIC VIPARANGE… …ENDVIPADYNAMIC

• The DVIPA is not active though until it is started in one of three different ways.

– Defining a single block of VIPA IP addresses makes the definition process easier, but provides less individual control. Since the smallest subnet consists of four IP addresses, defining a unique subnet for each DVIPA wastes three other IP addresses that could have been used for DVIPAs.



Activate Unique

Application-instance DVIPA• Unique Application-instance DVIPA may be activated in one of three

different ways:– Bind() Function Call

• DVIPA is created when the application instance issues a bind() function call and specifies an IP address that is not active.

• DVIPA is deleted when the application instance closes the socket.

– BIND Parameter on the PORT Statement• DVIPA is created when the application instance uses the BIND parameter on the PORT

reservation statement.

• DVIPA is deleted when the application closes the socket.

• If the BIND parameter and an IP address are specified on the PORT reservation statement for a server application, and the application binds a socket to that port and INADDR_ANY, TCP/IP will convert the bind to be specific to the IP address specified on the PORT reservation statement.

– Utility MODDVIPA (which issues SIOCSVIPA or SIOCSVIPA6 ioctl())• DVIPA is created when the application instance uses the utility MODDVIPAIPA to

activate the DVIPA.

• DVIPA is deleted when the application instance uses the utility MODDVIPAIPA to deactivate the DVIPA.

• MODDVIPA can be included in a JCL procedure or OMVS script to activate the DVIPA before initiating the application. As long as the same JCL or script is used to restart the application on another node in the event of a failure, the same DVIPA will be activated on the new stack.



VIPADYNAMIC VIPADEFINE, VIPARANGE

bind(), or VIPARANGE MODDVIPA?

• As per the IP Configuration Guide:– When should VIPADEFINE and VIPABACKUP be used to define a DVIPA?

• One or more applications bind to INADDR_ANY or IN6ADDR_ANY, and the application exist on multiple TCP/IPs.

• DVIPA takeover is desired.

• The DVIPA does not need to be deleted when the application is stopped.

– When should VIPARANGE and bind() or PORT BIND be used to define a DVIPA?• The application cannot bind to INADDR_ANY or IN6ADDR_ANY, or TCP/IP stack initiated DVIPA

takeover is not desired.

• For the bind(), the IP address to which the application binds can be controlled by the user. The application’s first explicit bind causes the listening socket to remain for the life of the application.

• Automatic deletion of the DVIPA when the application is stopped is acceptable.

• A specific DVIPA address must be associated with a specific application.

• The application does not need to be APF authorized, or run under a user ID with superuser authority.

– When should VIPARANGE and the MODDVIPA utility (ioctl command SIOCSVIPA or SIOCSVIPA6) be used to define a DVIPA?

• The application cannot bind to INADDR_ANY or IN6ADDR_ANY, or TCP/IP stack initiated DVIPA takeover is not desired.

• The IP address to which the application binds is known but cannot be controlled by the user.

• Automatic deletion of the DVIPA when the application is stopped is not acceptable.

• The MODDVIPA utility (or application issuing ioctl command) will be run from an APF authorized library and under a userid with superuser authority. Utility can be initiated from JCL or an OMVS script.

• You can restrict access to the MODDVIPA (EZBXFDVP) program by defining a RACF profile under the SERVAUTH class and specifying the user IDs that are authorized to execute the ioctl or the MODDVIPA utility program.



Introduction:

DVIPA Recoveries and

Failovers



DVIPA Recovery

1. VIPA1 comes up on primary stack z/OS1.

2. z/OS1 fails.• All current connections are dropped.

3. VIPA1 is moved to backup stack z/OS2.• Clients can immediately log back in.

4. z/OS1 recovers and VIPA1 moves back to z/OS1.• New sessions are setup on z/OS1.

• Data connections that are still active to z/OS2 are routed by z/OS1 to z/OS2 until those connections terminate.

TN3270BTN3270A

z/OS1 TCP/IP

VIPA1 10.2.1.101

OSA

Router

z/OS2 TCP/IP

OSAOSA OSA

Router

Session1 Session1

TN3270B

z/OS2 TCP/IP

VIPA1 10.2.1.101

TN3270A

z/OS1 TCP/IP

VIPA1 10.2.1.101

OSA

Router

OSAOSA OSA

Router

Session2 Session3

TN3270B

z/OS2 TCP/IP

VIPA1 10.2.1.101

TN3270A

z/OS1 TCP/IP

VIPA1 10.2.1.101

OSA

Router

OSAOSA OSA

Router

Session2



VIPARANGE

• The application causes the VIPA to be brought up on one and only one stack at a time.1. Appl1 is brought up on z/OS1.

2. Appl1 issues Bind() on z/OS1.

3. VIPA1 is brought up on z/OS1.

4. Appl1 goes down on z/OS1.

5. VIPA1 is deleted on z/OS1.

6. Appl1 is brought up on z/OS2.

7. Appl1 issues Bind() on z/OS2.

8. VIPA1 is brought up on z/OS2.

z/OS1 TCP/IP

VIPA1 10.2.1.101

Appl1

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS2 TCP/IP

OSA

10.1.1.13

OSA

10.1.1.14

z/OS1 TCP/IP

VIPA1 10.2.1.101

Appl1

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS2 TCP/IP

VIPA1 10.2.1.101

Appl1

OSA

10.1.1.13

OSA

10.1.1.14

• MODDVIPA(EZBXFDVP) utility to issue SIOCSVIPA IOCTL

- Bind-Specific Application- Bind-Inaddr_any Application

- If using MODDVIPA to create a DVIPA, then MODDVIPA would be needed to delete the DVIPA from the stack.

• See “DVIPA Creation Results” table in IP Configuration Guide.



Sysplex Distributor Failure (Distributor)

• When the Distributing stack fails, the backup Distributor receives routing tables

from the target stacks of all the current sessions so none of the sessions to the

Target systems fail.

• When the primary Distributor stack recovers the inbound session paths are

returned to the primary Distributor stack.

z/OS1 TCP/IP

VIPA 10.2.1.101

Sess1

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

z/OS2 TCP/IP

Sess2Sess3

z/OS1 TCP/IP

VIPA 10.2.1.101

Sess1

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

z/OS2 TCP/IP

VIPA 10.2.1.101

Sess2Sess3

z/OS1 TCP/IP

VIPA 10.2.1.101

Sess1

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

z/OS2 TCP/IP

VIPA 10.2.1.101

Sess2Sess3



Sysplex Distributor Failure (Target)

• When a Sysplex Distributor Target stack fails, any sessions currently setup to

that Target fail.

• No more connection requests are sent to that failed Target system.

z/OS1 TCP/IP

VIPA 10.2.1.101

Sess1

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

z/OS2 TCP/IP

Sess2Sess3

z/OS1 TCP/IP

VIPA 10.2.1.101

Sess1

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

z/OS2 TCP/IP

Sess2Sess3

z/OS1 TCP/IP

VIPA 10.2.1.101

Sess4

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

z/OS2 TCP/IP

Sess2Sess3



Introduction:

Source VIPA



Source IP Address Selection

• As per the IP Configuration Guide…

• TCP/IP determines the source IP address for a TCP outbound connection, or for a UDP or RAW outbound packet, using the following sequence, listed in descending order of priority.

1. Sendmsg( ) using the IPV6_PKTINFO ancillary option specifying a nonzero source address (RAW and UDP sockets only)

2. Setsockopt( ) IPV6_PKTINFO option specifying a nonzero source address (RAW and UDP sockets only)

3. Explicit bind to a specific local IP address

4. PORT profile statement with the BIND parameter

5. SRCIP profile statement (TCP connections only)

6. TCPSTACKSOURCEVIPA parameter on the IPCONFIG or IPCONFIG6 profile statement (TCP connections only)

7. SOURCEVIPA: static VIPA address from the HOME list or from the SOURCEVIPAINTERFACE parameter

8. HOME IP address of the link over which the packet is sent

• For a TCP connection, the source address is selected for the initial outbound packet, and the same source IP address is used for the life of the connection. For the UDP and RAW protocols, a source IP address selection is made for each outbound packet.



Outbound Connection Using IP

Address of LINK• Local z/OS Client and Remote Server

• Local client sends out a connection request.– Source in IP Header = z/OS client IP address

– Destination in IP Header = Remote server IP address

• Response is sent back to the client.– Source in IP Header = Remote server IP address

– Destination in IP Header = z/OS client IP address

• Source OSA IP Address– Connection Request Source=10.1.1.11 and



– An OSA outage may cause a connection drop.

• Resolver may be used to determine remote server IP address.





10.1.2.109

z/OS TCP/IP

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.1

Router1

10.1.2.1

OSA is

single point

of failure.



Outbound Connection Using

Source VIPA• Local z/OS Client and Remote Server





• Source VIPA IP Address– Connection Request Source=10.2.1.101 and Destination=10.1.2.109


– An OSA outage may NOT cause a connection drop.• For all the same reasons documented previously for Inbound Connections





10.1.2.109

z/OS TCP/IP

VIPA 10.2.1.101

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.1

Router1

10.1.2.1



Source VIPA Configurations

• IPCONFIG SOURCEVIPA and IPCONFIG6 SOURCEVIPA– Enables interface fault tolerance for z/OS clients that establish outbound connections.

– Static VIPAs can be used as the source IP address for outbound datagrams for TCP, RAW, UDP (except routing protocols), and ICMP requests.

– IPCONFIG SOURCEVIPA defines that an IPv4 static VIPA that precedes a physical IPv4 address in the HOME list is used as the source IP address, if not overridden.

– IPCONFIG6 SOURCEVIPA defines that an INTERFACE SOURCEVIPAINT keyword defines the IPv6 static VIPA to be used as the source IP address, if not overridden.

• IPCONFIG TCPSTACKSOURCEVIPA and IPCONFIG6 TCPSTACKSOURCEVIPA– Enables a single sysplex-wide source IP address for z/OS clients that establish outbound connections.

– The source IP address is defined on the TCPSTACKSOURCEVIPA statement.

– Any valid IP address (real interface, VIPA, DVIPA, or distributed DVIPA) in the HOME list may be defined.

– Outbound TCP datagrams only (no RAW or UDP support).

– Either specify single distributed DVIPA on all stacks in sysplex (SYSPLEXPORTS required), or define unique source IP address on each stack.

– Requires SOURCEVIPA.

– The recommendation is to use SRCIP instead.

• SRCIP/ENDSRCIP– Enables a source IP address for a job or destination address (destination new in z/OS V1.8).

– The source IP address is defined on the SRCIP/ENDSRCIP statement.

– Any valid IP address (real interface, VIPA, DVIPA, or distributed DVIPA) in the HOME list may be defined.

– Outbound TCP datagrams only (no RAW or UDP support).

– If a Distributed DVIPA is specified in the SRCIP block, then GLOBALCONFIG EXPLICITBINDPORTRANGE is required on all target stacks to guarantee that unique sysplex wide ephemeral ports are assigned. See Explicit Bind Port Range on a later page in this presentation.

– SOURCEVIPA not required.

• Order of precedence:1. SRCIP

2. TCPSTACKSOURCEVIPA

3. SOURCEVIPA

• SRCIP is Recommended Rather than TCPSTACKSOURCEVIPA– Because SRCIP provides all of the functionality of TCPSTACKSOURCEVIPA and additional granularity.

– Specifying JOBNAME * in a SRCIP profile statement provides the same result as specifying the TCPSTACKSOURCEVIPA parameter for implicit bind scenarios, and also applies to applications that issue a bind to the INADDR_ANY or IN6ADDR_ANY IP address.

• Sysplex Distributor– If a distributed DVIPA is used as the source IP address for outbound connections there is no load balancing being performed like what happens

for inbound connections.



Outbound Connection Using

TCP/IP Stack Source VIPA• Local z/OS Client and Remote Server





• Source VIPA IP Address– Connection Request Source=10.2.1.101 and



– An OSA outage may NOT cause a connection drop.• For all the same reasons documented previously for Inbound

Connections





• If a Distributed DVIPA is being used as a source IP address then SYSPLEXPORTS must be used to guarantee that unique sysplex wide ephemeral ports are assigned. (EZBEPORT CF structure)

z/OS TCP/IP

VIPA 10.2.1.101

OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2

Router2

10.1.2.2

10.1.1.1

Router1

10.1.2.1

z/OS TCP/IP

VIPA 10.4.1.104

OSA

10.1.1.13

OSA

10.1.1.14

Distributor Stack

z/OS TCP/IP

VIPA 10.2.1.101

z/OS TCP/IP

VIPA 10.2.1.101

z/OS TCP/IP

VIPA 10.2.1.101


10.1.2.109



TCP Stack Source VIPA

Distributed DVIPA and Sysplex Ports• If you specify the same distributed DVIPA interface for TCPSTACKSOURCEVIPA on multiple

target stacks, you should specify SYSPLEXPORTS on the VIPADISTRIBUTE statement. Otherwise connections might be disrupted because identical connections could be created from more than one stack.

– Multiple clients on Sysplex Distributor target stacks may be connecting to remote servers.– The FTP servers on Sysplex Distributor target stacks may support passive FTP connections.

• If SYSPLEXPORTS is defined, the coupling facility structure EZBEPORT coordinates sysplex-wide ephemeral port assignment for each distributed DVIPA when they are used for TCP connection requests.

• The stack maintains a list of allowable ephemeral ports– Any port number above 1023 that has not been reserved for TCP by a PORT or PORTRANGE statement.

• The coupling facility structure contains sysplex port assignment information. The name of this structure is in the format EZBEPORTvvtt.

– vv is the 2-character VTAM group ID suffix specified on the XCFGRPID start option.– tt is the TCP group ID suffix specified on the GLOBALCONFIG statement in the TCP/IP profile.

• If no VTAM group ID suffix is specified, but a TCP/IP group ID suffix is specified, vv is 01.• If no TCP/IP group ID suffix is specified, but a VTAM group ID suffix is specified, tt is not present.• If neither group ID suffix is specified, both vv and tt are not present.

– See the “z/OS Communications Server: SNA Network Implementation Guide, SC31-8777” for more information about the EZBEPORT structure.

• If your FTP server accepts connections on a distributed DVIPA, SYSPLEXPORTS must be specified for the distributed DVIPA if either of the following are true:

– The DVIPA is an IPv6 address.– Passive mode FTP is used.



Source IP Distributed DVIPA Based on

Destination IP and Explicit Bind Port Range

• z/OS V1.8 SRCIP DESTIP– Selects source IP address based on the

destination IP address of the connection.

– Does not support distributed DVIPA.

• Distributed DVIPA as source IP address for SRCIP z/OS V1.9 Enhancement

– Extends the destination-based source IP address selection to include a distributed DVIPA.

– Participating stacks will reserve a coordinated range of port numbers for this use with GLOBALCONFIG EXPLICITBINDPORTRANGE.

– If an application issues an explicit bind to INADDR_ANY or INADDR6_ANY and port 0, the stack has GLOBALCONFIG EXPLICITBINDPORTRANGE defined, and the stack has SRCIP rules, a port from this new range will be requested.

Connect to

customer A

(10.1.1.nnn)

and transfer

data.

Connect to

customer B

(10.1.2.nnn)

and transfer

data.

BATCHJOB

Allow source IP 9.85.115.1, deny all others!

Allow source IP 9.85.114.1, deny all others!

Customer A network

Customer B network

10.1.1.0/24

10.1.2.0/24

SRCIP

Jobname CUSTAJOB 9.55.112.1

Jobname CUSTBJOB 9.55.113.1

Jobname User1* 555:555::222

DESTIP 10.1.1.0/24 9.55.114.1

DESTIP 10.1.2.0/24 9.55.115.1

ENDSRCIP

CINET: Supported if only one stack configured – or multiple

stacks configured but all applications have stack affinity

Source IP address

9.85.114.1

Source IP address

9.85.115.1

– All TCP/IP stacks in the sysplex that participate in EXPLICITBINDPORTRANGE processing should have the same port range specified. To ensure this, specify the GLOBALCONFIG EXPLICITBINDPORTRANGE statement in a file that is specified in an INCLUDE statement in the TCP profile data sets of all the participating stacks.

– The port range defined on the EXPLICITBINDPORTRANGE parameter should not overlap any existing port reservations of any TCP/IP stacks in the sysplex. Any reserved ports that are within the range are excluded.

– The EXPLICITBINDPORTRANGE port range must be large enough, but if the size is too large, there are fewer ports available for local ephemeral port allocation.



DVIPA Details:

DVIPA Route and Alternate

Distribution Options



DVIPA Route

• VIPAROUTE on the VIPADYNAMIC statement is used to select a route from a distributing stack or a backup distributing stack to a target stack. This allows a path other than the XCF link to be used.

• Note that GRE (Generic Routing Encapsulation) is used for IPv4. IPv6 does not need GRE to have an outer IP header for encapsulation.

– The GRE encapsulation process increases the size of the forwarded packet by 28 bytes. As a result, if the size of the encapsulated GRE packets are larger than the maximum transmission unit (MTU) of the network interface that will be used for forwarding the packet, the TCP/IP stack might need to perform fragmentation.

z/OS1 TCP/IP

VIPA 10.2.1.101

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

Shared OSA

Session

Request

• TCP/IP Profile

– VIPADYNAMIC VIPAROUTE dynxcf_ip target_ip



Multi-Node Load Balance (MNLB)

• Cisco’s Original MNLB– MNLB defines multiple

real Server addresses as one or more server cluster addresses. Client traffic specifies the server cluster address as its destination address. Service Manager informs Forwarding Agents of best suited Server based on input from WorkLoad Manager.

• Current SysplexDistributor (SD) supported MNLB

– SD may become the Service Manager for MNLB.

WLM

z/OS

WLM

z/OS

WLM

z/OS

WLM

z/OS

Target Servers

Services

Manager

Local Director

L2 Switch L2 Switch

ForwardingAgent

ForwardingAgent

R1 R2 R3

IP NetworkR1, R2, R3Access Routers

ClientWorkstation



Load Balancing Advisor (LBA)• IBM’s Load Balancing Advisor

– Alternative load balancing solution to Sysplex Distributor

– Does not use DVIPAs.

– Requires at least one external load balancer that supports the Server/Application State Protocol (SASP).

– Client traffic specifies the server cluster address as its destination address. LBA Advisor informs LBA Balancer of best suited Server based on input from WorkLoad Manager.

– Different from sysplexdistributor and MNLB because the actual decision of where to route work is made outside of the sysplex.

– UDP is supported as well as TCP.

Load

Balancer

Load

Balancing

Advisor

z/OS

Load

Balancing

Agent

z/OS

Load

Balancing

Agent

z/OS

Load

Balancing

Agent

z/OS



DVIPA Details:

Sysplex Distributor


See the IP Configuration Reference manual for syntax rules.




• There are several things that influence the Sysplex Distributor Distribution:– Target Server Responsiveness (TSR)

– Distribution Method

– Weighted Distribution

– Local Target Preference

– Processor Type

– Importance Level and Crossover

– Timed (Target) Affinity

– QoS Policy Outbound Interface

– QoS Policy Performance Monitoring

– QoS Policy Total Connections Limit

– QoS Policy Maximum Connections Limit

– Share Port WLM



Target Server Responsiveness

(TSR)• Distributor uses several health metrics to monitor Target stacks and

Servers:– TSR (Target Server Responsiveness) fraction, is a compound health-metric

per Target Server (range from 0 (bad) to 100 (good)):• TCSR (Target Connectivity Success Rate) indicates the connectivity between the

Distributing stack and the Target stack. Are new connection requests reaching the Target? (0 is bad, 100 is good)

Client

Distributor

Target

Stack ServerSEF

TCSR

CER

• SEF (Server accept Efficiency Fraction) indicates the Target Server accept efficiency. Is the Server accepting new work? (0 is bad, 100 is good)

• QoS (Quality of Service) fractions take retransmits and packet loss into consideration. (0 is good, 100 is bad)

• Number of half-open connections also impact the TSR value.

• No configuration is required for TSR usage.– TSR is automatically calculated and used

when Sysplex Distributor is defined.

–CER (Connection Establishment Rate) indicates the network connectivity between Server and Client. Are new connections being established? (0 is bad, 100 is good)

• New in z/OS V1.11 CER no longer impacts the TSR value, but is still calculated and included in netstat displays



WLM Weights and DISTMETHOD

• WLM-based forwarding requires WLM goal mode on Distributing and Target systems.

• WLM-based forwarding requires TCP/IP Profile IPCONFIG SYSPLEXROUTING.– If you have not made the changes needed to enable WLM-based forwarding (SYSPLEXROUTING has not

been specified for all participating stacks, or all participating stacks are not configured for WLM GOAL mode), even if BASEWLM or SERVERWLM is specified, the distributing stack will use round-robin forwarding to distribute connections.

• TCPDATA HOSTNAME– The weights received from WLM are returned based on the value of HOSTNAME. The HOSTNAME value is

determined by the search path at initialization time. Verify that all systems have a unique HOSTNAME value to ensure proper distribution.

• WLM-based forwarding requires BASEWLM or SERVERWLM.

• TCP/IP Profile VIPADYNAMIC VIPADISTRIBUTE DISTMETHOD BASEWLM– Distributing stack forwards connections based upon the workload of each of the target stacks.

• WLM-based forwarding based on a comparison of available/displaceable capacity on each system.

• Relative WLM System Weight Recommendations are used.

• TCP/IP Profile VIPADYNAMIC VIPADISTRIBUTE DISTMETHOD SERVERWLM– Distributing stack forwards connections based on a WLM recommendation indicating how well each server is

executing on its system.• WLM-based forwarding based on:

– How well each server is meeting the goals of its service class

– A comparison of available/displaceable capacity on each system given the importance level of the service class.

• Server-specific WLM Weight Recommendations are used.

• TCP/IP Profile GLOBALCONFIG SYSPLEXWLMPOLL seconds (new in z/OS V1.8)– Determines how quickly the sysplex queries to WLM are done, which causes quicker reactions to workload

changes.



Round Robin

• TCP/IP Profile VIPADYNAMIC VIPADISTRIBUTE DISTMETHOD ROUNDROBIN– Causes round robin distribution to target systems rather than using

WLM or Policy information.

– When Round Robin is defined WLM and Policy do not influence distribution.

z/OS1 TCP/IP

VIPA 10.2.1.101

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

Session

Request

1

2 3



Sysplex Distributor (SD)

Distribution Prior to z/OS V1.9

• Target systems vary significantly in terms of capacity

– Small systems versus larger systems

• WLM recommendations may favor the larger systems significantly.

– The target application may not scale well to larger systems, being unable to take full advantage of the additional capacity on the larger systems.

– The result can be that these types of servers when running on larger systems get inflated WLM recommendations and as a result they get overloaded with work.

• SHAREPORT may have a different number of server instances

– One system may have three the other may have only one

• The current Round Robin or WLM recommendations do not change distribution based on the number of server instances on each target.

– Round Robin distributes 1 connection per target stack regardless of the number of shareport server instances on that stack.

– WLM Server-specific weights from a target stack with multiple server instances reflects the average weight.

• Customers may want to reserve some capacity on certain systems for batch type of workloads that get injected into the system during specific time periods and which have specific time window completion requirements.

• If that system is also a target for long running DDVIPA connections

– WLM recommendations will allow that available capacity to be consumed and thereby potentially impact the completion of the batch jobs

– Or vice versa; the connections on that system may suffer from a performance perspective when those jobs are running

LPAR

Capacity LPAR1

LPAR2

LPAR1 LPAR2

SHAREPORT

LPAR1 LPAR2

Fence

off spare

capacity



Weighted Distribution

• Sysplex Distributor (SD) Weighted Distribution z/OS V1.9 Enhancement

– Based on currently active connections

� Weights to balance active connections, not incoming connections.

– Weights configured on the VIPADISTRIBUTE statement per destination IP address

� Objective is to keep the number of active connections distributed according to the configured weights.

� More optimal than traditional round robin or weighted round robin algorithms.

above6030LPAR4

above2010LPAR3

above2010LPAR2

below3020LPAR1

StatusCurrent number of

active connections

Configured

weights

Case 1

LPAR1

LPAR2

LPAR4

SHAREPORT

LPAR3

Fence

off spare

capacity

below4030LPAR4

on target2010LPAR3

on target2010LPAR2

above6020LPAR1

StatusCurrent number of

active connections

Configured

weights

Case 2

• LPAR0 Distributing Stack

• TCP/IP Profile

VIPADYNAMIC VIPADISTRIBUTE

DISTMETHOD WEIGHTEDACTIVE DESTIP

dynxcf_ipaddr1 WEIGHT value2



dynxcf_ipaddr4 WEIGHT value4…

– Note: WLM and Policy information are not used.

LPAR0

• LPAR1 Target Stack

• TCP/IP Profile

IPCONFIG

DYNAMICXCF dynxcf_ipaddr1…


• TCP/IP Profile

IPCONFIG



• TCP/IP Profile

IPCONFIG



• TCP/IP Profile

IPCONFIG


weight2

weight1

weight1weight3



Local Target Preference (new in z/OS

V1.8)

• OPTLOCAL (new in z/OS V.18) on the VIPADYNAMIC VIPADISTRIBUTE statement causes target stacks to optimize sysplex connections for which both endpoints reside on the same stack. When this value is specified, target stacks should bypass sending connection requests to the sysplex distributor stack for connections to a distributed DVIPA and port pair that reside locally, and instead process the connection locally using local optimizations. The local target stack continues to favor the local stack unless conditions on the local stack become unfavorable as defined by the value specified.

– NOOPTLOCAL causes target stacks to send locally originating connection requests to distributor stack even when both endpoints reside on same target stack.

– A value of 0 indicates that connections originating from a target stack within sysplex should always bypass sending connection request to the sysplex distributor.

– A value of 1 indicates that connections originating from a target stack within the sysplex should always bypass sending the connection request to the sysplex distributor as long as the WLM weight for the server on the local stack is not 0.

– If a value in the range 2 - 16 is specified, the value is used as a multiplier against local target stack’s raw WLM weight to cause it to be favored over other targets.

– Regardless of the value specified on the OPTLOCAL parameter, if no local server is available, or the SEF is less than 75 or the abnormal transaction completions is greater than 250, or the health indicator is less than 75, connections are sent to the distributing stack.

– If round-robin distribution is configured, the OPTLOCAL value is forced to a value of 0.

– TIMEDAFFINITY and OPTLOCAL are mutually exclusive.

– TIMEDAFFINITY and OPTLOCAL take precedence over DISTMETHOD.

• As part of the OPTLOCAL feature, regardless of how a target is chosen (OPTLOCAL or NOOPTLOCAL configured), if the target and client are using the same stack, then the data traffic will stay local (distributor will not be part of the datapath).

z/OS1 TCP/IP

VIPA 10.2.1.101

z/OS

appl Server appl Server

OPTLOCALNOOPTLOCALappl Client

z/OSappl Server

appl Client

z/OS



Processor Type

– Sysplex Distributor

– z/OS Load Balancing Advisor

– (DNS/WLM)

• WLM (Work Load Monitor) z/OS V1.9 Enhancements– Allows Communications Server (CS) access to improved weight information in two different ways

– SERVERWLM - server-specific weights (Sysplex Distributor and Load Balancing Advisor)• Uses composite WLM weight that factors in capacity of specialty processors based on the portion they are used

by the specific server address space as determined by WLM.

• No configuration changes needed for SERVERWLM.

– BASEWLM - system weights (Sysplex Distributor and Load Balancing Advisor)

• Uses composite WLM weight that factors in capacity of specialty processors based on the portion they are used on each system (whole system rather than specific server address space) as determined by WLM.

• Uses separate WLM weights from all processor types and combines them based on CS configuration of how each processor type should impact the combined weights.� An option to configure the mix of processor types that is needed by the application workload is distributed to will be added to both

Sysplex Distributor and Load Balancing Advicer.

VIPADISTRIBUTE ... BASEWLM PROCTYPE(CP=3,zAAP=1,zIIP=0)

• Prior to z/OS V1.9 capacity of specialty processors, such

as zAAP and zIIP, has not been factored in by WLM when

calculating weights for:

System

Processor

Capacity System z9

General

CPs

zAAP

zIIP

General

CPs

zAAP

General

CPs

zSeries z990

zSeries z900



IL SUs

0 0

1 0

2 0

3 500

4 0

5 0

6 0

7 0

IL SUs

0 0

1 0

2 0

3 0

4 0

5 500

6 0

7 0

New

workload

at IL=2

(can

displace

IL=3 to

IL=7

workload)

LPAR1 LPAR2IL CP

SUs

zAAP

SUs

0 0 0

1 0 0

2 0 0

3 900 100

4 0 0

5 0 0

6 0 0

7 0 0

IL CP

SUs

zAAP

SUs

0 0 0

1 0 0

2 0 0

3 100 900

4 0 0

5 0 0

6 0 0

7 0 0

New

workload at

IL=2

(can

displace

IL=3 to IL=7

workload)

New workload

designed to use

90% zAAP and

10% CP

LPAR1 LPAR2

IL 0: High

IL 7: Low

� New in z/OS V1.11 Importance Level (IL) of workload influences distribution target choice.

� For example, if new workload at IL 2 arrives for distribution…– Both targets have 500 service units of

displaceable workload

– Prior to V1.11, they would be equal

� z/OS V1.11 takes IL of displaceable workload into consideration– LPAR2 will be preferred

� New in z/OS V1.11 Crossover Cost of workload being run on a particular type of processor influences distribution target choice.

� For example, if new workload at IL 2 arrives for distribution on 90% zAAP and 10% CP…– Both targets have equal amount of displaceable service units

– Prior to V1.11, they would be equal

� z/OS V1R11 takes crossover cost of workload being run on a CP instead of a zAAP– LPAR2 will be preferred since it has the least amount of

crossover cost

Importance Level and

Crossover



Timed Affinity

• TIMEDAFFINITY on the VIPADYNAMIC VIPADISTRIBUTE statement specifies whether or not a connection from a client to a particular server instance of several served by sysplex distributor shall establish an affinity for future connections from the same client (IP address) to the same Distributed DVIPA and ports.

– A value of 0 means that no affinity is established when a new connection request is distributed to a particular server application instance by sysplex distributor.

– A nonzero value means that when a connection from a client is routed to a particular server instance, any subsequent connectionsfrom the same client to the same Distributed DVIPA and ports are routed to the same server instance until the specified number of seconds have elapsed after the last such connection was closed.

• Timed Affinity is required when using TN3270E printer associations.

• TIMEDAFFINITY and OPTLOCAL are mutually exclusive.

• TIMEDAFFINITY and OPTLOCAL take precedence over DISTMETHOD.

z/OS1 TCP/IP

VIPA 10.2.1.101

z/OS

TN3270A

z/OS

TN3270B

z/OS

TN3270C

Session

Request

1

23



Quality of Service (QoS)

Policy Outbound InterfacePROFILE.TCPIPVIPADYNAMIC

VIPADEFINE 10.1.1.1

VIPADISTRIBUTE 10.1.1.1 PORT 2000 DESTIP ALL

ENDVIPADYNAMIC

Policy Agent DefinitionsOutboundInterface ip@1 ip@2 ip@3

ForLoadDistribution TRUE

Policy Agent DefinitionsOutboundInterface ip@1

ForLoadDistribution TRUE

Subnet 1Subnet 1

Subnet 2Subnet 2

Sysplex Distributor

Distributor Stack

Sysplex Distributor

Target Stack

ip@1

Sysplex Distributor

Target Stack

ip@2

Sysplex Distributor

Target Stack

ip@3

• Incoming connection requests can be distributed to

different target stacks within the sysplex by the

sysplex distributor distributing stack.



Outbound Interface Configuration

• QoS Outbound Interface routing requires:– WLM based routing

– Distributing stack Policy Agent configuration:• PolicyAction PolicyScope DataTraffic (or Both) OutboundInterface ip_addr

• PolicyRule ForLoadDistribution TRUE

– Where ip_addr specifies the DynamicXCF addresses of the Sysplex Distributor Targets to receive the connections.

– A value of 0 can be specified for the interface, which indicates to the Sysplex Distributor Distributing stack that if it cannot distribute the request to a target stack on one of the specified interfaces, then the request can be distributed to any of the other eligible target stacks.



QoS Policy Performance

Monitoring• Policy Agent configuration statement PolicyPerfMonitorForSDR Enable is required for Policy

Performance Monitoring.– Assigns cumulative and discrete weight fractions to the monitored policy performance data and sends them to the Sysplex Distributor Distributing stack as the

monitored data crosses defined thresholds.

• PolicyPerfMonitorForSDR parameters:– Enable | Disable Enables or disables the policy performance monitor function. The weight fractions

determined for the loss ratio and timeout ratio are added together to form a single weight fraction before being sent to the Sysplex Distributor Distributing stack. One weight fraction is generated for each DVIPA/port pair on a Sysplex Distributor Target stack.

– SamplingInterval Specifies the interval in seconds for sampling policy performance data.

– LossRatioAndWeightFr Specifies two numbers. The first is the unit ratio of retransmitted bytes (loss) over transmitted bytes, in tenths of a percent (1 - 1 000). The second number is the weight fraction to be returned to the Sysplex Distributor Distributing stack, in percentage (1 - 100). When present, this parameter results in creation of a threshold table.

– LossMaxWeightFr Specifies the max weight fraction to be assigned for the loss ratio factor.

– TimeoutRatioAndWeightFr Specifies two numbers. The first number is the unit ratio of the number of time-outs over transmitted packets, in tenths of a percent (1 - 1 000). The second number is the weight fraction to be returned to the Sysplex Distributor Distributing stack, in percentage (1 - 100).

– TimeoutMaxWeightFr Maximum weight fraction to be assigned for the timeout ratio factor.

– MaxConnWeightFr Specifies three percentages that are used in calculating the connection limit portion of the policy action (service level) weight fractions.

• Each percentage must be in the range 1-100, and each value must be greater than or equal to the preceding value.

• When calculating policy action weight fraction, the number of active connections to a target DVIPA/Port is compared with maximum connections allowed for the associated policy action as follows:

– When the number of active connections reaches the percentage of maximum connections specified by the first number, the policy action weight fraction is set to MAX (50%, current calculated value).

– When the number of active connections reaches the percentage of maximum connections specified by the second number, the policy action weight fraction is set to MAX (85%, current calculated value).

– When the number of active connections reaches the percentage of maximum connections specified by the third number, the policy action weight fraction is set to 100%.



QoS Cumulative and Discrete

Weight Fractions• Discrete Weight Fractions requires:

– Distributing Stack• WLM Distribution Method (BASEWLM or SERVERWLM)

– Distributing and Target Stacks• Policy Agent configuration

– PolicyPerfMonitorForSDR Enable

– PolicyAction name

– PolicyRule ForLoadDistribution TRUE

• Note: PolicyAction name must be the same on the

distributing and target stacks

– Target Stacks• Policy Agent configuration

– PolicyRule ForLoadDistribution FALSE

– Sent over TCP connections

– Separate for each Service Level

• Cumulative Weight Fractions are sent when:– Distributing Stack

• WLM Distribution Method (BASEWLM or SERVERWLM)

– Distributing and Target Stacks• Policy Agent configuration

– PolicyPerfMonitorForSDR Disable

– Sent over XCF connections

– Not separate by Service Level

• Service Level configured by– PolicyRule ApplicationPriority serv_lev

Sysplex

Distributor

Policy

Agent

Statistics

Stack

QoS Weight

Fraction

Sysplex Distributor

Target Stack

Sysplex

Distributor

Policy

Agent

Statistics

Stack

QoS Weight

Fraction

Sysplex Distributor

Target Stack

Sysplex Distributor

Distributing Stack

Sys Dist /

WLM

Interface

Policy

Agent

Policies

Stack

QoS Weight

Fraction

Coupling

Facility



Total Connections and

Maximum Connections Limits• QoS Policy Performance Monitoring may be used with a Total Connections Limit

policy and/or a Maximum Connections Limit.

• Total Connections Limit– PolicyAction PolicyScope TR TotalConnections tot_num

• Where tot_num is the number of concurrent connections allowed

• TR policy is considered CONSTRAINED when– Active number of connections > 90% tot_num

• When TR policy is CONSTRAINED– All policy action fractions, including the default, will be forced to 100%

• Maximum Connections Limits– PolicyAction PolicyScope DataTraffic MaxConnections max_num

• Where max_num is the number of concurrent connections allowed

– Using PolicyPerfMonitorForSDR default MaxConnWeightFr 70 85 95• When active connections reaches 70% (first number on MaxConnWeightFr parameter) then the weight

fraction is set to 50%.

• When active connections reaches 85% (first number on MaxConnWeightFr parameter) then the weight fraction is set to 85%.

• When active connections reaches 95% (first number on MaxConnWeightFr parameter) then the weight fraction is set to 100%.

• Recommendations– Use MaxConnections rather than PolicyScope TR because it has multiple throttling

percentages and more granularity.

– Avoid having multiple applications map to the same QoS policy action.



Share Port WLM

• PORT statement in the TCP/IP Profile is used to reserve a port for a specific job name.

• Parameter SHAREPORT or SHAREPORTWLM may be specified on the PORTstatement.

• When SHAREPORT is specified, TCP/IP allows multiple listeners to listen on the same combination of port and interface. Specification of this keyword causes incoming connection requests for the port to be distributed among the listeners using a weighted round-robin distribution method based on the servers’ accept Efficiency Fractions (SEFs) of the listeners sharing the port.

• By specifying SHAREPORTWLM on the PORT statement, connections are distributed in a weighted round-robin fashion based on the WLM server-specific recommendations, as modified by the Server accept Efficiency Fraction (SEF).

• SHAREPORT and SHAREPORTWLM are only valid for TCP ports.

• If the shared port is a Sysplex Distributed port and SERVERWLM is the distribution method that is being used, then SHAREPORTWLM should be coded on each target’s PORT statement to take advantage of the WLM server-specific recommendations when connections are received at the target; if it is not, new connections continue to be distributed using the existing SHAREPORT algorithm when they are received at the target.

• Server’s Accept Efficiency Fractions (SEF)– The server’s accept Efficiency Fractions (SEF) is a measure of the efficiency of the server application in

accepting new connection requests and managing its backlog queue.



DVIPA Details:

Sysplex Distributor

TN3270 LU Support



TN3270 Coordinated LU Names

• Active LU names must be unique across all LPARs in the Sysplex.

• New in z/OS V1.10 TN3270 LU names may be coordinated across the Sysplex.– One telnet server may be configured as a LU Name Server (LUNS) for multiple LU Name Requesters

(LUNR) in the sysplex so that the telnet port may be configured for Sysplex Distributor.• A shared set of LU names are defined as a shared name space.

– Generic LU requests pick an available LU name in the shared name space.

– Specific LU requests verify that the requested LU is not already in use in the Sysplex.

– Reconnect with specific LU name requests after temporary network failure is supported.• Manage timemark processing on existing connection(s) across sysplexed TN3270 servers:

– Terminate "stale" TN3270 connections

– Clean up and terminate associated SNA session

– Free LU name(s) for terminated connections

– Process reconnect on any TN3270 server in the Sysplex

VTAMSessMgt

LUTable A

Telnet A LUNR

CICS

LUCICSLUTable A+B

Telnet 1 LUNS

LUTable B

Telnet B LUNR

Local LUs

LUNS LUs

Some LUs are managed by the LUNS,

others are managed locally.

It depends on the LU group definition.



No LUNS, LUNS, or LUNR

• Three TN3270E Telnet server roles:– Classic Telnet

• Not a member of a Telnet XCF group

• No shared LU groups permitted– XCF Telnet

• Must Join XCF Group

• LUNR (LU Name Requestor) capable– Ports can use shared LU groups

– XCF LUNS (LU Name Server) Telnet

• Must Join XCF Group

• LUNR capable

• Coordinates shared LU name assignments

• One LUNS in control at any given time

– others are standby

• Manual or automated takeover by standby LUNS



DVIPA Details:

Fast Response Cache

Accelerator (FRCA)



Fast Response Cache

Accelerator (FRCA)• FRCA caches Web

pages within the TCP/IP stack.–Requests are handled

without traversing the full protocol stack up to the Web server.

–Significant performance improvements when compared to the Web server handling all requests.

Web server

Sockets layer

TCP

Layer

IP networking layer

Network interfaces

FRCA

Cache

FRCA

The Web server stores

static objects in the

cache

The Web server

provides simple exits

to parse URIs

Cached static

objects are served

from the cache

without the Web

server accepting the

TCP connection



FRCA use by WebSphere

Application Server (WAS)• Cached responses can be served with

high performance using a minimal amount of CPU cycles.

– Serve static requests from the FRCA cache.• Provide equivalent performance on WAS as is

possible with the FRCA cache on the web server.

– Serve dynamic content from the FRCA cache.• Serve the same content that the Dynamic Cache

serves but serve it from the FRCA cache.

– Record HTTP Access Log entries for requests served from the FRCA cache.

• New in WebSphere Application Server V7, WAS will start exploiting the FRCA cache.

– Amount of CPU time needed to process a request is dramatically reduced using FRCA as compared to Dynamic Caching.



Sysplex Distributor FRCA

Support• FRCA connections that are serviced by the cache will not be measured against

a server’s health.

• Half-open connections will not be weighed as heavily against the server’s health.– Half-open connections reflect more of a routing issue than a problem with the server.

• TCPSTACKSOURCEVIPA will no longer use MOVING DVIPAs (Dynamic VIPA).– Normal source IP address selection process will be used.

• Display TCPIP,tcpproc,NETSTAT,ALL to see the new ServerBacklog and FRCABacklog fields.– Established: Full three-way handshake has completed.

– Half-open: You have received SYN and responded with a SYN+ACK, but are waiting for the final ACK from the client.

Current Backlog

EstablishedHalf-open

Server BacklogFRCA Backlog



DVIPA Details:

QDIO Accelerator



HiperSockets Accelerator

• A single z/OS stack may be used as a “Router” (Accelerator) between remote devices and other LPARs.– The Accelerator stack is the only one that directly connects to the network via QDIO OSAs.

– The Accelerator stack is also connected to the other LPARs via HiperSockets LANs.

– The first packet will travel up thru QDIO to the Accelerator stack and down thru iQDIO device drivers to reach the backend LPAR IP address. After that first packet, all the rest of the packets flow via the accelerated path through the DLC layer, thus bypassing the IP layer in z/OS and reducing path length and improving performance.

• If a specific HiperSockets Acclerator entry is not used for 60 seconds, it is deleted from the iQDIORouting Table.

• Each entry in the iQDIORouting table contains: the backend LPAR IP address, first hop IP address beyond OSA, and OSA link name.

– IP time to live (TTL) Processing bypassed (considered no hop).

– HiperSockets accelerator is for unicast IP packets only.

– There are no packet trace functions available to HiperSockets-accelerator-forwarded packets.

• There may be multiple HiperSockets Acclerators on a single CEC supporting different backend LPARs.

• HiperSockets Accelerator is designed to route traffic into one OSA priority queue and does not honor TOS/DSCP setting in the IP header.– This was done for performance reasons to block as many packets as possible in one write operation out to the OSA-E adapter.

– Pagent's policy-based setting of the type of Service (TOS) will not be used by the OSA.

• Note: Some of the literature refers to HiperSockets Acceleration as "HSA." This acronym may causes confusion because it also stands for Hardware System Area.

System z

z/OS

TCP/IP

LPAR

z/OS

TCP/IP

LPAR

z/OS

TCP/IP

LPAR

z/OS

TCP/IP

LPAR

z/OS

TCP/IP

LPAR

z/OS

TCP/IP

LPAR

OSA OSA

HiperSockets LAN HiperSockets LAN

Remote

DeviceRemote

DeviceRemote

DeviceRemote

DeviceRemote

DeviceRemote

DeviceRemote

Device

• Accelerator stack TCP/IP Profile coding:

– IPCONFIG IQDIORouting QDIOPriority n

• IQDIORouting enables HiperSocketsAccelerator

• QDIOPriority may be coded to define an OSA QDIO outbound queue between 1 and 4.

– The default is 1, which is the high priority queue.

– IPCONFIG PATHMTUDiscovery

• May be desired because IP Fragmentation is not supported (VTAM Device Drivers do not fragment)

– IPCONFIG DATAGRAMFWD

• Required because the first packet to a backend LPAR will use the IP routing.

– PRIROUTER

• On the OSA DEVICE statement(s) so that packets to unknown destinations may be forwarded to the backend IP addresses.

– Do not code IPCONFIG IPSECURITY

• HiperSockets Accelerator and IPSec are mutually exclusive.

HiperSockets Accelerator was most helpful

because of OSA limits on number of stacks

per OSA. Since the introduction of

HiperSockets Accelerator OSA stack limits

have increased and OSA VMAC support

(announcement 206-190) has caused

HiperSockets Accelerator to be used less.



Renamed QDIO Accelerator

Transport protocols

Networking protocols

(IPv4/IPv6)

DLC protocols

OSA

Transport protocols

Networking protocols

(IPv4/IPv6)

DLC protocols

Traditional routing

Transport

protocols

Networking protocols (IPv4/IPv6)

DLC protocols

OSA

Transport

protocols

Networking protocols (IPv4/IPv6)

DLC protocols

QDIO accelerator

• New in z/OS V1.11

• Accelerator support expanded to include all combinations

of QDIO and iQDIO traffic

YesYesInbound

iQDIO

YesYesInbound

QDIO

Outbound

iQDIO

Outbound

QDIO

• Supports Sysplex Distributor (SD)

– When traffic to target stack is sent over HiperSockets

Dynamic XCF or QDIO as a result of VIPAROUTE

definition.



Multi-tier Applications and

non-z/OS Targets



Multi-tier Applications

• New in z/OS V1.11 Sysplex Distributor support for multi-tiered z/OS workloads – When choosing a tier 1 z/OS target server, SD considers the availability and

WLM recommendations of the tier 2 server that will be used on the same system.

• Increase the value of the existing optimized local support.

– Reduces cross-LPAR traffic for multi-tier z/OS workloads, which reduces final end-user response times.

VIPADEFINE

TIER1 DVIPA1

VIPADISTRIBUTE

TIER1 WASCICS

DVIPA1 Port 8080

DISTMETHOD SERVERWLM

DESTIP XCF1 XCF2 XCF3 XCF4

1

VIPADEFINE

TIER2 DVIPA2

VIPADISTRIBUTE

OPTLOCAL

TIER2 WASCICS

DVIPA2 Port 9001



2

CICS1

WLM 1

WAS1

WLM 10

TCPIP1

LPAR1

CICS2

WLM 10

WAS2

WLM 10

TCPIP2

LPAR2

Tier 1 SD

DVIPA11

Tier 2

GROUP1

Tier 1

GROUP1

CICS3

WLM 1

WAS3

WLM 10

TCPIP3

LPAR3

CICS4

WLM 10

WAS4

WLM 10

TCPIP4

LPAR4

Tier 2 SD

DVIPA22



CPCSCOPE

• New in z/OS V1.11 Sysplex Distributor support for CPC grouping of tier 1 and tier 2 workload.

VIPADEFINE

TIER1 DVIPA1

VIPADISTRIBUTE

TIER1 WASCICS

DVIPA1 Port 8080



1VIPADEFINE

TIER2 CPCSCOPE DVIPA2

VIPADISTRIBUTE

OPTLOCAL

TIER2 WASCICS

DVIPA2 Port 9001


DESTIP XCF1 XCF2

2

VIPADEFINE


VIPADISTRIBUTE

OPTLOCAL

TIER2 WASCICS

DVIPA3 Port 9001


DESTIP XCF3 XCF4

3

– To avoid cross CPC traffic, tier 2 servers may be limited to the same CPC as the tier 1 server.

– OPTLOCAL may be used to prefer the local LPAR for the tier 2 server.

– Tier 2 distributors may be defined with CPCSCOPE, one for each CPC with the same group name, ie. WASCICS.

• Eligible tier 2 target servers are limited to the same CPC as the tier 1 server.

• Tier 1 servers are configured differently on the two different CPCs.

– Tier 1 servers on CPC1 are configured to use tier 2 DVIPA2 while tier 1 servers on CPC2 are configured to use tier 2 DVIPA3.

VIPADEFINE

TIER1 DVIPA1

VIPADISTRIBUTE

TIER1 WASCICS

DVIPA1 Port 8080



1VIPADEFINE


VIPADISTRIBUTE

OPTLOCAL

TIER2 WASCICS

DVIPA2 Port 9001


DESTIP XCF1 XCF2

2

VIPADEFINE


VIPADISTRIBUTE

OPTLOCAL

TIER2 WASCICS

DVIPA3 Port 9001


DESTIP XCF3 XCF4

3

CICS1

WLM 1

WAS1

WLM 10

TCPIP1

LPAR1

CICS2

WLM 10

WAS2

WLM 10

TCPIP2

LPAR2

Tier 1 SD

DVIPA11

Tier 2

GROUP1

Tier 1

GROUP1

CICS3

WLM 1

WAS3

WLM 10

TCPIP3

LPAR3

CICS4

WLM 10

WAS4

WLM 10

TCPIP4

LPAR4

Tier 2 SD

DVIPA22

Tier 2 SD

DVIPA33

CPC1 CPC2

Backup path

remains within

same CPC

Client Client

– To avoid cross CPC traffic, tier 2 servers may be limited to the same CPC as the tier 1 server.

– OPTLOCAL may be used to prefer the local LPAR for the tier 2 server.

– Tier 2 distributors may be defined with CPCSCOPE, one for each CPC with the same group name, ie. WASCICS.

• Eligible tier 2 target servers are limited to the same CPC as the tier 1 server.

• Tier 1 servers are configured differently on the two different CPCs.

– Tier 1 servers on CPC1 are configured to use tier 2 DVIPA2 while tier 1 servers on CPC2 are configured to use tier 2 DVIPA3.

Example from IP

Configuration Guide



non-z/OS Targets

• New in z/OS V1.11 Sysplex Distributor supports non-z/OS targets –Initial candidate targets are XML appliances,

such as DataPower.• Requires DataPower support level

–Requires a SD specific agent on the target platform to provide weights, availability, and connection state information back to SD.

–Connection forwarding based on MAC-level forwarding using GRE encapsulation.

SD

Agent

DP@A1

SD

Agent

DP@A1

SD

Agent

DP@A1

Sys1

z/OS

SD Tier-1@1

Client

Connect to @1

Data

Power

Devices



non-z/OS Targets (DataPower) and

CPCSCOPE• When a sysplex spans two different sites CPCSCOPE may be used to cause

the tier 2 servers to be chosen in the same site as the tier 1 servers.

VIPADEFINE

TIER1 DVIPA1

VIPADISTRIBUTE

TIER1 WASCICS

DVIPA1 Port 8080



1

VIPADEFINE

TIER1 10.91.1.1

VIPADISTRIBUTE

TIER1 CICSGRP

GRE CONTROLPORT 5601

10.91.1.1 Port 8080

DISTM TARGCONTROLLED

DESTIP 201.81.10.1 201.81.10.2

201.81.20.1 201.81.20.2

1VIPADEFINE


VIPADISTRIBUTE

OPTLOCAL

TIER2 WASCICS

DVIPA2 Port 9001


DESTIP XCF1 XCF2

2

VIPADEFINE

TIER2 CPCSCOPE 10.81.1.1

VIPADISTRIBUTE

TIER2 CICSGRP

10.81.1.1 Port 9001


DESTIP ALL

VIPADEFINE CPCSCOPE 201.81.10.7

2

CICS1

TCPIP1

LPAR1

CICS2

TCPIP2

LPAR2

Tier 1 SD10.91.1.1Port 8080

1

CICS3

TCPIP3

LPAR3

CICS4

TCPIP4

LPAR4

CPC1 CPC2

Client Client

– To avoid cross site traffic, tier 2 servers may be limited to the same site as the tier 1 server.

– Tier 2 distributors may be defined with CPCSCOPE, one for each site with the same group name, ie. CICSGRP.

• Eligible tier 2 target servers are limited to the same site as the tier 1 server.

• Tier 1 servers are configured differently in the two different sites.

– Tier 1 servers on CPC1 are configured to use tier 2 DVIPA 10.81.1.1 while tier 1 servers on CPC2 are configured to use tier 2 DVIPA 10.81.2.2.

Site 1 Site 2

Tier 2 SD10.81.1.1Port 9001

2Tier 2 SD10.81.2.2Port 9001

3

Subnet201.81.10.0/24

Subnet201.81.20.0/24

DVIPA201.81.10.7DataP1201.81.10.1

DVIPA201.81.10.7DataP2201.81.10.2

DVIPA201.81.20.7DataP3201.81.20.1

DVIPA201.81.20.7DataP4201.81.20.2

VIPADEFINE


VIPADISTRIBUTE

OPTLOCAL

TIER2 WASCICS

DVIPA2 Port 9001


DESTIP XCF1 XCF2

2

VIPADEFINE

TIER2 CPCSCOPE 10.81.2.2

VIPADISTRIBUTE

TIER2 CICSGRP

10.81.2.2 Port 9001


DESTIP ALL

VIPADEFINE CPCSCOPE 201.81.20.7

3

Example from IP

Configuration Guide



DVIPA Details:

Sysplex Autonomics



Automatic Functions

• Sysplex health is always automatically monitored.

• Without Sysplex Autonomics sysplex problems which do not result in DVIPA termination, can prevent needed recovery.– When a participating stack “hangs”, recovery actions will not

occur.

– New connection requests will continue to be sent to the target. The stack may even appear lightly loaded since the DVIPA applications are not processing any new work.

• With Sysplex Autonomics:– If a problem is detected the stack may be automatically removed

from the sysplex.

– When a problem is resolved a stack may be automatically added back into the sysplex.



Monitor and Action

• Monitor Sysplex Health– Internal Resources and Conditions, ie.:

• Timely execution of sysplex functions

• Available Private Storage

• Common Storage

– External Resources, ie.:• Availability of VTAM

• Availability of OMPROUTE if it is being used

– Monitoring is always active.• Time period between monitoring snap shots is

configurable.

CS

TCP/IP

StackIP#1

CF

EZBTCPC$

Dynamic XCF

IP Network

CS

TCP/IP

Stack

IP#2

CS

TCP/IP

StackIP#3

Sick? – Better remove

myself from Sysplex!

? OMPROUTE

? VTAM

z/OS Sysplex

?

• Action When a Problem is Detected– Once a problem is detected a message is sent to the console. Further actions

depend on whether GLOBALCONFIG SYSPLEXMONITOR RECOVERY or NORECOVERY was specified.

– Each sysplex member can automatically leave the sysplex if it determines that it is no longer able to function correctly as a router, target, backup, or owner of a DVIPA.

• See the IP Configuration Guide, chapter “TCP/IP in a sysplex” for more details and a table that maps monitored problems and actions.



TCP/IP Startup

• Pre z/OS V1R6: – DVIPAs unavailable between T2 and T4– Availability issue especially if the DVIPAs were taken back from backup stacks

• z/OS V1R6, V1R7, and V1R8: – GLOBALCONFIG DELAYJOIN delays joining the Sysplex until after OMPROUTE has initialized– Solves the issue with taking back DVIPAs too early– Autologged servers now fail when they try to bind to their DVIPAs at T3, and need to be restarted after T5

• z/OS V1R9:– AUTOLOG entry option added to specify DELAYSTART to delay Autolog until after the stack has joined the Sysplex

Stack address

space starts

Stack joins

Sysplex and takes

back DVIPAs

Autologged server

binds to DVIPA

OMPROUTE comes

up and begins

advertising DVIPA

Stack address

space starts

Stack address

space starts

Stack joins

Sysplex and takes

back DVIPAs

Stack joins

Sysplex and takes

back DVIPAs

Autologged server

binds to DVIPA

Autologged server

binds to DVIPA

OMPROUTE comes

up and begins

advertising DVIPA

OMPROUTE comes

up and begins

advertising DVIPA

Pre z/OS V1.6

z/OS V1.6

z/OS V1.9

T1 T2 T3 T4 T5 T6



Monitor Sysplex Support

• MONSYSPLEX– You can identify which links and interfaces are critical for inbound network connectivity for a distributed

DVIPA or DVIPA-owning TCP/IP stack. Specify which links and interfaces are to be monitored by sysplex autonomics using the MONSYSPLEX parameter on the LINK and INTERFACE statements.

• LINK … MONSYSPLEX or NOMONSYSPLEX

• INTERFACE … MONSYSPLEX or NOMONSYSPLEX

• GLOBALCONFIG SYSPLEXMONITOR– The rest of the Sysplex Monitor and Automatic Recovery is defined by the GLOBALCONFIG

SYSPLEXMONITOR statement.

– New in z/OS V1.8• If SYSPLEXMONITOR is not specified, then the sysplex autonomics function uses a default timer interval of 60 seconds

and performs the NORECOVERY action if a sysplex problem is detected.

• GLOBALCONFIG SYSPLEXMONITOR MONINTERFACE considerations:– If the TCP/IP stack detects that all monitored interfaces are inactive as a result of stopping devices or

interfaces, the TCP/IP stack can issue messages regarding the problem and possibly trigger a recovery action. You can disable monitoring of these interfaces before stopping the devices or interfaces, using the VARY TCPIP,,OBEYFILE command and specifying the NOMONSYSPLEX keyword for the LINK or INTERFACE statement.

– If MONINTERFACE and DYNROUTE option is configured (or defaulted), and the first hop routers are brought down for maintenance, an inadvertent recovery action might be triggered. You can temporarily disable the monitoring of dynamic routes using the VARY TCPIP,,OBEYFILE command and specifying MONINTERFACE NODYNROUTE on the SYSPLEXMONITOR parameter of the GLOBALCONFIG statement.



TCP/IP Profile GLOBALCONFIG

SYSPLEXMONITOR Options• AUTOREJOIN | NOAUTOREJOIN specifies whether TCP/IP should automatically rejoin the TCP/IP sysplex group

when a detected problem is relieved after the stack has left the sysplex group.

• DELAYJOIN | NODELAYJOIN specifies whether TCP/IP should delay joining or rejoining the TCP/IP sysplex group (EZBTCPCS) during stack initialization, or rejoining the sysplex group following a VARY TCPIP,,OBEYFILE command.

– NODELAYJOIN Attempt to join the TCP/IP sysplex group.

– DELAYJOIN Delay joining the TCP/IP sysplex group and processing VIPADYNAMIC block or DYNAMICXCF statements during stack initialization until OMPROUTE is started and active.

• DYNROUTE | NODYNROUTE (new in z/OS V1.8) specifies whether TCP/IP should monitor the presence of dynamic routes over monitored network links or interfaces.

– NODYNROUTE The TCP/IP stack should not monitor the presence of dynamic routes over monitored network links or interfaces.

– DYNROUTE The TCP/IP stack should monitor the presence of dynamic routes over monitored network links or interfaces.

– This is useful in detecting problems that OMPROUTE is having in communicating with other routing daemons on the selected networkinterfaces.

– DYNROUTE cannot be specified when NOMONINTERFACE is configured (or is the default value).

– If DYNROUTE is specified, also specify DELAYJOIN to avoid a scenario where the stack leaves the TCP/IP sysplex group before OMPROUTE is started.

• MONINTERFACE | NOMONINTERFACE (new in z/OS V1.8)– NOMONINTERFACE The TCP/IP stack should not monitor the status of any network links or interfaces.

– MONINTERFACE The TCP/IP stack should monitor the status of specified network link or interfaces. The interfaces or links being monitored are those that are configured with the MONSYSPLEX keyword on the LINK or INTERFACE statement.

– This option can be useful in environments where the dynamic XCF interface is not configured as an alternate network path for this stack (for example, where no dynamic routes are advertised over dynamic XCF interfaces and no static or replaceable static routes are defined over those interfaces).

• RECOVERY | NORECOVERY specifies the action to be taken when a sysplex problem is detected.– NORECOVERY When a problem is detected, issue messages regarding the problem but take no further action.

– RECOVERY When problem is detected, issue messages, leave the TCP/IP sysplex group, and delete all DVIPA resources owned by this stack.

• TIMERSECS seconds determines how quickly the sysplex monitor reacts to problems with needed sysplex resources.



Deactivate and Reactivate

• Commands to allow TCP/IP to relinquish or reclaim ownership of a DVIPA– VARY TCPIP,,SYSPLEX,DEACTIVATE

• DVIPA is deactivated and a configured backup stack will takeover the DVIPA

• Backup DVIPA can be deactivated also, removing eligibility as a backup

TCP and UDP

IP

Interfaces

Stack 2

VIPADEFINE 255.255.255.0 192.168.1.3

VIPABACKUP 100 192.168.1.1

192.168.1.1

192.168.1.1

TCP and UDP

IP

Interfaces

Stack 1

VIPADEFINE 255.255.255.0 192.168.1.1

VIPABACKUP 100 192.168.1.3

v tcpip,,deactivate,dvipa=192.168.1.1

192.168.1.1

v tcpip,,reactivate,dvipa=192.168.1.1

192.168.1.1

– VARY TCPIP,,SYSPLEX,REACTIVATE• Original owner can regain ownership

• Can also reactivate a backup DVIPA that’s been deactivated

• These commands can’t be used on DVIPA’s created from VIPARANGE with bind, ioctl(), or the Modvipa utility



Quiesce and Resume• Commands to gracefully quiesce or resume a target system

or application (must be issued on target system):– VARY TCPIP,,SYSPLEX,QUIESCE,option

• TARGET - Quiesces all applications on target stack

• PORT=xxx - Quiesce all applications bound to the specified port on this stack

– JOBNAME=jobname - Allows quiesce of a single application in SHAREPORT group

– ASID=asid - Further qualify job being quiesced (i.e. deal with duplicate jobnames)

• Ability to quiesce a system/application prior to shutdown– No new TCP connections sent to the quiesced target (stack or application)

» For all Distributed DVIPAs that the entity is a target for

– Existing TCP connections are maintained (i.e. non-disruptive)

• Planned maintenance scenarios of the system/application– Allows existing systems/applications to drain work queue prior to shutdown

• Relieves temporary constraints of resources on target system

• Temporary - Does not affect Sysplex Distributor's permanent configuration

• Issued on target system being affected

– VARY TCPIP,,SYSPLEX,RESUME,option• TARGET|PORT|JOBNAME|ASID

• Allows identified target stacks and/or applications to once again be targets for distribution

Target 1 Target 2 Target 3

Sysplex

Distributor

Sysplex Distributor sends work to

three target stacks.

If Quiesce is issued for Target 2 then

no new work is sent to Target 2 but

all current connections continue.


Sysplex

Distributor

No

new work


Sysplex

Distributor

If Resume is issued for Target 2 then

new work is again sent to Target 2.



Quiesce V1.9 Enhancement

• VARY TCPIP,,SYSPLEX,QUIESCE,option

• Prior to z/OS V1.9– Quiesce for an individual application must identify a unique listener.

– If an application has multiple port listeners for a jobname, a quiesce must be issued for each port separately.

– If an application has multiple listeners for the same port, a quiesce for the target must be used.

– Quiesce Port• Effects matching listener, as long as there are not multiple Jobnames or ASIDs.

– Quiesce Port,Jobname• Effects matching listener, as long as there are not multiple ASIDs.

– Quiesce Port,Jobname,ASID• Effects matching listener.

• New in z/OS V1.9– Matching listeners must have the same jobname and ASID, but not port.

– Quiesce Jobname• Effects all matching listeners if they have the same ASID.

– Quiesce Jobname,ASID• Effects all matching listeners.



DVIPA Details:

Sysplex-Wide Security

Associations (SWSA)

See IPsec presentation at http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS2993 for more details.



IPsec Support

• IP Security Protocol (IPsec)– Defined by IETF (RFC 2401-2404,2406-2410,2451,3947-3948,4301-

4305,4308))• Provides authentication, integrity, and data privacy between two IP entities.

• It can protect a segment of the data path or the entire data path.

– Implemented in the Network Layer• May provide blanket protection for all IP applications.

• Does not require modification to existing applications.

Applications

TCP/UDP

IP/ICMP

Data Link

Applications

TCP/UDP

IP/ICMP

Data Link

IPsec



IPSec Security Associations

My PC(1.1.1.1)

My Company’s Main Computer

(2.2.2.2)

Destination 1.1.1.1 IP packet

Use Encryption

Use tunnel mode

SPI=60

Enc. Key

Combine IP packet and key

using ESP algorithm and place

into 1 tunnel mode IP Packet.


Use Encryption

Use tunnel mode

SPI=60

Enc. Key





Use Encryption

Use tunnel mode

SPI=89

Enc. Key





Use Encryption

Use tunnel mode

SPI=89

Enc. Key




SAs SAs

• IPSec SAs are unidirectional, meaning that if two devices are going to send and receive IPSec packets then 2 SAs

are needed (inbound and outbound) on each endpoint; between the 2 endpoints you have 4 SAs.

• Information contained in the SA includes:• IP addresses of the communicating parties

• Algorithms to be used for authentication and/or encryption

• Authentication and encryption keys and the key lifetimes

• A unique identifier known as the Security Parameter Index (SPI)

• Encapsulation mode (tunnel or transport)



IPsec for Distributed DVIPA

(Sysplex Distributor)

• Existing connections to target systems will be preserved if distributing stack fails and backup stack takes over distribution.

– When a DVIPA is moved during DVIPA takeover (planned or unplanned), SWSA automatically reestablishes new IPSec SAs with the same security service characteristics as the SAs that existed on the host that previously owned the DVIPA. The SA reestablishment is transparent to the client that owns the other end of the SA. That is, the SA reestablishment looks like a normal SA refresh.

• The distributing stack(s) require: IPCONFIG IPSECURITY, IKED, the filter/VPN rules for the IPsec traffic to/from the distributed DVIPA.

• Because the outbound traffic does not necessarily pass through the distributing stack SWSA has additional requirements on the target stacks: IPCONFIG IPSECURITY, and identical filter/VPN rules for the IPsec traffic on the target stacks as what is configured on the distributing stack(s). Stacks that are only distributed DVIPA targets do not need IKE, key exchange rules, or DVIPSEC.

• SWSA also requires the use of a coupling facility structure with a name in the form EZBDVIPAvvtt, where vv is the 2-digit VTAM group ID suffix specified on the XCFGRPID start option, and tt is the TCP group ID suffix specified on the GLOBALCONFIG statement in the TCP/IP profile.

• Sysplex-Wide Security Associations (SWSA) may be configured when z/OS-B is a distributing stack and z/OS-A is a target stack.

– IPv4 only.

• Host-to-Gateway: z/OS-A node is a data host and security endpoint both (host), connecting to a security endpoint (gateway) in front of a remote data host.

z/OS-Bz/OS-A

S S

z/OS-Bz/OS-A

SS

• Host-to-Host: z/OS-A node is data host and security endpoint both (host), connecting to a data host and security endpoint remote node (host).



Syntax, Commands, and

References: VIPA Syntax

IP

Configuration

Reference

SC31-8776



VIPA Definition Syntax• TCP/IP Profile DEVICE, LINK, and VIPADYNAMIC statements define static and dynamic VIPAs.

• Device and Link– DEVice dev_name VIRTual dev_num

– LINK link_name VIRTual adapt_addr dev_name

– Defines a static IPv4 VIPA.

– The adapt_addr must be an integer, but the value is ignored. This parameter is included for consistency with the LINK for other device types.

– The device_name or link_name must not start with VIPAD, as this is a restricted keyword.

– A virtual LINK cannot be coded on the START, BEGINROUTES, GATEWAY or TRANSLATE statements, but can be coded on a BSDROUTINGPARMS statement for interface characteristics such as subnet mask.

• INTERFACE…VIRTUAL6…– Use the INTERFACE statement to specify an IPv6 static virtual interface.

• VIPADynamic VIPABackup…ENDVIPADynamic– Designates one or more DVIPAs for which this stack provides automatic backup if the owning stack fails. Another stack is expected, but not

required, to have this same DVIPA defined with a VIPADYNAMIC VIPADEFINE statement.

• VIPADynamic VIPADEFine…ENDVIPADynamic– Designates one or more DVIPAs that this stack should initially own/support. Other stacks can provide backup for these VIPAs if this stack fails.

• VIPADynamic VIPADELete…ENDVIPADynamic– The VIPADELETE statement allows you to remove a DVIPA interface from the VIPADEFINE or VIPABACKUP statement in which it occurs.

• VIPADynamic VIPARange…ENDVIPADynamic– Defines or deletes a subnet for which DVIPA activation requests, by way of a BIND, SIOCSVIPA IOCTL, or SIOCSVIPA6 IOCT are honored.

– VIPARANGE definition statements that are common to more than one stack should be defined in a common file and included in the appropriate stack profiles.

• VIPADynamic VIPADISTribute…ENDVIPADynamic– Enables (VIPADISTRIBUTE DEFINE) or disables (VIPADISTRIBUTE DELETE) the sysplex distributor function for a DVIPA. VIPADYNAMIC

VIPADISTRIBUTE may designate up to 32 stacks and 64 ports where connections for a particular DVIPA can be distributed, including the stack where the DVIPA is defined.

• VIPADynamic VIPAROUTE…ENDVIPADynamic– A VIPAROUTE statement is used to select a route from a distributing stack or a backup distributing stack to a target stack. This route is used for

distribution of all DVIPAs for which a matching dynamic XCF address, or ALL, was specified on a VIPADISTRIBUTE statement. This route is also used for forwarding packets to existing connections on a stack that contains the DVIPA in MOVING status.

• VIPADynamic SMMCASTgroup…ENDVIPADynamic– Specifies the multicast address used for communications between the sysplex distributor and the Cisco routers acting as forwarding agents.

– Required when any VIPADEFINE or VIPABACKUP statement in the profile contains the SERVICEMGR keyword.

• MONSYSPLEX/NOMONSYSPLEX parameter on LINK and INTERFACE statements– Specifies whether or not sysplex autonomics should monitor the link’s/interface’s status.



IPCONFIG Statement• TCP/IP Profile IPCONFIG parameters that effect VIPA.

• NODYNAMICXCF/DYNAMICXCF– Indicates DynamicXCF support status. DynamicXCF required for DVIPA.

• IPSECURITY– Required for Sysplex-wide Security Associations (SWSA). SWSA supports IPv4 only.

– Activates IPv4 IP filtering and IPv4 IPSec tunnel support.

– Use this parameter so that the stack can function with the Communications Server IKE daemon, and for the stack to receive IPv4 IPSec policy information such as IP filter rules from the policy agent.

– IPSec functions can be activated only at initial activation of TCP/IP.

• NOSOURCEVIPA/SOURCEVIPA– NOSOURCEVIPA specifies that TCP/IP does not use the TCPSTACKSOURCEVIPA or the corresponding virtual IP address in the

HOME list as the source IP address for outbound datagrams. NOSOURCEVIPA is the default value.

– SOURCEVIPA specifies that TCP/IP use the TCPSTACKSOURCEVIPA address (if specified) or the corresponding virtual IP address in the HOME list as the source IP address for outbound datagrams that do not have an explicit source address.

• NOSYSPLEXROUTING/SYSPLEXROUTING– NOSYSPLEXRouting specifies that this TCP/IP host is not part of an MVS sysplex domain and therefore does not communicate

interface changes with workload manager (WLM). NOSYSPLEXROUTING is the default value.

– SYSPLEXRouting specifies that this TCP/IP host is part of an MVS sysplex domain and should communicate interface changes to the workload manager (WLM). This is required on all stacks (distributor stacks as well as all target stacks) if WLM-based forwarding is desired.

• NOTCPSTACKSOURCEVIPA/TCPSTACKSOURCEVIPA– NOTCPSTACKSOURCEVIPA specifies that TCP/IP does not use a stack-level IP address as the source address for outbound

TCP connections. The source IP address is governed by the IPCONFIG SOURCEVIPA setting.

– TCPSTACKSOURCEVIPA vipa_addr The IPv4 address (vipa_addr) is used as the source IP address for outbound TCP connections if SOURCEVIPA has been enabled. The vipa_addr value must be a static VIPA or an active dynamic VIPA (DVIPA).

– If you specify the same distributed DVIPA interface for TCPSTACKSOURCEVIPA on multiple target stacks, you also should specify SYSPLEXPORTS on the VIPADISTRIBUTE statement. Otherwise connections might be disrupted because identical connections could be created from more than one stack.

– TCPSTACKSOURCEVIPA is not used when an outbound TCP request is connecting to an IP address that is active in the Home list.

• If the TCP/IP stack is configured to be a sysplex distributor, datagrams destined to a sysplex-distributed dynamic VIPA are forwarded to stacks, whether or not forwarding (DATAGRAMFWD) is enabled.



IPCONFIG6 Statement

• TCP/IP Profile IPCONFIG6 parameters that effect VIPA.

• DYNAMICXCF … SOURCEVIPAINTERFACE– Indicates DynamicXCF support status. DynamicXCF required for DVIPA.

– SOURCEVIPAINTERFACE vipa_name specifies which static VIPA interface is to be used as the source IP address when IPCONFIG6 SOURCEVIPA is specified and outbound packets are sent over the dynamically generated XCF or Same Host interfaces. If the VIPA has multiple IP addresses, then the source VIPA address for outbound packets is selected from among these addresses according to the default source address selection algorithm.

• See the “z/OS CS IPv6 Network and Application Design Guide, SC31-8885” for the default source address selection algorithm.

• IPSECURITY– Activates IPv6 IP filtering and IPv6 IPSec tunnel support. This parameter requires the IPSECURITY parameter to be configured for

IPv4 on the IPCONFIG statement. This parameter must be specified for the stack to function with the Communications Server IKE daemon, and for the stack to receive IPv6 IPSec policy information such as IP filter rules from the policy agent.

– IPSec functions can be activated only at initial activation of TCP/IP.

• NOSOURCEVIPA/SOURCEVIPA– NOSOURCEVIPA specifies that TCP/IP does not use a VIPA address as the source IP address for outbound datagrams.

NOSOURCEVIPA is the default value.

– SOURCEVIPA specifies that TCP/IP use a VIPA assigned to TCPSTACKSOURCEVIPA or SOURCEVIPAINTERFACE as the source address for outbound datagrams that do not have an explicit source address. If multiple addresses are assigned, the source address is selected from among these addresses according to the default source address selection algorithm.

• NOTCPSTACKSOURCEVIPA/TCPSTACKSOURCEVIPA– NOTCPSTACKSOURCEVIPA specifies that TCP/IP does not use a stack-level IPv6 address as the source address for outbound

TCP connections.

– TCPSTACKSOURCEVIPA intf_name allows z/OS clients to specify a sysplex wide source IP address for TCP connections. Intf_name is the name of a static VIPA or a DVIPA interface. If the interface has multiple IP addresses, then the sourcevipa address for outbound packets is selected from among these addresses according to the default source address selection algorithm. If SOURCEVIPA has not been enabled for IPCONFIG6, IPCONFIG6 TCPSTACKSOURCEVIPA is ignored.

– If you specify the same distributed DVIPA interface for TCPSTACKSOURCEVIPA on multiple target stacks, you also should specify SYSPLEXPORTS on the VIPADISTRIBUTE statement. Otherwise connections might be disrupted because identical connections could be created from more than one stack.

– A DVIPA interface that is created by a VIPARANGE statement can have multiple dynamic VIPA addresses associated with it. The actual address chosen as the source IP for the outbound connection is not predictable or necessarily meaningful.



SOURCEVIPAINTERFACE

Parameter• TCP/IP Profile INTERFACE statements with parameter SOURCEVIPA:

– INTERFACE IPAQENET6• Specify an IPv6 OSA-Express QDIO Gigabit Ethernet or Fast Ethernet interface.

• The following OSA-Express features can be defined in QDIO mode for IPv6:– Fast Ethernet

– Gigabit Ethernet

– 1000BASE-T Ethernet

– 10G Ethernet

– INTERFACE IPAQIDIO6• Specify IPv6 HiperSockets connectivity.

– INTERFACE MPCPTP6• Specify IPv6 MPC Point-To-Point Data Link Control for ESCON channels, XCF links in

a sysplex, or simulated devices provided by the IUTSAMEH function in VTAM.

• SOURCEVIPAINTERFACE is optional and allows the customer to specify which static VIPA interface is to be used for SOURCEVIPA (when IPCONFIG6 SOURCEVIPA is in effect).

• If the VIPA has multiple IP addresses, then the sourcevipa address for outbound packets is selected from among these addresses according to the default source address selection algorithm.



SRCIP Statement

• TCP/IP Profile SRCIP statement designates source IP addresses to be used for outbound TCP connections initiated by specified jobs or destined for specified IP addresses, networks or subnets.

• These source IP addresses override source IP address specification based on the following:– The VIPA IP addresses in the HOME list or the SOURCEVIPAINTERFACE specification.

– The TCPSTACKSOURCEVIPA IP address.

• However, the use of the SRCIP block can also be overridden directly by an application through the use of specific socket API options.

• Only one SRCIP block should appear in a configuration data set. Any subsequent SRCIP blocks are ignored and an informational message is displayed. If a syntax error is encountered when processing this statement, an error message is displayed and the entire SRCIP-ENDSRCIP block is ignored (no entries are processed).

• The SRCIP block does not require that the SOURCEVIPA parameter be specified on IPCONFIG and/or IPCONFIG6.

• SRCIP JOBNAME and DESTINATION (destination new in z/OS V1.8) entries can appear in any order. If an outbound connection matches more than one JOBNAME or DESTINATION entry, the order of precedence is:

1. A match on the most specific JOBNAME entry with at least one, non-wildcard character (that is, excluding JOBNAME *)

2. A match on the most specific DESTINATION entry

3. JOBNAME * entry

• If a connection request matches both a jobname other than JOBNAME * and a SRCIP destination address, the matching JOBNAME entry is used. Otherwise, the matching DESTINATION entry is used.

• Restrictions– The source IP address does not need to be defined prior to the processing of the SRCIP block but it must be defined before the

first TCP connect request is issued for the associated destination, otherwise the connect request fails.

– A match to a SRCIP DESTINATION entry cannot be identified until a connect request is issued and the destination is known.

– Some applications establishing outbound TCP connections might issue an explicit bind socket API prior to issuing the connect request. The port assigned during this early bind processing might not be available across the sysplex for the destination-specific source IP address that is determined later at connect request time.

• This could result in an ambiguous situation where multiple outbound connections to the same destination IP address and port have the same source IP address and port.



GLOBALCONFIG EXPLICITBINDPORTRANGE

and SYSPLEXWLMPOLL

• TCP/IP Profile GLOBALCONFIG EXPLICITBINDPORTRANGE and SYSPLEXWLMPOLL parameters effect VIPA.

• GLOBALCONFig EXPLICITBINDPORTRANGE…– EXPLICITBINDPORTRANGE | NOEXPLICITBINDPORTRANGE

indicates whether this stack participates in the allocation of ports from a pool of ports guaranteed to be unique across the sysplex, when processing an explicit bind() of a TCP socket to the IPv4 INADDR_ANY address, or to the IPv6 unspecified address (in6addr_any), and port 0.

– This parameter defines the range used by all stacks participating in Explicit Port Range port allocation processing throughout the sysplex.

– Use this parameter so that you can specify distributed DVIPAs as the source IP address on DESTINATION or JOBNAME rules in a SRCIP block.

• GLOBALCONFig SYSPLEXWLMPoll…– Determines how quickly the sysplex distributor and its target servers poll

WLM for new weight recommendations. A short time results in quicker reactions to changes in target status.



GLOBALCONFIG

SYSPLEXMONitor• TCP/IP Profile GLOBALCONFIG SYSPLEXMONitor is used to define Sysplex Monitoring.

• GLOBALCONFig SYSPLEXMONitor AUTOJOIN– AUTOREJOIN | NOAUTOREJOIN specifies whether TCP/IP should automatically rejoin the TCP/IP sysplex group when a detected problem is relieved after the

stack has left the sysplex group.

– AUTOREJOIN cannot be configured when NORECOVERY is configured (or defaulted).

– AUTOREJOIN should be used when RECOVERY is configured to allow the stack to rejoin the sysplex group without operator intervention.

• GLOBALCONFig SYSPLEXMONitor DELAYJOIN– DELAYJOIN | NODELAYJOIN specifies whether TCP/IP should delay joining or rejoining the TCP/IP sysplex group (EZBTCPCS) during stack initialization, or

rejoining the sysplex group following a VARY TCPIP,,OBEYFILE command.

• GLOBALCONFig SYSPLEXMONitor DYNROUTE– DYNROUTE | NODYNROUTE specifies whether TCP/IP should monitor the presence of dynamic routes over monitored network links or interfaces.

– This level of monitoring is useful in detecting problems that OMPROUTE is having in communicating with other routing daemons on the selected network interfaces.

– DYNROUTE cannot be specified when NOMONINTERFACE is configured (or is the default value).

– Specify DYNROUTE only when OMPROUTE is configured and started; otherwise, the TCP/IP stack might be forced to leave the TCP/IP sysplex group if RECOVERY is coded.

– If DYNROUTE is specified, also specify DELAYJOIN to avoid a scenario where the TCP/IP stack leaves the TCP/IP sysplex group before OMPROUTE is started.

• GLOBALCONFig SYSPLEXMONitor MONINTERFACE– MONINTERFACE | NOMONINTERFACE specifies whether the TCP/IP stack should monitor the status of any network links or interfaces.

– The interfaces or links being monitored are those that are configured with the MONSYSPLEX keyword on the LINK or INTERFACE statement.

– This level of monitoring can further qualify the health of the TCP/IP stack by ensuring that at least one key interface is active and available.

– This option can be useful in environments where the dynamic XCF interface is not configured as an alternate network path for this stack.

• GLOBALCONFig SYSPLEXMONitor RECOVERY– RECOVERY | NORECOVERY specifies the action to be taken when a sysplex problem is detected.

– When a problem is detected, issue messages regarding the problem, leave the TCP/IP sysplex group, and delete all DVIPA resources owned by this stack.

– Recovery is the preferred method of operation because other members of the TCP/IP sysplex can automatically take over the functions of a member with no actions needed by an operator.

– IBM Health Checker for z/OS can be used to check whether the RECOVERY parameter has been specified when the IPCONFIG DYNAMICXCF or IPCONFIG6 DYNAMICXCF parameters have been specified.

• GLOBALCONFig SYSPLEXMONitor TIMERSECS…– Determines how quickly the sysplex monitor reacts to problems with needed sysplex resources.



Other Definitions

• Other TCP/IP Profile statements that effect VIPA.

• AUTOLog…DELAYSTART ENDAUTOLOG– DELAYSTART is an optional keyword that indicates that the procedure should not be started until the TCP/IP stack has joined the

sysplex group and has processed its DVIPA configuration.

– Do not specify the DELAYSTART parameter for your OMPROUTE procedure if you specify the DELAYJOIN parameter on the GLOBALCONFIG profile statement.

• DELETE LINK link_name– To delete a link, you must first delete any associated HOME entry by specifying a HOME statement that does not include the link.

– You do not need to (and cannot) stop the device when deleting a link for a virtual device.

• IPSEC DVIPSEC…– Use the IPSEC statement to define policy for the IPv4 security function that is enabled with the IPCONFIG IPSECURITY

parameter. The IPSEC statement is ignored if IPSECURITY is not specified on the IPCONFIG statement. If you also enable IPv6 Security with the IPCONFIG6 IPSECURITY parameter, then use the IPSEC statement to also define policy for IPv6 IP security.

– DVIPSEC indicates that IPsec tunnels associated with IPv4 dynamic VIPA addresses are eligible to be distributed if the dynamic VIPA address is being distributed. The IPsec tunnels are also eligible to be moved during dynamic VIPA takeover or giveback.

• PORT…SHAREPORTWLM BIND ip_addr…– SHAREPORTWLM, like SHAREPORT, causes incoming connections to be distributed among a set of TCP listeners; however,

unlike SHAREPORT, the listener selection is based on WLM server-specific recommendations, modified by the SEF values for each listener.

– BIND associates a job name with the IP address, ip_addr. When a job with the designated name binds to the IPv4 INADDR_ANY address, or to the IPv6 unspecified address (in6addr_any), the bind is intercepted and converted to a bind to the IP address specified by ip_addr.

• Note:– VIPA links are not allowed on the GATEWAY statement.

– The START statement is not valid for virtual devices or interfaces.

– A virtual device or interface cannot be stopped.



Telnet

• TELNETGLOBALS XCFGROUP defines the telnet XCF group.– JOIN | NOJOIN causes the telnet to join the XCF group.

– SUBPLEX suffix defines the subplex subset of the sysplex.

– XCFMONITOR sec defines the XCF monitor interval.

– CONNECTTIMEOUT sec defines the connect timeout interval.

– RECOVERYTIMEOUT sec defines the recovery timeout interval.

– LUNS ipaddr port defines the LUNS IP address and port.• PRIMARY RANK num defines this telnet to be the primary LUNS.

• BACKUP RANK num defines this telnet to be the backup LUNS.

• On an LUNR, without the PRIMARY or BACKUP parameter this statement defines the LUNS to use.

• BEGINVTAM– SDEFAULTLUS defines the shared terminal LU names mapped to the NULL client identifier for generic LU

telnet terminal connection requests.

– SDEFAULTLUSSPEC defines the shared terminal LU names mapped to the NULL client identifier for specific LU telnet terminal connection requests.

– SDEFAULTPRT defines the shared printer LU names mapped to the NULL client identifier for generic LU printer telnet connection requests.

– SDEFAULTPRTSPEC defines the shared printer LU names mapped to the NULL client identifier for specific LU printer telnet connection requests.

– SLUGROUP defines a group of shared terminal LU names. This SLUGROUP may then be used on an LUMAP statement.

– SPRTGROUP defines a group of shared printer LU names. This SPRTGROUP may then be used on a PRTMAP statement.




References: VIPA

Commands

IP

System

Administrator’s

Commands

SC31-8781



VIPA Commands

• Display TCPIP,tcp_proc,SYSplex,VIPADyn…– Displays the DVIPAs defined to the stack, their status, their backup rank, and if they are a distributing or target stack.

• Vary TCPIP,tcp_proc,SYSplex…– Causes stack to leave or join the sysplex group, deactivate or activate a DVIPA, or quiesce or resume applications.

• NETSTAT SRCIP– Displays the source IP address information for the stack including the source VIPA information.

• NETSTAT VCRT…– Displays the DVIPA Connection Routing Table used for sysplex distributor and DVIPA support.

• On a sysplex distributor stack, it displays all connections being routed through that distributor.

• On the stack taking over the distributed DVIPA, it displays every connection that is being routed to the DVIPA.

• On stacks where the DVIPA connection endpoints are, the report displays every connection ending in that stack for the DVIPA.

• When sysplex distribution is not being used (no VIPADISTRIBUTE) the report on the stack taking over the DVIPA will display all existing connections to the stack giving up the DVIPA, as well as the connections ending in this stack.

• NETSTAT VDPT…– Displays the DVIPA destination port table information. The destination port table exists only on distributing stacks.

• NETSTAT VIPADCFG…– Displays the DVIPA configuration for a local host.

• NETSTAT VIPADyn…– Displays the current DVIPA and VIPAROUTE information for a local host.

• PING remote_host (Srcip src_ipaddr

– Sends an echo request to a foreign node (remote_host)(IP address or host name) to determine whether the node is accessible.

– src_ipaddr specifies the source IP address.




References: Documents



IBM Product Manuals and

Redbooks• z/OS Communications Server

– IP Configuration Guide, SC31-8775

– IP Configuration Reference, SC31-8776

– IP System Administrator's Commands, SC31-8781

– IP Diagnosis Guide, GC31-8782

• z/OS SNA– Network Implementation Guide, SC31-8777

– SNA Resource Definition Reference, SC31-8778

– SNA Operation, SC31-8779

– SNA Diagnosis Volume 1, Techniques and Procedures, GC31-6850

• OSA– OSA-E Customer’s Guide and Reference, SA22-7476

• Redbook– Communications Server for z/OS TCP/IP Implementation Volume 1: Base Functions, Connectivity, and

Routing

– Communications Server for z/OS TCP/IP Implementation Volume 3: High Availability, Scalability, and Performance, SG24-7341

– Communications Server for z/OS TCP/IP Implementation, Volume 4: Policy-Based Network Security, SG24-7342

– Redpiece DB2 for z/OS Data Sharing: Distributed Load Balancing and Fault Tolerant Configuration, REDP-4449

– IBM System z Connectivity Handbook, SG24-5444

– OSA-E Implementation Guide, SG24-5948

Redbook

Product

Manual



Web Information

• IBM ATS Technical Documents:– http://www.ibm.com/support/techdocs

• z/OS Communications Server– http://www.ibm.com/software/network/commserver/zos

• IBM Information Center– http://www.ibm.com/support/documentation/us/en

• IBM Education Assistant– http://www.ibm.com/software/info/education/assistant

• z/OS Communications Server Publications– http://www.ibm.com/systems/z/os/zos/bkserv

• IBM Redbooks– http://www.redbooks.ibm.com

• System z main web site:– http://www.ibm.com/systems/z/hardware/



Appendices



Appendices

• Appendix A: Sysplex and Subplex Support

• Appendix B: Details of Static VIPA Backup

and Load Balance

• Appendix C: Details of DVIPA Backup and

Load Balance



Appendix A: Sysplex and

Subplex Support



VTAM & TCP/IP Sysplex Support

• Each VTAM in the Sysplex:– Joins ISTXCF and ISTCFS01 Sysplex groups

– Establishes Dynamic XCF Connectivity (controlled by XCFINIT start option)

– Can access Generic Resource and MNPS structures (specified by STRGR & STRMNPS start options)

• Each TCP/IP stack in the Sysplex:– Joins EZBTCPCS Sysplex group

– Exchanges IP address information

– Coordinates DVIPA movement

– Can be a Sysplex Distributor target or SysplexDistributor Backup

– Can access SWSA and Sysplexports structures

– Sets up dynamic connectivity• Using Dynamic IUTSAMEH - if stacks are in the same

LPAR

• Using Dynamic HiperSockets (iQDIO) - if stacks are on the same CEC & use the same CHPID

• Using VTAM's Dynamic XCF Connectivity

CEC

LPAR

VTAM

joins groups

ISTXCF & ISTCFS01

TCP/IP

joins group

EZBTCPCS

TCP/IP

joins group

EZBTCPCS

LPAR

VTAM

joins groups

ISTXCF & ISTCFS01

TCP/IP

joins group

EZBTCPCS

TCP/IP

joins group

EZBTCPCS

CEC

LPAR

VTAM

joins groups

ISTXCF & ISTCFS01

TCP/IP

joins group

EZBTCPCS

TCP/IP

joins group

EZBTCPCS

LPAR

VTAM

joins groups

ISTXCF & ISTCFS01

TCP/IP

joins group

EZBTCPCS

TCP/IP

joins group

EZBTCPCS

Sysplex with no subplexing.



Subplex Support

• Partitions the Sysplex into Subplexes to restrict use of Sysplex functions in a Sysplex connected to multiple security areas.

• A Subplex is a subset of VTAM nodes or TCP stacks in the Sysplex.– All the members of the subset can establish dynamic connectivity through the

Sysplex with other members of the subset.

– Members of the subset cannot establish dynamic connectivity through the Sysplex with non-members of the subset.

• Functions restricted to within a Subplex (Subplex Scope):– VTAM Generic Resources (GR) & Multi-Node Persistent Session (MNPS) resources

– Automatic connectivity - IP connectivity and VTAM connectivity over XCF (including dynamic IUTSAMEH and dynamic HiperSockets based on Dynamic XCF for IP)

– XCF communications about DVIPAs, target stack server availability, etc.• DVIPA movement candidates

• Sysplex Distributor target candidates

• HiperSockets VLANID– Allow a Virtual LAN ID (VLANID) to be specified on a user-defined HiperSockets

definition.

– DynamicXCF may be configured to use a HiperSockets LAN so VLAN support forHiperSockets is required for DynamicXCF support if subplexes are configured.



Subplex Restrictions

• Each VTAM can belong to only one subplex– The subplex is defined by the name of the Sysplex

group that VTAM joins

– One VTAM Subplex per z/OS

• Each TCP/IP stack can belong to only one subplex– The subplex is defined by the name of the Sysplex

group that TCP/IP joins

– A TCP/IP Subplex cannot span multiple VTAM Subplexes

– Different TCP/IP stacks in a z/OS may belong to different TCP/IP Subplexes



Subplex and VLANID Configuration

• VTAM Subplex Support– Start Option XCFGRPID vv

• where vv is a number between 2 and 31

– VTAM joins ISTXCFvv and ISTCFSvv Subplex groups

– STRGR and STRMNPS CF structure names are suffixed with vv

• TCP/IP Subplex Support– Profile GLOBALCONFIG statement

• XCFGRPID tt (new in z/OS V1.8) to subplex TCP/IP– where tt is a numeric value between 2 and 31

• IQDVLANID nn (new in z/OS V1.8) to subplex HiperSockets for Dynamic XCF connectivity

– where nn is a numeric value between 1 and 4094

– TCP/IP will join Sysplex group EZBTvvtt• where vv is the VTAM subplex number

– SWSA and Sysplexports structure names will be suffixed by vvtt• EZBDVIPAvvtt and EZBEPORTvvtt

– Profile keyword on LINK and INTERFACE statements for HiperSockets• VLANID nn

– where nn is numeric value between 1 and 4094



No Subplex vs. Subplex

• VTAM Start Option XCFGRPID=vv

• VTAM XCF (cross-system coupling facility) group name is ISTXCF..

• VTAM CFS (coupling facilities services) group name is ISTCFS..

• TCP/IP profile GLOBALCONFIG XCFGRPID=tt

• TCP/IP XCF group name is EZBT….

CEC

LPAR

VTAM

XCFGRPID=11

joins groups

ISTXCF11 & ISTCFS11

TCP/IP

XCFGRPID=12

joins group

EZBT1112

TCP/IP

XCFGRPID=12

joins group

EZBT1112

LPAR

VTAM

XCFGRPID=11

joins groups

ISTXCF11 & ISTCFS11

TCP/IP

XCFGRPID=12

joins group

EZBT1112

TCP/IP

XCFGRPID=12

joins group

EZBT1112

CEC

LPAR

VTAM

XCFGRPID=21

joins groups

ISTXCF21 & ISTCFS21

TCP/IP

XCFGRP=22

joins group

EZBT2122

TCP/IP

XCFGRP=22

joins group

EZBT2122

LPAR

VTAM

XCFGRPID=21

joins groups

ISTXCF21 & ISTCFS21

TCP/IP

XCFGRP=23

joins group

EZBT2123

TCP/IP

XCFGRP=23

joins group

EZBT2123

STRGRSTRGRSTRGRvvSTRGRvvSTRGR CF (coupling

facility) structure for

generic resource name

STRMNPSSTRMNPSSTRMNPSvvSTRMNPSvvSTRMNPS CF structure

for MNPS (multi-node

persistent sessions)

EZBTCPCSEZBTCPttEZBTvvCSEZBTvvttTCP/IP XCF Group Name

ISTXCFISTXCFISTXCFvvISTXCFvvVTAM XCF Group Name

EZBEPORTEZBEPORT01ttEZBEPORTvvEZBEPORTvvttEZBEPORT CF structure

for sysplex TCP/IP Stack

Source VIPA

EZBDVIPAEZBDVIPA01ttEZBDVIPAvvEZBDVIPAvvttEZBDVIPA CF structure

for SWSA (Sysplex Wide

Security Associations for

IPsec)

ISTCFS01ISTCFS01ISTCFSvvISTCFSvvCFS Group Name

VTAM

XCFGRPID

not defined

and TCP/IP

XCFGRPID

not defined

VTAM

XCFGRPID not

defined and

TCP/IP

XCFGRPID=tt

VTAM

XCFGRPID=vv

and TCP/IP

XCFGRPID not

defined

VTAM

XCFGRPID=vv

and TCP/IP

XCFGRPID=tt

VTAM Subplex 11

VTAM Subplex 21

TCP/IP

Subplex 12TCP/IP

Subplex 12

TCP/IP

Subplex 22

TCP/IP

Subplex 23



Appendix B:

Details of Static VIPA

Backup and Load Balance



ARP Fail-over with VIPA and Static Routing

z/OS TCP/IP

VIPA 10.1.1.30

VIPA 10.1.1.30OSA

10.1.1.11

OSA

10.1.1.12

10.1.1.2Router210.1.2.2

10.1.1.1Router110.1.2.1

z/OS TCP/IP

VIPA 10.1.1.30

VIPA 10.1.1.30

OSA

10.1.1.11

VIPA 10.1.1.30

OSA

10.1.1.11

10.1.1.12

10.1.1.2Router210.1.2.2

10.1.1.1Router110.1.2.1

• If static routing is defined between the z/OS system and the first hop routers, then a VIPA in the same subnet as the OSA attachments is "owned" by one of the OSAs at any given time (randomly/unpredictable). "Owned" means that the OSA will respond to ARP for that VIPA. If the OSA that "owns" the VIPA goes down then one of the other OSAs will not only send out a gratuitous ARP for the failed OSA IP address but also for the VIPA.

– Note that this does require that both (or multiple) OSAs originally come up so that the stack marked them as parallel to the same network.

– Outbound traffic could be "load balanced" using IPCONFIG MULTIPATH but inbound traffic with the VIPA destination would all be sent to the VIPA "owning" OSA.

• The same failover support exists for IPv6 where gratuitous neighbor advertisements are sent rather than gratuitous ARPs.

• Prior to z/OS V1.10, the stack updates OSA to perform ARP processing for all VIPAs.– This causes many unnecessary gratuitous ARPs which can cause confusion in routers and sniffer traces.

• z/OS V1.10 with the IPv4 INTERFACE and /num_bits, causes the stack to only update OSA for a VIPA if it the VIPA is in the same subnet as the OSA.

– This eliminates superfluous gratuitous ARPs.



Static VIPA, non-QDIO OSAs, and Static

Routing

• Requires:– VIPA in different subnet than OSAs

– IPCONFIG MULTIPATH for outbound load balancing

– Static Routing Dead Gateway Detection when a first hop router fails:



• Therefore with Static Routing HSRP/VRRP should be used between first hop routers

• One of the following is required:– PRIROUTER must be defined on all the

OSAs.• VIPA does not need to be defined in the OATs.

– OSAs must not be shared with other LPARs.• PRIROUTER is defaulted.

• VIPA does not need to be defined in the OATs.

z/OS TCP/IP

VIPA

OSA OSA

RouterRouter

• Support:– Outbound Load Balance

• Yes, across multiple OSAs.

– Inbound Load Balance• Yes, if routers support multipath and

have multiple static routes defined.

– Dynamic OSA Backup• No, because inbound traffic will still

be sent to a failed OSA.



Static VIPA, non-QDIO OSAs, and Dynamic

Routingz/OS TCP/IP

VIPA

OSA OSA

RouterRouter



– Inbound Load Balance• Yes, across multiple OSAs with OSPF.

• No with RIP.

– Dynamic OSA Backup• Yes, dynamic routing will route around a failed OSA or first hop router.



• One of the following is required:– PRIROUTER must be defined on all the OSAs.






Static VIPA, QDIO OSAs, and Static Routing

z/OS TCP/IP

VIPA

OSA OSA

RouterRouter



– Inbound Load Balance• Yes, if routers support multipath and

have multiple static routes defined.

– Dynamic OSA Backup• Yes, OSA QDIO Gratuitous ARP

Fail-over will move OSA and VIPA IP addresses when an OSA fails.

• Requires:– VIPA in same subnet as OSAs






• PRIROUTER is not required because VIPA is dynamically downloaded into the OAT

– The VIPA could be on either OSA after an IP stack startup. It will be “owned” by whichever OSA happens to come up first but the order of the OSAs in the “start device” statements in the profile does not guarantee the start order.



Static VIPA, QDIO OSAs, and Dynamic

Routingz/OS TCP/IP

VIPA

OSA OSA

RouterRouter




• No with RIP.


• Requires:

– VIPA in different subnet than OSAs

– IPCONFIG MULTIPATH for outbound load

balancing

• PRIROUTER is not required because VIPA

is dynamically downloaded into the OAT of

both OSAs.



Appendix C:

Details of DVIPA

Backup and Load Balance



DVIPA, non-QDIO OSAs, and Static Routing



– Inbound Load Balance

• Yes, if routers support multipath and have multiple static routes defined.

– Dynamic OSA Backup

• No, because inbound traffic will still be sent to a failed OSA.

– Dynamic TCP/IP Stack Backup

• No

z/OS TCP/IP

VIPA

OSA OSA

RouterRouter

z/OS TCP/IP

OSA OSA







• The following is required:– The OSAs must be shared between the primary VIPA

stack and the backup VIPA stack.

– PRIROUTER must be defined on all the OSAs for the primary VIPA stack.

– SECROUTER must be defined on all the OSAs for the backup VIPA stack.



DVIPA, non-QDIO OSAs, and Dynamic

Routing




• No with RIP.


– Dynamic TCP/IP Stack Backup• Yes


– PRIROUTER or non-shared OSA• VIPA is not defined in the OAT


• One of the following is required:– PRIROUTER must be defined on all the OSAs.




z/OS TCP/IP

VIPA

OSA OSA

RouterRouter

z/OS TCP/IP

OSA OSA



DVIPA, QDIO OSAs, and Static Routing


• Yes, across multiple OSAs.– Inbound Load Balance

• Yes, if routers support multipath and have multiple static routes defined.

– Dynamic OSA Backup• Yes, OSA QDIO Gratuitous ARP

Fail-over will move OSA and VIPA IP addresses when an OSA fails.

– Dynamic TCP/IP Stack Backup• Yes, OSA QDIO Gratuitous ARP will

be sent when VIPA moves.

• Requires:– VIPA in same subnet as OSAs



• TCP connection will eventually timeout (3 to 10 minutes) and TCP will redrive the route selection algorithm and hopefully get a successful connection.


• Therefore with Static Routing HSRP/VRRP should be used between first hop routers.

• PRIROUTER is not required because VIPA is dynamically downloaded into the OAT.

– The VIPA could be on either OSA after an IP stack startup. It will be “owned” by whichever OSA happens to come up first but the order of the OSAs in the “start device” statements in the profile does not guarantee the start order.

z/OS TCP/IP

VIPA

OSA OSA

RouterRouter

z/OS TCP/IP

OSA OSA



DVIPA, QDIO OSAs, and Dynamic Routing


• Yes, across multiple OSAs

– Inbound Load Balance• Yes, across multiple OSAs with OSPF.• No with RIP


– Dynamic TCP/IP Stack Backup• Yes, dynamic routing will advertise VIPA when it moves.

• Requires:

– VIPA in different subnet than OSAs

– IPCONFIG MULTIPATH for outbound load

balancing

• PRIROUTER is not required because VIPA

is dynamically downloaded into the OAT of

both OSAs.

z/OS TCP/IP

VIPA

OSA OSA

RouterRouter

z/OS TCP/IP

OSA OSA



The End