Shorewall and Multiple Internet Connections

Tom Eastep

Copyright 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 Thomas M. Eastep

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU FreeDocumentation License, Version 1.2 or any later version published by the Free Software Foundation; with noInvariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in thesection entitled GNU Free Documentation License.

2012/12/09

Table of Contents

Multiple Internet Connection SupportOverview/etc/shorewall/providers FileWhat an entry in the Providers File Does/etc/shorewall/masq and Multi-ISPMartiansLegacy ExampleRouting a Particular Application Through a Specific InterfacePort ForwardingMore than 2 ProvidersApplications running on the Firewall -making them use a particular provider/etc/shorewall/rtrules (formerly /etc/shorewall/route_rules)

Routing RulesColumns in the rtrules fileMulti-ISP and VPNExamples

/etc/shorewall/routes FileLooking at the routing tablesUSE_DEFAULT_RT

DHCP with USE_DEFAULT_RTAn alternative form of balancingGateway Monitoring and Failover

SWPINGLink Status Monitor (LSM)

Two Providers Sharing an InterfaceA Complete Working Example

IPv4 ConfigurationIPv6 Configuration

Warning

This document describes the Multi-ISP facility in Shorewall 4.4.26 and later. If you are running anearlier release, please see the documentation for that release.

Warning

Reading just Shorewall documentation is probably not going to give you enough background to usethis material. Shorewall may make iptables easy but the Shorewall team doesn't have the resources tobe able to spoon-feed Linux policy routing to you (please remember that the user's manual for atractor doesn't teach you to grow corn either). You will likely need to refer to the following additionalinformation:

The LARTC HOWTO: http://www.lartc.org

1 de 32

Output of man ip

Output of ip route help and ip rule help

Multiple Internet Connection Support

Shorewall includes limited support for multiple Internet connections. Limitations of this support are as follows:

It utilizes static routing configuration. If there is a change in the routing topopogy, Shorewall must be restarted.

The routing changes are made and the route cache is purged when Shorewall is started and when Shorewall isrestarted (unless you specify the "-n" option to shorewall restart). Ideally, restarting the packet filter shouldhave no effect on routing.

For most routing applications, Quagga is a better solution although it requires that your ISPs offer routingprotocol support.

Overview

Let's assume that a firewall is connected via two separate Ethernet interfaces to two different ISPs.[1] as in thefollowing diagram.

eth0 connects to ISP1. The IP address of eth0 is 206.124.146.176 and the ISP's gateway router has IP address206.124.146.254.

eth1 connects to ISP 2. The IP address of eth1 is 130.252.99.27 and the ISP's gateway router has IP address130.252.99.254.

2 de 32

eth2 connects to the local LAN. Its IP configuration is not relevant to this discussion.

Each of these providers is described in an entry in the file /etc/shorewall/providers.

Entries in /etc/shorewall/providers can specify that outgoing connections are to be load-balanced between thetwo ISPs. Entries in /etc/shorewall/tcrules and /etc/shorewall/rtrules can be used to direct particularoutgoing connections to one ISP or the other. Use of /etc/shorewall/tcrules is not required for /etc/shorewall/providers to work, but in most cases, you must select a unique MARK value for each provider so Shorewall canset up the correct marking rules for you.

When you use the track option in /etc/shorewall/providers, connections from the Internet are automaticallyrouted back out of the correct interface and through the correct ISP gateway. This works whether the connection ishandled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall.

Shorewall will set up the routing and will update the /etc/iproute2/rt_tables to include the table names andnumbers of the tables that it adds.

Caution

This feature uses packet marking to control the routing. As a consequence, there are some restrictionsconcerning entries in /etc/shorewall/tcrules:

Packet marking for traffic control purposes may not be done in the PREROUTING table forconnections involving providers with 'track' specified (see below).

You may not use the SAVE or RESTORE options unless you also setHIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET > 0 with Shorewall 4.4.26 and later) in/etc/shorewall/shorewall.conf.

You may not use connection marking unless you also set HIGH_ROUTE_MARKS=Yes(PROVIDER_OFFSET > 0 with Shorewall 4.4.26 and later) in /etc/shorewall/shorewall.conf.

The /etc/shorewall/providers file can also be used in other routing scenarios. See the Squid documentation foran example.

/etc/shorewall/providers File

Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if youdon't want to enter any value.

NAME

The provider name. Must begin with a letter and consist of letters and digits. The provider name becomes thename of the generated routing table for this provider.

NUMBER

A number between 1 and 252. This becomes the routing table number for the generated table for this provider.

MARK

A mark value used in your /etc/shorewall/tcrules file to direct packets to this provider. Shorewall will also markconnections that have seen input from this provider with this value and will restore the packet mark in thePREROUTING CHAIN. Mark values must be in the range 1-255.

Alternatively, you may set HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET > 0 with Shorewall 4.4.26and later) in /etc/shorewall/shorewall.conf. This allows you to:

Use connection marks for traffic shaping, provided that you assign those marks in the FORWARD chain.

Use mark values > 255 for provider marks in this column.

3 de 32

With HIGH_ROUTE_MARKS=Yes (PROVIDER_OFFSET=8), these mark values must be amultiple of 256 in the range 256-65280 (hex equivalent 0x100 - 0xFF00 with the low-order 8 bitsbeing zero); or

Set WIDE_TC_MARKS=Yes in shorewall.conf (5) (PROVIDER_OFFSET=16), and use markvalues in the range 0x10000 - 0xFF0000 with the low-order 16 bits being zero.

This column may be omitted if you dont use packet marking to direct connections to a particular provider.

DUPLICATE

Gives the name or number of a routing table to duplicate. May be 'main' or the name or number of a previouslydeclared provider. For most applications, you want to specify 'main' here. This field should be be specified as '-'when USE_DEFAULT_RT=Yes in shorewall.conf

INTERFACE

The name of the interface to the provider. Where multiple providers share the same interface, you must followthe name of the interface by a colon (":") and the IP address assigned by this provider (e.g.,eth0:206.124.146.176). See below for additional considerations.

The interface must have been previously defined in shorewall-interfaces (5). In general, that interface shouldnot have the proxyarp option specified unless loose is given in the OPTIONS column of this entry.

GATEWAY

The IP address of the provider's Gateway router.

You can enter detect here and Shorewall will attempt to automatically determine the gateway IP address.

Hint: "detect" is appropriate for use in cases where the interface named in the INTERFACE column isdynamically configured via DHCP etc. Be sure, however, that you don't have stale dhcp client state files in/var/lib/dhcpcd or /var/lib/dhclient-*.lease because Shorewall may try to use those stale files todetermine the gateway address.

The GATEWAY may be omitted (enter '-') for point-to-point links.

OPTIONS

A comma-separated list from the following:

track

Important

Beginning with Shorwall 4.3.3, track defaults to the setting of the TRACK_PROVIDERSoption in shorewall.conf (5). To disable this option when you have specifiedTRACK_PROVIDERS=Yes, you must specify notrack (see below).

If specified, connections FROM this interface are to be tracked so that responses may be routed backout this same interface.

You want to specify 'track' if Internet hosts will be connecting to local servers through this provider. Anytime that you specify 'track', you will normally want to also specify 'balance' (see below). 'track' will alsoensure that outgoing connections remain stay anchored to a single provider and don't try to switchproviders when route cache entries expire.

Use of this feature requires that your kernel and iptables include CONNMARK target and connmarkmatch support (Warning: Until recently, standard Debian and Ubuntu kernels lacked that support.Both Lenny and Jaunty do have the proper support).

Important

4 de 32

If you are running a version of Shorewall earlier than 4.4.3 and are using/etc/shorewall/providers because you have multiple Internet connections, werecommend that you specify track even if you don't need it. It helps maintain long-termconnections in which there are significant periods with no traffic.

balance

The providers that have balance specified will get outbound traffic load-balanced among them.Balancing will not be perfect, as it is route based, and routes are cached. This means that routes tooften-used sites will always be over the same provider.

By default, each provider is given the same weight (1) . You can change the weight of a given providerby following balance with "=" and the desired weight (e.g., balance=2). The weights reflect the relativebandwidth of the providers connections and should be small numbers since the kernel actually createsadditional default routes for each weight increment.

Important

If you are using /etc/shorewall/providers because you have multiple Internetconnections, we recommend that you specify balance even if you don't need it. You canstill use entries in /etc/shorewall/tcrules and /etc/shorewall/rtrules to force alltraffic to one provider or another.

Note

If you don't heed this advice then please read and follow the advice inFAQ 57 and FAQ 58.

loose

Do not generate routing rules that force traffic whose source IP is an address of the INTERFACE to berouted to this provider. Useful for defining providers that are to be used only when the appropriatepacket mark is applied.

Shorewall makes no attempt to consolidate the routing rules added when loose is not specified. So, if youhave multiple IP addresses on a provider interface, you may be able to replace the rules that Shorewallgenerates with one or two rules in /etc/shorewall/rtrules. In that case, you can specify loose tosuppress Shorewall's rule generation. See the example below.

notrack

Added in Shorewall 4.4.3. This option turns off the track option.

optional

Note

This option is deprecated in favor of the optional interface option. That option performsthe same function.

Shorewall will determine if this interface is up and has a configured IP address. If it is not, a warning isissued and this provider is not configured.

Note

optional is designed to detect interface states that will cause shorewall start orshorewall restart to fail; just because an interface is in a state that Shorewall can[re]start without error doesn't mean that traffic can actually be sent through the interface.

You can supply an 'isusable' extension script to extend Shorewall's interface statedetection. See also the Gateway Monitoring and Failover section below.

5 de 32

src=source-address

Specifies the source address to use when routing to this provider and none is known (the local client hasbound to the 0 address). May not be specified when an address is given in the INTERFACE column. Ifthis option is not used, Shorewall substitutes the primary IP address on the interface named in theINTERFACE column.

mtu=number

Specifies the MTU when forwarding through this provider. If not given, the MTU of the interface namedin the INTERFACE column is assumed.

fallback[=weight]

Indicates that a default route through the provider should be added to the default routing table (table253). If a weight is given, a balanced route is added with the weight of this provider equal to thespecified weight. If the option is given without a weight, a separate default route is added through theprovider's gateway; the route has a metric equal to the provider's NUMBER.

Prior to Shorewall 4.4.24, the option is ignored with a warning message if USE_DEFAULT_RT=Yes inshorewall.conf.

Warning

If you set this option on an interface, you must disable route filtering on the interface.Include 'routefilter=0,logmartions=0' in the OPTIONS column of shorewall-interfaces(5).

For those of you who are confused between track and balance:

track governs incoming connections (but is also useful for binding long-running connections to the sameinterface).

balance governs outgoing connections.

COPY

A comma-separated list of interface names. Wildcards specified using an asterisk ("*") are permitted (e.g., tun*).

When you specify an existing table in the DUPLICATE column, Shorewall copies all routes through theinterface specified in the INTERFACE column plus the interfaces listed in this column. Normally, you will listall interfaces on your firewall in this column except those Internet interfaces specified in the INTERFACEcolumn of entries in this file.

Note

Beginning with Shorewall 4.4.15, provider routing tables can be augmeted with additional routesthrough use of the /etc/shorewall/routes file.

What an entry in the Providers File Does

Adding another entry in the providers file simply creates an alternate routing table for you (see the LARTC Howto).The table will usually contain two routes:

A host route to the specified GATEWAY through the specified INTERFACE.1.

A default route through the GATEWAY.2.

Note that the first route is omitted if "-" is specified as the GATEWAY; in that case, the default route does not specifya gateway (point-to-point link).

If the DUPLICATE column is non-empty, then routes from the table named in that column are copied into the new

6 de 32

table. By default, all routes (except default routes) are copied. The set of routes copied can be restricted using theCOPY column which lists the interfaces whose routes you want copied. You will generally want to include all localinterfaces in this list. You should exclude the loopback interface (lo) and any interfaces that do not have an IPconfiguration. You should also omit interfaces like tun interfaces that are created dynamically. Traffic to networkshandled by those interfaces should be routed through the main table using entries in /etc/shorewall/rtrules (seeExample 2 below) or by using USE_DEFAULT_RT=Yes.

In addition:

Unless loose is specified, an ip rule is generated for each IP address on the INTERFACE that routes trafficfrom that address through the associated routing table.

1.

If you specify track, then connections which have had at least one packet arrive on the interface listed in theINTERFACE column have their connection mark set to the value in the MARK column. In the PREROUTINGchain, packets with a connection mark have their packet mark set to the value of the associated connectionmark; packets marked in this way bypass any prerouting rules that you create in /etc/shorewall/tcrules.This ensures that packets associated with connections from outside are always routed out of the correctinterface.

2.

If you specify balance, then Shorewall will replace the 'default' route with weight 100 in the 'main' routingtable with a load-balancing route among those gateways where balance was specified. So if you configuredefault routes, be sure that their weight is less than 100 or the route added by Shorewall will not be used.

3.

That's all that these entries do. You still have to follow the principle stated in the Shorewall Routing documentation:

Routing determines where packets are to be sent.1.

Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowedto go there and controls rewriting of the SOURCE IP address (SNAT/MASQUERADE).

2.

The bottom line is that if you want traffic to go out through a particular provider then you must mark that traffic withthe provider's MARK value in /etc/shorewall/tcrules and you must do that marking in the PREROUTINGchain; or, you must provide the appropriate rules in /etc/shorewall/rtrules.

/etc/shorewall/masq and Multi-ISP

If you masquerade a local network, you will need to add masquerade rules for both external interfaces. Referring tothe diagram above, if each of the interfaces has only a single IP address and you have no systems with public IPaddresses behind your firewall, then I suggest the following simple entries:

#INTERFACE SOURCE ADDRESSeth0 0.0.0.0/0 206.124.146.176eth1 0.0.0.0/0 130.252.99.27

If you have a public subnet (for example 206.124.146.176/30) behind your firewall, then use exclusion:

#INTERFACE SOURCE ADDRESSeth0 !206.124.146.176/29 206.124.146.176eth1 0.0.0.0/0 130.252.99.27

Note that exclusion is only used on the interface corresponding to internal subnetwork.

If you have multiple IP addresses on one of your interfaces, you can use a similar technique -- simplY exclude thesmallest network that contains all of those addresses from being masqueraded.

Warning

Entries in /etc/shorewall/masq have no effect on which ISP a particular connection will be sentthrough. That is rather the purpose of entries in /etc/shorewall/tcrules and /etc/shorewall/rtrules.

Martians

7 de 32

One problem that often arises with Multi-ISP configuration is 'Martians'. If your Internet interfaces are configuredwith the routefilter option in /etc/shorewall/interfaces (remember that if you set that option, you should alsoselect logmartians), then things may not work correctly and you will see messages like this:

Feb 9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1 Feb 9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00

The above message is somewhat awkwardly phrased. The source IP in this incoming packet was 64.86.88.116 andthe destination IP address was 206.124.146.176. Another gotcha is that the incoming packet has already had thedestination IP address changed for DNAT or because the original outgoing connection was altered by an entry in/etc/shorewall/masq (SNAT or Masquerade). So the destination IP address (206.124.146.176) may not have beenthe destination IP address in the packet as it was initially received.

There a couple of common causes for these problems:

You have connected both of your external interfaces to the same hub/switch. Connecting multiple firewallinterfaces to a common hub or switch is always a bad idea that will result in hard-to-diagnose problems.

1.

You are specifying both the loose and balance options on your provider(s). This can cause individualconnections to ping-pong back and forth between the interfaces which is almost guaranteed to cause problems.

2.

You are redirecting traffic from the firewall system out of one interface or the other using packet marking inyour /etc/shorewall/tcrules file. A better approach is to configure the application to use the appropriatelocal IP address (the IP address of the interface that you want the application to use). See below.

3.

If all else fails, remove the routefilter option from your external interfaces. If you do this, you may wish to add rulesto log and drop packets from the Internet that have source addresses in your local networks. For example, if the localLAN in the above diagram is 192.168.1.0/24, then you would add this rule:

#ACTION SOURCE DESTDROP:info net:192.168.1.0/24 all

Be sure the above rule is added before any other rules with net in the SOURCE column.

Legacy Example

This section describes the legacy method of configuring multiple uplinks. It is deprecated in favor of theUSE_DEFAULT_RT=Yes configuration described below.

The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers asfollows.

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPYISP1 1 1 main eth0 206.124.146.254 track,balance eth2ISP2 2 2 main eth1 130.252.99.254 track,balance eth2

Other configuration files go something like this:

/etc/shorewall/interfaces:

#ZONE INTERFACE BROADCAST OPTIONSnet eth0 detect net eth1 detect

/etc/shorewall/policy:

#SOURCE DESTINATION POLICY LIMIT:BURSTnet net DROP

/etc/shorewall/masq:

#INTERFACE SOURCE ADDRESSeth0 0.0.0.0/0 206.124.146.176

8 de 32

eth1 0.0.0.0/0 130.252.99.27

Routing a Particular Application Through a Specific Interface

This continues the example in the preceding section.

Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. You wouldmake this entry in /etc/shorewall/tcrules (and if you are running a version of Shorewall earlier than 3.0.0, you wouldset TC_ENABLED=Yes in /etc/shorewall/shorewall.conf).

#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST# PORT(S)2:P 0.0.0.0/0 tcp 25

Note that traffic from the firewall itself must be handled in a different rule:

#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST# PORT(S)2 $FW 0.0.0.0/0 tcp 25

Port Forwarding

Shorewall provides considerable flexibility for port forwarding in a multi-ISP environment.

Normal port forwarding rules such as the following will forward from both providers.

/etc/shorewall/rules:

#ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE ORIGINAL# PORTS(S) DESTDNAT net loc:192.168.1.3 tcp 25

Continuing the above example, to forward only connection requests from ISP 1, you can either:

Qualify the SOURCE by ISP 1's interface:

#ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE ORIGINAL# PORTS(S) DESTDNAT net:eth0 loc:192.168.1.3 tcp 25

or

1.

Specify the IP address of ISP 1 in the ORIGINAL DEST column:

#ACTION SOURCE DEST PROTO DEST PORT(S) SOURCE ORIGINAL# PORTS(S) DESTDNAT net loc:192.168.1.3 tcp 25 - 206.124.146.176

2.

More than 2 Providers

When there are more than two providers, you need to extend the two-provider case in the expected way:

For each external address, you need an entry in /etc/shorewall/masq to handle the case where a connectionusing that address as the SOURCE is sent out of the interfaces other than the one that the address is configuredon.

1.

For each external interface, you need to add an entry to /etc/shorewall/masq.2.

If we extend the above example to add eth3 with IP address 16.105.78.4 with gateway 16.105.78.254, then:

/etc/shorewall/providers:

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY

9 de 32

ISP1 1 1 main eth0 206.124.146.254 track,balance eth2ISP2 2 2 main eth1 130.252.99.254 track,balance eth2ISP3 3 3 main eth3 16.105.78.254 track,balance eth2

/etc/shorewall/masq:

#INTERFACE SUBNET ADDRESSeth0 0.0.0.0/0 206.124.146.176eth1 0.0.0.0/0 130.252.99.27eth3 0.0.0.0/0 16.105.78.4

Applications running on the Firewall -making them use a particular provider

As noted above, separate entries in /etc/shorewall/tcrules are required for traffic originating from the firewall.

Experience has shown that in some cases, problems occur with applications running on the firewall itself. This isespecially true when you have specified routefilter on your external interfaces in /etc/shorewall/interfaces (seeabove). When this happens, it is suggested that you have the application use specific local IP addresses rather than 0.

Examples:

Squid: In squid.conf, set tcp_outgoing_address to the IP address of the interface that you want Squid to use.

In OpenVPN, set local (--local on the command line) to the IP address that you want the server to receiveconnections on.

Note that some traffic originating on the firewall doesn't have a SOURCE IP address before routing. At least oneShorewall user reports that an entry in /etc/shorewall/rtrules with 'lo' in the SOURCE column seems to be themost reliable way to direct such traffic to a particular ISP.

Example:

#SOURCE DEST PROVIDER PRIORITYlo - shorewall 1000

/etc/shorewall/rtrules (formerly /etc/shorewall/route_rules)

The rtrules file allows assigning certain traffic to a particular provider just as entries in the tcrules file. Thedifference between the two files is that entries in rtrules are independent of Netfilter.

Routing Rules

Routing rules are maintained by the Linux kernel and can be displayed using the ip rule ls command. When routing apacket, the rules are processed in turn until the packet is successfully routed.

gateway:~ # ip rule ls0: from all lookup local

An ip address (network or host) that matches the source IP address in a packet. May also be specified as aninterface name optionally followed by ":" and an address. If the device 'lo' is specified, the packet mustoriginate from the firewall itself.

DEST (Optional)

An ip address (network or host) that matches the destination IP address in a packet.

If you choose to omit either SOURCE or DEST, place "-" in that column. Note that you may not omit bothSOURCE and DEST.

PROVIDER

The provider to route the traffic through. May be expressed either as the provider name or the providernumber.

PRIORITY

The rule's priority which determines the order in which the rules are processed.

1000-1999 Before Shorewall-generated 'MARK' rules

11000- 11999 After 'MARK' rules but before Shorewall-generated rules for ISP interfaces.

26000-26999 After ISP interface rules but before 'default' rule.

Rules with equal priority are applied in the order in which they appear in the file.

MARK (Optional - added in Shorewall 4.4.25)

Mark and optional mask in the form mark[/mask]. For this rule to be applied to a packet, the packet's markvalue must match the mark when logically anded with the mask. If a mask is not supplied, Shorewall supplies asuitable provider mask.

Multi-ISP and VPN

For those VPN types that use routing to direct traffic to remote VPN clients (including but not limited to OpenVPN inrouted mode and PPTP), the VPN software adds a host route to the main table for each VPN client. The bestapproach is to use USE_DEFAULT_RT=Yes as described below. If that isn't possible, you must add a routing rule inthe 1000-1999 range to specify the main table for traffic addressed to those clients. See Example 2 below.

If you have an IPSEC gateway on your firewall, be sure to arrange for ESP packets to be routed out of the sameinterface that you have configured your keying daemon to use.

Examples

Example 1: You want all traffic entering the firewall on eth1 to be routed through Comcast.

#SOURCE DEST PROVIDER PRIORITYeth1 - Comcast 1000

With this entry, the output of ip rule ls would be as follows.

gateway:~ # ip rule ls0: from all lookup local1000: from all iif eth1 lookup Comcast10001: from all fwmark 0x1 lookup Blarg10002: from all fwmark 0x2 lookup Comcast20000: from 206.124.146.176 lookup Blarg20256: from 24.12.22.33 lookup Comcast32766: from all lookup main32767: from all lookup defaultgateway:~ #

11 de 32

Note that because we used a priority of 1000, the test for eth1 is inserted before the fwmark tests.

Example 2: You use OpenVPN (routed setup w/tunX) in combination with multiple providers. In this case you haveto set up a rule to ensure that the OpenVPN traffic is routed back through the tunX interface(s) rather than throughany of the providers. 10.8.0.0/24 is the subnet chosen in your OpenVPN configuration (server 10.8.0.0255.255.255.0).

#SOURCE DEST PROVIDER PRIORITY- 10.8.0.0/24 main 1000

/etc/shorewall/routes File

Beginning with Shorewall 4.4.15, additional routes can be added to the provider routing tables using the/etc/shorewall/routes file.

The columns in the file are as follows.

PROVIDER

The name or number of a provider defined in shorewall-providers (5).

DEST

Destination host address or network address.

GATEWAY (Optional)

If specified, gives the IP address of the gateway to the DEST.

DEVICE (Optional)

Specifies the device route. If neither DEVICE nor GATEWAY is given, then the INTERFACE specified for thePROVIDER in shorewall-providers (5).

Assume the following entry in /etc/shorewall/providers:

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPYComcast 1 - xxx eth2 ....

The following table gives some example entries in the file and the ip route command which results.

#PROVIDER DEST GATEWAY DEVICE | Generated CommandComcast 172.20.1.0/24 - eth0 | ip -4 route add 172.20.1.0/24 dev eth0 table 1Comcast 192.168.4.0/24 172.20.1.1 | ip -4 route add 192.168.1.0/24 via 172.20.1.1 tabComcast 192.168.4.0/24 | ip -4 route add 192.168.4.0/24 dev eth2 table 1

Looking at the routing tables

To look at the various routing tables, you must use the ip utility. To see the entire routing configuration (includingrules), the command is shorewall show routing. To look at an individual provider's table use ip route ls tableprovider where provider can be either the provider name or number.

Example:

lillycat:- #ip route ls144.77.167.142 dev ppp0 proto kernel scope link src 144.177.121.19971.190.227.208 dev ppp1 proto kernel scope link src 71.24.88.151192.168.7.254 dev eth1 scope link src 192.168.7.1192.168.7.253 dev eth1 scope link src 192.168.7.1192.168.7.0/24 dev eth1 proto kernel scope link src 192.168.7.1192.168.5.0/24 via 192.168.4.2 dev eth0192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.223192.168.1.0/24 via 192.168.4.222 dev eth0default

12 de 32

nexthop dev ppp1 weight 2 nexthop dev ppp0 weight 1lillycat: #ip route ls table 1144.77.167.142 dev ppp0 proto kernel scope link src 144.177.121.199 192.168.5.0/24 via 192.168.4.2 dev eth0 192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.223 192.168.1.0/24 via 192.168.4.222 dev eth0 default dev ppp0 scope link lillycat: #

USE_DEFAULT_RT

USE_DEFAULT_RT is an option in shorewall.conf (5).

One of the drawbacks of the Multi-ISP support as described in the preceding sections is that changes to the maintable made by applications are not added to the individual provider tables. This makes route rules such as describedin one of the examples above necessary.

USE_DEFAULT_RT=Yes works around that problem by passing packets through the main table first rather than last.This has a number of implications:

Both the DUPLICATE and the COPY columns in the providers file must remain empty or contain "-". Theindividual provider routing tables generated when USE_DEFAULT_RT=Yes contain only a host route to thegateway and a default route via the gateway.

1.

The balance option is assumed for all interfaces that do not have the loose option. When you want bothbalance and loose, both must be specified.

2.

The default route generated by Shorewall is added to the default routing table (253) rather than to the mainrouting table (254).

3.

Packets are sent through the main routing table by a routing rule with priority 999. The priority range 1-998may be used for inserting rules that bypass the main table.

4.

You should disable all default route management outside of Shorewall. If a default route is inadvertently addedto the main table while Shorewall is started, then all policy routing will stop working except for those routingrules in the priority range 1-998.

5.

For ppp interfaces, the GATEWAY may remain unspecified ("-"). For those interfaces managed by dhcpcd ordhclient, you may specify 'detect' in the GATEWAY column; Shorewall will use the dhcp client's database todetermine the gateway IP address. All other interfaces must have a GATEWAY specified explicitly.

6.

The configuration in the figure at the top of this section would be specified in /etc/shorewall/providers asfollows.

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPYISP1 1 1 - eth0 206.124.146.254 track -ISP2 2 2 - eth1 130.252.99.254 track -

The remainder of the example is the same.

Although 'balance' is automatically assumed when USE_DEFAULT_RT=Yes, you can easily cause all traffic to useone provider except when you explicitly direct it to use the other provider via shorewall-rtrules (5) or shorewall-tcrules (5).

Example (send all traffic through the 'shorewall' provider unless otherwise directed).

/etc/shorewall/providers:

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONSlinksys 1 1 - wlan0 172.20.1.1 track,balance=1,optionalshorewall 2 2 - eth0 192.168.1.254 track,balance=2,optional

/etc/shorewall/rtrules:

13 de 32

#SOURCE DEST PROVIDER PRIORITY- - shorewall 11999

Tuomo Soini describes the following issue when using USE_DEFAULT_RT=Yes.

He has a /27 network (let.s call it 70.90.191.0/27) from his primary ISP and his secondary ISP supplies him with adynamic IP address on the 91.156.0.0/19 network. From the output of shorewall show routing:

999: from all lookup main 10000: from all fwmark 0x100 lookup ISP1 10001: from all fwmark 0x200 lookup ISP2

Note that the main routing table is consulted prior to the marks for his two provlders. When clients in the large /19network connected to his /27 (through ISP1), the responses were routed out of the ISP2 interface because the mainrouting table included a route to the /19.

The solution was to add an additional entry to rtrules:

#SOURCE DEST PROVIDER PRIORITY70.90.191.0/27 91.156.0.0/19 ISP1 900

With this additional entry, the routing rules are as below and traffic from the /27 is returned via ISP1.

900: from 70.90.191.0/27 to 91.156.0.0/19 lookup ISP1 999: from all lookup main 10000: from all fwmark 0x100 lookup ISP1 10001: from all fwmark 0x200 lookup ISP2

DHCP with USE_DEFAULT_RT

When USE_DEFAULT_RT=Yes, you don't want your DHCP client inserting a default route into the main routingtable.

Debian

In this Debian-specific example, eth0 is managed by dhcpcd.

/etc/default/dhcpcd:

# Config file for dhcpcd. Note that you have to edit the interface# name below, or duplicate the configuration for different interfaces.# If you are editing this file just to get DNS servers set by DHCP,# then you should consider installing the resolvconf package instead.

case ${INTERFACE} ineth0)

# Uncomment this to allow dhcpcd to set the DNS servers in /etc/resolv.conf# If you are using resolvconf then you can leave this commented out.#SET_DNS='yes'

# Uncomment this to allow dhcpcd to set hostname of the host to the# hostname option supplied by DHCP server.#SET_HOSTNAME='yes'

# Uncomment this to allow dhcpcd to set the NTP servers in /etc/ntp.conf#SET_NTP='yes'

# Uncomment this to allow dhcpcd to set the YP servers in /etc/yp.conf#SET_YP='yes'

# Add other options here, see man 8 dhcpcd-bin for details.OPTIONS=(--nogateway --nodns --nontp --script /etc/shorewall/dhcpcd.sh);;

# Add other interfaces here*)

14 de 32

;;

esac

/etc/shorewall/start:

cat

This method works when there are multiple links to the same ISP where both links have the same default gateway.

The key features of this method are:

Providers to be balanced are given a load factor using the load= option in shorewall-providers (5).1.

A load factor is a number in the range 0 < number

You specify the optional option in /etc/shorewall/interfaces:

#ZONE INTERFACE BROADCAST OPTIONSnet eth0 detect optional net eth1 detect optional

SWPING

Shorewall includes a sample monitoring script swping. The swping file is available in the main directory contained inthe Shorewall-common tarball and is included in the Shorewall-common documentation directory in the Shorewall-common RPM. The script is inspired by Angsuman Chakraborty's gwping script.

Important

These samples are offered as is they work for me but I don't make any claim that they will work foranyone else. But if you have a need for automated link monitoring, they offer you a place to start.

Important

If you have installed Shorewall-init, you should disable its ifup/ifdown/NetworkManager integration(set IFUPDOWN=0 in the Shorewall-init configuration file).

The script should be copied to a directory on root's PATH such as /usr/local/sbin/.

The script works by sending pings to target IP addresses through each external interface. These targets must notdepend on any routes other than those that are present in the main routing table. That ensures that a route is availableto the target even when the target's interface is not working and Shorewall has omitted it from the routingconfiguration. An interface is assumed to be up when a specified number (UP_COUNT) of consecutive pingoperations succeed. Similarly, an interface is assumed to be down when a specified number (DOWN_COUNT) ofconsecutive ping operations fail. You can specify the interval between pings (PING_INTERVAL).

The script monitors two interfaces but it is a trivial exercise to extend it to more than two. At the top are a number ofvariables to set:

## IP family -- 4 or 6#FAMILY=4## The commands to run when the status of a line changes. Multiple commands may be specified # when separated by semicolons (";")#COMMAND=

...

## Interfaces to monitor -- you may use shell variables from your params file#IF1=eth0IF2=eth1## Sites to Ping. Must depend only on routes in the 'main' routing table. If not specified,# the interface is assumed to be managed by dhcpcd and the script uses the gateway address# from /var/lib/dhcpcd/dhcpcd-${IFx}.info#TARGET1=TARGET2=## How often to ping#PING_INTERVAL=5## Value for ping's -W option#PING_TIMEOUT=2## This many successive pings must succeed for the interface to be marked up when it is down

17 de 32

#UP_COUNT=5## This many successive pings must fail for the interface to be marked down when it is up#DOWN_COUNT=2

If you leave COMMAND empty, the script sets its value automatically depending on whether Shorewall-lite isinstalled.

When the status of an interface changes:

For each interface, a file is placed in ${VARDIR} (normally /var/lib/shorewall) to record the status of theinterface: either 0 (UP) or 1 (DOWN). The name of the file is interface.status where interface is theinterface (e.g., eth0.status).

Important

Beginning with Shorewall 4.5.0, the generated script automatically maintains this .status file.

A shorewall -f restart command is executed (shorewall-lite restart, if Shorewall-lite is installed).

The contents of the main routing table are displayed.

The .status files are intended to be used with the following /etc/shorewall/isusable script.

local status=0

[ -f ${VARDIR}/${1}.status ] && status=$(cat ${VARDIR}/${1}.status)

return $status

The above script is installed in /etc/shorewall in Shorewall releases 4.3.11 - 4.5.0. Beginning with Shorewall 4.5.1,it is no longer installed in /etc/shorewall, but may be copied there from /usr/share/shorewall/configfiles.

Also included is a sample init script (swping.init) to start the monitoring daemon. Copy it to /etc/init.d/swping and use your distribution's SysV init tools to cause it to be run at boot. It works on OpenSuSE 11.0 --YMMV. Modify the PROG and STATEDIR variables as needed.

As an alternative to using the init script, you can add the following to /etc/shorewall/started:

if [ "$COMMAND" = start ]; then killall -9 swping 2> /dev/null #be sure that there are none left running /usr/local/sbin/swping &fi

and add this to /etc/shorewall/stopped.

if [ "$COMMAND" = stop -o "$COMMAND" = clear ]; then killall -9 swping 2> /dev/nullfi

This simple script has a number of limitations:

It only works on IPv4 or IPv6 but not both at once. So if you want to monitor both IPv4 and IPv6, you need toclone the script are run two copies; one for IPv4 and one for IPv6.

1.

It can only detect the gateway for interfaces managed by dhcpcd.2.

It's method of determining whether an interface is up or down is crude. You will normally specify the defaultgateway for each provider as the sites to ping and being able to ping the default gateway is not a surefireindication that the provider is usable. The method of determining whether a site is up or down is also crude.

3.

Because of the crudeness of the algorithm, hysteresis may occur.4.

18 de 32

It is tricky to configure a system such that the system works correctly when one of its providers is down unlessyou largely don't care which interface is used.

5.

Link Status Monitor (LSM)

Link Status Monitor was written by Mika Ilmaranta and performs more sophisticated monitoringthan the simple swping script described in the preceding section.

Important

If you have installed Shorewall-init, you should disable its ifup/ifdown/NetworkManager integration(set IFUPDOWN=0 in the Shorewall-init configuration file) before installing LSM.

Like many Open Source products, LSM is poorly documented. It's main configuration file is normally kept in/etc/lsm/lsm.conf, but the file's name is passed as an argument to the lsm program so you can name it anythingyou want.

The sample lsm.conf included with the product shows some of the possibilities for configuration. One feature that isnot mentioned in the sample is that an "include" directive is supported. This allows additional files to be sourced infrom the main configuration file.

LSM monitors the status of the links defined in its configuration file and runs a user-provided script when the statusof a link changes. The script name is specified in the eventscript option in the configuration file. Key arguments tothe script are as follows:

$1

The state of the link ('up' or 'down')

$2

The name of the connection as specified in the configuration file.

$4

The name of the network interface associated with the connection.

$5

The email address of the person specified to receive notifications. Specified in the warn_email option in theconfiguration file.

It is the responsibility of the script to perform any action needed in reaction to the connection state change. Thedefault script supplied with LSM composes an email and sends it to $5.

I personally use LSM here at shorewall.net (configuration is described below). I have set things up so that:

Shorewall [re]starts lsm during processing of the start and restore commands. I don't have Shorewall restartlsm during Shorewall restart because I restart Shorewall much more often than the average user is likely to do.

Shorewall starts lsm because I have a dynamic IP address from one of my providers (Comcast); Shorewalldetects the default gateway to that provider and creates a secondary configuration file (/etc/lsm/shorewall.conf) that contains the link configurations. That file is included by /etc/lsm/lsm.conf.

The script run by LSM during state change (/etc/lsm/script) writes a ${VARDIR}/xxx.status file whenthe status of an interface changes. Those files are read by the isusable extension script (see below).

Below are my relevant configuration files.

Warning

These files only work with Shorewall-perl 4.4 Beta 2 and later.

19 de 32

/etc/shorewall/isusable:

local status=0## Read the status file (if any) created by /etc/lsm/script#[ -f ${VARDIR}/${1}.status ] && status=$(cat ${VARDIR}/${1}.status)

return $status

/etc/shorewall/lib.private:

################################################################################ Create /etc/lsm/shorewall.conf# Remove the current interface status files# Start lsm###############################################################################start_lsm() { # # Kill any existing lsm process(es) # killall lsm 2> /dev/null # # Create the Shorewall-specific part of the LSM configuration. This file is # included by /etc/lsm/lsm.conf # # Avvanta has a static gateway while Comcast's is dynamic # cat > /var/log/lsm}

eth0 has a dynamic IP address so I need to use the Shorewall-detected gateway address ($SW_ETH1_GATEWAY). Isupply a default value to be used in the event that detection fails.

Note

In Shorewall 4.4.7 and earlier, the variable name is ETH1_GATEWAY.

/etc/shorewall/started:

################################################################################### [re]start lsm if this is a 'start' command or if lsm isn't running##################################################################################if [ "$COMMAND" = start -o -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then start_lsmfi

/etc/shorewall/restored:

20 de 32

################################################################################### Start lsm if it isn't running##################################################################################if [ -z "$(ps ax | grep 'lsm ' | grep -v 'grep ' )" ]; then start_lsmfi

/etc/lsm/lsm.conf:

## Defaults for the connection entries#defaults { name=defaults checkip=127.0.0.1 eventscript=/etc/lsm/script max_packet_loss=20 max_successive_pkts_lost=7 min_packet_loss=5 min_successive_pkts_rcvd=10 interval_ms=2000 timeout_ms=2000 warn_email=you@yourdomain.com check_arp=0 sourceip= ttl=0}

include /etc/lsm/shorewall.conf

/etc/lsm/script

#!/bin/sh## (C) 2009 Mika Ilmaranta # (C) 2009 Tom Eastep ## License: GPLv2#

STATE=${1}NAME=${2}CHECKIP=${3}DEVICE=${4}WARN_EMAIL=${5}REPLIED=${6}WAITING=${7}TIMEOUT=${8}REPLY_LATE=${9}CONS_RCVD=${10}CONS_WAIT=${11}CONS_MISS=${12}AVG_RTT=${13}

if [ -f /usr/share/shorewall-lite/lib.base ]; then VARDIR=/var/lib/shorewall-lite STATEDIR=/etc/shorewall-liteelse VARDIR=/var/lib/shorewall STATEDIR=/etc/shorewallfi

[ -f ${STATEDIR}/vardir ] && . ${STATEDIR}/vardir

cat

device = ${DEVICE}warn_email = ${WARN_EMAIL}

Packet counters:replied = ${REPLIED} packets repliedwaiting = ${WAITING} packets waiting for replytimeout = ${TIMEOUT} packets that have timed out (= packet loss)reply_late = ${REPLY_LATE} packets that received a reply after timeoutcons_rcvd = ${CONS_RCVD} consecutively received replies in sequencecons_wait = ${CONS_WAIT} consecutive packets waiting for replycons_miss = ${CONS_MISS} consecutive packets that have timed outavg_rtt = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating t

Your LSM Daemon

EOM

# Uncomment the next two lines if you are running Shorewall 4.4.x or earlier

# [ ${STATE} = up ] && state=0 || state=1# echo $state > ${VARDIR}/${DEVICE}.status

/sbin/shorewall restart -f >> /var/log/lsm 2>&1

/sbin/shorewall show routing >> /var/log/lsm

exit 0

#EOF

Beginning with Shorewall 4.4.23, it is not necessary to restart the firewall when an interface transitions between theusable and unusable states./etc/lsm/script

#!/bin/sh## (C) 2009 Mika Ilmaranta # (C) 2009 Tom Eastep ## License: GPLv2#

STATE=${1}NAME=${2}CHECKIP=${3}DEVICE=${4}WARN_EMAIL=${5}REPLIED=${6}WAITING=${7}TIMEOUT=${8}REPLY_LATE=${9}CONS_RCVD=${10}CONS_WAIT=${11}CONS_MISS=${12}AVG_RTT=${13}

if [ -f /usr/share/shorewall-lite/lib.base ]; then VARDIR=/var/lib/shorewall-lite STATEDIR=/etc/shorewall-liteelse VARDIR=/var/lib/shorewall STATEDIR=/etc/shorewallfi

[ -f ${STATEDIR}/vardir ] && . ${STATEDIR}/vardir

cat

device = ${DEVICE}warn_email = ${WARN_EMAIL}

Packet counters:replied = ${REPLIED} packets repliedwaiting = ${WAITING} packets waiting for replytimeout = ${TIMEOUT} packets that have timed out (= packet loss)reply_late = ${REPLY_LATE} packets that received a reply after timeoutcons_rcvd = ${CONS_RCVD} consecutively received replies in sequencecons_wait = ${CONS_WAIT} consecutive packets waiting for replycons_miss = ${CONS_MISS} consecutive packets that have timed outavg_rtt = ${AVG_RTT} average rtt, notice that waiting and timed out packets have rtt = 0 when calculating t

Your LSM Daemon

EOM

if [ ${STATE} = up ]; then# echo 0 > ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier ${VARDIR}/firewall enable ${DEVICE}else# echo 1 > ${VARDIR}/${DEVICE}.status # Uncomment this line if you are running Shorewall 4.4.x or earlier ${VARDIR}/firewall disable ${DEVICE}fi

/sbin/shorewall show routing >> /var/log/lsm

exit 0

#EOF

Two Providers Sharing an Interface

Shared interface support has the following characteristics:

Only Ethernet (or Ethernet-like) interfaces can be used. For inbound traffic, the MAC addresses of thegateway routers are used to determine which provider a packet was received through. Note that only routedtraffic can be categorized using this technique.

a.

You must specify the address on the interface that corresponds to a particular provider in the INTERFACEcolumn by following the interface name with a colon (":") and the address.

b.

Entries in /etc/shorewall/masq must be qualified by the provider name (or number).c.

This feature requires Realm Match support in your kernel and iptables.d.

You must add rtrules entries for networks that are accessed through a particular provider.e.

If you have additional IP addresses through either provider, you must add rtrules to direct traffic FROMeach of those addresses through the appropriate provider.

f.

You must manually add MARK rules for traffic known to come from each provider.g.

You must specify a gateway IP address in the GATEWAY column of /etc/shorewall/providers; detect isnot permitted.

h.

Taken together, b. and h. effectively preclude using this technique with dynamic IP addresses.

Example:

This is our home network circa fall 2008. We have two Internet providers:

Comcast -- Cable modem with one dynamic IP address.1.

Avvanta -- ADSL with 5 static IP addresses.2.

Because the old Compaq Presario that I use for a firewall only has three PCI slots and no onboard Ethernet, itdoesn't have enough Ethernet controllers to support both providers. So I use a Linksys WRT300n pre-N router as a

23 de 32

gateway to Comcast. Note that because the Comcast IP address is dynamic, I could not share a single firewallinterface between the two providers directly.

On my personal laptop (ursa), I have 9 virtual machines running various Linux distributions. It is the Shorewallconfiguration on ursa that I will describe here.

Below is a diagram of our network:

The local wired network in my office is connected to both gateways and uses the private (RFC 1918) network172.20.1.0/24. The Comcast gateway has local IP address 172.20.1.1 while the Avvanta gateway has local IP address172.20.1.254. Ursa's eth0 interface has a single IP address (172.20.1.130).

This configuration uses USE_DEFAULT_RT=Yes in shorewall.conf (see above).

Here is the providers file:

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPYcomcast 1 1 - eth0:172.20.1.130 172.20.1.1 track,loose,balance,optionalavvanta 2 2 - eth0:172.20.1.130 172.20.1.254 track,optional,loosewireless 3 3 - wlan0 172.20.1.1 track,optional

Several things to note:

172.20.1.130 is specified as the eth0 IP address for both providers.1.

24 de 32

Both wired providers have the loose option. This prevents Shorewall from automatically generating routingrules based on the source IP address.

2.

Only comcast has the balance option. With USE_DEFAULT_RT=yes, that means that comcast will be thedefault provider. While balance is the default, with USE_DEFAULT_RT=Yes, it must be specified explicitlywhen loose is also specified.

3.

I always disable the wireless interface when the laptop is connected to the wired network.4.

I use a different Shorewall configuration when I take the laptop on the road.5.

Here is the rtrules file:

#SOURCE DEST PROVIDER PRIORITY- 206.124.146.176/31 avvanta 1000- 206.124.146.178/31 avvanta 1000- 206.124.146.180/32 avvanta 1000

Those rules direct traffic to the five static Avvanta IP addresses (only two are currently used) through the avvantaprovider.

Here is the tcrules file (MARK_IN_FORWARD_CHAIN=No in shorewall.conf):

#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS # PORT(S) 2 $FW 0.0.0.0/0 tcp 212 $FW 0.0.0.0/0 tcp - - - - - - 2 $FW 0.0.0.0/0 tcp 119

These rules:

Use avvanta for FTP.

Use avvanta for NTTP

The remaining files are for a rather standard two-interface config with a bridge as the local interface.

zones:

#ZONE IPSEC OPTIONS IN OUT# ONLY OPTIONS OPTIONSfw firewallnet ipv4kvm ipv4

policy:

net net NONEfw net ACCEPTfw kvm ACCEPTkvm all ACCEPTnet all DROP infoall all REJECT info

interfaces:

#ZONE INTERFACE BROADCAST OPTIONS GATEWAY#net eth0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignorenet wlan0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optionalkvm br0 detect routeback #Virtual Machines

Note

wlan0 is the wireless adapter in the notebook. Used when the laptop is in our home but not connectedto the wired network.

25 de 32

masq:

#INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSECeth0 192.168.0.0/24wlan0 192.168.0.0/24

Note

Because the firewall has only a single external IP address, I don't need to specify the providers in themasq rules.

A Complete Working Example

This section describes the network at shorewall.net in late 2012. The configuration is as follows:

Two providers:

ComcastC -- A consumer-grade Comcast cable line with a dynamic IP address.

ComcastB -- A Comcast Business-class line with 5 static IP addresses.

A local network consisting of wired and wireless client systems. A wireless-N router is used as an access pointfor the wireless hosts.

A DMZ hosting a two servers (one has two public IP addresses - one for receiving email and one for sending)and a system dedicaed to running irssi (usually via IPv6)

The network is pictured in the following diagram:

26 de 32

IPv4 Configuration

The Business Gateway manages a gigabit local network with address 10.0.1.1/24. So The firewall is given address10.0.1.11/24 and the gateway is configured to route the public IP block via that address. The gateway's firewall isonly enabled for the 10.0.1.0/24 network.

Because the business network is faster and more reliable, the configuration favors sending local network traffic viathat uplink rather than the consumer line.

Here are the key entries in /etc/shorewall/params:

LOG=NFLOG

INT_IF=eth2

27 de 32

TUN_IF=tun+COMB_IF=eth1COMC_IF=eth0

STATISTICAL=PROXY=FALLBACK=PROXYDMZ=SQUID2=

The last three variables are used to configure the firewall differently to exercise various Shorewall features.

Here are the key entries in /etc/shorewall/shorewall.conf:

################################################################################ F I R E W A L L O P T I O N S###############################################################################

...

ACCOUNTING_TABLE=mangle

...

AUTOMAKE=Yes

BLACKLISTNEWONLY=Yes

...

EXPAND_POLICIES=No

EXPORTMODULES=Yes

FASTACCEPT=No

..

KEEP_RT_TABLES=Yes #This is necessary when both IPv4 and IPv6 Multi-ISP are used

LEGACY_FASTSTART=Yes

LOAD_HELPERS_ONLY=Yes

...

MARK_IN_FORWARD_CHAIN=No

MODULE_SUFFIX=ko

MULTICAST=No

MUTEX_TIMEOUT=60

NULL_ROUTE_RFC1918=Yes

OPTIMIZE=31

OPTIMIZE_ACCOUNTING=No

REQUIRE_INTERFACE=No

RESTORE_DEFAULT_ROUTE=No

RETAIN_ALIASES=No

ROUTE_FILTER=No

SAVE_IPSETS=

TC_ENABLED=No

TC_EXPERT=No

TC_PRIOMAP="2 3 3 3 2 3 1 1 2 2 2 2 2 2 2 2"

28 de 32

TRACK_PROVIDERS=Yes

USE_DEFAULT_RT=Yes

USE_PHYSICAL_NAMES=Yes

ZONE2ZONE=-

################################################################################# P A C K E T M A R K L A Y O U T################################################################################

TC_BITS=8

PROVIDER_BITS=2

PROVIDER_OFFSET=16

MASK_BITS=8

ZONE_BITS=0

I use USE_DEFAULT_RT=Yes and since there are only two providers, two provider bits are all that are required.

Here is /etc/shorewall/zones:

fw firewallloc ip #Local Zonenet ip #Internetsmc:net ip #10.0.1.0/24vpn ip #OpenVPN clientsdmz ip #LXC Containers

/etc/shorewall/interfaces:

#ZONE INTERFACE OPTIONSloc INT_IF dhcp,physical=$INT_IF,required,wait=5,routefilter,nets=172.20.1.0/24net COMB_IF optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMB_IF,upnp,nosmurfs,net COMC_IF optional,sourceroute=0,routefilter=0,arp_ignore=1,proxyarp=0,physical=$COMC_IF,upnp,nosmurfs,vpn TUN_IF+ physical=tun+,ignore=1dmz br0 routeback,proxyarp=1- lo ignore

/etc/shorewall/hosts:

#ZONE HOST(S) OPTIONSsmc COMB_IF:10.1.10.0/24

/etc/shorewall/providers:

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY?if $FALLBACKComcastB 1 0x10000 - COMB_IF 70.90.191.126 loose,fallbackComcastC 2 0x20000 - COMC_IF detect loose,fallback?elsif $STATISTICALComcastB 1 0x10000 - COMB_IF 70.90.191.126 loose,load=0.66666667ComcastC 2 0x20000 - COMC_IF detect loose,load=0.33333333?elseComcastB 1 0x10000 - COMB_IF 70.90.191.126 loose,balance=2ComcastC 2 0x20000 - COMC_IF detect loose,balance?endif?if $PROXY && ! $SQUID2Squid 3 - - lo - tproxy ?endif

Notice that in the current balance mode, as in the STATISTICAL mode, the business line is favored 2:1 over theconsumer line.

Here is /etc/shorewall/rtrules:

29 de 32

#SOURCE DEST PROVIDER PRIORITY70.90.191.121 - ComcastB 100070.90.191.123 - ComcastB 1000&COMC_IF - ComcastC 1000172.20.1.145 - ComcastC 1000172.20.1.146 - ComcastC 1000br0 - ComcastB 11000

For reference, this configuration generates these routing rules:

root@gateway:~# ip rule ls0: from all lookup local 999: from all lookup main 1000: from 70.90.191.121 lookup ComcastB 1000: from 70.90.191.123 lookup ComcastB 1000: from 67.170.121.6 lookup ComcastC 1000: from 172.20.1.145 lookup ComcastC 1000: from 172.20.1.146 lookup ComcastC 10000: from all fwmark 0x10000/0x30000 lookup ComcastB 10001: from all fwmark 0x20000/0x30000 lookup ComcastC 11000: from all iif br0 lookup ComcastB 32765: from all lookup balance 32767: from all lookup default root@gateway:~#

/etc/shorewall/tcrules is not used to support Multi-ISP:

#MARK SOURCE DEST PROTO DEST SOURCE # PORT(S) PORT(S)FORMAT 2TTL(+1):P INT_IF -SAME:P INT_IF - tcp 80,443?if $PROXY && ! $SQUID2 DIVERT COMB_IF - tcp - 80 DIVERT COMC_IF - tcp - 80 DIVERT br0 172.20.1.0/24 tcp - 80 TPROXY(3129,172.20.1.254) INT_IF - tcp 80 ?if $PROXYDMZ TPROXY(3129,172.20.1.254) br0 - tcp 80 ?endif?endof

IPv6 Configuration

The IPv6 configuration has two separate sub-nets, both services through 6in4 tunnels from Hurricane Electric. Theyare both configured through the Business IPv4 uplink. I originally had the sit2 tunnel configured through theconsumer uplink but Comcast (Xfinity) decided to start blocking HE IPv6 tunnels on their consumer network,preferring their own 6to4 IPv6 solution.

One HE tunnel handles the servers and one tunnel handles the local network.

Here are the key entries in /etc/shorewall6/shorewall6.conf:

################################################################################ F I R E W A L L O P T I O N S###############################################################################

...

FASTACCEPT=No

FORWARD_CLEAR_MARK=Yes

IMPLICIT_CONTINUE=No

IP_FORWARDING=Keep

KEEP_RT_TABLES=Yes #Required when both IPv4 and IPv6 Multi-ISP are used

...

30 de 32

TRACK_PROVIDERS=No

USE_DEFAULT_RT=Yes

ZONE2ZONE=-

...

################################################################################# P A C K E T M A R K L A Y O U T################################################################################

TC_BITS=8

PROVIDER_BITS=8

PROVIDER_OFFSET=8

MASK_BITS=8

ZONE_BITS=0

Here is /etc/shorewall6/zones:

#ZONE TYPE OPTIONSfw firewallnet ipv6loc ipv6dmz ipv6

/etc/shorewall/interfaces:

#ZONE INTERFACE OPTIONSnet sit1 forward=1,sfilter=2001:470:b:227::40/124,optionalnet sit2 forward=1,sfilter=2001:470:b:227::40/124,optionalnet sit3 forward=1,sfilter=2001:470:b:227::40/124,optionalloc eth2 forward=1dmz br0 routeback,forward=1,required

/etc/shorewall/providers:

#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPYLOC 4 0x100 - sit2 - track,balance,looseDMZ 5 0x200 - sit1 - track,fallback,loose6to4 6 0x300 - sit3 ::192.88.99.1 track,fallback,loose

Notice that the provider numbers are disjoint from those in the IPv4 configuration. This allows for unique providernames in /etc/iproute2/rt_tables:

## reserved values#255 local254 main253 default250 balance0 unspec## local#1 ComcastB2 ComcastC3 TProxy4 LOC5 DMZ6 6to4

The /etc/shorewall6/rtrules file is straight-forward:

#SOURCE DEST PROVIDER PRIORITY

31 de 32

2001:470:B:227::1/64 ::/0 DMZ 110002001:470:B:787::1/64 ::/0 LOC 110002002:465a:bf79::1/64 ::/0 6to4 11000

This results in the following routing rules:

root@gateway:~# ip -6 rule ls0: from all lookup local 999: from all lookup main 10003: from all fwmark 0x100/0xff00 lookup LOC 10004: from all fwmark 0x200/0xff00 lookup DMZ 10005: from all fwmark 0x300/0xff00 lookup 6to4 11000: from 2001:470:b:787::1/64 lookup LOC 11000: from 2001:470:b:227::1/64 lookup DMZ 11000: from 2002:465a:bf79::1/64 lookup 6to4 32765: from all lookup balance 32767: from all lookup default root@gateway:~#

[1] While we describe a setup using different ISPs in this article, the facility also works with two uplinks from thesame ISP.

32 de 32