2015 miercom unified threat management …the spirent avalanche tool provides multi-10 gbps...
TRANSCRIPT
2015 Miercom
Unified Threat Management
Throughput Performance
DR150514P
June 2015
Miercom
www.miercom.com
WatchGuard Firebox M300 UTM 2 DR150514P
Copyright © 2015 Miercom 9 June 2015
Contents
1.0 Executive Summary ............................................................................................................................... 3
2.0 How We Did It ......................................................................................................................................... 4
3.0 Throughput Test Results ................................................................................................................... 10
4.0 Fair Test Notification ........................................................................................................................... 18
5.0 About Miercom ..................................................................................................................................... 18
6.0 Use of This Report ............................................................................................................................... 18
WatchGuard Firebox M300 UTM 3 DR150514P
Copyright © 2015 Miercom 9 June 2015
1.0 Executive Summary
Miercom was engaged by WatchGuard to validate the performance of its Firebox M300
Unified Threat Management (UTM) appliance, as well as the comparative performance
of Vendor A’s products, the Fortinet FortiGate 100D, Vendor B’s product, and the
Sophos SG 210.
A series of six tests were applied to verify throughput performance of the firewall
operating alone and with other policies enabled.
This report summarizes the throughput of the appliances. Each appliance was
configured with vendor defaults enabled to ensure a baseline configuration. Features
were then enabled incrementally and the resulting throughput rates were recorded.
Key Findings:
The Firebox M300 had excellent throughput performance for any enabled
feature, whether it was tested alone or in combination
WatchGuard’s provides firewall and antivirus protection over encrypted traffic
while outperforming the competitive average throughput performance by 385%
With full UTM enabled, WatchGuard had over a 45% lead in throughput than its
best competitor, against HTTP traffic and performed 218% better than the
industry average
The WatchGuard Firebox M300 UTM appliance out-classed the competitors in almost
every test. The M300 is one of the best mid-sized and distributed enterprise class UTM
appliances on the market, given competing products in the same price range
and scale.
Based on the impressive results of the testing, we proudly
award the Miercom Performance Verified Certification to
WatchGuard, having delivered outstanding performance
by the WatchGuard Firebox M300 UTM.
Robert Smithers
CEO
Miercom
WatchGuard Firebox M300 UTM 4 DR150514P
Copyright © 2015 Miercom 9 June 2015
2.0 How We Did It
About the Test
Unified Threat Management (UTM) appliances are intended to operate as a combination
of firewall, intrusion prevention, and antivirus functionalities. Testing evaluated the
throughput performance of various threat management functionalities, in isolation and
in combination, for each vendor.
These functionalities protect a network by evaluating traffic through abstract layers of
the Open Systems Interconnection (OSI) model. In order to simulate a real-world
environment, tests were performed across multiple layers to portray how real traffic
would travel and be handled.
Two layers of the standardized OSI model were of focus for this product evaluation: the
transport layer and the application layer. A majority of the pertinent network traffic
processed by the appliances under test operates on these layers, making them
significant layers to verify throughput performance.
The following six tests were used to validate performance:
Test Function Parameter
1. Firewall
2. Firewall (FW)
These tests establish line speed on each appliance and
ensure that results are within a reasonable range of
those claimed by vendors.
UDP, UDP IMIX, HTTP,
HTTPS
3. FW+ IPS
4. FW + AV
5. FW+ IPS + AV
6. Full UTM: FW +
IPS + AV + AppCtrl
These tests establish proper functioning and expected
traffic analysis of each appliance.
These tests are the primary focus since they describe the
throughput of appliances with real-world deployment.
HTTP, HTTPS w/ deep
packet inspection
The first two tests were performed for two purposes: to ensure appliances were properly
configured and processing traffic, and to compare appliances based on throughput
speed with only firewall enabled.
Four different types of traffic were used in the throughput tests: user datagram protocol
(UDP), UDP IMIX, hypertext transfer protocol (HTTP), and HTTP Secure (HTTPS). UDP
operates on the transport layer and is intended for fast data transmission. It simply
includes the source and destination ports, the length (bytes), and the “checksum”
parameter to ensure data is intact when it does arrive. UDP Internet Mix, or IMIX, is used
to resemble real-world internet traffic conditions. Standardized IMIX profiles are based
on statistical sampling of traffic passed through routers and is used for testing to certify
WatchGuard Firebox M300 UTM 5 DR150514P
Copyright © 2015 Miercom 9 June 2015
comparable results for each vendor. HTTP operates on the application layer and serves
as request-response communication protocol between clients and servers. HTTPS is
when the data being communicated is encrypted and travels on the secure socket layer
(SSL). HTTPS implies a more complex process where certificates must be authenticated,
and session keys conveyed to continue encrypted transmission. Throughput is expected
to drop considerably during SSL traffic, for firewall and subsequent tests.
Both user datagram and hypertext transfer protocols allow the comparison of respective
throughput speeds of each UTM appliance while under various real-world network
traffic scenarios.
The second and following throughput tests were conducted using both HTTP and HTTPS
traffic. HTTP is unencrypted hypertext transfer protocol traffic. HTTPS traffic is encrypted
with secure socket layer (SSL) enabled and provides more realistic network conditions to
test deep packet inspection (DPI) functionality of each device, which is implemented
when IPS and antivirus functionality is enabled.
The third test, FW + IPS, validated HTTP and HTTPS throughput speeds for each
appliance with both the firewall and the intrusion prevention system (IPS) functions
enabled. IPS functionality can be deployed on both HTTP traffic and encrypted HTTPS
traffic, so the test was run for each traffic type. IPS tends to be rule or signature-based,
and monitors network traffic and/or system activity to actively detect, prevent and block
intrusions.
The fourth test, FW + AV, compared throughput speeds for each appliance with firewall
and antivirus functionality enabled. This allows an accurate record of throughput speeds
while the UTM appliance is performing virus scanning activities on HTTP and HTTPS
traffic.
Since Unified Threat Management tends to involve at a bare minimum: FW + IPS + AV
functionality, the fifth test compares throughput speed in a real-life UTM deployment
scenario. This test generated results as if each UTM appliance was acting as a next-
generation firewall with IPS and AV systems both inspecting and acting upon
unencrypted (HTTP) and encrypted (HTTPS) traffic.
The sixth and final test, full UTM: FW + IPS + AV + AppCtrl, compared throughput rates
for each appliance with all functionality enabled. It measured throughput in a
deployment scenario where the device under test is operating as a next-generation
firewall with IPS, AV, and application control functionality enabled. This would describe a
situation involved signature-based monitoring of network connections, DPI functionality
for HTTPS (SSL) decryption and analysis, as well as a fine-tuned application layer security
control scheme.
WatchGuard Firebox M300 UTM 6 DR150514P
Copyright © 2015 Miercom 9 June 2015
Products Tested
WatchGuard Firebox M300
This appliance aims to provide security for mid-sized enterprises using its unified
approach which includes the following features: firewall, application control, antivirus,
and IPS. These features are also offered to operate with HTTPS inspection enabled. In
addition to these core features, it is also capable of additional subscriptions which
enable advanced persistent threat blocker, data loss prevention, reputation enabled
defense, spam blocker, and URL filtering.
Vendor A Product One
This appliance intends to deliver a wide scope of security for medium enterprises,
without compromising its performance. This product is claimed to have industry leading
SSL-decryption rates to block malware, an authentication server to enforce policies for
application control, and its NSA Series integration of firewall and intrusion prevention,
all manageable by a single console.
Vendor A Product Two
This appliance supports security for distributed enterprises and remote offices, all
managed by a central office. Its deep packet inspection examines traffic across all ports,
with integrated intrusion prevention engine, anti-virus, anti-spyware, and application
controls, and web filtering over both unsecure and encrypted SSL connections.
Fortinet FortiGate 100D
This appliance is used to protect medium enterprises or remote branches with one
management console, capable of multiple technologies for detecting network attacks.
Features include: firewall, IPsec and SSL-VPN, application control, intrusion prevention,
anti-malware, anti-spam, P2P security, and web filtering. These features can be enabled
alone or combined for a full unified threat management system.
Vendor B
This appliance offers security to mid-sized enterprises by providing application visibility,
antivirus, IPS, data filtering, modern malware protection, URL filtering, and mobile
security. It is capable of policy-based decryption and inspection to ensure that SSL and
SSH encrypted traffic is pertinent to business purposes only.
Sophos SG 210
This appliance offers a consolidated security solution which protects a medium sized
business or organization for both standard and secured internet traffic. This unified
threat management consists of firewall, intrusion prevention, and antivirus with high
rates of throughput claimed for each.
WatchGuard Firebox M300 UTM 7 DR150514P
Copyright © 2015 Miercom 9 June 2015
Test Bed Setup
Figure 1: Test Setup Diagram
All hardware appliances were tested in an identical environment.
Bidirectional test traffic from each load generator was routed to each appliance using
three port pairs and back to the generator for Layer 4 testing. Packet size consisted of
1518 bytes.
Unidirectional test traffic from each load generator was routed to each appliance using
three port pairs for Layer 7 testing. Protocols used were port 80 HTTP and port
443 HTTPS.
Source: Miercom May 2015
WatchGuard Firebox M300
Vendor A Product One
Fortinet FortiGate 100D
Sophos SG 210
Vendor B
Management
Workstation
Vendor A Product Two
WatchGuard Firebox M300 UTM 8 DR150514P
Copyright © 2015 Miercom 9 June 2015
List of Equipment
Name Function Version
WatchGuard Firebox M300 UTM Appliance V5.518 Build
11.9.6.B475461
Vendor A Product One Next Generation Firewall XXX
Vendor A Product Two Next Generation Firewall XXX
Fortinet FortiGate 100D UTM Appliance V5.2.3 Build 670
Sophos SG 210 UTM Appliance 9.310-11
Vendor B UTM Appliance XXX
Spirent TestCenter Traffic Generator 4.43
Spirent Avalanche Traffic Generator 4.43
The Spirent TestCenter tool delivers comparative analysis of devices or services with
deterministic traffic during product development cycles or vendor comparisons.
The Spirent Avalanche tool provides multi-10 Gbps capacity, security, and performance
testing for network infrastructures.
WatchGuard Firebox M300 UTM 9 DR150514P
Copyright © 2015 Miercom 9 June 2015
Configurations
All appliances were capable of firewall, antivirus, application control and IPS. They were
tested in-line with default features and up-to-date firmware. All of the devices were
tested with three port pairs enabled.
WatchGuard Firebox M300
This product was configured for this testing primarily as a high-capacity, next generation
firewall.
Vendor A Product One
This product was a Next Generation Firewall appliance.
Vendor A Product Two
This product was a Next Generation Firewall appliance.
Fortinet FortiGate 100D
This product was configured as a NGFW appliance with default features enabled.
However, it did not support TLC v1.2, so RC4-MD5 and RC4-SHA were used instead.
Vendor B
This product was configured with default features enabled. However, it had a packet
fragmentation issue that required a reduction in packet size from the standard 1518
bytes to 1500 bytes.
Sophos SG 210
This product was configured as a NGFW appliance with default features enabled.
WatchGuard Firebox M300 UTM 10 DR150514P
Copyright © 2015 Miercom 9 June 2015
3.0 Throughput Test Results
Summary of Results
Throughput results by vendor and traffic:
Throughput (Mbps)
WatchGuard Firebox M300
Vendor A Product One
Vendor A Product Two
Fortinet FortiGate
100D Vendor B
Sophos SG 210
UDP Firewall
UDP 1518 byte 4000 1398 1170
1440 414 6000
UDP IMIX 2160 370 660 532 414 2212
HTTP/S Firewall (FW)
HTTP 2663 940 1000 1147 975 3000
HTTPS 1500 982 1000 1309 975 2200
FW + IPS
HTTP 2513 903 1000 1130 217 233
HTTPS 1231 795 1000 1216 975 2246
FW + AV
HTTP 1149 152 742 538 976 675
HTTPS w/ DPI 987 103 144 256 152 361
FW + IPS + AV
HTTP 847 178 343 438 215 185
HTTPS w/ DPI 417 103 139 218 142 443
Full UTM: FW + IPS + AV + AppCtrl
HTTP 778 150 252 423 215 183
HTTPS w/ DPI 393 103 123 207 142 374
WatchGuard Firebox M300 UTM 11 DR150514P
Copyright © 2015 Miercom 9 June 2015
WatchGuard shows a consistent decrease in throughput as more featured are applied, in
comparison to other vendors which change much more drastically even upon the first
additional feature enabled.
WatchGuard had a much more steady decrease as more features were enabled,
maintaining its baseline throughput better than any vendor for full UTM enabled.
0 1000 2000 3000 4000
SophosSG 210
Vendor B
FortinetFortiGate 100D
Vendor A ProductTwo
Vendor A ProductOne
WatchGuardFirebox M300 UTM
Throughput Performance over HTTP Traffic
Baseline Firewall
FW + IPS
FW + AV
FW + IPS + AV
FW + IPS + AV + AppCtrl
Source: Miercom May 2015
0 500 1000 1500 2000 2500
SophosSG 210
Vendor B
FortinetFortiGate 100D
Vendor A ProductTwo
Vendor A ProductOne
WatchGuardFirebox M300 UTM
Throughput Performance over HTTPS Traffic
Baseline Firewall
FW + IPS
FW + AV
FW + IPS + AV
FW + IPS + AV + AppCtrl
Source: Miercom May 2015
WatchGuard Firebox M300 UTM 12 DR150514P
Copyright © 2015 Miercom 9 June 2015
The following charts graphically represent the comparative throughput performance for
each appliance, by traffic type and features enabled.
Firewall Performance on Transport Layer
Firewall throughput with bidirectional UDP and UDP IMIX test traffic.
The purpose of this test was to ensure each appliance was configured correctly and
processing traffic before applying the baseline and subsequent tests.
Figure 2: Firewall Performance with UDP traffic
WatchGuard outpaced all other vendors for both types of UDP traffic except for Sophos.
Vendor B’s device was only configurable to handle UDP 1500 byte traffic due to high
packet loss, implying a possible issue regarding fragmentation.
0
1000
2000
3000
4000
5000
6000
7000
WatchGuardFireboxM300
Vendor AProduct
One
Vendor AProduct
Two
FortinetFortiGuard
100D
Vendor B SophosSG 210
Thro
ugh
pu
t (M
bp
s)
Firewall with UDP traffic
UDP 1518 byte
UDP iMix
Source: Miercom May 2015
WatchGuard Firebox M300 UTM 13 DR150514P
Copyright © 2015 Miercom 9 June 2015
Firewall Performance on Application Layer
Firewall throughput with unidirectional HTTP and HTTPS traffic generated by the
Spirent TestCenter and Spirent Avalanche test system. HTTP and HTTPS traffic
provided a more realistic test environment for the UTM appliances.
This test served as a baseline test. Subsequent tests increase levels of protection and
are expected to have a decreasing performance in throughput.
Figure 3: Firewall Performance with Encrypted and Unencrypted HTTP traffic
Sophos displayed the highest baseline throughput for both unencrypted and encrypted
traffic. WatchGuard displayed the second highest throughput for both. These throughput
results will be the values for comparison against those for other tests where features are
enabled to determine how much supplementing features deter performance.
0
500
1000
1500
2000
2500
3000
3500
WatchGuardFireboxM300
Vendor AProduct
One
Vendor Aproduct
Two
FortinetFortiGuard
100D
Vendor B Sophos SG210
Thro
ugh
pu
t (M
bp
s)
Firewall with HTTP and HTTPS traffic
HTTP
HTTPS
Source: Miercom May 2015
WatchGuard Firebox M300 UTM 14 DR150514P
Copyright © 2015 Miercom 9 June 2015
Intrusion Prevention System Performance on Application Layer
Intrusion Prevention System (IPS) throughput with unidirectional HTTP and HTTPS
traffic generated by the Spirent TestCenter and Spirent Avalanche test system.
The purpose of this test was to verify any degradation to throughput when an
additional feature was enabled.
Figure 4: Firewall and IPS Performance with Encrypted and Unencrypted
HTTP Traffic
WatchGuard predictably had lower throughput for both secure and unsecured HTTP traffic
when its IPS feature was additionally enabled. However, it maintained the highest
throughput of all vendors for unsecured traffic. Vendor A Product One and Fortinet showed
between 1-19% decrease in performance for either HTTP or HTTPS. Vendor A Product Two
showed no change in throughput from its baseline. Vendor B’s throughput dropped by 78%
for HTTP but not at all for HTTPS. Sophos saw a huge drop of 92% for HTTP but actually
increased throughput by 2% for secured traffic.
0
500
1000
1500
2000
2500
3000
WatchGuardFireboxM300
Vendor AProduct
One
Vendor AProduct
Two
FortinetFortiGuard
100D
Vendor B Sophos SG210
Thro
ugh
pu
t (M
bp
s)
Firewall and IPS with HTTP and HTTPS Traffic
HTTP
HTTPS
Source: Miercom May 2015
WatchGuard Firebox M300 UTM 15 DR150514P
Copyright © 2015 Miercom 9 June 2015
Antivirus Performance on Application Layer
Antivirus (AV) throughput with unidirectional HTTP and HTTPS traffic generated by
the Spirent TestCenter and Spirent Avalanche test system.
Deep packet inspection (DPI) filters packets during data transactions. DPI examines
the packet’s raw data and possibly header as it passes through the appliance’s
inspection point. It is here that it searches for viruses, spam, and other predefined
criteria before it is passed on through the network, or rerouted to another
destination. Performance of the antivirus feature with HTTPS traffic under DPI, rather
than with solely encrypted traffic, yields a more realistic throughput evaluation.
Figure 5: Firewall and Antivirus Performance with Encrypted and Unencrypted
HTTP Traffic
WatchGuard had highest throughput over HTTP and HTTPS than any other vendor. As
expected, WatchGuard’s throughput decreased with antivirus enabled, however it only
decreased slightly for both types of traffic. Vendor A Product One had significant
degradation from its baseline, showing 84% and 90% decrease for HTTP and HTTPS traffic,
respectively. Vendor A Product Two degraded slightly over HTTP and by 86% over secured
traffic. Fortinet had a major shift from its baseline, and even the previous test, having a
degradation of 53% and 80% for HTTP and HTTPS, respectively. Vendor B displayed no
change for HTTP, but dropped significantly for HTTPS by 84% from its baseline. Sophos
experienced a drop from its baseline, but an increase from the previous test for IPS enabled
over HTTP. However, it fell by 84% from its baseline over HTTPS.
0
200
400
600
800
1000
1200
1400
WatchGuardFireboxM300
Vendor AProduct
One
Vendor AProduct
Two
FortinetFortiGuard
100D
Vendor B Sophos SG210
Thro
ugh
pu
t (M
bp
s)
Firewall and Antivirus with HTTP and HTTPS Traffic
HTTP
HTTPS w/DPI
Source: Miercom May 2015
WatchGuard Firebox M300 UTM 16 DR150514P
Copyright © 2015 Miercom 9 June 2015
UTM: Firewall, IPS, and Antivirus Combined Performance on Application Layer
Throughput of AV and IPS features of each device with unidirectional HTTP and
HTTPS traffic generated by the Spirent TestCenter and Spirent Avalanche test system.
Figure 6: Firewall, IPS, and Antivirus Performance with Encrypted and
Unencrypted HTTP Traffic
WatchGuard had the highest throughput of all vendors for AV and IPS features additionally
enabled over HTTP, and trailed only slightly behind Sophos in throughput over HTTPS by
26 Mbps. Its throughput decreased from its baseline by 76% for HTTP and 72% for HTTPS,
its biggest drop yet. Vendor A Product One maintained approximately the same
throughput as the previous test with only firewall and antivirus enabled. Vendor A Product
Two, Fortinet, Vendor B and Sophos degraded in performance by at least 62% for HTTP
and 80% for HTTPS. Sophos displayed the largest drop in HTTP throughput of 94% from its
baseline.
0
100
200
300
400
500
600
700
800
900
WatchGuardFireboxM300
Vendor AProduct
One
Vendor AProduct
Two
FortinetFortiGuard
100D
Vendor B SophosSG 210
Thro
ugh
pu
t (M
bp
s)
Firewall, IPS, and Antivirus with HTTP and HTTPS traffic
HTTP
HTTPSw/ DPI
Source: Miercom May 2015
WatchGuard Firebox M300 UTM 17 DR150514P
Copyright © 2015 Miercom 9 June 2015
UTM: Application Control, Antivirus, and IPS Combined Performance on Application Layer
Throughput of full unified threat management: application control (AppCtrl), AV, and
IPS features of each device on the application with unidirectional HTTP and HTTPS
traffic generated by the Spirent TestCenter and Spirent Avalanche test system.
Application control relates to the data transactions pertaining to specific, individual
applications and is important for protecting endpoint users. DPI is relevant because
the packets of these data transactions need to be carefully analyzed for adept
detection and protection.
Figure 7: Firewall, IPS, Antivirus, and Application Control Performance with
Encrypted and Unencrypted HTTP Traffic
WatchGuard had the highest throughput of all vendors for UTM throughput over both
HTTP and HTTPS, and maintained approximately the same throughput as the previous test
where application control was not yet enabled. Every other vendor had degradation in
throughput by at least 63% for HTTP traffic and 83% for HTTPS. Sophos showed the most
significant drop, by 94% over HTTP as it did when application control was not enabled.
0
100
200
300
400
500
600
700
800
900
WatchGuardFireboxM300
Vendor AProduct
One
Vendor AProduct
Two
FortinetFortiGuard
100D
Vendor B SophosSG 210
Thro
ugh
pu
t (M
bp
s)
Firewall, IPS, AV, and AppCtrl with HTTP and HTTPS traffic
HTTP
HTTPSw/ DPI
Source: Miercom May 2015
WatchGuard Firebox M300 UTM 18 DR150514P
Copyright © 2015 Miercom 9 June 2015
4.0 Fair Test Notification
All vendors with products featured in this report were afforded the opportunity before,
during, and after testing was complete to comment on the results and demonstrate the
performance of their product(s). Any vendor with a product tested by Miercom in one of
our published studies that disagrees with our findings is extended an opportunity for a
retest and to demonstrate the performance of the product(s) at no charge to the
vendor. All vendors are welcome to demonstrate their performance on their own to
Miercom. Miercom will update these results if new data presents itself.
5.0 About Miercom
Miercom has published hundreds of network product analyses in leading trade
periodicals and other publications. Miercom’s reputation as the leading, independent
product test center is undisputed. Private test services available from Miercom include
competitive product analyses, as well as individual product evaluations. Miercom
features comprehensive certification and test programs including: Certified
Interoperable, Certified Reliable, Certified Secure and Certified Green. Products may also
be evaluated under the Performance Verified program, the industry’s most thorough
and trusted assessment for product usability and performance.
6.0 Use of This Report
Every effort was made to ensure the accuracy of the data contained in this report but
errors and/or oversights can occur. The information documented in this report may also
rely on various test tools, the accuracy of which is beyond our control. Furthermore, the
document relies on certain representations by the vendors that were reasonably verified
by Miercom but beyond our control to verify to 100 percent certainty.
This document is provided “as is,” by Miercom and gives no warranty, representation or
undertaking, whether express or implied, and accepts no legal responsibility, whether
direct or indirect, for the accuracy, completeness, usefulness or suitability of any
information contained in this report.
No part of any document may be reproduced, in whole or in part, without the specific
written permission of Miercom or Websense All trademarks used in the document are
owned by their respective owners. You agree not to use any trademark in or as the
whole or part of your own trademarks in connection with any activities, products or
services which are not ours, or in a manner which may be confusing, misleading or
deceptive or in a manner that disparages us or our information, projects
or developments.