PERFORMANCE VERIFICATION OF NEXT-GENERATION FIREWALLS IN POST-PRODUCTION ENVIRONMENTS
OVERVIEW Regular software upgrades for firewalls deployed in the production network are essential to the success of enterprise information and data security. With advanced features and functionalities, next-generation firewalls (NGFWs) are able to perform data inspection, application identification, content identification, user recognition, and treat traffic accordingly based on the rule and policy set defined by the security manager. NGFW performance degradation vs. advanced features enabled has become a vital and sensitive issue for enterprises. Each software upgrade released by NGFW vendors includes new features and functions, bug fixes, improvements, etc. Before security engineers upgrade the deployed NGFWs in the network, intensive software performance testing is required. The firewall software should be put to test under both modeled traffic as well as realistic application traffic in order to verify the specific performance of the rule and policy sets used by the enterprise as well as the general performance. Xena provides a great variety of test solutions for such NGFW performance
WHITE PAPER
WHITE PAPER
“Using realistic
application traffic
mixes to verify the performance and
behavior of NGFWs
after each software
upgrade is essential in post-production environments.”
verification for post-production environments. TCP tweaking helps engineers to test firewall failover latency. With XenaAppMix, you are able to carry out application traffic test, protocol traffic test, traffic profile test, and customize mixes for specific
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
Contents Introduction ......................................................................................................................... 3 Why NGFWs are Widely Deployed to Replace Traditional Firewalls?................................. 4 What Can NGFWs Do? ......................................................................................................... 5 NGFW Can Become the Performance Bottleneck ............................................................... 7 Important to Verify Performance after Software Update ................................................... 7 Use XenaAppMix and Capture-Replay for Thorough Performance Verification ................. 9 Conclusion ......................................................................................................................... 13
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
PERFORMANCE VERIFICATION OF NEXT-GENERATION FIREWALLS (NGFWs) IN POST-PRODUCTION ENVIRONMENTS
Many enterprises rely on firewalls to protect their network and data, as shown in Figure 1. Firewall deployment will usually go through a number of phases, including vendor selection, performance verification, comparison, training, device configuration, pre-production testing, evaluation, production deployment, etc. After being deployed, the firewall will examine the traffic coming in and out the protected networks in order to secure the enterprise data according to the configured rules and policies. In order for firewalls to protect enterprise networks and information effectively, firewall equipment vendors often release software upgrades to introduce new functions and features, bug fixes, system performance improvements, etc. Thus, it is vital to validate and verify the system and its performance after each software upgrade, because of the great importance of firewalls to the network. Since network security devices such as firewalls are located inline of the network, which means that both signaling traffic and data traffic need to pass the firewall for security examination, traffic performance bottleneck is often one of the top concerns of the enterprise. A network may achieve 10 Gigabits per second of throughput without a firewall and could endure a reduction to only half, or even less, of the original throughput with advanced features and functions enabled. Since firewalls are stateful devices, there is always a trade-off between performance and functionality. Different from a stateless device such as a switch, a stateful device requires more participation because it retains certain information extracting from packets in order to decide whether the traffic should pass or be blocked. With more features enabled, more CPU and system resources will be allocated to process the packets, consuming more time and causing throughput reduction. With the same set of security rules unchanged, a software upgrade may also bring down the performance of the security
DMZ
device due to the new feature introduced. Additionally, the reliability of the upgrade is another concern to most of the IT departments.
Untrusted LAN
Due to the reasons mentioned above, enterprises always
Firewall
Figure 1. Firewall protects trusted networks from the untrusted.
verify the performance of the
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
INTRODUCTION
change will take effect in the real production environment, potentially affecting users and data security with millions of dollars value.
WHY NGFWS ARE WIDELY DEPLOYED TO REPLACE TRADITIONAL FIREWALLS? Enterprises deploy firewalls in their networks to consolidate information and network security. As networking attacks are becoming sophisticated, application-layer oriented, and more targeted and focused, traditional firewalls that rely on simple port filtering are largely replaced by more advanced next-generation firewalls (NGFWs) in enterprises. Distributed Denial-of-Service (DDoS) attacks are in constant growth, both in attack duration and in attack volume, because it is easy to make and requires little hacking or networking skills. In addition to the network itself, websites are also being hammered, which could affect many enterprises. Attacks on websites and servers are application-layer oriented, which means that these attacks hide themselves inside HTTP/HTTPS traffic to gain unauthorized access to data and files, such as SQL injection (SQLi) and Local File Inclusion (LFI). The philosophy of these application-layer attacks are quite different from DDoS in that they do not intend to bring the server down but browse thousands of information in stealth mode without the host knowing. This is why there are so many attacks over HTTPS, a secure HTTP communication. The reason is simple. HTTPS encrypts the traffic exchanged between a client and the server, but there is no guarantee that the intention of the traffic is conforming. Malicious traffic, as shown in Figure 2, that intends to gain unauthorized access to certain files are also encrypted and protected by the protocol. Due to this shift of attack methods, NGFW and other advanced network security devices are widely deployed in enterprise networks due to the content-aware and application-aware capability of these devices.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
firewall every time a new upgrade is installed. The process is usually done carefully because any
WHAT CAN NGFWS DO? NGFW began to emerge in 2009. Implemented differently by vendors, NGFWs now have the following major features that protect the enterprise network and information from attacks and malicious traffic: •
High-Performance
•
Content Identification
•
Application Identification
•
User Identification
•
Policy Control
Since a NGFW device is located inline in the network, performance is one of the baseline metrics enterprises are concerning about. NGFWs are expected to provide real-time analysis and protection with no performance degradation, i.e. low latency, high throughput and no packet loss. Malicious traffic does not come to the enterprise alone. On the contrary, they always hide themselves in the benign traffic. With the resource-intensive security-related features (such as SSL/TLS offloading, malware signature scanning, file filtering, application identification, etc.) to process the tremendous traffic volume that constantly blasting today’s security infrastructure, NGFW should deliver high performance under heavy loads so that QoS can be guaranteed, especially for those latency-sensitive applications. Due to the reason that malicious traffic could potentially be protected by the encryption technology, such as SSL/TLS, it is necessary for NGFWs to examine the traffic and perform threat prevention, URL filtering, file and data filtering. Closely related to application identification,
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
Figure 2. Malicious traffic can hide in the encrypted tunnel threating servers.
data filtering. The threat prevention component prevents malware and exploits from infecting the enterprise network, regardless of what application traffic the content is carried on, e.g. email or HTTP. This involves several functions such as application decoding, malware scanning, threat signature matching, zero-day protection, and cloud-based intelligence. URL filtering allows IT administrators to block or allow specific URLs/websites to different users accordingly based on the information provided by user identification. File and data filtering leverages the technologies of in-depth application inspection and reduces the risk of malware propagation. It is also vital for NGFW to identify applications regardless of protocol, port, evasive techniques, or SSL encryption. Unlike traditional firewalls where applications are identified based on the port and protocol pair, the NGFW should identify applications regardless of protocol, port, evasive techniques, or SSL encryption used by the application. NGFW should detect the application protocol, for instance HTTP, IMAP, SMTP, DNS, XML-RPC, SSL/TLS, etc. If SSL/TLS is in use, the NGFW should offload the session so that the content inside the secure tunnel can be inspected. Traffic should be re-encrypted after the inspection in order to be transparent on the client-server path. NGFW should also decode the application and detect signatures. It is very likely that some XML-RPC malicious payload are hidden in the HTTP payload, and can cause server down if not decoded. In addition to content and application identification, NGFWs are expected to accurately identify users and user identity information as an attribute for policy control. Usually, there are many users in an enterprise in terms of network and IT services. Users are either located centrally in a building and rely on LAN to exchange information or distributed among several locations and use VPN techniques to work remotely. User identification controls network activity on a per-user basis, which is supported by integration with LDAP directories. This feature allows the IT department to gain more control over how users use their applications. With the ability to the identify traffic content on the network, NGFWs provide visibility and granular, policy-based control over applications, including individual application functions. Policy control occurs after a full inspection has been performed to the traffic. Given a complete picture of what application the traffic is using, who is using it, and what the purpose of the traffic is, NGFW applies the pre-defined policies to the traffic rather than simply block or allow the traffic like traditional firewalls. Policy control options are for instance, allow/deny, allow based on users or groups, allow but scan for exploits, viruses and threats, decrypt and inspect, apply traffic shaping, apply policy-based forwarding, allow/deny certain types of file transfer, etc.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
content identification examines the traffic and perform threat prevention, URL filtering, file and
Although NGFWs are much more advanced than traditional port filtering firewalls, enterprises usually do not enable all the features or do not examine all traffic passing through the firewall. This is simply because of the performance issue. With advanced security features, a NGFW device requires more processing complexity than a traditional firewall. This is obvious since the NGFW can do much more inspection work not only on per packet basis but also on individual application session. This may seem trivial to a few hundred packets but when the firewall is deployed in an enterprise network where millions of sessions traverse, the bottleneck effect will become an issue. The IT security administrator often configure a set of rules and policies, which define which traffic should be deep-inspected, from which IP addresses the traffic should be handled with maximum security level, etc. It is unlikely to see that all the traffic is selected for close observation and examination. This makes sense since each company has its own information security policies and concerns and the network administrator is usually responsible to provide enough throughput/bandwidth to the users.
IMPORTANT TO VERIFY PERFORMANCE AFTER SOFTWARE UPDATE Firewall rules and policies are configured and defined to ensure the data and information security of the enterprise. At the same time, as mentioned previously, the performance of the protected network should be adequate to support the services running over the network. From time to time, firewall vendors release software upgrades to introduce new functions and features, bug fixes, system performance improvements, etc. Even if the defined rules and policies are not changed by the IT administrator, performance or system reliability may vary due to the upgrade. Thus, for each enterprise it is crucial to verify how well the upgrade performs and how stable the system is before installing the software upgrade in their production network. Testing and verification of firewall performance and reliability is often taken place in the labs, a controlled environment where engineers carry out various sorts of test plans over days or even months. The test object in the lab is usually the software because it can be costly to uninstall a working firewall from the production network and test the hardware every time there is a need for software upgrade.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
NGFW CAN BECOME THE PERFORMANCE BOTTLENECK
use traffic generator to test the new software of the NGFW, as illustrated in Figure 3. Among all the requirements, the engineers must verify the performance of the NGFW software, including metrics such as maximum concurrent TCP sessions, maximum TCP connections per second, maximum concurrent HTTP sessions, maximum HTTP connections per second, etc. The NGFW software should not only be under the test of simulated traffic where engineers define the load profile and use dummy data as payload, but also be tested with realistic traffic, including PCAP replay and application emulation. This is because NGFWs are able to recognize modelling traffic and start generating fixed process mechanisms to handle the traffic, which will compromise the test purpose. In order to maintain a high degree of traffic dynamics, engineers should either use pre-capture traffic (PCAP files) and replay it onto the NGFW or use application emulation where a pre-define application library is offered. By using these two kinds of traffic, engineers can have a better view of how this software upgrade reacts to realistic traffic and handles various applications.
Performance verification of new software upgrades
DMZ
LAN
Installation of tested upgrades
sync
Master
Redundancy
Slave
Untrusted Untrusted Untrusted
Figure 3. Lab testing the NGFW software before installing in the production network. It is common for a network to have redundant firewalls in case of failure or maintenance, as shown in Figure 3. During the installation of the new software release, there will be a maintenance window. Most enterprises, if the upgrade process is similar as the above, will carry out the upgrade when the traffic load on the network is at its minimum, for maintenance should in no circumstance affect the normal operation of services. While the master NGFW equipment if
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
To compare the results, engineers will apply the same firewall rules and policies in the lab and
NGFW in order to ensure a functioning network. For any redundant firewall to take over the traffic as quickly as possible before the running applications and services are disrupted, it is necessary for the master and slave firewalls to sync information such as keep-alive, CPU load, system status, etc. The latency between the traffic switching should be as low as possible. Otherwise, some TCP sessions with small timeout values may time out before the network is converged back to its original state. This should also be put on the list of lab testing, where engineers configure different TCP and HTTP sessions in order to test the latency tolerance. For each firewall upgrade, one should be prepared to see performance variation from version to version, as a simple example shows in Figure 4. For the same rule and policy set, different versions may generate different performance results. Test engineers may need to adjust how they define and configure the rules and policies in order to regain the lost throughput from the NGFW. Thus, thorough tests using various traffic profile mixes, which cover as many scenarios as
DUT Performance
possible, become vital for all enterprises.
Version 1.0 Version 1.4 Version 2.1
Set 1
Set 2
Set 3
Rules and Policies Enabled
Figure 4. Performance variation vs. different NGFW software upgrades.
USE XENAAPPMIX AND CAPTURE-REPLAY FOR THOROUGH PERFORMANCE VERIFICATION It is important to verify your firewall with performance traffic, i.e. millions of TCP connections, UDP flows, and HTTP sessions, etc. RFC 3511 has defined a set of procedure to verify the firewall performance. However, the standard was published in 2003 and it is not enough for NGFW testing. Since NGFW protects network up to the application layer, it is significant to see how the
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
undergoing the software upgrade, the network should route the traffic onto the backup/slave
traffic contains much more information than you think. NGFWs and other advanced network security devices can dynamically detect traffic patterns related to the use of a particular application instead of just detecting the service and protocol used. Inspecting the payload of the traffic is not port-dependent and applications can be detected even on non-standard ports. Application identification first identifies the protocol and then compares the traffic to a protocol-specific context that matches the pattern to identify the applications. Take a simple HTTP request and response shown in Figure 5 for example. When a user browses a web page, the browser sends HTTP requests asking for content from the web server. The time interval between the HTTP requests is highly correlated with the application. NGFW will notice this correlation and know the traffic is not from a bot or a malicious software. This is a very simple example and yet it shows that the traffic pattern is very difficult to model and simulate and thus using realistic application traffic for testing becomes extremely important for those who take network and information security issues seriously.
Suspicious Suspicious
Figure 5. Application traffic pattern detected and compared XenaAppMix is a powerful yet easy-to-use solution for users to submerge their firewall in a mix of realistic application traffic for performance test. With an up-to-date application library, XenaAppMix constantly brings new applications, protocols, and traffic mixes onto the palette to generate stateful traffic with customizable load percentage for various test scenarios and needs, as illustrated in Figure 6.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
firewall behaves when it handles realistic traffic. Why? The reason is simple: realistic application
Application-Oriented Library
Traffic Mixes
Figure 6. XenaAppMix provides protocol and application traffic library as well as customizable traffic mix templates for application emulation. NGFWs from different vendors can have large performance difference when tested with various realistic application mixes. Evaluating NGFW only from the datasheet is far from enough or thorough. Additionally, regular software update to the firewall requires retesting before put into the production network. Testing the firewall with XenaAppMix to verify how the upgrade reacts to realistic application traffic provides better foresight. Testing with different traffic mixes as shown in the examples below can provide an in-depth understanding how the NGFW device performs under different network environments: •
Enterprise Mix
•
Datacenter Mix
•
Finance Mix
•
Web Mix
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
Protocol-Oriented Library
Datacenter
Finance
Web
Figure 7. XenaAppMix offers traffic mix templates of different network environments. Thousands of applications, if not millions, are being used every day by enterprises, financial institutions, governments, school, etc. It is nearly impossible for to include everything into a predefined library. Thus, Xena provides a flexible capture-and-replay feature for users to fill up the gap in order to make sure that the network is comprehensively tested and ready to defend against potential attacks. Capturing real-world application traffic is not difficult. There are many tools can do the job and generate industry the standard file, a PCAP file. One PCAP file is a snapshot of the network status of a certain period and contains plenty of sessions, as shown in Figure 8. Users can combine several PCAP files to create a customized test scenario, or
PCAP
generate millions of sessions with one PCAP file to test the performance of the firewall. With this great flexibility, unique applications and protocols can
Figure 8. One PCAP file is a snapshot of the network status containing many application sessions.
be captured and then replayed onto the network for stability testing.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
Enterprise
NGFW vendors from time to time release software upgrades, which include performance improvement, new features and functions, bug fixes, etc. Enterprises upgrade their NGFW software in order to consolidate data and information security and protect the network from cyber threats. Different from deploying new NGFW into pre-production networks, when the firewalls are in fact up and running in the production networks, any upgrade should be handled carefully. The software must be thoroughly tested with not only modeled traffic but also realistic application traffic in order to check whether the performance degrades with the same set of rules and policies, whether the system is as stable as before, etc. This verification process could take up to weeks or even months before the enterprise has enough confidence to upgrade their NGFWs in the production environment. Issues such as maintenance windows, fallback latency, and adjustment of NGFW configurations should be considered by the engineers. With XenaAppMix and the powerful L4-7 testing platforms, Xena offers effective and convincing test solutions for NGFW performance verification in a post-production environment for enterprises. The solutions cover from modeled traffic with extreme performance stress testing to rich library for application emulation and capture-replay. Engineers are free to capture realistic traffic and replay to test the NGFW software, as well as use different traffic mixes to inspect the performance fluctuation. For measuring and verifying the performance of NGFW, Xena has particular strengths that facilitate configuration and testing: •
Client and server emulation of application protocols.
•
Customizable pre-defined application traffic mixes for various network environments (such as enterprise, SMBs, financial institutes, and web).
•
Scalable replay of user captured network traffic with great flexibility.
•
True stateful TCP with extreme performance.
•
Customizable TCP traffic pattern and payload for performance test (such as TCP capacity test, CPS test, and HTTP TPS test).
•
Configurable TCP and IP parameters for performance verification.
•
Flexible load test and load profile creation.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
CONCLUSION
