THE CASE FOR 25 AND 50 GIGABIT ETHERNET OVERVIEW The demand for cloud-based solutions has grown rapidly, leaving many datacenters hungry for bandwidth. This has created a demand for cost-effective solutions that can deliver extra bandwidth and fully utilize the high capacity of the new generations of CPUs being used in the servers in the data centers. The new CPUs can easily support data rates higher than the 10 Gigabit Ethernet, which for a long time has been used for connections in the datacenter environment. But upgrading to 40 Gigabit Ethernet comes with an extra price tag due to the capital expenditure and power consumption. This is why the industry has pushed to get cost effective 25 Gigabit Ethernet and 50 Gigabit Ethernet solutions standardized and ready for deployment. To evaluate and benchmark the performance of any new 25GE and 50GE equipment, data centers will need test equipment that support the new speeds. To meet this need, Xena has launched a new 5-speed test module – the Loki-100G-5S1P - which can generate traffic at 100GE/50GE /40GE/25GE/10GE. Being able to test all five speeds with just one test module represents a significant cost saving.
WHITE PAPER
WHITE PAPER
Datacenter
optimization
drives the need
for new Ethernet rates – 25GE and
50GE. Xena makes it easy to test the new speeds.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
Contents OVERVIEW ........................................................................................................................................ 1 Introduction ...................................................................................................................................... 3 Top-of-Rack Datacenter Design .................................................................................................... 4 25 Gigabit Ethernet (25GE) and 50 Gigabit Ethernet (50GE) Standardization ................................. 5 Protocol Stack ................................................................................................................................... 6 25GE and 50GE switches .................................................................................................................. 7 25GE and 50GE NICs ......................................................................................................................... 8 Available Electrical and Optical Modules.......................................................................................... 8 25GE and 50GE Test Requirements .................................................................................................. 9 Xena Networks 25GE and 50GE Test Solutions .............................................................................. 10 Unique Eye Diagram ................................................................................................................... 10 Testing Above Layer 1 ................................................................................................................ 11 Testing from 10 Gigabit/s to 100 Gigabit/s ................................................................................ 12 Conclusion ...................................................................................................................................... 13
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
The Case for 25 and 50 Gigabit Ethernet
For a number of years the upgrade path for higher speed Ethernet has been 10GE -> 40GE/100GE, which has been well accepted for many applications. However a couple of new speeds have recently been added between 10 Gigabit/s and 100 Gigabit/s: 25 Gigabit/s (25GE) and 50 Gigabit/s (50GE). As there an ongoing development of standards for Ethernet speeds higher than 100 Gigabit, it may seem a bit odd that lower speeds are not only discussed but in fact also standardized. The drive for 25GE and 50GE comes from datacenters, where multiple 10GE connections are used to interface to the servers in the datacenters. In high density datacenters numerous 10GE “Top of Rack” (ToR) switches are required to connect to the datacenter servers. Each switch and each switch port has a cost of its own. To this adds the cost of requirements for space, maintenance, power and cooling of the switches. By introducing 25GE single lane connections instead of the 10GE connections, the datacenters will get 2.5 times more capacity per connection, while equipment cost and space, power, maintenance and cooling requirements will be reduced compared with datacenters using only 10GE to provide the same bandwidth, simply because the datacenter with 25GE connections will be built with less equipment. In addition the CPUs in datacenter servers can now be so powerful that they easily can fill up a 10GE connection. To provide more transmission capacity the next step has until now been to upgrade to 40GE, which is considerably more expensive than a 10GE connection as 40GE is based on sending the data on 4 10 Gigabit/s lanes in parallel. As an alternative to 40GE an upgrade to 25GE will be a cost effective solution because fewer lanes are used. The general expectation in the industry is that 25GE deployment will grow rapidly in the coming years. Analysts forecast that by 2020 25GE and 50GE will together have a market share of approx. 25% in the high speed NIC market (from 10GE to 100GE) (based on data from Crehan Research). Using 25 Gigabit lanes for transport of Ethernet is not new: Most current 100GE implementations use four 25 Gigabit lanes to transport the Ethernet signals. Therefore the 25GE standardization bodies have standardized a 25 Gigabit lane based on the 100GE technology. The current standardization of 50GE is based on using two 25 Gigabit lanes. This will be an alternative to the current 40G implementations, which use four 10 Gigabit lanes to transport the Ethernet signal. The reduction of lanes will reduce the cost of the network equipment needed in the datacenter, which leads to reduced CAPEX and OPEX. In the future the Ethernet upgrade path may change to 10GE -> 25GE -> 50GE -> 100GE. In any case it will be easy and cost effective to upgrade datacenters from multiple 25GE lanes to 50GE or 100GE networks.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
INTRODUCTION
Top-of-Rack Datacenter Design A popular datacenter design is the ”Top-of-Rack” (ToR) design: Each server rack is equipped with one Ethernet switch (or two for redundancy). The switch could be placed in the top of the rack, which has given the design its name; the switch may however be placed elsewhere in the rack. Another design is the “End-of-Rack” design, where one Ethernet switch, placed outside the server racks, connects to all the servers in the datacenter. In both cases with the increased need for bandwidth you will either need more switch ports or more bandwidth per port. Going from 10 GE ports to 25 GE ports will provide the increased bandwidth in a more cost effectively compared with adding 10 GE ports.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
Figure 1: Current and future high speed Ethernet upgrade path
25 GIGABIT ETHERNET (25GE) AND 50 GIGABIT ETHERNET (50GE) STANDARDIZATION 25 Gigabit Ethernet (25GE) is a new standard for Ethernet connectivity in a datacenter environment, developed by the IEEE 802.3by 25 Gb/s Ethernet Task Force. The work was started in the IEEE standards body with a successful Call for Interest (CFI) in July, 2014. In the CFI the 25 Gigabit Ethernet Motivation was outlined: • •
Provide cost optimized server capability beyond 10G Provide a 25Gbit/s MAC rate that: o Leverages single-lane 25Gbit/s physical layer technology developed to support 100GE o Maximize efficiency of server to access switch interconnect
The IEEE 802.3by standard is based on technology defined for 100 Gigabit Ethernet implemented as four 25-Gbit/s lanes. As the main application of 25GE are connections inside a datacenter and inside a server rack in the data center the IEEE 802.3by standard focus on very short range connections. The IEEE 802.3by standard objectives (approved November 2014) included: • • • • •
Support a MAC data rate of 25 Gb/s Define a single‐lane 25 Gb/s PHY for operation over a printed circuit board backplane Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 3m Define a single-lane 25 Gb/s PHY for operation over links consistent with copper twin axial cables, with lengths up to at least 5m Define a single-lane 25 Gb/s PHY for operation over MMF consistent with IEEE P802.3bm Clause 95
The IEEE 802.3by standard was approved in June 2016.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
Figure 2: Top-of-Rack datacenter design
The following table is based on a table in the 25G Ethernet Consortium specification showing a summary of the two solutions: Port Mode
Lane Count
Speed per Lane (GHz)
PCS Lane Bonding
Autonegotiation
A. Single Lane 25G
1
25.78125
N/A
Yes
B. Two lane 50G
2
25.78125
MLD4 or RS FEC (see end of next section)
Yes
Table 1: 25G and 50G solutions summary
In 2016 IEEE initiated work on defining 50 Gigabit Ethernet by setting up the IEEE 802.3cd task force. The objective of the task force is however to define 50 Gigabit Ethernet over a single lane instead of the dual lane solution as outlined in the 25G Ethernet Consortium specification as a single lane technology is expected to be more cost effective than the dual lane technology. To achieve this it will be necessary to use a Pulse Amplitude Modulation technique PAM-4, where 4 different pulse amplitudes are used to transfer the information. The IEEE 802.3cd task force target is to complete the 50 Gigabit Ethernet single lane standard in the 2018-2019 time frame.
PROTOCOL STACK The Ethernet protocol stack is based on the seven-layer OSI model of computer networking: Layer 7
Application layer
Layer 6
Presentation layer
Layer 5
Session layer
Layer 4
Transport layer
Layer 3
Network layer
Layer 2
Data link layer
Layer 1 Physical layer Table 2: The OSI model of computer networking The 25G Ethernet Consortium specification for 25 Gigabit Ethernet (25GE) and 50 Gigabit Ethernet (50GE) addresses Layer 1, which contains the following sublayers:
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
In addition to the IEEE 802.3by 25 Gb/s Ethernet Task Force an industry consortium (the 25G Ethernet Consortium) was formed in July 2014 by Arista, Broadcom, Google, Mellanox and Microsoft to support the specification of single lane 25 Gigabit/s Ethernet and dual lane 50 Gigabit/s Ethernet technology. The 25G Ethernet Consortium draft specification was completed in September 2015.
PCS (Physical Coding Sublayer)
FEC (Forward Error Correction Sublayer)
PMA (Physical Medium Attachment Sublayer) PMD (Physical Medium Dependent Sublayer) Table 3: Layer 1 sublayers
Functions • auto-negotiation • determine a functional link has been established (Link Training) • coding (64b/66b encoding in the case of 25 Gigabit and 50 Gigabit Ethernet) • optional for 25 and 50 Gigabit Ethernet • the 25G Ethernet Consortium specification lists the following possible FECs: o BASE-R FEC “Fire code” (IEEE 802.3 Clause 74) o Reed Solomon FEC (IEEE 802.3 Clause 91) o No FEC In addition to the FECs above IEEE 802.3by includes Reed Solomon FEC (IEEE 802.3 Clause 108) auto- negotiation can be used to determine which FEC is used • framing • octet synchronization/detection • scrambling/descrambling • the transceiver for the physical medium
The 25G Ethernet Consortium specification in many cases refers to clauses in IEEE 802.3 for other rates to define the layer 1 functions for 25GE and 50GE. For 50GE a Multi-Lane Distribution (MLD) technique is required to distribute data into two lanes. When RS FEC is used for the 50GE solution MLD is handled by the interface between the PCS sublayer and the FEC sublayer. Otherwise MLD4, defined for 40G systems, is used as a part of the functionality in the PMA sublayer. In both cases the data from the upper layers are distributed into 4 “logical” (or virtual) 12.89063 Gigabit/s lanes, two of each sent on the physical 25.78125 Gigabit/s lanes. Alignment markers are inserted into the data stream to handle the multi-lane distribution, one unique alignment marker for each virtual lane.
25GE AND 50GE SWITCHES Several vendors offer switches with 25GE and 50GE capabilities, typically implemented as very flexible solutions supporting 100GE, 50GE, 40GE, 25GE and in most cases also 10GE from one port. Some examples are:
Product Arista 7260CX-64 Broadcom StrataXGS® Tomahawk BCM56960 Series Broadcom StrataXGS® Tomahawk II BCM56970 Series Cavium XPliant® CNX88091 Cisco Nexus 3232C Mellanox SN2700 Table 4: Switches supporting 25GE and 50GE
Maximum number of ports
100GE 64 32 64 32 32 32
50GE 128 64 128 64 64 64
40GE 64 64 128 32 32 32
25GE 256 128 256 128 128 64
WHITE PAPER
Sublayer
10GE 258 N/A N/A 128 128 64
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
A Network Interface Controller (NIC) (or network interface card, network adapter) connects a computer (e.g. servers in a datacenter) to a network. NIC products supporting 25GE and 50GE include: • • •
Broadcom NetXtreme® C-Series: Dual port 10/25G Ethernet Mellanox ConnectX® -4 Lx EN 10/25/40/50 Gigabit Ethernet Adapter: Single and dual port versions QLogic® FastLinQ™ 45000 Series Intelligent Ethernet Adapters: will include adapters that support 10GE, 25GE, 40GE, 50GE, 100GE
AVAILABLE ELECTRICAL AND OPTICAL MODULES For deployment of 25GE vendors like Finisar, Avago and Innolight offer SFP28 modules, which support one lane with bandwidth up to 28 Gigabit/s. SFP28 modules have the same form factor as SFP and SFP+ modules. Available modules include: •
•
25 Gigabit/s copper o Direct Attach Cables Cobber twinax cables (DACs) terminated with SFP28 modules in each end, 0.5 to 5 meters in length. 25 Gigabit/s fiber optics o 850 nm SR supporting distances up to 100 meters on OM4 multimode cable or distances up to 70 m OM3 multimode cable o 1310 nm LR supporting single mode fiber and distances from 5 to 20 km
Another module that will be relevant for 25GE is QSFP28, which supports four 28 Gigabit/s lanes in the QSFP+ form factor. The QSFP28 module was originally designed to support 100 Gigabit/s connections. However for datacenters with 25GE connections, direct attach cobber twinax cables (DACs) connecting a QSFP28 100GbE port to four independent 25GE SFP28 connections are available. The DACs will have a QSFP28 module in one end breaking out to four cables, each terminated with a SFP28 module. This means high capacity Ethernet switch (ToR) with 100 Gigabit/s ports can interface to 4 servers with 25 Gigabit/s ports through one 100 Gigabit/s port in the ToR. For optical interfaces 100 Gigabit/s QSFP28 to 4 x 25Gigabit/s SFP28 breakout Active Optical Cables (AOC) are available. In addition passive optical breakout cables are available splitting four 25 Gigabit/s optical signals from a QSFP28 optical module with MPO connector into four individual cable pairs to be connected to SFP28 optical modules with LC connectors.
Figure 3: 100 GE to 4*25 GE break-out cable
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
25GE AND 50GE NICS
Figure 5: 100 Gigabit/s QSFP28 to 4 x 25 Gigabit/s SFP28 break-out Active Optical Cable (AOC)
Figure 6: MTP to 4 dual LC passive optical break-out cable
25GE AND 50GE TEST REQUIREMENTS Testing Ethernet systems is required to ensure the proper functioning and quality of the systems. This of course also applies to the new 25GE and 50GE solutions as they are developed and deployed. To test the layer 1 functions defined in the 25G Ethernet Consortium specification verification of functions like auto-negotiation, Link Training and FEC must be performed. In addition an eye diagram of the signal should be checked to identify physical layer issues, which may be caused e.g. by noise and signals that are too high, too low, too long or too short.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
Figure 4: 100 Gigabit/s QSFP28 to 4 x 25 Gigabit/s SFP28 DAC electrical breakout cable
XENA NETWORKS 25GE AND 50GE TEST SOLUTIONS The Xena Loki-100G-5S-1P 5-speed dual-media test module provides powerful testing capabilities to test and verify the layer 1 functions defined for 25GE and 50GE.
Figure 7: The powerful and versatile Xena Loki-100G-5S-1P 5-speed dual-media test module You can set the Loki-100G-5S-1P to perform auto-negotiation so when connected to a Device Under Test (DUT) the Loki-100G-5S-1P can verify that the DUT reacts correctly to the autonegotiation process. The Loki-100G-5S-1P will also do Link Training with the DUT, verifying the correct functioning of this feature. Signals can be configured with or without FEC, verifying that the DUT use the correct FEC. Loki-100G-5S-1P supports the FECs specified by IEEE 802.3by and
by the 25G Ethernet Consortium: • • •
BASE-R FEC “Fire code” (IEEE 802.3 Clause 74) Reed Solomon FEC (IEEE 802.3 Clause 91) Reed Solomon FEC (IEEE 802.3 Clause 108)
Unique Eye Diagram The Loki-100G-5S-1P includes the “eye diagram”, a unique feature for analyzing signal quality. For this the Loki-100G-5S-1P controls and monitors the link at the physical level and creates bit-errorrate (BER) eye diagrams, estimates the link BER from the vertical and horizontal BER bathtub curves and controls the PHY tuning in the transmit and the receive directions. The eye-data table provides an estimate of several parameters of the eye, including width, height and jitter.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
As ToR switches typically are implemented as very flexible products that support 100GE, 50GE, 40GE, 25GE and 10GE from one port it is of course also important to test Ethernet at all rates supported by the port under is test.
Testing Above Layer 1 Ethernet testing with the Loki-100G-5S-1P is of course not limited to the layer 1 tests described in the previous sections. Advanced layer 2 and layer 3 test scenarios can be performed using the free test applications for the Loki-100G-5S-1P: •
XenaManager-2G test software is used to configure and generate streams of Ethernet traffic between Xena test equipment and Devices Under Test (DUTs) and analyze the results.
•
Xena2544 offers full support for the 4 test-types specified in RFC2544 o Throughput o Latency o Frame loss o Back-to-back frames o Jitter (Frame Delay Variation) is also supported
•
Xena3918 makes it easy to create, edit and execute all test-types specified in RFC 3918. RFC3918 describes tests for measuring and reporting the throughput, forwarding, latency and Internet Group Management Protocol (IGMP) group membership characteristics of devices that support IP multicast protocols.
•
Xena2889 is an application for benchmarking the performance of Layer 2 LAN switches. The following RFC 2889 test types are supported:
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
Figure 8: The unique eye diagram you get with Xena 100G test modules
•
All Throughput and Forwarding rate tests (both Fully and Partially meshed) Congestion Control Address Caching Capacity Address Learning Rate Broadcast Frame Forwarding and Latency Forward Pressure and Maximum Forwarding Rate
Xena1564 provides full support for both the configuration and performance test types described in Y.1564 for complete validation of Ethernet service-level agreements (SLAs) in a single test
Another free application for the Loki-100G-5S-1P (and all other Xena test modules): •
XenaScripting - a powerful and easy-to-use command-line-interface (CLI) scripting API that makes test automation easier for test engineers.
Testing from 10 Gigabit/s to 100 Gigabit/s
Figure 9: Loki-100G-5S-1P can test four 25GE ports through a 100 Gigabit/s QSFP28 transceiver and a break out cable terminated with four SFP28 transceivers or two 25GE ports connected directly to 25 Gigabit/s transceivers in the Loki-100G-5S-1P In addition 25GE and 50GE testing Loki-100G-5S-1P also supports test of 100GE, 40GE and 10GE, so both the new and the existing Ethernet rates in the range from 10 Gigabit/s to 100 Gigabit/s can be tested. This is needed to test and verify 25GE and 50 GE capable switch ports. The unique test module lets users dynamically choose between two different physical transceiver cages and
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
o o o o o o
The unique combination of five different Ethernet network speeds and multiple physical optical transceiver form factors makes the Loki-100G-5S-1P a versatile solution for performance and functional testing of network infrastructure and Ethernet equipment such as taps, switches, routers, NICs, packet-brokers, and backhaul platforms.
CONCLUSION New Ethernet data rates – 25GE and 50GE - are coming, driven by the need for cost effective expansion of datacenter capacity to fulfil the huge increase in bandwidth requirements from cloud based solutions. Therefore the deployment of 25GE and 50GE is expected to grow rapidly in the coming years. Datacenter switches with 25GE and 50 GE capabilities are typically implemented as very flexible solutions that support 100GE, 50GE, 40GE, 25GE and 10GE from one port. To test the product all rates need to be tested. With the Xena Loki-100G-5S-1P 5-speed dual-media test module the user gets powerful and versatile Ethernet test capabilities up to layer 3 to test and verify both new and existing Ethernet rates in the range from 10 Gigabit/s to 100 Gigabit/s: 10GE, 25GE, 40GE, 50GE and 100GE.
Xena Networks – Global Price/Performance Leaders in Gigabit Ethernet Testing – www.xenanetworks.com
WHITE PAPER
form factors. The first is a single QSFP28/QSFP+ transceiver cage, and the second is two SFP28/SFP+ transceiver cages.
