VXLAN versus GENEVE (NSX-V vs. NSX-T)

 

August 14th, 2021

Image-Network-Graphic

With the ramp-up of NSX-T overlay networks and transition away from NSX-V overlay networks, it's a good time to look at one of the fundamental differences between them. NSX-V uses VXLAN as its encapsulation protocol while NSX-T uses the more recent GENEVE encapsulation protocol. Each require the physical networking devices have their MTU adjusted to 1600 bytes or greater. We'll take a detailed look at why that is. First, the basics:

VXLAN is:


GENEVE is:


From the prospective of the physical network, an overlay network is essentially an application. NSX-T is an application using well known UDP port 6081. Switching perspectives to the overlay's view of the physical network, you can think of the physical network performing the same role as the switch backplane in a single physical switch. In a physical switch, packets go into the backplane, some logic is applied, and the packet comes out of the correct port. For an overlay, packets go into the overlay, some logic is applied, packets come out of the overlay at the correct location. With NSX-T, your complete datacenter network effectively functions as a single switch backplane! 


Before we dive into overlay packets and MTU, lets look at how MTU impacts traditional 1500 byte network packets:

TCP Packet
TCP Packet

20 + 20 + 1460 = 1500


UDP Packet
UDP Packet

20 + 8 + 1472 = 1500

MTU is measured in bytes and defines the number of bytes that can be carried in an Ethernet frame. When measuring MTU, the Ethernet header and Frame Check Sequence are not included in the calculation. This is why if you packet capture an Ethernet frame which contains the standard limit of 1500 bytes, the actual packet will have 1518 bytes. The Ethernet frame is 14 bytes and the FSC is 4 bytes. 

Now let's see why the NSX overlay requires an MTU of 1600 bytes or greater in the physical network:

VXLAN vs GENEVE Packet

(click image to see a more detailed view)

Both encapsulation protocols add an outer Ethernet header, IP header, UDP header and VXLAN/GENEVE header. The original Ethernet frame is maintained, but this time since its inside of another Ethernet frame, it is included in the outer frame's MTU calculation. Refer to the image for the math bits. You will notice that VXLAN and GENEVE encapsulated packets are larger than 1500 bytes. They will be dropped by unmodified networking devices. VMware recommends 1600 byte MTU as a safe round number to apply to support NSX. Technically, you could set your MTU to 1560 and GENEVE encapsulation would still be forwarded with no problem. However, you should stick to 1600 MTU or jumbo 9000 MTU frames if your network devices can handle that.  

VXLAN Packet In Wireshark (zero byte ICMP payload)GENEVE Packet

GENEVE Packet In Wireshark (zero byte ICMP payload)VXLAN Packet

Putting this into the context of a real world scenario, when would this information matter? Let's consider an example from one of my customer interactions. Oracle's RAC (Real Application Cluster) requires a network to support 8922 bytes of payload per packet. Add to that 20 bytes of IP header and 8 bytes of UDP header for 8950 bytes of data and overhead prior to VXLAN encapsulation. VXLAN will add 50 bytes of overhead creating a 9000 byte packet. Even with jumbo frames, this application is sending the largest possible frame that most physical networks can forward. Now migrate your application to NSX-T! Swap the 8 byte VXLAN header for 16 byte GENEVE header and what do you get? You get a broken application attempting to send a 9008 byte packet! In the case of my customer, we decided to lower the VM's interface MTU by 8 bytes. Fortunately we only had to change a handful of interface MTU settings!

The GENEVE and VXLAN protocols are often dismissed and not given much consideration until something in the datacenter breaks. This post is one that I wish I had when confronted overlay packet size problems in the datacenter. Hopefully this helps someone master their datacenter network with the sophistication modern networks require!


Oracle RAC:

https://support.oracle.com/knowledge/Oracle%20Cloud/341788_1.html





Comments

Post a Comment

Popular posts from this blog

"Twice NAT" with NSX-T T0 Gateway

Image
  Network address translation, or more commonly NAT, is most often used to do source NAT or destination NAT. In rare instances, SNAT or DNAT isn't enough to get ip packets to their destination. Enter twice nat. While twice NAT isn't a function I've needed often, it was a function drilled into me while studying for various Cisco exams. I used to watch Christian Matei explain it over at INE and it really helped me to understand what twice NAT is and how to configure it on Cisco devices. Previous to my time working with NSX-T, I'd only utilized twice NAT once. A few years back, a customer wanted to build and IPsec tunnel from the organization HQ to a branch office. The private ip space used by HQ and the branch used overlapping ip space. We used twice NAT to NAT the source and destination ip address in a single NAT rule ( similar to this guy ) before shoving it all into an IPsec tunnel. Previous to working with NSX, circumstances requiring twice NAT were uncommon.  While w...
Read more

Packet Capture Network Traffic Inside ESXi Hypervisor

Image
The three components of computer infrastructure are considered compute, storage, and networking. When applications don't work properly, its often the case that networking is the first component to receive blame for failed application performance. In the case of virtualized workloads (i.e. applications running on virtual machines), we are fortunate to have a powerful troubleshooting tool which allows us to packet capture network traffic in an ESXi hypervisor. For this blog, I'm going to use an Hands On Lab (HOL) from the VMware website to demonstrate the process of doing a quick live packet capture. The output of the packet capture will be sent to the shell of the hypervisor. There are ways to sent the network traffic to a file so it can be extracted and analyzed by a packet analyzer tool, but that will be for another post. First, lets start with heading over to the HOL website:  https://labs.hol.vmware.com/ For this lab, I'm going to type NSX in the search bar to find...
Read more