NSX - Distributed Logical Router Packet Walk Lab / Tutorial

This lab / tutorial will show you the path of an IP packet that is routed through an NSX distributed logical router (DLR). This is often one of the most difficult components to conceptualize, particularly for network engineers who have experience with traditional routing using physical routers. My hope is that by going through this lab, it will help clarify the ways in which a DLR is the same as traditional routing, and the ways in which it is different from traditional routing in physical routing devices. Let's get started!

Step 1: Head over to http://labs.hol.vmware.com/ and type nsx in the search bar. Click to enroll in the lab below.

Use this HOL:

Step 2: Run powershell script 3 & 4. This removes some Distributed Firewall (DFW) configuration that exists to show users how to manage the DFW. Here we are not concerned about the DFW:

Step 3: Open the browser. It should load the vSphere web client to the login screen. login with the
following credentials:
administrator@corp.local
VMware1!

This is a diagram of the virtual topology we are working with:

Step 4:
We need to identify 2 VMs that are on distinct logical switches and connected to the same DLR. The first VM I'll use is web-01a_corp.local and the second is hr-db-01a_corp.local. You'll need to open a web console to both of the VMs to collect some information. We also want to validate that the two VMs can communicate before we set up a continuous ping for packet capturing. The login credentials for the VM web console is as follows:
username: root
password: VMware1!

Here are some tables showing the data I've collected in my lab. You'll need to do something like this so you can identify the packets during the packet capture phase. I've already collected this data in my lab. You'll want to do the same in you lab since I cannot guarantee that your lab will have the same mac addressing, nor can I guarantee that your lab will have VMs on the same esxi hosts.

Step 4a:
To get to the ESXi Host shell, use Putty:

In order to identify some of the information needed for the packet walk / packet capture, you'll need to know the following commands:
-At an ESXi shell prompt, type this to view what dvSwitch port number the VM's vNic is connected to: net-stats -l

-To find the DLR dataplane interface (vdr port): net-vdr -C -l

-Lastly, use this command to find out what physical nic is supporting the virtual interfaces. This nic will be used when ip traffic needs to leave the host: esxtop ; push "n" to view network view; push "q" to quit from that view

Step 5:
It may seem tedious, but we want to build the packet capture commands in a notepad here. You should be able to paste these commands directly from your computer into the lab (ctr + c in the lab should bring up a text box asking if you want to paste into the lab). Use my commands here as a template:

Packet Capture Commands for VM1's vNic: (top command captures transmitted packets, bottom captures received packets. You can only do one command at a time)
--------------------------------------
pktcap-uw --switchport 50331661 --capture VnicTx -o - |tcpdump-uw -envvvr - | grep ICMP

pktcap-uw --switchport 50331661 --capture VnicRx -o - |tcpdump-uw -envvvr - | grep ICMP



Packet Capture Commands for VM2's vNic:
--------------------------------------
pktcap-uw --switchport 33554444 --capture VnicTx -o - |tcpdump-uw -envvvr - | grep ICMP

pktcap-uw --switchport 33554444 --capture VnicRx -o - |tcpdump-uw -envvvr - | grep ICMP




Capture packets at dvSwitch port connected to vdr port of the DLR (esx-01a)
---------------------------------------------------------------------------
#Packet received from DVS, VM1 on vni 5003, mac address destined to vdr port
packet-uw --switchport 50331658 --vni 5003 --capture PortOutput -o - |tcpdump-uw -envvvr - | grep ICMP


#Packet received from DVS, packet has bee routed from vni 5003 to 5001, destination mac is VM2's mac
packet-uw --switchport 50331658 --vni 5001 --capture PortInput -o - |tcpdump-uw -envvvr - | grep ICMP


Capture packets at dvSwitch port connected to vdr port of the DLR (esx-03a)
---------------------------------------------------------------------------
#Packet received from DVS, VM1 on vni 5001, mac address destined to vdr port
packet-uw --switchport 50331658 --vni 5001 --capture PortOutput -o - |tcpdump-uw -envvvr - | grep ICMP


#Packet received from DVS, packet has bee routed from vni 5001 to 5003, destination mac is VM2's mac
packet-uw --switchport 50331658 --vni 5003 --capture PortInput -o - |tcpdump-uw -envvvr - | grep ICMP



Step 6:
Now for the actual packet capturing. Use the web console on VM 1 to start a continuous ping to VM2:

The highlighted parts show what we are interested in. VM1 is sending ICMP Echo request. The red highlight shows that VM1 is sending other traffic that we don't care about. This is why we made the table earlier so we can distinguish the traffic we want from the traffic we don't want. To stop a packet capture, push ctrl + c.

The packet has been routed by the DLR now and is headed toward ESXi host 3 and subsequently, VM2:

At this point the packet will be logically switched. To unpack that at bit, it will be encapsulated in a vxlan packet and tagged with vni 5001, then sent to host ESXi03. Those details are more in line with a post discussing logical switching, so i'm not going to cover that here. It can be its own rabbit hole. I want to keep the topic here on logical routing. All you need to know is that routing has completed and the packet will now go to VM2. To see it received, you will packet capture at the vNic of VM2.


Now, we will look at the route occur as VM2 sends the ICMP reply:

And now the packet has been routed and is being sent back to ESXi host 1 and VM1:

So there you have it. DLR routing occurs in the ESXi host that receives the traffic. The DLR gateway interface lives on all hosts that are in the same transport zone.