In the most recent post we discussed ethernet frames in detail. In that post I provided a diagram depicting a NetApp FAS controller, configured with a VLAN trunk to a ethernet switch. What was unique about this configuration was that the NetApp controller was supporting multiple VLANs, two of which had jumbo frames enabled and one using standard size frames. A expanded version of that diagram is provided and this post will discuss the configuration required on the NetApp controller and on a switch running Cisco IOS.
If you read the last post you will see that we have filled out the diagram with some additional information and we have introduced a redundant connection for the NetApp controller. It is important to note that this configuration is focusing on the configuration of a single NetApp controller and ethernet switch for simplicity purposes. In your production infrastructure you would likely include a second storage controller and ethernet switch with redundant connections from each server.
The Server Configuration
Each server in this particular example is connected via 10Gbps Ethernet. These 10Gbps Ethernet connections are exclusively IP so it would be possible for these servers to communicate with our NetApp storage controller via iSCSI, CIFS, HTTP or NFS. We will provide a FCoE example configuration in a future post but it is important to note that FCoE does not run over IP, it runs over Ethernet, more on that another time.
The NetApp Controller Configuration
This particular controller is one of our FAS3100 Series controllers. These controllers include four PCIe expansion slots for I/O or intelligent cache (PAM) expansion. We have chosen to use NetApp’s X1008A-R6 dual port 10Gbps Ethernet Module. For the purposes of this example we will only be using one of the 10Gbps ports on the expansion card. We will also be using both of the onboard 1Gbps Ethernet ports. Our configuration will bond the two onboard 1Gbps ports into a LACP EtherChannel. We will configure the system to favor the 10Gbps Ethernet link and only use the LACP EtherChannel in the event that the 10Gbps connection has a loss of link. In NetApp documentation this is referred to as a Second Level VIF. The first level VIF will be the VIFs that we layer each physical interface into. The Second level VIF will be the virtual interface that we layer each virtual interface into.
Controller RC File
# Manually Edited Controller RC File, by Trey Layton
hostname netapp01
vif create single ntapvif01 e1a
vif create lacp ntapvif02 -b ip e0a e0b
vif create single seclevelvif ntapvif01 ntapvif02
vif favor ntapvif01
vlan create seclevelvif 1 2 3
ifconfig seclevelvif-1 10.10.1.5 netmask 255.255.255.0 mtusize 9000
ifconfig seclevelvif-2 10.10.2.5 netmask 255.255.255.0 mtusize 9000
ifconfig seclevelvif-3 10.10.3.5 netmask 255.255.255.0 mtusize 1500
route add default 10.10.3.1 1
routed on
options dns.domainname demo.netapp.com
options dns.enable on
options nis.enable off
savecore
There are a few items to note about the configuration above.
Second Level VIFs
If you research the second level VIF configurations in Data ONTAP documentation you will find the common use case for second level VIFs is when you have two multimode VIFs that you would like to use as active and passive between two physical switches. While the documented example in ONTAP guides is a common use of second level VIFs I provided this configuration to demonstrate that you may also layer a single mode VIF into a second level VIF. In this example we have actually layered one single mode vif and one dynamic multimode (LACP) VIF into the second level VIF.
802.1q VLAN Trunking to NetApp
You will notice that no where in the configuration do we reference 802.1q, this is because NetApp only supports 802.1q VLAN trunking thus any VLAN created is automatically going to use the 802.1q VLAN tagging feature.
The first step in enabling 802.1q is to create the VLANs that you wish to address on the controller. The VLAN creation is local to the NetApp controller and is simply instructing the VIF to tag VLANs which are identified in the “vlan create command”. You will notice that when we specify the VLAN create command we do so on the VIF we want these tagged frames to be transmitted out. In our case it is our second level VIF as this is going to be our VLAN trunk. The second level VIF feature will ensure that if the 10Gbps e1a interface goes down our traffic will float to the LACP interface and operations will continue.
The final piece for NetApp VLAN configuration is assigning the tag and IP address to the VIF. You will notice that we have 3 ifconfig statements, each of which represent the configuration for each individual VLAN we are addressing on the controller. The tag is created by immediately following the vif name with a hyphen and the VLAN number (ex: seclevelvif-10 (for VLAN 10)).
Default Route
A NetApp controller can only support one active default route. This is actually common practice in systems as the default route is the last resort destination, if the address is not locally known. When you attach a NetApp controller to a Network where the controller is addressed on multiple networks, you must consider how routing is working in the network. In most cases what you will find is that you will have one network interface which you consider the publicly addressable interface for the controller. This interface is usually the interface which is advertised in WINS or DNS as the target for any user to attach to the controller. The other VLANs which are configured are typically purpose built VLANs for a dedicated NFS or iSCSI Network. The intent for creating these isolated networks is to provide local direct access, in the data center for those protocols. I mention this because when you specify your default route you want that default route to be on the network which you consider your publicly advertised interface. If you require route definitions for the other VLANs addressed on the controller then you may configure static routes. Static routes will allow you to statically define a more direct path to the network destination. Once you specify the static route that route will be used instead of the default route for that particular defined destination. A future post will describe static routes in detail.
The Cisco IOS Configuration
The configuration for the switch is actually quite similar to the configuration defined in the Multimode VIF Survival Guide. The difference in this configuration will be the specification of a Jumbo Frame MTU and the conversion of the switch ports from a standard access port to a VLAN trunked interface when connecting to the NetApp controller.
interface TenGigabitEthernet 1/1
description VLAN 2 Server
switchport
switchport mode access
switchport access vlan 2
spanning-tree portfast
mtu 9216
interface TenGigabitEthernet 1/2
description VLAN 3 Server
switchport
switchport mode access
switchport access vlan 3
spanning-tree portfast
interface TenGigabitEthernet 1/3
description VLAN 1 Server
switchport
switchport mode access
switchport access vlan 1
##### NOTE: IOS defaults a port to VLAN 1 so this command is not necessary but placed here for simplicity.
spanning-tree portfast
mtu 9216
interface TenGigabitEthernet 1/4
description NetApp 10Gbps 802.1q Trunk
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1-3
switchport mode trunk
switchport nonegotiate
spanning-tree guard loop
spanning-tree portfast trunk
mtu 9216
interface Port-channel 1
description 2 Port LACP Channel to NetApp
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1-3
switchport mode trunk
switchport nonegotiate
spanning-tree guard loop
spanning-tree portfast trunk
mtu 9216
interface GigabitEthernet 2/1
description NetApp e0a 2x 1Gbps LACP 802.1q Trunk
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1-3
switchport mode trunk
switchport nonegotiate
spanning-tree guard loop
spanning-tree portfast trunk
channel-protocol lacp
channel-group 1 mode active
mtu 9216
interface GigabitEthernet 2/2
description NetApp e0b 2x 1Gbps LACP 802.1q Trunk
switchport
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1-3
switchport mode trunk
switchport nonegotiate
spanning-tree guard loop
spanning-tree portfast trunk
channel-protocol lacp
channel-group 1 mode active
mtu 9216
IOS Configuration Notes
The above configuration is very consistent with those that you find in the MultiMode VIF Survival Guide. There are a few items that we want to point out specifically.
switchport trunk allowed vlan 1-3 - This configuration is a optional configuration parameter. It is one that I always use as the 802.1q specification supports 4096 VLANs and within the infrastructure in your own network you likely have in production more VLANs than you are enabling on the NetApp controller. This command instructs the Cisco switch to only allow traffic from VLANs 1-3 over this link. This prevents broadcasts from the other VLANs in your network consuming bandwidth on your connections to your storage array. It should be noted that if you ever introduce a new VLAN to the NetApp then you must modify this parameter to ensure that VLAN is passed to the NetApp.
spanning-tree portfast trunk - Portfast is a feature which is used for non-infrastructure network devices to prevent the physical switch port from having to calculate spanning-tree when link is negotiated. Spanning-Tree calculations are done to prevent loops in the switching topology and the process by which a port goes through spanning-tree actually will block traffic until the calculation is complete. The time it takes to calculate spanning-tree ranges from a minimum of 30 seconds to as much as 2 minutes. In a controller failover event you do not want spanning-tree to prevent access to the storage controller so I advise you to always use this parameter when connecting devices that are not acting as switches or ethernet bridges.
In most cases you would simply type the command “spanning-tree portfast”. However, the interfaces we have defined in this configuration are 802.1q VLAN trunks. In the past it was expected that a VLAN trunk was connected to another switch therefore there was not an ability to enable portfast for those connections. Since it has become common practice for servers and other endpoints to use VLAN trunking to switches, Cisco introduced the portfast trunk command to accommodate fast activation of ports which were 802.1q trunks to non-switching platforms.
Summary
This completes a in depth configuration example of using 802.1q VLAN trunking with mixed Jumbo and Standard Frames using second level VIFs.
I hope this helps in your deployment efforts,
Trey
