Connectivity Issue with regards to routing between a Nexus 5548 Switch and a 3rd party Firewall for VoIP/SIP traffic

[jdelrosario]

Hello Cisco Community,

    I have a vexing issue regarding Layer 3 between a Nexus 5000 switch I'm using in my customer's core network and a Firewall (Mikrotik Cloud Router Switch CRS109) from a 3rd party Vendor who my customer is wanting to test a new VoIP/SIP service through. We wanted to isolate the voice traffic from any data traffic on the network so we have 2 VLANs defined (VLAN 10 for Data and VLAN 60 for Voice). We have a new Internet circuit with this SIP/VoIP provider that I have isolated just the VLAN 60 voice traffic on. So on the Nexus Switch I have the following config:

     vlan 60
        name NEW-Voice-VLAN

    interface Ethernet1/5
      switchport access vlan 60
      speed 1000
     duplex full
     no shutdown

   interface Vlan60
     no shutdown
     ip address 172.16.60.254/24
     hsrp 60
     ip 172.16.60.1

   The HSRP is active. The VLAN 60 interface is in an UP/UP state. The Ethernet 1/5 interfaces is also UP/UP and the Duplex and Speed are forced at Full-Duplex/1000 Mbps. 

    I have Layer 1 and Layer 2 up between Ethernet 1/5 and the Firewall Interface (Port 3). Port 3 also has connectivity and is configured with the following IP address:

        ip address =172.16.60.2/24

   I had plugged an IP phone on the Firewall Port 3 and it is getting it's config from the Provider's configuration server and registering correctly against their registration server. I'm also able to PING the FW interface that it is directly connected to (172.16.60.2)

    Now, when I connect the Firewall interface to the Nexus switch and try a PING test from the Nexus switch, I get the following:

<Switch-Name># ping 172.16.60.2
PING 172.16.60.2 (172.16.60.2): 56 data bytes
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 0 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 1 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 2 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 3 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 4 timed out

--- 172.16.60.2 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss

When I do a "show ip route" I have these routes that pertain to this VLAN:

172.16.60.0/24, ubest/mbest: 1/0, attached
*via 172.16.60.254, Vlan60, [0/0], 6w5d, direct
172.16.60.1/32, ubest/mbest: 1/0
*via 172.16.60.1, Vlan60, [0/0], 6w5d, hsrp
172.16.60.254/32, ubest/mbest: 1/0, attached
*via 172.16.60.254, Vlan60, [0/0], 6w5d, local

From my understanding of basic routing, shouldn't I be able to ping the FW interface? (the provider says they don't have any ICMP restriction rules and I verified that when I plugged an IP phone directly into Port 3 on the Provider's FW and was able to ping the Default GW of 172.16.60.2 after it got its config from the Configuration server). Since they are directly attached (no other device in between), they should be able to PING each other correct? I don't understand why there is a "Destination Host Unreachable" error when they are Directly Connected/Attached to each other... if it was a TIMEOUT, then I can see the FW blocking the return packets but I don't see that as the problem. 

   The only other thing that may be an issue is this route:

    0.0.0.0/0, ubest/mbest: 1/0
    *via 10.1.0.12, [1/0], 1y0w, static

   Currently the 10.1.0.12 IP address is not defined anywhere on the customer network and all the Data devices and current IP phones have different default gateways defined that go to another firewall that is connected to their current Internet circuit. 

    So I am puzzled with this one. Am I just missing something basic? I'm going to isolate the Nexus and use a spare 2960-x switch configured for routing that my customer has to see if is an issue with the Nexus. The code is old on the Nexus and needs to be upgraded but I won't be able to do that with the customer's permission until January at the earliest. 

Software
BIOS: version 3.6.0
Power Sequencer Firmware:
Module 1: v5.0
Module 2: v5.0
Microcontroller Firmware: version v1.2.0.1
QSFP Microcontroller Firmware:
Module not detected
CXP Microcontroller Firmware:
Module not detected
kickstart: version 7.1(4)N1(1)
system: version 7.1(4)N1(1)
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:///n5000-uk9-kickstart.7.1.4.N1.1.bin
kickstart compile time: 9/2/2016 10:00:00 [09/02/2016 08:37:35]
system image file is: bootflash:///n5000-uk9.7.1.4.N1.1.bin
system compile time: 9/2/2016 10:00:00 [09/02/2016 10:16:21]

Hardware
cisco Nexus5548 Chassis ("O2 32X10GE/Modular Supervisor")
Intel(R) Xeon(R) CPU with 8253848 kB of memory.
Processor Board ID <REMOVED>

Device name: <REMOVED>
bootflash: 2007040 kB

Kernel uptime is 367 day(s), 2 hour(s), 32 minute(s), 44 second(s)

Last reset at 154465 usecs after Sun Nov 12 13:27:50 2017

Reason: Reset Requested by CLI command reload
System version: 7.1(4)N1(1)
Service:

plugin
Core Plugin, Ethernet Plugin

Any help would be greatly appreciated. 

 Thanks!

      Joe

[Francesco Molino]

Hi
Can you check your arp table to see if the firewall mac address is there?
The firewall is connected on your Nexus port 5 directly?
In terms of hsrp, are you seeing one active and the other standby?
If you have the firewall mac address, try to set the arp manually and test again please.
When trying to ping, if you run a capture, do you see arp requests and replies?

[jdelrosario]

Francesco, 

    I forgot I had the looked at the ARP table after I tried the ICMP Echos... I got the INCOMPLETE message in the ARP table after I did a "show ip arp"

    172.16.60.2     00:00:15  INCOMPLETE      Vlan60

   For HSRP, the primary router is Active for the HSRP group:

       Vlan60 - Group 60 (HSRP-V1) (IPv4)
       Local state is Active, priority 100 (Cfged 100)
       Forwarding threshold(for vPC), lower: 1 upper: 100
       Hellotime 3 sec, holdtime 10 sec
       Next hello sent in 2.646000 sec(s)
       Virtual IP address is 172.16.60.1 (Cfged)
       Active router is local
       Standby router is 172.16.60.253 , priority 100 expires in 5.231000 sec(s)
       Authentication text "cisco"
       Virtual mac address is 0000.0c07.ac3c (Default MAC)
       2 state changes, last state change 7w0d
        IP redundancy name is hsrp-Vlan60-60 (default)

    I'll get the firewall MAC address and setup as a static ARP entry in the MAC address table... what is the exact command I need to use to set this up? 

     I'll report more after I get in later this morning to the customer site to try this... 

     Thanks Francesco!

         Joe

[Richard Burts]

Joe

It is important information that arp is not resolving the address of the firewall. And this is certainly the cause of your problem. Perhaps you can turn on debug for arp and try the ping again and post any debug output? You might also ask the folks with the firewall if they know of any reason why they would not respond to the arp request?

I am wondering about the possibility of a mismatch in the mask for the address on the firewall. I am not sure about that firewall but I know in terms of Cisco that it could be a problem. If a Cisco device was configured as 172.16.60.2 with a 255.255.255.128 mask and it received an arp request from address 172.16.60.254 it would treat that as an error, would complain in debug about "wrong cable" and would not respond.

HTH

Rick

[jdelrosario]

It's a Mikrotik Cloud Router Switch (CRS109-8G-2HnD-IN).. unfortunately, since this is provided by a 3rd party VoIP hosting provider, I don't have access into the device... I found some articles within the Cisco Community about issues with Cisco devices and Mikrotik devices regarding ARP and the MAC address of the Mikrotik device not showing up in the Cisco ARP table... I'll try to create static arp entries for the Mikrotik firewall and see if that fixes the issue.. I know that's not the normal thing to do, but seems to have worked with people having the same issue. 

I'll let you and Francesco know how it turns out when I get to my customer site in a little bit!

Thanks for the advice!

     Joe

[Richard Burts]

Joe

Interesting that there does seem to be some history of issues about arp between Cisco and Mikrotik. Please do keep us updated as you go forward trying to solve this issue.

HTH

Rick

[jdelrosario]

Francesco/Richard,

   What should I be creating for the static ARP entry for the Mikrotik firewall? Trying to figure out from CLI... do I need to use the "mac address-table" command? When I see about creating a static ARP entry, that's already assuming that the port has been configured for route mode instead of switch mode (no switchport)...

    Also, I've got a VLAN interface defined on this switch:

           interface Vlan60
             no shutdown
             ip address 172.16.60.254/24
             hsrp 60
             ip 172.16.60.1

     Shouldn't this work with the Mikrotic router? Or do I need to change my Ethernet1/5 port from Switching to Routing mode (no switchport) and configure an IP address on that interface? I think this device is more of a router with some firewall capabilities than a firewall so I may have to do it that way... thoughts? 

         Joe

[Richard Burts]

Joe

You ask an interesting question about whether the port connecting to the Mikrotik should be configured as a routed port rather than as a switch port. My first reaction was pretty positive, thinking that perhaps taking the switching processing (BPDUs etc) off the interface might help the Mikrotik process the connection as another layer 3 host. Then I thought about what you have told us about your environment. It seems particularly important to me that you have configured your switch with 2 vlans. You have a vlan for phones/SIP which you want to keep separate from the vlan for data/PCs etc. If you keep the port connecting to the Mikrotic as a switch port in vlan 60 then all of the devices in that vlan can have the IP of the Mikrotik as their default gateway, and it does make the phone vlan very separate from the data vlan (since your switch will not provide routing for the phone vlan/subnet). If you make the port connecting to Mikrotik into a routed port, then the devices in vlan 60 will need to specify your switch as their default gateway, and this makes it more difficult to keep vlan 60 phones separate from vlan 50 data.

If there is something that I have not understood correctly about your environment then please clarify what it is.

HTH

Rick

[jdelrosario]

Rick,

     You are correct.. My whole intention is to separate the Voice traffic (VLAN 60) from the Data traffic (VLAN 10). My customer brought in another circuit specifically for VoIP/SIP hosted service and we wanted to separate this traffic from the Data traffic. 

     Since the Nexus Ethernet 1/5 interface is directly connected to Port 3 on the Mikrotik router/firewall, and they're both on VLAN 60 (Mikrotik is 172.16.60.2, I have VLAN 60 with an HSRP address of 172.16.60.1 and Interface IP of 172.16.60.254) that they are Directly connected interfaces and I shouldn't have to create any specific static routes. I've got the default route in place, but I know it's not correct since 10.1.0.12 doesn't exist in this network. 10.1.0.1 is the IP address of the customer firewall and all devices on their Data VLAN are pointing to that Firewall as their default gateway. 

   the IP phones that will be in VLAN 60 and will eventually be pointing to 172.16.60.2 as it's default gateway once it can get an IP from the DHCP server (currently on the Mikrotik router but will eventually be moved to the customer's Active Directory DCs once the phones are working correctly).

   I did try to make the Ethernet 1/5 a routed port but using the VLAN interface IP but got the same Destination Host Unreachable error... so I configured it back to a Switchport... I also tried to create a static MAC address table entry with the MAC address of Prort 3 on the Mikrotik router, but got the same error message. 

   Could the default route going to 10.1.0.12 be the problem? Like I said, since the interfaces on both sides are directly connected and in the same VLAN, I don't think packets would go out this default route... 

   I'm currently configuring a 2960-X switch and will add Layer 3 to it and connect it to the Mikrotik router taking the Nexus (in the customer core) out of the equation for now.. if I can get Layer 3 working there, I'll plug an IP phone in and see if that works... if all that works, then it is a Nexus issue... if the Layer 3 doesn't work, it is a Mikrotik issue... 

    I just don't know what the Mikrotik is doing with ARP requests? There are some ARP features that can be enabled on it, but I won't know and can't test until I hear back from the provider that manages the Mikrotik router... 

   Hope this helps to clarify my thought process!

    Thanks!

        Joe

[Francesco Molino]

Try adding a static arp and mac-address static like this:

int vlan 60
ip arp 172.16.60.1 0011.2233.4455
!
mac address-table static 0011.2233.4455 vlan 60 interface Ethernet1/5

I believe it's e1/5 where your FW is connected to (sorry replying through my smartphone)

[jdelrosario]

Francesco,

  I did put the mac-address table command exactly like you have but that didn't work... I didn't configure the IP ARP command with the mac-address table command... why is that needed and why the 172.16.60.1 IP address with the Mikrotik Firewall MAC address? 

   Thanks,

        Joe

[jdelrosario]

Tried your suggestions for that static ARP entry and static MAC entry... still have the same Destination Host Unreachable...

Thanks!

Joe

[Francesco Molino]

Ok then can you span the port where the firewall is connected to and see if you have arp exchanges when you're trying to ping it.

Your firewall is connected directly on the Nexus. Is it the primay or secondary?

[Richard Burts]

Joe

I have a couple of thoughts. First and most important is to clarify how you want to separate the vlans. As I see it you have two choices. 1) you could treat vlan 60 simply as a layer 2 vlan with ports on the Nexus in the vlan and the vlan connected to Mikrotik. Devices in vlan 60 would use Mikrotik as their default gateway. In this case you do not want to have interface vlan 60 on the Nexus. The advantage of this approach is that it is simple and effective. It immediately and completely separates the two vlans. 2) you could treat vlan 60 as layer 3 as well as layer 2. In this case you do need to have interface vlan 60 with an IP address and devices in vlan 60 would use the Nexus IP as their default gateway. You would then need some routing logic to get separation of the vlans. You might need some access lists, and perhaps something like Policy Based Routing.

When I read that down the road that you might want the phones to access some resources in your network such as DHCP it points toward the second solution of treating vlan 60 as both layer 2 and layer 3. Think about the alternatives and decide which one is better for your requirements.

You asked if something about the default route might be the problem. At this point I am comfortable in saying no it is not. There might or might not be issues with the default route. But if the Nexus sends arp requests and does not receive any response then this problem is not at all related to default routes.

You mention that you are configuring HSRP. Is there a second switch in your network providing redundancy for vlan 60? Does that mean that there are two Mikrotik firewalls operating as a redundant pair? Is it possible that the attempt at redundancy is contributing to the issue? Perhaps (temporarily) removing the second switch, removing HSRP, and connecting to a single Mikrotik might be worth trying?

I am wondering if there is communication between the Nexus and Mikrotik given that arp is not working. If you look in the mac address table on Nexus is there any learned mac address on the port connecting to Mikrotik?

HTH

Rick

[paul driver]

Hello

---

@jdelrosario wrote:

   I had plugged an IP phone on the Firewall Port 3 and it is getting it's config from the Provider's configuration server and registering correctly against their registration server. I'm also able to PING the FW interface that it is directly connected to (172.16.60.2)

    Now, when I connect the Firewall interface to the Nexus switch and try a PING test from the Nexus switch, I get the following:

---

The ports on some nx-os software have midx disabled by default 

Did your ip phone have hard coded speed-duplex settings like your switch port has?

Have you tried to replicate the ip phone  port settings also try auto-negociatie on the switchport with mdix turned on with stp portfast applied 

Lastly - Have you checked your cabling?