Connectivity Issue with regards to routing between a Nexus 5548 Switch and a 3rd party Firewall for VoIP/SIP traffic

[jdelrosario]

Hello Cisco Community,

    I have a vexing issue regarding Layer 3 between a Nexus 5000 switch I'm using in my customer's core network and a Firewall (Mikrotik Cloud Router Switch CRS109) from a 3rd party Vendor who my customer is wanting to test a new VoIP/SIP service through. We wanted to isolate the voice traffic from any data traffic on the network so we have 2 VLANs defined (VLAN 10 for Data and VLAN 60 for Voice). We have a new Internet circuit with this SIP/VoIP provider that I have isolated just the VLAN 60 voice traffic on. So on the Nexus Switch I have the following config:

     vlan 60
        name NEW-Voice-VLAN

    interface Ethernet1/5
      switchport access vlan 60
      speed 1000
     duplex full
     no shutdown

   interface Vlan60
     no shutdown
     ip address 172.16.60.254/24
     hsrp 60
     ip 172.16.60.1

   The HSRP is active. The VLAN 60 interface is in an UP/UP state. The Ethernet 1/5 interfaces is also UP/UP and the Duplex and Speed are forced at Full-Duplex/1000 Mbps. 

    I have Layer 1 and Layer 2 up between Ethernet 1/5 and the Firewall Interface (Port 3). Port 3 also has connectivity and is configured with the following IP address:

        ip address =172.16.60.2/24

   I had plugged an IP phone on the Firewall Port 3 and it is getting it's config from the Provider's configuration server and registering correctly against their registration server. I'm also able to PING the FW interface that it is directly connected to (172.16.60.2)

    Now, when I connect the Firewall interface to the Nexus switch and try a PING test from the Nexus switch, I get the following:

<Switch-Name># ping 172.16.60.2
PING 172.16.60.2 (172.16.60.2): 56 data bytes
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 0 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 1 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 2 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 3 timed out
36 bytes from 172.16.60.254: Destination Host Unreachable
Request 4 timed out

--- 172.16.60.2 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss

When I do a "show ip route" I have these routes that pertain to this VLAN:

172.16.60.0/24, ubest/mbest: 1/0, attached
*via 172.16.60.254, Vlan60, [0/0], 6w5d, direct
172.16.60.1/32, ubest/mbest: 1/0
*via 172.16.60.1, Vlan60, [0/0], 6w5d, hsrp
172.16.60.254/32, ubest/mbest: 1/0, attached
*via 172.16.60.254, Vlan60, [0/0], 6w5d, local

From my understanding of basic routing, shouldn't I be able to ping the FW interface? (the provider says they don't have any ICMP restriction rules and I verified that when I plugged an IP phone directly into Port 3 on the Provider's FW and was able to ping the Default GW of 172.16.60.2 after it got its config from the Configuration server). Since they are directly attached (no other device in between), they should be able to PING each other correct? I don't understand why there is a "Destination Host Unreachable" error when they are Directly Connected/Attached to each other... if it was a TIMEOUT, then I can see the FW blocking the return packets but I don't see that as the problem. 

   The only other thing that may be an issue is this route:

    0.0.0.0/0, ubest/mbest: 1/0
    *via 10.1.0.12, [1/0], 1y0w, static

   Currently the 10.1.0.12 IP address is not defined anywhere on the customer network and all the Data devices and current IP phones have different default gateways defined that go to another firewall that is connected to their current Internet circuit. 

    So I am puzzled with this one. Am I just missing something basic? I'm going to isolate the Nexus and use a spare 2960-x switch configured for routing that my customer has to see if is an issue with the Nexus. The code is old on the Nexus and needs to be upgraded but I won't be able to do that with the customer's permission until January at the earliest. 

Software
BIOS: version 3.6.0
Power Sequencer Firmware:
Module 1: v5.0
Module 2: v5.0
Microcontroller Firmware: version v1.2.0.1
QSFP Microcontroller Firmware:
Module not detected
CXP Microcontroller Firmware:
Module not detected
kickstart: version 7.1(4)N1(1)
system: version 7.1(4)N1(1)
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:///n5000-uk9-kickstart.7.1.4.N1.1.bin
kickstart compile time: 9/2/2016 10:00:00 [09/02/2016 08:37:35]
system image file is: bootflash:///n5000-uk9.7.1.4.N1.1.bin
system compile time: 9/2/2016 10:00:00 [09/02/2016 10:16:21]

Hardware
cisco Nexus5548 Chassis ("O2 32X10GE/Modular Supervisor")
Intel(R) Xeon(R) CPU with 8253848 kB of memory.
Processor Board ID <REMOVED>

Device name: <REMOVED>
bootflash: 2007040 kB

Kernel uptime is 367 day(s), 2 hour(s), 32 minute(s), 44 second(s)

Last reset at 154465 usecs after Sun Nov 12 13:27:50 2017

Reason: Reset Requested by CLI command reload
System version: 7.1(4)N1(1)
Service:

plugin
Core Plugin, Ethernet Plugin

Any help would be greatly appreciated. 

 Thanks!

      Joe

[jdelrosario]

 Thanks to everyone for your replies!

 Been meaning to update everyone. I decided to do a test and take the Nexus Switches out of the picture. My customer had a spare Catalyst 2960-X and I configured it for IP routing and with the same information as the Nexus Switch (VLAN 60 for Voice, VLAN 10 for Data), forced Full-Duplex /1000 Mbps on the switchport I was connecting to the MikroTik device. Got the same problems with the MAC address of the MikroTik device not showing up in the MAC-Address table or the ARP table when I send ICMP requests to it. I also tried to create static MAC-Address entry and a Static IP ARP entry to no avail. I then turned on some of the ARP Debug features and verified that I was NOT seeing any MAC-address from the MikroTik device being sent back to my Catalyst switch. All the issues were exactly the same whether it was the Nexus Switch or the Catalyst switch. so that definitely points to an issue with the MikroTik device. 

   For the cabling, I didn't switch cables because I was seeing UP/UP on my interface connected to the MikroTik device. I know if there is a Line Protocol issue, that it could be cabling or speed mismatch or duplex mismatch or something with autoconfiguration on one or both sides, but I didn't see any of those issues when I connected both the Catalyst 2960-X switch and the Nexus Switch... got UP/UP on the physical interface in both scenarios... 

  We're trying to get the Voice provider to check the MikroTik router and just take it out and replace it with a firewall they provide (Sophos XG)... what's frustrating is that I was able to get 2 of the IP phones to work a few weeks ago through my customer's firewall (Sophos UTM) and configuring SIP parameters and DHCP giving out IPs to Phones in the Voice VLAN... but any subsequent phones after that were not working... so this MikroTik router is configuration #2.... 

   I also didn't want to connect this circuit directly into my customer's Nexus core and just use access lists on it... wanted an intermediate security device (like a firewall) in between to secure the circuit... 

   any other suggestions? 

   Thanks in advance,

         Joe

[paul driver]

Hello

Can you post a topology of this network if possible.

[Richard Burts]

When you were running debug for arp did you see any arp requests from the Mikrotik?

I am interested in your statement that you do not find the mac address of the Mikrotik in the mac address table. Even though the interface shows as up/up I am wondering if there is some problem with communication to Mikrotik.

HTH

Rick

[paul driver]

Hello

---

@jdelrosario wrote:
Debug features and verified that I was NOT seeing any MAC-address from the MikroTik device being sent back to my Catalyst switch. All the issues were exactly the same whether it was the Nexus Switch or the Catalyst switch. so that definitely points to an issue with the MikroTik device

---

Can you apply the following on the L3 interface of the switch and test again

int x/x
ip proxy-arp
ip local-proxy-arp

[jdelrosario]

At customer site and I had done the ip proxy-arp and ip local-proxy-arp commands to no avail. Tried that last week and got the same issues... just tried again and got the same problem...

WOrkign with the VoIP provider to see if he can enable any debug features or some of the similar ARP commands on the MikroTik device and see if they see the same ARP issues from their side...

[jdelrosario]

Latest Update..

Been meaning to update since this was solved a week ago. It turns out that the provider had configured the port on the Mikrotik Switch as a trunk port instead of an access port. I had configured both my Nexus switch and Catalyst switch as an Access Port since I was only passing traffic from the voice network (VLAN 60). For some reason, he had a trunk port configured although I had told him I was only passing traffic from one network and not multiple networks. As soon as he changed it to an Access Port, Layer 3 connectivity was there (ICMP Echo Requests/Responses worked as intended). I didn't think of checking for this previously, but it makes sense since the Trunk port was looking for either untagged traffic or native vlan tagged traffic. I don't think the Mikrotik had either as a Trunk port, so it was dropping any packets coming from my Nexus and/or Catalyst switch even though Layer 1 and Layer 2 (UP/UP on the physical interfaces) were present. 

Thanks to all of you for your assistance with this. Very frustrating, but at least I was able to prove it wasn't anything that I had configured on my end! :-)

    Joe

[Richard Burts]

Joe

Thanks for the update informing us that the issue is solved and that the problem was that Mikrotik was configured as a trunk port. +5 for solving the problem. In my most recent response I questioned whether there was a problem that was preventing communication between your switch and Mikrotik. Their configuration as a trunk was indeed preventing communication. Glad to know that now things are working as expected.

HTH

Rick