Today I had to turn up a few T1's for a Windstream MPLS that we are pushing voice over. All Windstream voice runs over 172.16.X.0/24 private networks, so the diagram is basically the HQ which holds the 172.16.5.0/24 network. The destination subnet is 172.16.6.0/24.
There is a voicemail server that has two NICs one on the 172.16.5.0/24 network with no gateway and a NIC on 10.170.2.0/24 network (Data) with a gateway of 10.170.2.1 (HSRP VIP of 5k's). The two Links connected from the Nexus 5k's to the Verizon MPLS router are peering iBGP with Verizon and learning different 10.X.X.X networks from corporate private MPLS. I have verified that BGP is not learning any 172.16.1.0/24 through 172.16.7.0/24 networks from the corporate network. The route table only shows 2 networks being learned in that subnet area and they are 172.16.14.0/24 and 172.16.22.0/24.
So I have interface vlan 30 on each Nexus, and at first I entered static routes into each Nexus that states to get to networks 172.16.1.0/24 to 172.16.7.0/24 use 172.16.5.1 as the gateway. After this from the Nexus, either one, I would do a ping and traceroute to 172.16.6.1 using the source of 172.16.5.252 for Nexus 1 and 172.16.5.253 for Nexus 2. The ping failed, and the traceroute showed the traffic going out the layer 3 links connected to the Verizon MPLS router. No clue why. The route table shows me all my 10.X.X.X networks being learned via BGP, shows me the statics that I entered for the 172.16 networks and a default route that points to 10.170.105.20, which is the ASA to the internet.
So my fix for this, because I was clueless as to what was going on and why, was I created a VRF and called it windstream, associated it with Interface vlan 30 and entered all the static routes into that VRF. Now when I try to ping 172.16.6.1 and source it from VRF Windstream I can get to my destination. Now the issue was the Voice mail server that was dual homed had a gateway of 10.170.2.1 which is Interface Vlan 2 and it could not ping anything 172.16.5.0/24 or 172.16.6.0/24. Now I understand why it cant reach 172.16.6.0/24, but why cant it ping 172.16.5.1/24? The Nexus holds both SVI's so shouldn't it just route? Of does the SVI for vlan 30 being in its own VRF isolate it? To fix this for now I just put a persistant route into Voicemail Windows server route table that points it to 172.16.5.1 to get to anything 172.16.
So why was the original configuration pushing 172.16.6.0/24 traffic to the verizon MPLS router where it was not learning this network, and therefore if static routes didnt exist, it would have atleast gone to the default route which would have taken it to the ASA and dropped the traffic. With the static routes entered into the routing table with all the BGP networks learned, shouldn't the Nexus have know to push that traffic back to the Windstream Router to get to those networks?
Did I miss something?
