ESXi -> Cisco 3850 -> Upstream Router routing Not Working

[hbuchal]

Please see attached diagram.

I currently have "router on the stick" setup and I am moving to SVIs on Cisco 3850 stack. I have moved VLAN100 as a start.  I can ping each of the directly connected devices (i.e. 3850 and 2911 router). I can't seem to ping a VM on vlan 100 from the router and vice versa. Here is what is working what is not working.

Working in both directions

VM (172.16.100.51) <-> GW on SVI (172.16.100.254) 

VM (172.16.100.51) <-> Another SVI (172.16.230.254)

VM (172.16.100.51) <-> L3 Int on 3850 (10.2.2.2)

L3 int on 3850 (10.2.2.2) <-> L3 int on 2911 (10.2.2.1)

SVI on 3850 (172.16.100.254) <-> L3 int on 2911 (10.2.2.1)

Not Working in either direction:

VM (172.16.100.51) <-> L3 interface on 2911 (10.2.2.1)

VM (172.16.100.51) <-> Anything else NOT routed on 3850

I have following routes on 2911 and 3850.

3850:
ip route 0.0.0.0 0.0.0.0 10.2.2.1

2911:

ip route 172.16.100.0 255.255.255.0 10.2.2.2

ip route 172.16.230.0 255.255.255.0 10.2.2.2

So In theory anything coming from 172.16.100.51 not local to 3850 should be forwarded to 10.2.2.1 since it's default route on 3850.

I suspect this to be a licensing issue. I do have IP Base feature set license on 3850 stack.I have verified it using show license and show version commands.

As per this Cisco FAQ, http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3850-series-switches/qa_c67-722110.html, routing should be working as I don't have more than 16 static routes and I am only using basic L3 routing features.

I am at a loss here. What's going on? Can someone please confirm?

I had bought WS-C3850-24T-S,

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3e/release_notes/OL3262101.html#pgfId-950711

thinking I would be able to use SVIs and keep all traffic from going to upstream routers as our older switches were only L2.

It looks like an upgrade to IP Services feature set is possible,

https://cisco3850.wordpress.com/2015/04/22/licensing-for-cisco-catalyst-3850-series-switches/.

Do I need to upgrade the image as well or can I just switch the license using the built-in commands described here,

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3se/system_management/configuration_guide/b_sm_3se_3850_cg/b_sm_3se_3850_cg_chapter_0100.html#concept_83A11E6B66E349A0A9090DBD37F28602

I hope I don't have to reboot the switches as this setup is currently using this stack as core and distribution.

Any help is appreciated.

Thanks

[hbuchal]

Even configured SPAN on upstream port on the switch with continuous PINGs from 172.16.230.51/100.51 --> 10.2.2.1 but nothing in wireshark. It only shows communication to/from switch's configured L3 interfaces.

I am banging my head against the wall here. This is simple Network.

[Reza Sharifi]

I completely understand.

Can you turn off routing and then turn it back on.

no ip routing

ip routing

If this does not work, can you try configuring a simple OSPF area0 between the switch and the router?

HTH

[hbuchal]

That was it. Wow...

[Reza Sharifi · Accepted solution]

Turning off and on "IP routing" did it?

[hbuchal]

Yes. As soon I did

no ip routing

ip routing

PINGS started to work. Now everything is working as expected.

I even had debugging on but nothing was showing up.

do you have any insights into why this made it working? I suspect this caused the routing processes to reinitialize. I am guessing.

But i really appreciate your help.

[Reza Sharifi]

I think the routing process was stuck and so resting it did it.

Can you rate all helpful posts and close the post, so other members can benefit from it?

Good Luck!

[hbuchal]

It's not VMware issue. 

I connected a Windows host directly to an Access Port on 3850.

I am getting IP from DHCP on 3850 stack.

I can NOT ping anything not on 3850.

Are you 100% sure I don't need IP Services feature set to route devices connected to 3850 to upstream routers?

Th Mgmt VRF is enabled by default I think. Do I need a separate VRF?

I am at a loss here.

[Reza Sharifi]

Are you 100% sure I don't need IP Services feature set to route devices connected to 3850 to upstream routers?

No, you don't need IP Services license for simple static routing. I have used many 3850s with IP Based license and static config is no issue.

What is the output of sh license right-to use?

Th Mgmt VRF is enabled by default I think.

That is correct.

Do I need a separate VRF?

No, just global config.

do you have a different switch you can try.

HTH

[hbuchal]

Unfort. I don't have a spare switch to test. These two switches in a stack have been in production since last year w/o any issues.

Core#sh license right-to-use usage
Slot# License Name Type usage-duration(y:m:d) In-Use EULA
-----------------------------------------------------------------------
1 ipservices permanent 0 :0 :0 no no
1 ipservices evaluation 0 :0 :0 no no
1 ipbase permanent 0 :11:28 yes yes
1 ipbase evaluation 0 :0 :0 no no
1 lanbase permanent 0 :0 :0 no no
1 apcount evaluation 0 :0 :0 no no
1 apcount base 0 :0 :0 no no
1 apcount adder 0 :0 :0 no no

Slot# License Name Type usage-duration(y:m:d) In-Use EULA
-----------------------------------------------------------------------
2 ipservices permanent 0 :0 :0 no no
2 ipservices evaluation 0 :0 :0 no no
2 ipbase permanent 0 :11:28 yes yes
2 ipbase evaluation 0 :0 :0 no no
2 lanbase permanent 0 :0 :0 no no
2 apcount evaluation 0 :0 :0 no no
2 apcount base 0 :0 :0 no no
2 apcount adder 0 :0 :0 no no