Replaced two old 3560's and a 2960 with a new Catalyst 3650-X and two 2960-X's. Routing seems OK for the most part, but there are some issues getting to the SAN VLAN.
There is a trunk port between the core switch (3650) and the SAN switch (2960), and the routes are below, but then I started uncovering (not my network design) hairpinning in an ASA 5510 for one of the SAN subnets (172.15.20.0/24) going to the core switch IP.
So in the ASA firewall (10.10.10.3) you have: route insite 172.15.20.0/24 to 10.10.10.250 (core switch)
At the core switch I have an IP default gateway defined for a 2811 router: 10.10.10.1
Then static routes are as follows:
ip route 0.0.0.0 0.0.0.0 10.10.10.1
ip route 0.0.0.0 0.0.0.0 10.10.10.2 250
ip route 10.2.8.0 255.255.255.240 10.10.10.3
ip route 10.4.8.0 255.255.255.0 10.10.10.3
ip route 10.6.8.0 255.255.255.0 10.10.10.3
ip route 10.8.8.0 255.255.255.0 10.10.10.3
But also in the core switch is int vlan 2 for 172.15.20.1
Previously 172.15.20.1 could be reach from the SAN switch. But the SAN switch (again, a 2960) has no static routes, ip routing is not one, just a Trunk port back to the core, and a few VLAN 2 ports for the SAN, but an ip default gateway defined to: 10.10.10.1 (the router!)
I am thinking that needs to be fixed to point to the core switch instead, but doing so didn't totally fix it. And I am thinking maybe a static route for 172.15.20.0/24 in the core switch...? Since pinging 172.15.20.1 from the core switch goes to the ASA and back to the core and creates a bunch of ICMP errors..
