Weird inter-VLAN issue - VLAN is local, can't route?

[ITguy84]

Went on vacation for a week, came back and one of my SAN management addresses isn't working.

Coles notes;

Management VLAN 633 - 10.133.63.192/26 - DG 10.133.63.193

10.133.63.200 and 10.133.63.202 are the management addresses for the SAN.

If I do a ping from the switch the SAN is plugged into I get these results;

3750x-XXXX(config)#do ping 10.133.63.200

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.133.63.200, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/5/16 ms
3750x-XXXX(config)#

Show ip route displays this;

C        10.133.63.192/26 is directly connected, Vlan633
L        10.133.63.193/32 is directly connected, Vlan633

show arp displays this;

Internet  10.133.63.193           -   44d3.ca17.c676  ARPA   Vlan633
Internet  10.133.63.194          60   4cd9.8fc1.a515  ARPA   Vlan633
Internet  10.133.63.200          20   4cd9.8fc1.a515  ARPA   Vlan633
Internet  10.133.63.202          20   4cd9.8fc1.afc1  ARPA   Vlan633

If I do a traceroute from the same VLAN (633) I get a reply;

3750x-XXXX#traceroute
Protocol [ip]:
Target IP address: 10.133.63.200
Source address: 10.133.63.193
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to sn289795.Ultra-XXXX.ca (10.133.63.200)

  1 sn289795.Ultra-XXXX.ca (10.133.63.200) 0 msec 8 msec 0 msec
3750x-XXXX#

If I do a traceroute from the same switch, but a different VLAN I get this;

3750x-XXXX#traceroute
Protocol [ip]:
Target IP address: 10.133.63.200
Source address: 10.133.0.1
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to sn289795.Ultra-XXXX.ca (10.133.63.200)

  1  *  *  *
  2  *  *  *
  3  *  *
3750x-XXXX#

I don't get it! VLAN 633 is directly connected, and ip routing is turned on. Why will the VSI's not talk to one another??? Because it's local, I can't make a rule or it complains about being on the same switch.

Here are some of the key parts of the running config;

vlan 633
 name Server_rm_SW5


interface GigabitEthernet4/0/23
 switchport access vlan 633
 switchport mode access
 switchport voice vlan 541
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
!
interface GigabitEthernet4/0/24
 switchport access vlan 633
 switchport mode access
 switchport voice vlan 541
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
!

interface Vlan633
 description Server Room Switch 5
 ip address 10.133.63.193 255.255.255.192
 ip helper-address 10.133.1.1
 ip helper-address 10.133.1.2
 ip helper-address 10.133.2.13
!

router ospf 2019
 router-id 3.3.3.1
 redistribute static
 network 10.133.48.0 0.0.7.255 area 0
 network 10.133.61.128 0.0.0.127 area 0
 network 10.133.62.0 0.0.0.127 area 0
 network 10.133.62.128 0.0.0.127 area 0
 network 10.133.63.0 0.0.0.127 area 0
 network 10.133.63.128 0.0.0.63 area 0
 network 10.133.63.192 0.0.0.63 area 0
 network 172.20.25.0 0.0.0.255 area 0
 network 172.20.26.0 0.0.0.255 area 0
 network 172.20.27.0 0.0.0.255 area 0
 network 172.20.28.0 0.0.0.255 area 0
 network 172.20.29.0 0.0.0.255 area 0
 network 172.20.30.0 0.0.0.31 area 0
 network 172.31.0.0 0.0.0.7 area 0
 network 172.31.0.8 0.0.0.3 area 0
 network 172.31.0.20 0.0.0.3 area 0
 network 172.31.0.24 0.0.0.3 area 0
 network 172.31.0.32 0.0.0.3 area 0
 network 172.31.0.40 0.0.0.7 area 0
 network 172.31.0.48 0.0.0.3 area 0
 network 172.31.0.56 0.0.0.3 area 0
 network 192.168.1.0 0.0.0.255 area 0
 network 192.168.26.0 0.0.1.255 area 0
 network 192.168.31.0 0.0.0.255 area 0
 network 192.168.32.0 0.0.15.255 area 0
 network 192.168.150.0 0.0.0.255 area 0
 default-information originate always
!
ip default-gateway 10.133.0.100
ip forward-protocol udp 5246
no ip http server
ip http authentication local
ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 10.133.255.233
ip route 0.0.0.0 0.0.0.0 10.133.0.100
ip route 10.1.110.19 255.255.255.255 172.31.0.4
ip route 10.133.2.18 255.255.255.255 172.31.0.49
ip route 10.133.53.10 255.255.255.255 10.133.0.3
ip route 10.133.134.0 255.255.255.0 10.133.0.100
ip route 10.133.137.21 255.255.255.255 172.31.0.1
ip route 10.133.147.0 255.255.255.128 172.31.0.49
ip route 10.242.0.0 255.255.0.0 172.31.0.1
ip route 10.242.2.0 255.255.255.0 172.31.0.1
ip route 172.20.0.0 255.255.255.0 172.31.0.1
ip route 172.20.64.0 255.255.255.0 192.168.31.6
ip route 172.20.71.0 255.255.255.0 192.168.31.6
ip route 172.21.5.0 255.255.255.0 172.21.0.2
ip route 172.21.6.0 255.255.255.0 172.21.0.2
ip route 172.21.8.0 255.255.255.0 172.21.0.2
ip route 172.21.9.0 255.255.255.0 172.21.0.2
ip route 172.21.16.0 255.255.255.0 172.21.0.2
ip route 172.31.0.4 255.255.255.255 GigabitEthernet1/0/48

Never had this happen before, likely something silly I'm overlooking after being off for a while, but any advice would be appreciated.

Also worth, I'm mid-flattening.... the previous sysadmin had pushed L3 down to the access layer and I'm undoing 10 years of debauchery by myself, while supporting 400 users and multiple projects.

Please be nice, I have about 2 good nerves left....  

[balaji.bandi]

why do you have 2 MAC address same in the ARP you listed ? is this expected ? try remove one of them and check ?

[paul driver · Accepted solution]

Hello
Seeing though you cannot ping sourced from a different  svi but you can via a direclty connected svi ----check the default-gateway on the end hosts

[ITguy84]

Hey Paul, I don't know how but you're correct. Somehow the DG was off by 1 digit..... logs don't go back far enough to determine how it happened. Ended up having to direct connect to the SAN management port and access it that way, and sure enough there was a .64 instead of a .63.....

Mystery....