InterVLAN Routing w/ Two Layer 3 Switches

[WorstNetAdminEver]

I'll preface this by saying I'm not a network admin.  I haven't setup LAN switching in many years.  However, I've been tasked with this, so here we are. 

Fairly simple network.  I have two 9200's, in two different buildings, trunked together.  I have devices on 2 VLANs.  All devices in the same VLAN can communicate.  Devices on different VLANs, even on the same switch, cannot communicate.  However, devices CAN ping the VLAN interface IPs for the VLAN they're not in. 

I've attached the IOS configs.  They're not pretty.  Feel free to rip me a new one.  Been working this issue for a week now.  Also should note - This network is VERY remote and I don't have on-site support.  So if I mess up and nuke my connection to the switches, it's a big problem.  Has made troubleshooting difficult. 

[Georg Pauwen]

Hello,

your problem basically comes down to the fact that you have two layer 3 switches with the same IP address space. In order to tell you how to fix this, and keeping in mind that you need to do this remotely, with no room for error, how are you connecting to these switches ?

[WorstNetAdminEver]

I'm connecting via SSH from a VM.   The VM is connected through a Cisco ASA, which is connected to commercial ISP.  This backdoor is (for the most part) independent of the network.  So I could possibly nuke one VLAN and still connect to the switches.  But that's the only room for error. 

Thanks for the reply BTW. 

[balaji.bandi]

Ok, we understand your config now.

what VLAN are you trying to test? VLAN 21 independently works? and VLAN 122 independently works?

from VLAN 21 to VLAN 122 (both sides not working ?)

can you post below output also :

show ip interface brief

show ip route

show ip arp

show vlan

show vtp status

[WorstNetAdminEver]

---

what VLAN are you trying to test? VLAN 21 independently works? and VLAN 122 independently works?

from VLAN 21 to VLAN 122 (both sides not working ?)

---

This entire statement is correct.  InterVLAN between devices does not work.

The results of the requested commands are attached.  Thank you. 

[paul driver]

Hello
The issue is your users don’t have a definitive default-gateway to use, both switches have ip routing enabled and have L3 interfaces (SVI's) for the same vlans.

You need to either append a first hop routing protocal such as (HSRP, VRRP)  to both switch svis interfaces or remove the ip routing from one of the switches and make the other the L3 switch for both vlans.

However it may be that you dont even require these two switches to run ip routing at all, it depends on how the rest of your network is setup, Can you elaborate on this please and confirm what device is designated to perform the L3 inter-vlan routing for your site, Then we can provide a correct solution for you.

[WorstNetAdminEver]

Here's a quick napkin drawing of basics of the network.  

To answer your question, I have both 9200s doing the L3 routing.  Which I now gather is the basis of my issue.  

[paul driver]

Hello

---

@WorstNetAdminEver wroteTo answer your question, I have both 9200s doing the L3 routing.  Which I now gather is the basis of my issue.  

---

Yes it is,
So if the fortigate fw isnt perfroming the L3 intervlan routing then do the following making BLDG1 switch the root switch and vlan 21 the MGT vlan for both switches and fw.

Provide bldg1 switch a default route towards the fw.and static routes for vlan 21-122 on the fw towards buld1 switch for return wan traffic.

Lasty users in vlan 21 and 122 have their default-gateways point to
D/G vlan 21 =10.21.64.249
D/G vlan 122 =10.2.64.72

Note- log on to bldg2 switch  via bldg1 switxh to make its changes that way you want lose remote connection.

BLDG2 switch
conf t
ip default-gateway 10.21.64.249
no ip routing
no int vlan 122

vtp domain xxx
vtp mode transparent
vtp mode client

BLDG1 switch
conf t
spanning-tree vlan 21 priorty 0
spanning-tree vlan 121 priorty 0
vtp domain xxx
vlan 21, 122
exit

ip route 0.0.0.0 0.0.0.0 x.x.x.x (fw ip)

fortigate
ip route 10.2.64.0 255.255.192.0 10.21.64.249

[Scott Leport]

Hi,

Assuming BLDG1 is the switch that connects to the Fortigate and based on the config you've supplied, it looks like you'll also need to configure a default route towards your Fortigate from this switch. It's not clear how interconnected hosts to this switch will get Internet access.

However, make the suggested changes first and then address this, if you haven't already.

[balaji.bandi]

Looking at your Diagram i take Cat 9200 conected to Fortigate is BLDG1,   what configuration or IP address configured on Fortigate ?

one of the switch need to go as VTP Transparent or Client, (i would suggested both should be Transparent), You STP Root should be the one connected to Fortigate-  same was suggested other poster too.

From switch are you able to ping each other ? and from switch are you able to ping Hosts ?

[Georg Pauwen · Accepted solution]

Hello,

below is what the revised configs should look like (important parts are marked in bold). I have tested this in a lab, since you are doing this remotely, there is a command 'reload in' which reloads the device after the amount of time specified. Don't save the changes to the running configuration, if you make a mistake, the device will reload with the original config.

The default route on the BLDG1 switch points to the IP address of the Fortigate.

hostname BLDG1
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
enable password
!
no aaa new-model
switch 1 provision c9200l-24t-4g
!
--> ip routing
!
login on-success log
no device-tracking logging theft
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
crypto pki trustpoint TP-self-signed-611622380
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-611622380
revocation-check none
rsakeypair TP-self-signed-611622380
!
crypto pki certificate chain SLA-TrustPoint
certificate ca 01
30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030
32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363
6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934
3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305
43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720
526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030
82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D
CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520
1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE
4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC
7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188
68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7
C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191
C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44
DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201
06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85
4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500
03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905
604EDCDE FF4FED2B 77FC460E CD636FDB DD44681E 3A5673AB 9093D3B1 6C9E3D8B
D98987BF E40CBD9E 1AECA0C2 2189BB5C 8FA85686 CD98B646 5575B146 8DFC66A8
467A3DF4 4D565700 6ADF0F0D CF835015 3C04FF7C 21E878AC 11BA9CD2 55A9232C
7CA7B7E6 C1AF74F6 152E99B7 B1FCF9BB E973DE7F 5BDDEB86 C71E3B49 1765308B
5FB0DA06 B92AFE7F 494E8A9E 07B85737 F3A58BE1 1A48A229 C37C1E69 39F08678
80DDCD16 D6BACECA EEBC7CF9 8428787B 35202CDC 60E4616A B623CDBD 230E3AFB
418616A9 4093E049 4D10AB75 27E86F73 932E35B5 8862FDAE 0275156F 719BB2F0
D697DF7F 28
quit
crypto pki certificate chain TP-self-signed-611622380
certificate self-signed 01
3082032E 30820216 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 36313136 32323338 30301E17 0D323131 30313332 31353432
355A170D 33313130 31333231 35343235 5A303031 2E302C06 03550403 1325494F
532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3631 31363232
33383030 82012230 0D06092A 864886F7 0D010101 05000382 010F0030 82010A02
82010100 9E3489B0 9F54CAC7 ECD6B1EF 99080409 7FC66C60 FC21CE66 88EBC946
9CFC491C 66604C25 8E611062 7601840E B1B33FBC E8306807 FEAF7447 10417B8D
EEC74B01 272F7427 5EC9A375 0F8728DC A0F1F51D D4318445 7FFCBA5E BD95FB26
22290AC0 29CA9534 092F2663 F779734C 2194931E 2B9570B0 44A380D3 1B17CC94
973ACB4F 620D83A7 CA45B160 EDFBE2CF 3918C1A2 302BAA01 3340812A BB959EC6
D74A154D 91672C8A A6164011 2E7B8102 96CE90E3 E62A1694 89B3B273 B18AA4F6
20EA81B9 CA9B0CA3 4500663F 293F7952 916C8D00 853BEEC4 2D39D44A D8416597
E47F2CF1 F24ED1E4 C910CBC6 13C8F708 34D6EE33 FC0948F6 DA5B7109 FB8EDA4A
A25EB963 02030100 01A35330 51300F06 03551D13 0101FF04 05300301 01FF301F
0603551D 23041830 1680147A BF6A0C39 F2087620 2628FE97 120691E3 AB806E30
1D060355 1D0E0416 04147ABF 6A0C39F2 08762026 28FE9712 0691E3AB 806E300D
06092A86 4886F70D 01010505 00038201 01006AA0 64B2BA43 762C2E45 006E31EF
7D8AD926 0B34313D 67348019 7982705D 2941AAA0 FD58B5D2 A3555555 F6A1E3F9
A74C4032 40A38AE7 B6239CE5 101AED5A D5853316 44056FB9 F1667B52 76EE9221
E33C2D35 C61C0293 DF1E1940 FE1B374E 108D1575 283A41A8 6C46CF3F 09F834B6
BA06CD7E AE30FE00 4CF659BA 121FFB95 BFC8E1CE 6D342AA8 A7D1CF56 2AD7C6AF
D2D319C6 28FF395C 25AA4B43 D1249B81 FF9F1759 93F865B3 DD9B64BF E0FDF987
EEEA2332 73B3B6D9 D0A3EE36 6072BDF8 7B328AA7 1E913524 C959C370 B0B67B0A
CF607142 E54846D2 371B67F9 FF6AE5BB 4E76E459 527F7550 313CDD07 92691A34
B2C3E05F 069F2E2D FA209C56 C6F9DA06 2EBB
quit
!
license boot level network-essentials addon dna-essentials
!
diagnostic bootup level minimal
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
memory free low-watermark processor 10308
!
redundancy
mode sso
!
transceiver type all
monitoring
!
class-map match-any system-cpp-police-ewlc-control
description EWLC Control
class-map match-any system-cpp-police-topology-control
description Topology control
class-map match-any system-cpp-police-sw-forward
description Sw forwarding, L2 LVX data packets, LOGGING, Transit Traffic
class-map match-any system-cpp-default
description EWLC data, Inter FED Traffic
class-map match-any system-cpp-police-sys-data
description Openflow, Exception, EGR Exception, NFL Sampled Data, RPF Failed
class-map match-any system-cpp-police-punt-webauth
description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
description L2 LVX control packets
class-map match-any system-cpp-police-forus
description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
description MCAST END STATION
class-map match-any system-cpp-police-high-rate-app
description High Rate Applications
class-map match-any system-cpp-police-multicast
description MCAST Data
class-map match-any system-cpp-police-l2-control
description L2 control
class-map match-any system-cpp-police-dot1x-auth
description DOT1X Auth
class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
description Stackwise Virtual OOB
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
description Routing control and Low Latency
class-map match-any system-cpp-police-protocol-snooping
description Protocol snooping
class-map match-any system-cpp-police-dhcp-snooping
description DHCP snooping
class-map match-any system-cpp-police-ios-routing
description L2 control, Topology control, Routing control, Low Latency
class-map match-any system-cpp-police-system-critical
description System Critical and Gold Pkt
class-map match-any system-cpp-police-ios-feature
description ICMPGEN,BROADCAST,ICMP,L2LVXCntrl,ProtoSnoop,PuntWebauth,MCASTData,Transit,DOT1XAuth,Swfwd,LOGGING,L2LVXData,ForusTraffic,ForusARP,McastEndStn,Openflow,Exception,EGRExcption,NflSampled,RpfFailed
!
policy-map system-cpp-policy
!
interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
no ip address
shutdown
negotiation auto
!
interface GigabitEthernet1/0/1
description // Removed //
switchport access vlan 122
switchport mode access
!
interface GigabitEthernet1/0/2
description // Removed //
switchport access vlan 122
switchport mode access
!
interface GigabitEthernet1/0/3
description // To Removed //
switchport access vlan 122
switchport mode access
!
interface GigabitEthernet1/0/4
description // Removed //
switchport access vlan 122
switchport mode access
!
interface GigabitEthernet1/0/5
description // Removed //
switchport access vlan 122
switchport mode access
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
description // Removed //
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/0/14
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/0/15
description // Removed //
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
description // Removed //
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
description // Trunk to 9200 in BLDG 2 //
switchport mode trunk
!
interface GigabitEthernet1/1/1
!
interface GigabitEthernet1/1/2
!
interface GigabitEthernet1/1/3
!
interface GigabitEthernet1/1/4
!
interface Vlan1
no ip address
shutdown
!
interface Vlan21
--> description Uplink to Fortigate 10.21.64.250
ip address 10.21.64.249 255.255.255.0
!
interface Vlan122
description // Plan 122 //
ip address 10.2.64.72 255.255.192.0
!
--> ip route 0.0.0.0 0.0.0.0 10.21.64.250
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
!
control-plane
service-policy input system-cpp-policy
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0 4
password
login
transport input all
line vty 5 15
login
transport input ssh
!
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http
!
end

Current configuration : 9873 bytes
!
! Last configuration change at 21:17:31 UTC Wed Nov 17 2021
!
version 17.3
service timestamps debug datetime msec
service timestamps log datetime msec
service call-home
platform punt-keepalive disable-kernel-core
!
hostname BLDG2
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
enable password
!
no aaa new-model
switch 1 provision c9200l-24t-4g
!
--> no ip routing <-- this won't show up in the running config)
!
login on-success log
no device-tracking logging theft
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
crypto pki trustpoint TP-self-signed-3553538736
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3553538736
revocation-check none
rsakeypair TP-self-signed-3553538736
!
crypto pki certificate chain SLA-TrustPoint
certificate ca 01
30820321 30820209 A0030201 02020101 300D0609 2A864886 F70D0101 0B050030
32310E30 0C060355 040A1305 43697363 6F312030 1E060355 04031317 43697363
6F204C69 63656E73 696E6720 526F6F74 20434130 1E170D31 33303533 30313934
3834375A 170D3338 30353330 31393438 34375A30 32310E30 0C060355 040A1305
43697363 6F312030 1E060355 04031317 43697363 6F204C69 63656E73 696E6720
526F6F74 20434130 82012230 0D06092A 864886F7 0D010101 05000382 010F0030
82010A02 82010100 A6BCBD96 131E05F7 145EA72C 2CD686E6 17222EA1 F1EFF64D
CBB4C798 212AA147 C655D8D7 9471380D 8711441E 1AAF071A 9CAE6388 8A38E520
1C394D78 462EF239 C659F715 B98C0A59 5BBB5CBD 0CFEBEA3 700A8BF7 D8F256EE
4AA4E80D DB6FD1C9 60B1FD18 FFC69C96 6FA68957 A2617DE7 104FDC5F EA2956AC
7390A3EB 2B5436AD C847A2C5 DAB553EB 69A9A535 58E9F3E3 C0BD23CF 58BD7188
68E69491 20F320E7 948E71D7 AE3BCC84 F10684C7 4BC8E00F 539BA42B 42C68BB7
C7479096 B4CB2D62 EA2F505D C7B062A4 6811D95B E8250FC4 5D5D5FB8 8F27D191
C55F0D76 61F9A4CD 3D992327 A8BB03BD 4E6D7069 7CBADF8B DF5F4368 95135E44
DFC7C6CF 04DD7FD1 02030100 01A34230 40300E06 03551D0F 0101FF04 04030201
06300F06 03551D13 0101FF04 05300301 01FF301D 0603551D 0E041604 1449DC85
4B3D31E5 1B3E6A17 606AF333 3D3B4C73 E8300D06 092A8648 86F70D01 010B0500
03820101 00507F24 D3932A66 86025D9F E838AE5C 6D4DF6B0 49631C78 240DA905
604EDCDE FF4FED2B 77FC460E CD636FDB DD44681E 3A5673AB 9093D3B1 6C9E3D8B
D98987BF E40CBD9E 1AECA0C2 2189BB5C 8FA85686 CD98B646 5575B146 8DFC66A8
467A3DF4 4D565700 6ADF0F0D CF835015 3C04FF7C 21E878AC 11BA9CD2 55A9232C
7CA7B7E6 C1AF74F6 152E99B7 B1FCF9BB E973DE7F 5BDDEB86 C71E3B49 1765308B
5FB0DA06 B92AFE7F 494E8A9E 07B85737 F3A58BE1 1A48A229 C37C1E69 39F08678
80DDCD16 D6BACECA EEBC7CF9 8428787B 35202CDC 60E4616A B623CDBD 230E3AFB
418616A9 4093E049 4D10AB75 27E86F73 932E35B5 8862FDAE 0275156F 719BB2F0
D697DF7F 28
quit
crypto pki certificate chain TP-self-signed-3553538736
certificate self-signed 01
30820330 30820218 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33353533 35333837 3336301E 170D3231 31303133 32313036
32335A17 0D333131 30313332 31303632 335A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 35353335
33383733 36308201 22300D06 092A8648 86F70D01 01010500 0382010F 00308201
0A028201 0100C0A9 74BD8ABA B69B5F42 A2B99395 D824AA82 90045862 E99EB542
6AFA59DD E3A025EE 8536003C 7DC11FF2 2B493514 0DF0076F DF0C33FD 3CD2635D
0990586D 90CDBFF3 37EDCE0C C64D8418 AA182F59 A974E6B3 91168EB0 0697DB6E
911FAA96 5A75CAF1 537DC036 483E8234 6EB0A450 7CBE4FD9 C6E8E20A 6023A20D
2A129EC1 8B5BDF42 311F5C18 1BFC6CFE BAD23B78 76AA20D0 DE6CFF3C 81EC5654
B3891AC7 788742FB 3EEF3584 7391BCBA E15522C0 A98EAAA8 5B6B9605 DE5E0A99
D7407CD2 39FBFD2F 0859E699 C51E28B3 7E82E79D 3327D168 51330EC6 6F4B80A5
ED67C26C 0A1F8D9E 391D4A10 EC0232F6 3806FBA8 19E155BA 29C36FE2 6C8C5702
263C8A4E 84250203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF
301F0603 551D2304 18301680 14792501 61C683AE 747FCB04 C7712112 76C2ADB9
D4301D06 03551D0E 04160414 79250161 C683AE74 7FCB04C7 71211276 C2ADB9D4
300D0609 2A864886 F70D0101 05050003 82010100 A7B2992D DC0F832D BF95D960
D634BA58 F0C943CE AA7AFC6B 12E3005A 47D570B3 51A62675 DA7F063D 1A75D4B1
80C01F4E B987A2F3 3853B05D 072F9840 EF47C0E4 6F62B560 A4016D0F E214EC7C
9ECB36D3 98D123A1 CF89FCFA 09314FD0 5151CA86 A1A10494 348A5C1D 1E1E6C55
0DC9EF0D 10B6FB1B 49B1A571 9CAB3EF8 924703C5 2CF0B4D1 551656F8 5DFC4186
A9517394 AEB0BC1B 938060D2 89FD0953 0120ECF7 D20E8702 70A04D9C DCC8F4D5
CA491CF6 DF40343D 92B10E74 9D08276A F2948E13 9E23CCEC 6F04447B 452A2873
E1A47C65 DEA09120 C76DC421 35069B7C 47528866 B74189A1 2B3483CE 888CCF11
AD2936D6 88E54069 25724173 0C757905 E18A4FD2
quit
!
license boot level network-essentials addon dna-essentials
!
diagnostic bootup level minimal
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
memory free low-watermark processor 10308
!
redundancy
mode sso
!
transceiver type all
monitoring
!
class-map match-any system-cpp-police-ewlc-control
description EWLC Control
class-map match-any system-cpp-police-topology-control
description Topology control
class-map match-any system-cpp-police-sw-forward
description Sw forwarding, L2 LVX data packets, LOGGING, Transit Traffic
class-map match-any system-cpp-default
description EWLC data, Inter FED Traffic
class-map match-any system-cpp-police-sys-data
description Openflow, Exception, EGR Exception, NFL Sampled Data, RPF Failed
class-map match-any system-cpp-police-punt-webauth
description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
description L2 LVX control packets
class-map match-any system-cpp-police-forus
description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
description MCAST END STATION
class-map match-any system-cpp-police-high-rate-app
description High Rate Applications
class-map match-any system-cpp-police-multicast
description MCAST Data
class-map match-any system-cpp-police-l2-control
description L2 control
class-map match-any system-cpp-police-dot1x-auth
description DOT1X Auth
class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
description Stackwise Virtual OOB
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
description Routing control and Low Latency
class-map match-any system-cpp-police-protocol-snooping
description Protocol snooping
class-map match-any system-cpp-police-dhcp-snooping
description DHCP snooping
class-map match-any system-cpp-police-ios-routing
description L2 control, Topology control, Routing control, Low Latency
class-map match-any system-cpp-police-system-critical
description System Critical and Gold Pkt
class-map match-any system-cpp-police-ios-feature
description ICMPGEN,BROADCAST,ICMP,L2LVXCntrl,ProtoSnoop,PuntWebauth,MCASTData,Transit,DOT1XAuth,Swfwd,LOGGING,L2LVXData,ForusTraffic,ForusARP,McastEndStn,Openflow,Exception,EGRExcption,NflSampled,RpfFailed
!
policy-map system-cpp-policy
!
interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
no ip address
shutdown
negotiation auto
!
interface GigabitEthernet1/0/1
switchport access vlan 21
!
interface GigabitEthernet1/0/2
switchport access vlan 21
!
interface GigabitEthernet1/0/3
switchport access vlan 21
!
interface GigabitEthernet1/0/4
switchport access vlan 21
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
!
interface GigabitEthernet1/0/13
description // Removed //
switchport access vlan 122
switchport mode access
!
interface GigabitEthernet1/0/14
!
interface GigabitEthernet1/0/15
!
interface GigabitEthernet1/0/16
!
interface GigabitEthernet1/0/17
!
interface GigabitEthernet1/0/18
!
interface GigabitEthernet1/0/19
!
interface GigabitEthernet1/0/20
!
interface GigabitEthernet1/0/21
!
interface GigabitEthernet1/0/22
!
interface GigabitEthernet1/0/23
!
interface GigabitEthernet1/0/24
description // Trunk to Bldg 1 //
switchport mode trunk
!
interface GigabitEthernet1/1/1
description // Removed //
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/1/2
description // Removed //
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/1/3
description // Removed //
switchport access vlan 21
switchport mode access
!
interface GigabitEthernet1/1/4
description // Removed //
switchport access vlan 21
switchport mode access
!
interface Vlan1
no ip address
shutdown
!
--> interface Vlan21
--> ip address 10.21.64.248 255.255.255.0
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
--> ip default-gateway 10.21.64.249
!
control-plane
service-policy input system-cpp-policy
!
line con 0
stopbits 1
line aux 0
stopbits 1
line vty 0
exec-timeout 40 0
password
logging synchronous
login
transport input telnet
line vty 1 4
password
login
transport input telnet
line vty 5 15
login
transport input ssh
!
call-home
! If contact email address in call-home is configured as sch-smart-licensing@cisco.com
! the email address configured in Cisco Smart License Portal will be used as contact email address to send SCH notifications.
contact-email-addr sch-smart-licensing@cisco.com
profile "CiscoTAC-1"
active
destination transport-method http
!
end

[WorstNetAdminEver]

Thank you George, Paul, and others.  This 100% worked.  InterVLAN is full op now.  I appreciate your time - Really pulled me out of a bind.  

Also - Good tip about the reload command.