Unable to ping HSRP'd interface subnet if it's not local over l3 link

[AdvantageProfessional]

So came into remote office to find that for some reason connect to specific subnet over in remote site. Few weeks ago we've redone some of our networking and as part of that HSRP was put in place between 2 l3 switches, all worked fine and dandy.

But today found that I can't ping/route any traffic over to subnets which aren't currently hsrp active local on that switch.

To give more details

offsite sw1 is 10.10.21.10 255.255.255.252

dc sw1 is 10.10.21.9 255.255.255.252

vlan 5 (.13) and 105 (.3) exist in DC, and have standy set between 2 switches with x.x.x.1 as gateway's

everything works and pings fine except for routing to the subnets/vlans which aren't local, but the actual local ip standby for that hsrp pings fine, so it's almost like switch isn't able to reach that global ip becuase it's not directly connected? if that makes sense.

S1_WAN_SW01#sh standby br
P indicates configured to preempt.
|
Interface Grp Pri P State Active Standby Virtual IP

Vl5 5 100 P Active local 10.20.13.15 10.20.13.1
Vl105 105 100 P Standby 10.20.3.15 local 10.20.3.1

S1_WAN_SW01#ping 10.10.21.9 source 10.20.3.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.21.9, timeout is 2 seconds:
Packet sent with a source address of 10.20.3.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
S1_WAN_SW01#ping 10.10.21.10 source 10.20.3.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.10.21.10, timeout is 2 seconds:
Packet sent with a source address of 10.20.3.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 10/10/12 ms

S1_WAN_SW01#ping 10.20.3.110 source 10.10.21.9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.3.110, timeout is 2 seconds:
Packet sent with a source address of 10.10.21.9
.....
Success rate is 0 percent (0/5)
S1_WAN_SW01#ping 10.20.13.1 source 10.10.21.9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.13.1, timeout is 2 seconds:
Packet sent with a source address of 10.10.21.9
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
S1_WAN_SW01#ping 10.20.3.6 source 10.10.21.9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.3.6, timeout is 2 seconds:
Packet sent with a source address of 10.10.21.9
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
S1_WAN_SW01#ping 10.20.3.1 source 10.10.21.9
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.3.1, timeout is 2 seconds:
Packet sent with a source address of 10.10.21.9
.....
Success rate is 0 percent (0/5)

Let me know if you want more info, for now I'll probably force priority on sw1 to be higher to resume traffic but really confused as for why this doesn't work as expected.

[Reza Sharifi]

Can you post the output of "sh run" and "sh route" from both switches running HSRP? 

What is the switch model?

HTH

[AdvantageProfessional]

Hey,

It's a c9300-24T (sw1) and c3650e-24TD (sw2)

with some details removed like passwords and public ip's, it's a bit of a mess but it's what I inherited and slowly fixing and removed obsolete when I get around to it.

sh run sw1;

S1_WAN_SW01#sh run
Building configuration...

Current configuration : 13717 bytes
!
! Last configuration change at 14:27:48 UTC Fri Apr 16 2021 by corpadmin
! NVRAM config last updated at 14:29:14 UTC Fri Apr 16 2021 by corpadmin
!
version 16.6
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no platform punt-keepalive disable-kernel-core
!
hostname S1_WAN_SW01
!
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
enable secret
!
no aaa new-model
switch 1 provision c9300-24t
!
!
!
!
ip routing
!
ip name-server 8.8.8.8 8.8.4.4 10.20.13.50 10.20.13.100
no ip domain lookup
ip domain name
!
!
!
!
!
!
!
!
!
!
cpp system-default
!
crypto pki trustpoint TP-self-signed-371332197
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-371332197
revocation-check none
rsakeypair TP-self-signed-371332197
!
!
crypto pki certificate chain TP-self-signed-371332197
certificate self-signed 01
3082032E 30820216 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33373133 33323139 37301E17 0D313831 30313931 34303231
315A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F
532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3337 31333332
31393730 82012230 0D06092A 864886F7 0D010101 05000382 010F0030 82010A02
82010100 9343D368 CF41B667 295F6BD0 B7F06E3D 813AEC3C 3E4FD34D 4D620F11
551EA771 1676CC20 A6899B18 6A5C317C CD4B34E6 6E5C6C8F 05B17DC8 2E15E48F
DCE72908 61C0D4BF EF66F7D8 94E770B8 E2A46D23 9EC522AE 8FC0872A C637E94C
C82B196B A65BF3BA 79A41BA1 023DD119 F6A5EEC8 C3DB4C7F EA0EE35E 55DDAA1C
EA2665D8 7A847035 923BF220 7A0D4CD4 FE54B703 A808B0FE 87995C9B D8E0DD54
3D98820C 67B127FD 778FAD14 6B004924 7395FD10 C3744FEE 43DBDEF3 5E990238
87DC8131 054E0FA0 240AA8A6 2182DB8C F2E07CC9 E40ACC82 E55DF566 2ED2790C
DBAB4CDB 19CE3129 67E9462B 37599E9D 4D07A87F 1363736C 4ECFF875 0BFB46C2
DC643945 02030100 01A35330 51300F06 03551D13 0101FF04 05300301 01FF301F
0603551D 23041830 1680141D 4D8A6A09 9716B45B DDAC9EE1 35CF3039 C1211C30
1D060355 1D0E0416 04141D4D 8A6A0997 16B45BDD AC9EE135 CF3039C1 211C300D
06092A86 4886F70D 01010505 00038201 01001BDC C89BD51A 55BACEDB 922A9648
908EE2E6 E7CFEEDE 1A2F0ABC 3DB7E87D 471790CC C35987BF 86976FA0 41F1CC45
55E6C6EE 4D568238 077AF879 AF9E6DE4 8399C4B3 1DD0564B 527BE14C 3EC7528A
3C079717 E1C93FFF 47002CAF 57D3D8AD C26E19C4 08121659 C5B9481E 1D4E74F6
785F7217 6F14D83D D70418E1 26A48EAC 156A6D8B C8FD11CA 541E8FC1 78F4BB98
415CC961 D00EBADF 36653F3A 620C37B9 568E9D4D FD424D3F 4B297F2E 3420266B
EC318DD0 58125B48 3C307BB8 A410EB2B CA3FDEFE 0C1DF8FC FAEE4080 6D655525
33952C77 BC0FA438 2375E65B 604D2051 8768736F 360C3D3E 00EA6EF8 519C6756
8915D651 1BCC60D6 F157E342 862342EB 68D9
quit
!
!
!
diagnostic bootup level minimal
spanning-tree mode rapid-pvst
spanning-tree extend system-id
!
username corpadmin privilege 15 secret
!
redundancy
mode sso
!
!
!
class-map match-any system-cpp-police-topology-control
description Topology control
class-map match-any system-cpp-police-sw-forward
description Sw forwarding, L2 LVX data, LOGGING
class-map match-any system-cpp-default
description DHCP Snooping, EWLC control, EWCL data
class-map match-any system-cpp-police-sys-data
description Learning cache ovfl, Crypto Control, Exception, EGR Exception, NFL SAMPLED DATA, Gold Pkt, RPF Failed
class-map match-any system-cpp-police-punt-webauth
description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
description L2 LVX control packets
class-map match-any system-cpp-police-forus
description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
description MCAST END STATION
class-map match-any system-cpp-police-multicast
description Transit Traffic and MCAST Data
class-map match-any system-cpp-police-l2-control
description L2 control
class-map match-any system-cpp-police-dot1x-auth
description DOT1X Auth
class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
description Stackwise Virtual
class-map match-any system-cpp-police-control-low-priority
description General punt
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
description Routing control
class-map match-any system-cpp-police-protocol-snooping
description Protocol snooping
!
policy-map system-cpp-policy
class system-cpp-police-data
police rate 600 pps
class system-cpp-police-sys-data
police rate 100 pps
class system-cpp-police-sw-forward
police rate 1000 pps
class system-cpp-police-multicast
police rate 500 pps
class system-cpp-police-multicast-end-station
police rate 2000 pps
class system-cpp-police-punt-webauth
police rate 1000 pps
class system-cpp-police-l2-control
police rate 2000 pps
class system-cpp-police-stackwise-virt-control
police rate 8000 pps
class system-cpp-police-routing-control
police rate 5400 pps
class system-cpp-police-control-low-priority
police rate 200 pps
class system-cpp-police-l2lvx-control
police rate 1000 pps
class system-cpp-police-topology-control
police rate 13000 pps
class system-cpp-police-dot1x-auth
police rate 1000 pps
class system-cpp-police-protocol-snooping
police rate 2000 pps
class system-cpp-police-forus
police rate 1000 pps
class system-cpp-default
police rate 1000 pps
!
!
!
!
!
!
!
!
!
!
!
!
!
!
interface Port-channel1
switchport access vlan 105
switchport mode access
!
interface Port-channel4
switchport access vlan 106
switchport mode access
!
interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
no ip address
shutdown
speed 1000
negotiation auto
!
interface GigabitEthernet1/0/1
description IT Admin local cab interface
switchport access vlan 5
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/2
no switchport
ip address 10.10.20.22 255.255.255.252
!
interface GigabitEthernet1/0/3
description "Connects to Sale - S5-FL01-STACK Port Ge1/0/1"
no switchport
ip address 10.10.21.9 255.255.255.252
!
interface GigabitEthernet1/0/4
!
interface GigabitEthernet1/0/5
!
interface GigabitEthernet1/0/6
description Link to S1_RT01
no switchport
ip address 10.8.10.1 255.255.255.252
!
interface GigabitEthernet1/0/7
!
interface GigabitEthernet1/0/8
!
interface GigabitEthernet1/0/9
!
interface GigabitEthernet1/0/10
!
interface GigabitEthernet1/0/11
!
interface GigabitEthernet1/0/12
description OLD LINK to C7000 BE01
no switchport
ip address 10.8.10.13 255.255.255.252
!
interface GigabitEthernet1/0/13
description DarktraceMGMT
switchport access vlan 5
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/14
description Exagrid NIC 2
switchport access vlan 105
switchport mode access
channel-group 1 mode active
spanning-tree portfast
!
interface GigabitEthernet1/0/15
description Exagrid NIC 1
switchport access vlan 105
switchport mode access
channel-group 1 mode active
spanning-tree portfast
!
interface GigabitEthernet1/0/16
description Darktrace1
switchport access vlan 5
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/17
description darktrace3
switchport access vlan 5
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/18
description Exagrid NIC 4
switchport access vlan 106
switchport mode access
channel-group 4 mode active
spanning-tree portfast
!
interface GigabitEthernet1/0/19
description HSM
switchport access vlan 10
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet1/0/20
description Exagrid NIC 3
switchport access vlan 106
switchport mode access
channel-group 4 mode active
spanning-tree portfast
!
interface GigabitEthernet1/0/21
description UKS1SEC-ASA (primary)
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet1/0/22
description UKS1SEC-ASA (secondary)
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet1/0/23
description LINK to BE02
switchport mode trunk
!
interface GigabitEthernet1/0/24
description Link to S1_WAN_SW02
no switchport
ip address 10.8.10.9 255.255.255.252
!
interface GigabitEthernet1/1/1
!
interface GigabitEthernet1/1/2
!
interface GigabitEthernet1/1/3
!
interface GigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/1
!
interface TenGigabitEthernet1/1/2
!
interface TenGigabitEthernet1/1/3
!
interface TenGigabitEthernet1/1/4
!
interface TenGigabitEthernet1/1/5
!
interface TenGigabitEthernet1/1/6
!
interface TenGigabitEthernet1/1/7
!
interface TenGigabitEthernet1/1/8
!
interface FortyGigabitEthernet1/1/1
!
interface FortyGigabitEthernet1/1/2
!
interface Vlan1
no ip address
shutdown
!
interface Vlan2
description S1-Workstation
ip address 10.20.10.16 255.255.255.0
ip helper-address 10.20.13.25
standby 2 ip 10.20.10.1
standby 2 preempt
!
interface Vlan3
description S1-Voice
ip address 10.20.11.16 255.255.255.0
ip helper-address 10.20.13.25
standby 3 ip 10.20.11.1
standby 3 preempt
!
interface Vlan4
description S1-Security
ip address 10.20.12.16 255.255.255.0
standby 4 ip 10.20.12.1
standby 4 preempt
!
interface Vlan5
description S1-Core_1
ip address 10.20.13.16 255.255.255.0
standby 5 ip 10.20.13.1
standby 5 preempt
!
interface Vlan6
description S1-Core_2
ip address 10.20.14.16 255.255.255.0
standby 6 ip 10.20.14.1
standby 6 preempt
!
interface Vlan7
ip address 10.20.15.3 255.255.255.0
standby 7 ip 10.20.15.1
standby 7 preempt
!
interface Vlan8
no ip address
!
interface Vlan9
ip address 10.20.16.16 255.255.255.0
ip helper-address 10.20.13.50
standby 9 ip 10.20.16.1
standby 9 preempt
!
interface Vlan10
ip address 10.20.17.16 255.255.255.0
standby 10 ip 10.20.17.1
standby 10 preempt
!
interface Vlan40
no ip address
!
interface Vlan105
ip address 10.20.3.6 255.255.255.0
standby 105 ip 10.20.3.1
standby 105 priority 150
standby 105 preempt
!
interface Vlan106
ip address 10.20.4.16 255.255.255.0
ip helper-address 10.20.3.201
standby 106 ip 10.20.4.1
standby 106 preempt
!
interface Vlan109
ip address 10.20.7.16 255.255.255.0
standby 109 ip 10.20.7.1
standby 109 preempt
!
interface Vlan400
no ip address
!
!
router eigrp 100
network 0.0.0.0
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
ip ftp username
ip ftp password
ip route 0.0.0.0 0.0.0.0 10.20.12.10 permanent
ip route 10.1.0.0 255.255.0.0 10.8.10.14
ip route 10.15.10.0 255.255.255.0 10.20.102.1
ip route 10.20.3.0 255.255.255.0 10.20.3.1
ip route 10.20.20.0 255.255.255.0 10.10.20.9 180
ip route 10.20.21.0 255.255.255.0 10.10.20.9 180
ip route 10.20.22.0 255.255.255.0 10.10.20.9 180
ip route 10.20.23.0 255.255.255.0 10.10.20.9 180
ip route 10.20.40.0 255.255.255.0 10.20.12.10 180
ip route 10.20.41.0 255.255.255.0 10.20.12.10 180
ip route 10.20.42.0 255.255.255.0 10.20.12.10 180
ip route 10.20.43.0 255.255.255.0 10.20.12.10 180
ip route 10.20.44.0 255.255.255.0 10.20.12.10 180
ip route 10.20.45.0 255.255.255.0 10.20.12.10 180
ip route 10.20.50.0 255.255.255.0 10.20.12.10 180
ip route 10.20.51.0 255.255.255.0 10.20.12.10 180
ip route 10.20.52.0 255.255.255.0 10.20.12.10 180
ip route 10.20.53.0 255.255.255.0 10.20.12.10 180
ip route 10.20.54.0 255.255.255.0 10.20.12.10 180
ip route 10.20.55.0 255.255.255.0 10.20.12.10 180
ip route 10.20.80.0 255.255.255.0 10.20.12.10 180
ip route 10.20.81.0 255.255.255.0 10.20.12.10 180
ip route 10.20.82.0 255.255.255.0 10.20.12.10 180
ip route 10.20.83.0 255.255.255.0 10.20.12.10 180
ip route 10.20.84.0 255.255.255.0 10.20.12.10 180
ip route 10.20.85.0 255.255.255.0 10.20.12.10 180
ip route 10.96.33.0 255.255.255.0 10.10.20.1
ip route 172.16.2.0 255.255.255.0 10.20.12.10
ip route 172.20.64.0 255.255.192.0 10.20.12.10
ip route 172.20.120.0 255.255.255.0 10.20.12.10
ip route 172.30.0.0 255.255.0.0 10.8.10.14
ip route 192.168.101.0 255.255.255.0 10.20.12.10
ip route 192.168.103.0 255.255.255.0 10.20.32.10
ip route 192.168.106.0 255.255.255.0 10.20.62.10
ip route 192.168.252.0 255.255.255.0 10.20.32.10
ip route 199.19.190.0 255.255.255.0 10.20.62.10
ip route <public> 255.255.255.255 10.20.12.10
!
!
!
!
snmp-server community public RO
snmp-server community 5w1tch RO
snmp-server host 10.20.34.185 5w1tch
!
!
control-plane
service-policy input system-cpp-policy
!
banner login ^C
THIS DEVICE IS PART OF A PRIVATE NETWORK
*************************************************
* Unauthorised access or use of this equipment *
* is prohibited and constitutes an offence under*
* the Computer Misuse Act 1990. *
* This system is being monitored and logs will *
* be used as evidence in court. *
* If you are not authorised to use this system, *
* Terminate this session now! *
*************************************************
^C
!
line con 0
password
logging synchronous
login
stopbits 1
line vty 0 4
login local
transport input telnet ssh
line vty 5 15
login local
transport input telnet ssh
!
!
monitor session 1 source interface Gi1/0/1 - 15 , Gi1/0/18 - 24
monitor session 1 destination interface Gi1/0/16
monitor session 2 destination interface Gi1/0/17
monitor session 2 source remote vlan 111
ntp server 81.168.77.149
ntp server 194.35.252.7 prefer
!
wsma agent exec
!
wsma agent config
!
wsma agent filesys
!
wsma agent notify
!
!
end

sw1 sh ip route;

S1_WAN_SW01#sh route
% Ambiguous command: "sh route"
S1_WAN_SW01#sh ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 10.20.12.10 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 10.20.12.10
10.0.0.0/8 is variably subnetted, 53 subnets, 3 masks
D 10.1.66.0/24 [90/3072] via 10.20.12.10, 4d13h, Vlan4
C 10.8.10.8/30 is directly connected, GigabitEthernet1/0/24
L 10.8.10.9/32 is directly connected, GigabitEthernet1/0/24
D 10.10.20.16/30 [90/28416] via 10.8.10.10, 3w5d, GigabitEthernet1/0/24
C 10.10.21.8/30 is directly connected, GigabitEthernet1/0/3
L 10.10.21.9/32 is directly connected, GigabitEthernet1/0/3
D 10.18.1.0/24 [90/3072] via 10.20.12.10, 4d13h, Vlan4
D 10.19.1.0/24 [90/3072] via 10.20.12.10, 4d13h, Vlan4
C 10.20.3.0/24 is directly connected, Vlan105
L 10.20.3.6/32 is directly connected, Vlan105
D EX 10.20.3.180/32 [170/3072] via 10.20.12.10, 4d13h, Vlan4
C 10.20.4.0/24 is directly connected, Vlan106
L 10.20.4.16/32 is directly connected, Vlan106
C 10.20.7.0/24 is directly connected, Vlan109
L 10.20.7.16/32 is directly connected, Vlan109
C 10.20.10.0/24 is directly connected, Vlan2
L 10.20.10.16/32 is directly connected, Vlan2
C 10.20.11.0/24 is directly connected, Vlan3
L 10.20.11.16/32 is directly connected, Vlan3
C 10.20.12.0/24 is directly connected, Vlan4
L 10.20.12.16/32 is directly connected, Vlan4
C 10.20.13.0/24 is directly connected, Vlan5
L 10.20.13.16/32 is directly connected, Vlan5
C 10.20.14.0/24 is directly connected, Vlan6
L 10.20.14.16/32 is directly connected, Vlan6
C 10.20.15.0/24 is directly connected, Vlan7
L 10.20.15.3/32 is directly connected, Vlan7
C 10.20.16.0/24 is directly connected, Vlan9
L 10.20.16.16/32 is directly connected, Vlan9
C 10.20.17.0/24 is directly connected, Vlan10
L 10.20.17.16/32 is directly connected, Vlan10
D EX 10.20.20.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.21.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.22.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.23.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.40.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.41.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.42.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.43.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.44.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 10.20.45.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D 10.20.50.0/24 [90/3072] via 10.10.21.10, 3w5d, GigabitEthernet1/0/3
D 10.20.51.0/24 [90/3072] via 10.10.21.10, 3w5d, GigabitEthernet1/0/3
D 10.20.52.0/24 [90/3072] via 10.10.21.10, 3w5d, GigabitEthernet1/0/3
D 10.20.53.0/24 [90/3072] via 10.10.21.10, 3w5d, GigabitEthernet1/0/3
D 10.20.54.0/24 [90/3072] via 10.10.21.10, 3w5d, GigabitEthernet1/0/3
D 10.20.55.0/24 [90/3072] via 10.10.21.10, 3w5d, GigabitEthernet1/0/3
S 10.20.80.0/24 [180/0] via 10.20.12.10
S 10.20.81.0/24 [180/0] via 10.20.12.10
S 10.20.82.0/24 [180/0] via 10.20.12.10
S 10.20.83.0/24 [180/0] via 10.20.12.10
S 10.20.84.0/24 [180/0] via 10.20.12.10
S 10.20.85.0/24 [180/0] via 10.20.12.10
172.16.0.0/24 is subnetted, 1 subnets
S 172.16.2.0 [1/0] via 10.20.12.10
172.20.0.0/16 is variably subnetted, 2 subnets, 2 masks
S 172.20.64.0/18 [1/0] via 10.20.12.10
S 172.20.120.0/24 [1/0] via 10.20.12.10
192.168.50.0/32 is subnetted, 15 subnets
D EX 192.168.50.184 [170/3072] via 10.20.12.10, 10:47:50, Vlan4
D EX 192.168.50.185 [170/3072] via 10.20.12.10, 10:34:32, Vlan4
D EX 192.168.50.186 [170/3072] via 10.20.12.10, 10:14:03, Vlan4
D EX 192.168.50.188 [170/3072] via 10.20.12.10, 00:06:56, Vlan4
D EX 192.168.50.194 [170/3072] via 10.20.12.10, 09:36:08, Vlan4
D EX 192.168.50.195 [170/3072] via 10.20.12.10, 05:04:18, Vlan4
D EX 192.168.50.197 [170/3072] via 10.20.12.10, 09:30:53, Vlan4
D EX 192.168.50.199 [170/3072] via 10.20.12.10, 09:30:11, Vlan4
D EX 192.168.50.205 [170/3072] via 10.20.12.10, 09:07:02, Vlan4
D EX 192.168.50.206 [170/3072] via 10.20.12.10, 09:00:24, Vlan4
D EX 192.168.50.207 [170/3072] via 10.20.12.10, 00:05:38, Vlan4
D EX 192.168.50.211 [170/3072] via 10.20.12.10, 07:55:35, Vlan4
D EX 192.168.50.215 [170/3072] via 10.20.12.10, 04:38:44, Vlan4
D EX 192.168.50.219 [170/3072] via 10.20.12.10, 04:06:08, Vlan4
D EX 192.168.50.221 [170/3072] via 10.20.12.10, 02:33:20, Vlan4
S 192.168.101.0/24 [1/0] via 10.20.12.10
D EX 192.168.103.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
D EX 192.168.106.0/24 [170/3072] via 10.20.12.10, 4d13h, Vlan4
<public>/32 is subnetted, 1 subnets
S <public> [1/0] via 10.20.12.10

SW 2 sh run;

S1_WAN_SW02#sh run
Building configuration...

Current configuration : 6955 bytes
!
! Last configuration change at 14:27:21 BST Fri Apr 16 2021 by corpadmin
! NVRAM config last updated at 11:57:52 BST Fri Apr 16 2021 by corpadmin
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname S1_WAN_SW02
!
boot-start-marker
boot-end-marker
!
enable secret
!
username corpadmin privilege 15 secret
username sup-tbase privilege 15 secret
no aaa new-model
clock timezone gmt 0
clock summer-time BST recurring last Sun Mar 2:00 last Sun Oct 2:00
system mtu routing 1500
ip subnet-zero
ip routing
no ip domain-lookup
ip domain-name
!
!
!
!
crypto pki trustpoint TP-self-signed-3107739776
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-3107739776
revocation-check none
rsakeypair TP-self-signed-3107739776
!
!
crypto pki certificate chain TP-self-signed-3107739776
certificate self-signed 01
30820254 308201BD A0030201 02020101 300D0609 2A864886 F70D0101 04050030
31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 33313037 37333937 3736301E 170D3933 30333031 30303031
32305A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D33 31303737
33393737 3630819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
8100BB0C 217424CC 046A7659 13A0BBB5 A05CC71F DD0DAF5E B24A1290 FAA744F8
6CF0346E 654C8D89 A6300038 1E28F06D CA760E46 B94E2639 30FE4E61 71A0F6B1
46AB58F0 FA26D456 9F11E20C 766AD9A1 EE5225EE E27BA627 F9122F68 9A877E70
50724303 00DAA45E 2B51FD9C B09CA379 116389FF 0773A3F0 BF34F3CE 8BE09E38
16C50203 010001A3 7C307A30 0F060355 1D130101 FF040530 030101FF 30270603
551D1104 20301E82 1C53315F 57414E5F 53573032 2E726563 636F7270 2E696E74
65726E61 6C301F06 03551D23 04183016 8014E5E1 282FE65E 44C2B99B 4834557B
5ECA2E7D C6B9301D 0603551D 0E041604 14E5E128 2FE65E44 C2B99B48 34557B5E
CA2E7DC6 B9300D06 092A8648 86F70D01 01040500 03818100 34E20F71 F76FC7F1
69C00AE0 9D2F94D3 B3E782EC A99783EB 87E1CCAE 5366FCD2 07DAEE82 EBDE6280
A1AFAA4A 7B91F2FB 0A5BD5B9 566532CB 67CB6460 8C60884B D6D8499C 7570A2AA
AB756DB2 989F7A76 F079CFD9 D3FD6FA5 0345F989 A6E19EFE EA47B9C6 2AC82E4F
3D077097 0FA8E4AA 12360C81 8E1828A2 0B1D1D86 5A128333
quit
!
!
!
!
!
archive
path ftp://backup
write-memory
time-period 1440
spanning-tree mode pvst
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
!
!
interface FastEthernet0
no ip address
no ip route-cache cef
no ip route-cache
no ip mroute-cache
shutdown
!
interface GigabitEthernet0/1
description
no switchport
ip address 10.10.21.2 255.255.255.252
delay 2
shutdown
!
interface GigabitEthernet0/2
shutdown
!
interface GigabitEthernet0/3
shutdown
!
interface GigabitEthernet0/4
shutdown
!
interface GigabitEthernet0/5
shutdown
!
interface GigabitEthernet0/6
description Link to S1_RT01
no switchport
ip address 10.8.10.5 255.255.255.252
!
interface GigabitEthernet0/7
shutdown
!
interface GigabitEthernet0/8
shutdown
!
interface GigabitEthernet0/9
shutdown
!
interface GigabitEthernet0/10
shutdown
!
interface GigabitEthernet0/11
shutdown
!
interface GigabitEthernet0/12
description Link to S1_BE01_STACK
no switchport
ip address 10.8.10.17 255.255.255.252
!
interface GigabitEthernet0/13
shutdown
!
interface GigabitEthernet0/14
shutdown
!
interface GigabitEthernet0/15
shutdown
!
interface GigabitEthernet0/16
shutdown
!
interface GigabitEthernet0/17
shutdown
!
interface GigabitEthernet0/18
description Link to S6 - S6_FL06_SW01
no switchport
ip address 10.10.20.18 255.255.255.252
speed 100
duplex full
!
interface GigabitEthernet0/19
shutdown
!
interface GigabitEthernet0/20
shutdown
!
interface GigabitEthernet0/21
shutdown
!
interface GigabitEthernet0/22
shutdown
!
interface GigabitEthernet0/23
shutdown
!
interface GigabitEthernet0/24
description Link to S1_WAN_SW01
no switchport
ip address 10.8.10.10 255.255.255.252
!
interface GigabitEthernet0/25
description Link1 to S13_WAN_SW01
no switchport
ip address 10.8.10.73 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet0/26
description Link2 to S13_WAN_SW01
no switchport
ip address 10.8.10.77 255.255.255.252
speed nonegotiate
!
interface GigabitEthernet0/27
shutdown
!
interface GigabitEthernet0/28
shutdown
!
interface TenGigabitEthernet0/1
shutdown
!
interface TenGigabitEthernet0/2
shutdown
!
interface Vlan1
no ip address
shutdown
!
!
router eigrp 100
no auto-summary
network 0.0.0.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.20.12.10 permanent
ip route 10.1.0.0 255.255.0.0 10.8.10.18
ip route 10.20.40.0 255.255.255.0 10.20.12.10 180
ip route 10.20.41.0 255.255.255.0 10.20.12.10 180
ip route 10.20.42.0 255.255.255.0 10.20.12.10 180
ip route 10.20.43.0 255.255.255.0 10.20.12.10 180
ip route 10.20.44.0 255.255.255.0 10.20.12.10 180
ip route 10.20.45.0 255.255.255.0 10.20.12.10 180
ip route 10.20.80.0 255.255.255.0 10.20.12.10 180
ip route 10.20.81.0 255.255.255.0 10.20.12.10 180
ip route 10.20.82.0 255.255.255.0 10.20.12.10 180
ip route 10.20.83.0 255.255.255.0 10.20.12.10 180
ip route 10.20.84.0 255.255.255.0 10.20.12.10 180
ip route 10.20.85.0 255.255.255.0 10.20.12.10 180
ip route 172.16.2.0 255.255.255.0 10.20.12.10
ip route 172.20.64.0 255.255.192.0 10.20.12.10
ip route 172.20.120.0 255.255.255.0 10.20.12.10
ip route 192.168.101.0 255.255.255.0 10.20.12.10
ip route 192.168.103.0 255.255.255.0 10.20.32.10
ip route 192.168.106.0 255.255.255.0 10.20.62.10
ip route 192.168.252.0 255.255.255.0 10.20.32.10
ip route 199.19.190.0 255.255.255.0 10.20.62.10
ip route <public> 255.255.255.255 10.20.12.10
ip http server
ip http secure-server
!
!
logging trap notifications
logging 10.20.32.22
logging 10.20.3.57
!
snmp-server community 5w1tch RO
snmp-server host 10.20.34.185 5w1tch
!
control-plane
!
banner login ^C
THIS DEVICE IS PART OF A PRIVATE NETWORK
*************************************************
* Unauthorised access or use of this equipment *
* is prohibited and constitutes an offence under*
* the Computer Misuse Act 1990. *
* This system is being monitored and logs will *
* be used as evidence in court. *
* If you are not authorised to use this system, *
* Terminate this session now! *
*************************************************
^C
!
line con 0
login local
line vty 0 4
login local
transport input telnet ssh
line vty 5 15
login local
transport input telnet ssh
!
!
monitor session 2 source interface Gi0/1 - 28
monitor session 2 destination remote vlan 111
ntp clock-period 36027365
ntp server 81.168.77.149
ntp server 194.35.252.7 prefer
end

sh ip route;

S1_WAN_SW02#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.20.12.10 to network 0.0.0.0

D EX 192.168.106.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
213.212.125.0/32 is subnetted, 1 subnets
S 213.212.125.48 [1/0] via 10.20.12.10
172.16.0.0/24 is subnetted, 1 subnets
S 172.16.2.0 [1/0] via 10.20.12.10
172.20.0.0/16 is variably subnetted, 2 subnets, 2 masks
S 172.20.120.0/24 [1/0] via 10.20.12.10
S 172.20.64.0/18 [1/0] via 10.20.12.10
10.0.0.0/8 is variably subnetted, 40 subnets, 3 masks
C 10.8.10.8/30 is directly connected, GigabitEthernet0/24
C 10.10.20.16/30 is directly connected, GigabitEthernet0/18
D EX 10.20.22.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.23.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.20.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.21.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D 10.20.16.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.17.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.14.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.15.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.12.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.13.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.10.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.11.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.19.1.0/24 [90/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D 10.20.7.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.18.1.0/24 [90/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D 10.20.4.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.20.3.0/24 [90/3072] via 10.8.10.9, 3w5d, GigabitEthernet0/24
D 10.10.21.8/30 [90/3072] via 10.8.10.9, 3w6d, GigabitEthernet0/24
D 10.20.54.0/24 [90/3328] via 10.8.10.9, 3w6d, GigabitEthernet0/24
D 10.20.55.0/24 [90/3328] via 10.8.10.9, 3w6d, GigabitEthernet0/24
D 10.20.52.0/24 [90/3328] via 10.8.10.9, 3w6d, GigabitEthernet0/24
D 10.20.53.0/24 [90/3328] via 10.8.10.9, 3w6d, GigabitEthernet0/24
D 10.20.50.0/24 [90/3328] via 10.8.10.9, 3w6d, GigabitEthernet0/24
D 10.20.51.0/24 [90/3328] via 10.8.10.9, 3w6d, GigabitEthernet0/24
D EX 10.20.44.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.45.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.42.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.43.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.40.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D EX 10.20.41.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
D 10.1.66.0/24 [90/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
S 10.20.84.0/24 [180/0] via 10.20.12.10
S 10.20.85.0/24 [180/0] via 10.20.12.10
S 10.20.82.0/24 [180/0] via 10.20.12.10
S 10.20.83.0/24 [180/0] via 10.20.12.10
S 10.20.80.0/24 [180/0] via 10.20.12.10
S 10.20.81.0/24 [180/0] via 10.20.12.10
D EX 10.20.3.180/32 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
192.168.50.0/32 is subnetted, 15 subnets
D EX 192.168.50.219 [170/3328] via 10.8.10.9, 04:10:54, GigabitEthernet0/24
D EX 192.168.50.221 [170/3328] via 10.8.10.9, 02:38:06, GigabitEthernet0/24
D EX 192.168.50.211 [170/3328] via 10.8.10.9, 08:00:21, GigabitEthernet0/24
D EX 192.168.50.215 [170/3328] via 10.8.10.9, 04:43:30, GigabitEthernet0/24
D EX 192.168.50.206 [170/3328] via 10.8.10.9, 09:05:10, GigabitEthernet0/24
D EX 192.168.50.207 [170/3328] via 10.8.10.9, 00:10:23, GigabitEthernet0/24
D EX 192.168.50.205 [170/3328] via 10.8.10.9, 09:11:49, GigabitEthernet0/24
D EX 192.168.50.194 [170/3328] via 10.8.10.9, 09:40:55, GigabitEthernet0/24
D EX 192.168.50.195 [170/3328] via 10.8.10.9, 05:09:04, GigabitEthernet0/24
D EX 192.168.50.199 [170/3328] via 10.8.10.9, 09:34:58, GigabitEthernet0/24
D EX 192.168.50.197 [170/3328] via 10.8.10.9, 09:35:40, GigabitEthernet0/24
D EX 192.168.50.186 [170/3328] via 10.8.10.9, 10:18:50, GigabitEthernet0/24
D EX 192.168.50.184 [170/3328] via 10.8.10.9, 10:52:37, GigabitEthernet0/24
D EX 192.168.50.185 [170/3328] via 10.8.10.9, 10:39:18, GigabitEthernet0/24
D EX 192.168.50.188 [170/3328] via 10.8.10.9, 00:11:41, GigabitEthernet0/24
D EX 192.168.103.0/24 [170/3328] via 10.8.10.9, 4d13h, GigabitEthernet0/24
S 192.168.101.0/24 [1/0] via 10.20.12.10
S* 0.0.0.0/0 [1/0] via 10.20.12.10

sh standby br looks correct as well, with both switches talking as expected, I've since added the prio 150 to sw1 so it forces itself for vlan105 active and that restored connectivity to vlan 105 from offsite (gi1/0/3 on sw1, on 10.10.21.8/30)

thanks for looking at it.