Symptoms
ACI uses shared l3out to provide shared services for multiple consumer vrfs. This is achieved via leaking the external subnet from provider vrf. When OSPF is selected to exchange the route between border leaf and exteranl router in consumer vrf, where that provider vrf is also deployed on the same leaf, from the consumer leaf perspective, the external subnet is treated as eBGP. Since eBGP had administrative distance 20 which is preferred than OSPF even the same subnet is learned over OSPF from a different source, shared l3out leaked route would provide a backdoor internally so that external implemented path is bypassed.
Further, it could cause an OSPF flapping due to LSA flush via maxage in a certain use case below. When 192.168.103.1/32 become the DR o f the 192.168.103.0/24 subnet for common_vrf, it would genrate type-2 LSA to firewall. In parallel, because of the shared l3out, 192.168.103.1/32 could also be leaked to user_vrf via shared-l3out, because 192.168.103.1/32 is reachable locally via another VRF, URIB would tag this particular route 192.168.103.1/32 as local. Hence Rtr-ID: 5.103.103.1 would believe it-self as the originator of the type-2 LSA generated by 3.103.103.1. However because the type-2 LSA received from North is more recent because higher age, 5.103.103.1 is flushing the type-2 LSA via max-age and flood back. That would trigger firewall remove all the path to DR and routes learned via DR.
Diagnosis
1. How does the type-2 LSA look like:
leaf103# show ip ospf database network 192.168.103.1 detail vrf common:common_vrf
OSPF Router with ID (4.0.3.16) (Process ID default VRF welkin:inside)
Network Link States (Area 0.0.0.15)
LS age: 1369
Options: 0x2 (No TOS-capability, No DC)
LS Type: Network Links
Link State ID: 192.168.103.1 (Designated Router address)
Advertising Router: 3.103.103.1
LS Seq Number: 0x8000002e
Checksum: 0x2029
Length: 36
Network Mask: /24
Attached Router: 3.103.103.1
Attached Router: 3.104.104.1
Attached Router: 192.168.103.3
2. user_vrf is max-aging the LSA
The type-2 LSA originated by common vrf is treated as sef-orginated by user_vrf so it was maxaged because the recieved one is more recent, this is wrong.
2018 Jan 5 09:55:28.921944 ospf default [10356]: TID 11119:ospf_ha_lsdb_update:4144:(user:user_vrf-base) ObjStore entry for LSA 192.168.103.1(0x2)3.103.103.1 (0x80000111) (0x570e) (3600) area 0.0.0.15 (if-none) updated
2018 Jan 5 09:55:28.921876 ospf default [10356]: TID 11119:ospf_ha_lsdb_update:4144:(user:user_vrf-base) ObjStore entry for LSA 192.168.103.1(0x2)3.103.103.1 (0x80000111) (0x570e) (3600) area 0.0.0.15 (if-none) updated
2018 Jan 5 09:55:28.921796 ospf default [10356]: TID 11119:ospf_ha_lsdb_update:4144:(user:user_vrf-base) ObjStore entry for LSA 192.168.103.1(0x2)3.103.103.1 (0x80000111) (0x570e) (7) area 0.0.0.15 (if-none) updated
2018 Jan 5 09:55:28.921722 ospf default [10356]: TID 11119:ospf_update_lsdb_entry:1661:(user:user_vrf-base) LSA already exists, updating
2018 Jan 5 09:55:28.921713 ospf default [10356]: TID 11119:ospfv2_overrule_stale_self_originated_lsa:1292:(user:user_vrf-base) Received self-originated LSA
2018 Jan 5 09:55:28.921673 ospf default [10356]: TID 11119:ospfv2_process_newer_lsa:1922:(user:user_vrf-base) LSA is more recent
2018 Jan 5 09:55:28.921665 ospf default [10356]: TID 11119:ospfv2_process_incoming_lsa:2330:(user:user_vrf-base) LSA 192.168.103.1(0x2)3.103.103.1 (0x80000111) (0x570e) (7) from 192.168.105.3
The problem is caused because 192.168.103.1/32 is local from the user:user_vrf, which has mis-leading the ospf believe 5.103.103.1 as the originator.
leaf103# vsh -c "show ip route 192.168.103.0 vrf user:user_vrf"
192.168.103.0/24, ubest/mbest: 1/0, attached, direct
*via 192.168.103.1%common:common_vrf, Vlan5, [20/0], 01:18:24, bgp-65000, external, tag 65000
leaf103# vsh -c "show ip route 192.168.103.1/32 vrf user:user_vrf"
192.168.103.1/32, ubest/mbest: 1/0, attached
*via 192.168.103.1%common:common_vrf, Vlan5, [1/0], 01:18:50, local, attached-export, local
3. Why OSPF believe 5.103.103.1 as the originator
If the LSA's header 192.168.103.1/32 belong to router-id or any local network, OSPF would think itself as the originator flush the LSA. This has made 5.103.103.1 believe himself was the originator and kicked in the LSA-flush.
Solution:
We will need prevent ACI from leaking the OSPF SVI subnet into user vrf via shared-list, this is an unexpected configuration which can be addressed by:
1. stop aggregate share subnet 0.0.0.0.0/0
2. limit the contract provider and consumer.
