vPC + HSRP + OSPF load-balancing issue

[satish.txt1]

I am playing with new cisco cml 2.0 lab simulator and seeing strange behavior not sure if its real issue or limitation of cisco lab.

dist1 and dist2 is my vPC (NSOX 9000v) peers, and tor1 is IOSv switch configured for port-channel.

dist1

interface Vlan100
  description **** Public VLAN ****
  no shutdown
  no autostate
  no ip redirects
  ip address 70.70.70.2/23
  no ipv6 redirects
  ip router ospf 1 area 0.0.0.0
  hsrp version 2
  hsrp 100 
    preempt 
    priority 110
    ip 70.70.70.1

dist2

interface Vlan100
  description **** Public VLAN ****
  no shutdown
  no autostate
  no ip redirects
  ip address 70.70.70.3/24
  no ipv6 redirects
  ip router ospf 1 area 0.0.0.0
  hsrp version 2
  hsrp 100 
    preempt 
    ip 70.70.70.1

vpc config (same config on both dist1/2 except peer-keepalive IP)

vpc domain 1
  peer-switch
  peer-keepalive destination 172.29.1.2 source 172.29.1.1
  peer-gateway
  ip arp synchronize

interface port-channel10
  vpc 10

interface port-channel999
  vpc peer-link

show arp (on both dist1/2)

dist1# show mac address-table address 0000.0c9f.f064
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
G  100     0000.0c9f.f064   static   -         F      F    sup-eth1(R)

dist2 (its pointing to vPC Peer-Link(R) is that correct?)

dist2# show mac address-table address 0000.0c9f.f064
Legend: 
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
G  100     0000.0c9f.f064   static   -         F      F    vPC Peer-Link(R)

core1 and core2 (connected to dist1/2 over L3 using OSPF, point to point link)

when server1 ping internet IP in that case my packet always take dist1 path only, it never goes over dist2 (until unless i make it active HSRP using priority), i did change port-channel load-balancing method to src-dst-ip and also changed my server-1 IP address to see if it makes any difference but no.

what could be wrong here?

when i shutdown interface on tor1 connected to dist1 to see behavior and i found now my packet going to dist2 and from dis2 to dist1 and then going out to core1 and core1, so still dist1 is in path (is this because dist1 is active HSRP?)

This is the path its taking: 

server---->tor1---->dist2---->dist1---->[core1/2 ECMP]

But its interesting when i increase OSFP cost on dist1 interfaces connected to core1 and core2 then my traffic going from dist2 to core1/2 like following, what is the roles here of OSFP influnacing HSRP vPC load-balancing. 

server---->tor1---->dist2----->[core1/2 ECMP]

[Sergiu.Daniluk]

Hi @satish.txt1 

Few things that I want to ask you:

+ is TOR1 connected to dist1 and dist2 using vPC?

+ is the OSPF adjacency over a vPC enabled port-channel?

• if yes, then you need to configure "layer3 peer-router" under the vpc domain.

+ you do not need preempt on vPC SVIs. It does not have any effect since the vPC peers will be active/active in forwarding.

+ "server---->tor1---->dist2---->dist1---->[core1/2 ECMP]" does not look good. How does the routing table looks like on both dist switches?

+ reg the HSRP vMAC, that looks ok since it has the G flag. This means that it will locally route the traffic destined to the vMAC.

Regards,

Sergiu

[satish.txt1]

I have similar design running on production with live traffic and exact same configuration command-to-command i have verify and i can see all good, my packets from server to internet doing load-balancing between dist1/2. 

I am trying to deploy same production model on CML 2.0 lab to do more hands on fun which i can't do on production. 

+ is TOR1 connected to dist1 and dist2 using vPC?

- Yes, TOR1 is configured for vPC (MLAG)

+ is the OSPF adjacency over a vPC enabled port-channel?

- I have vlan100 configured with HSRP on both switches for active-active function + i have same VLAN 100 configured on vpc-peerlink between two switches and i can see OSPF showing those IPs in neighboor relation ship, so i can see that its using vpc-peerlink for OSPF adjacency (In production i don't have layer3 peer-router command so wonderting how things working there?)  what will happened if i don't use that command?

+ "server---->tor1---->dist2---->dist1---->[core1/2 ECMP]" does not look good. How does the routing table looks like on both dist switches?

- I am also surprised with that behavior, may be CML 2.0 isn't smart enough or possible glitch in software, i have default route on both switches toward core1/2.  

[paul driver]

Hello

---

@satish.txt1 wrote:

But its interesting when i increase OSFP cost on dist1 interfaces connected to core1 and core2 then my traffic going from dist2 to core1/2 like following, what is the roles here of OSFP influnacing HSRP vPC load-balancing. 

server---->tor1---->dist2----->[core1/2 ECMP]

---

Obviously as you have manipulated the ospf cost the routing path decision now takes dist2 from the dist vPC , But I think what you are expecting to see is the path to take   server---->tor1---dist1---->dist2-->  because of that node being the hsrp active device - correct?

But I dont think this would be necessarily the case because both of those nodes will can respond to its advertised hsrp group vip mac address for which tor1 (in a vpc) has in its arp table as shown in you post showing both vip macd having the G flag set.

---

@satish.txt1 wrote:

dist1# show mac address-table address 0000.0c9f.f064

---------+-----------------+--------+---------+------+----+------------------
G  100     0000.0c9f.f064   static   -         F      F    sup-eth1(R)

G  100     0000.0c9f.f064   static   -         F      F    vPC Peer-Link(R)

So as you have manually changed the ospf costing its possible tor1 has recalculated its hashing to chose dist2.

Lastly on a side note, Ive been running CML2.0 since it came out and TBH i'm not that impressed with it, Getting to many anomalies, weird outputs, features not working correctly etc.. and now i'm at a point of not trusting it as a POC for lab simulation tool

[satish.txt1]

I am similar solution deployed in production and everything works correct and you gave me confident that CML 2.0 has some glitch (I am totally agree with you, i have seens some strange behavior which i can't explain here, i thought i am doing stupid but look like this software isn't matured)

I have eve-ng LAB also, i will try to reconfigure same setup there and will see if getting same behavior or not. it would be good to find out.

[paul driver]

Hello

Please share the results on your testing, would be interesting to see.

---