Nexus 7010 mgmt0 useage opinion

[Adrian Jenkinson]

As a Senior Network Engineer I have entered into a bit of a debate with our Architect about the use of the mgmt0 interfaces on the nexus 7010 switch (dual-sups, M2 and F2 linecards).

I would like to know opinion of the Cisco support network.

I believe the mgmt0 interface should left alone for control plane traffic only and Out Of Band management access (ie ssh).  At the moment I have made a subnet for all VDCs with the mgmt0 (vrf management) sitting in a common subnet.  The physical mgmt0 interfaces from both SUPs are connected a management hand off switch.  The mgmt0s also serves as our control plane for VPCs. The VPC peer-link however is using main interfaces of the line-cards.

The opinions;

- The Architect thinks we should use all the mgmt0 interfaces for snmp, ntp, tacacs netflow-analysis and switch management.

- However, I think I should use a traditional Loopback to perform these functions within the linecards.  The mgmt0 should only be used if traditional restricted switch access has failed.

My Basis;

the Loopback never goes down, uses multiple paths (the OOB hand off switch could fail closing switch management access completely).  The mgmt0 should be used as a last resort of management access to CMP.

Thoughts please - Cheers

[Marvin Rhoads]

Hmm,

I'd be more with your architect on this one. I'd recommend the mgmt0 port for VPC peer-keepalive plus all the other traditional management features cited as attributed to your architect above. That approach is also the Cisco recommendation per the Smart Business Architecture guide series. (Reference)

[Adrian Jenkinson]

I already popped the VPC keep-alives in the mgmt0 interfaces - as per recommended guidelines.  However, I'm not happy about it - as the OOB switch for mgmt0 SUP interface is singular. 

In order to mitigate this I did;

switch1

SUP0 mgmt0 > OOB1 fa0/1

SUP1 mgmt0 > OOB2 fa0/1

switch2

SUP0 mgmt0 > OOB1 fa0/2

SUP1 mgmt0 > OOB2 fa0/2

Is their a way of tracking mgmt0 interface state? If SUP0 mgmt0 interface goes down (OOB1 failure)

the redundant (SUP1) unit takes over?

[Marvin Rhoads]

I see your point about wanting to mitigate the impact of losing the OOB switch. I don't think the mgmt0 interface going down is considered the level of failure that will trigger a Supervisor switchover though. That's the way I read the Nexus 7000 HA whitepaper (and what I've seen based on some limited experience with taking apart a 7k pair).

So, no the 7k can't send you an SNMP trap or syslog message if it's configured management path is offline. Mitigation of that could be via your NMS polling the devices's mgmt0 addresses. No response = trouble in paradise. Investigation step would be to log into the 7ks using the loopback IP and local authentication since your TACACS source-interface (mgmt0) is offline and going from there.

The handful I've built (mostly 5k setups) I go for a Cat 3k switch with dual power supplies as the OOB switch. Once one of those is setup and seen not to be DOA, it's generally going to stay up until someone goes in and uplugs it or initiates a system reload.