6880-x and 6800IA snmp monitoring of down switch stacks

Ethan Grinnell · ‎01-20-2016

Hello,

We are deploying 6880-X switches with stacks of 6800IAs attached on FEXs. We've found that the stacking snmp MIBs that we use to monitor our stacks of 3750 switches are not supported on the 6880s for the 6800IA switches. The 3750s use the CISCO-STACKWISE-MIB. On Cisco's MIB browser I see that the the 6880s do not support the CISCO-STACKWISE-MIB, however, they are supposed to support CISCO-STACK-MIB, but I can't seem to find the commands to enable it on the 6880-X switches. Has anyone been able to monitor stacks of 6800IA switches? "snmp-server enable traps stackwise" is not present at the command line. Is there a different location in the configuration that I need to enable snmp for the IA switch stacks?

I've tried turning on most of the traps with "snmp-server enable traps" and used snmpwalk to parse the entire tree. Then compared the results with diff to see what changes when a switch stack is up versus down. Nothing really great was there.

The best method I've found is to compare the results of these two commands , if they don't match there is a switch stack down:

snmpwalk -v2c -c community x.x.x.x ifDescr | grep -e "GigabitEthernet1"[0123456789][0123456789] -e mgmt -e TenGigabitEthernet | wc -l
snmpwalk -v2c -c community x.x.x.x dot3StatsIndex | wc -l

But it's inefficient and only works for our particular implementation. If different modules were added to the switches I'd need to change the query.

There's got to be a better way, what am I missing?

Thanks,

Ethan

Joe Clarke · ‎02-04-2016

I don't have any experience with IA management, but on modern versions of Cat6K IOS, the ENTITY-MIB and its companions should be used over the STACK-MIB. If you walk the entPhysicalTable on the 6880-X, do you see the IAs as having rows within that table?

Ethan Grinnell · ‎02-06-2016

Hello Joe,

The 6880 does support the ENTITY-MIB and the entPhysicalTable. The 6800IAs that are online show up in the list with all their associated ports. However, what is missing are the ones that are not up. To see if this should be an error condition I'd have to know how many IAs/Ports should be there or somehow track the previous state(s). When I did my testing I couldn't find an indication in the tables for how many devices were configured or expected. The information could very well be there, but I was parsing 29k entries. I could have easily missed something.

The thought process in my Linux command strings are that the first one determines how many physical interfaces are present on the switch. Each of those should have Ethernet statistics. The second one determines how many sets of Ethernet statistics actually exist. When an IA is configured but not online it has no statistics, the entry for it doesn't exist. With this logic if there is a difference in configured interfaces and actual stats then something's not right. As far as I saw in the simple testing this meant a stack of IA's was either not up entirely or had a missing device. It works, but honestly it sucks ;)

The CISCO-STACKWISE-MIB is so much better. With that you can see if a device is configured but not online. Then it's status shows as "Provisioned". With a single SNMP query for each configured stack member you can see if it is operational. Also, with that set of MIBS you can look into the stack to see if the stack ports are up as well. With support like what the 3750 has show outputs below of a 6800IA stack could generate an error.

FEX-138#sh switch st
FEX-138#sh switch stack-p
FEX-138#sh switch stack-ports
Switch #    Port 1       Port 2
--------    ------       ------
    1           Ok           Ok
    2           Ok          Down
    3           Ok           Ok
    4          Down          Ok
    5           Ok           Ok

FEX-138#reload slot 5
Proceed with reload? [confirm]

FEX-138#sh switch
Switch/Stack Mac Address : ac7e.8acd.4200
                                           H/W   Current
Switch# Role   Mac Address     Priority Version State
----------------------------------------------------------
*1       Master ac7e.8acd.4200     1      4       Ready
2       Member 0000.0000.0000     0      0       Removed
3       Member dceb.9468.0b80     1      4       Ready
4       Member dceb.9468.0880     1      4       Ready
5       Member 0000.0000.0000     0      0       Removed

I can more or less catch the second one, but I have no idea how to catch the first one.

As is, I took those two Linux commands and built a Nagios module, it works, and is better than nothing. I think I've expressed how much I like my methods though.

Thanks for the reply,

Ethan

Joe Clarke · ‎02-06-2016

I had thought the IAs might be included if they were provisioned, even if they were offline. Again, I haven't done anything with IAs.

Enabling the "fru-cntl" traps will give you a heads up when an IA goes away via the CISCO-ENTITY-FRU-CONTROL-MIB. There is a bug, however, that a trap will also be generated for every PoE port that goes down on the IAs. This will be fixed in the next 6880-X release (CSCut52427).

Ethan Grinnell · ‎02-06-2016

It doesn't look like this is the case.

Thank you for the suggestion, i'll tinker with that to see if I can use that to do some of what I'm after. I also have a TAC case open, hopefully they can show me how to do it, or submit a software request for it if it isn't already possible.

Thanks,

Ethan

Ethan Grinnell · ‎02-07-2016

Searching the bug reports I found mention of a feature request to support the stackwise MIBs on the 6800IAs. I guess it's on the way.

https://tools.cisco.com/bugsearch/bug/CSCur01082

oscar.finra · ‎10-03-2016

Ethan,

Have you by chance found a solution for monitoring IA6880 stack cable status? Also having issues monitoring down ports:

FEX#sh switch stack-ports
Switch # Port 1 Port 2
-------- ------ ------
1 Ok Ok
2 Ok Ok
3 Ok Down
4 Down Ok

Thanks

Ethan Grinnell · ‎10-03-2016

I was recently told by TAC that they don't want to implement the Stackwise-MIB, but they can finally put some resources into this and they will add monitoring of the FEX devices through ENTITY-MIB/CISCO-ENTITY-FRU-CONTROL-MIB https://bst.cloudapps.cisco.com/bugsearch/bug/CSCva32946/?referring_site=bugquickviewreIm I'm not sure if this will also include stack port status though.