Issues with SNMP
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2017 08:00 AM - edited 03-01-2019 05:18 AM
Hi
I am trying to get snmp monitoring setup so that I can monitor the controllers, leafs and spines. Getting it working for spines and leafs seems to be straightforward but not so for the controllers. I need this to work using OOB management and I have created an oob contract with the udp/161 filter, as suggested in various documents I have read, but I then lose access to the APICs.
What I'm looking for is a definitive step-by-step guide to setting up snmp for both controllers and leaf/spines. Is there such a thing? If not, can anyone share how they have setup snmp for their APICs and leafs/spines?
Many thanks
Roy
- Labels:
-
Cisco ACI
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2017 08:34 AM
Roy,
Check out my document here:
Technote: SNMP in the ACI Fabric
https://supportforums.cisco.com/document/12961146/technote-snmp-aci-fabric
T.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2017 11:46 AM
Tomas
Thanks for the links. I had already looked at these and although they are comprehensive and I have followed them, I still have issues.
Does it make any difference if snmp is configured to go through the oob management or is inband management better? I have set my oob contract with filters for udp/161 and udp/162 but still nothing. The leaf/spines are all fine but the controllers just will not talk to my management server.
Thanks
Roy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2017 06:46 PM
Management preference for you and your environment is a decision best made by your networking team. That said, what ever management strategy that you choose does matter.
In regards to INB vs. OOB, remember INB & OOB is for “management” of ACI nodes. That is APICs, Leaf nodes, Spine Nodes, etc…
External Data collectors like Syslog, SNMP, and Callhome features communicate to the ACI nodes via INB, OOB, or Both.
If both in-band and out-of-band management policies for ACI nodes (APIC\Leaf\Spine) are configured correctly & successfully deployed; the in-band management interface is the preferred management interface. Even if you configure a policy to use the out-of-band management EPG, the ACI node will still use the in-band management interface.
The exceptions to the in-band management connectivity preference are:
* The Destination IP address for the policy resides in the out-of-band management network.
* The ACI Node receives a request packet destined to the node's IP address of the out-of-band management interface. The ACI Node will reply to the request using the out-of-band management interface.
Originally, there was no way to change this management connectivity preference for ACI nodes. So in the latest versions ACI firmware, a feature was added to toggle between in-band and out-of-band connectivity on the APIC server for management access to devices such as authentication servers or SNMP servers external to the ACI fabric. A requirement for this feature for the toggling between in-band and out-of-band connectivity to have effect, Both in-band or out-of-band management must already be configured.
Note: As noted above, this toggling between in-band and out-of-band connectivity feature will only change the management connectivity preference on the APIC controllers not the leaf or spine nodes. The leaf & spine nodes will continue to use the in-band management connectivity preference.
In troubleshooting your issue, here are some things to try:
- if you are using INBAND management with OOB, disable INB management by simply unconfiguring INB static node management addresses for all of the nodes. Then make sure the management EPG for your SNMP configuration is set to OOB.
- For OOB & INB, make sure that you have Static node management addresses configured for ALL nodes including the APICs. Most ACI Administrators forget to configure static node management addresses for APICs since they configured them during the setup script.
- For contracts for the OOB, use the "default" common filter to allow all for the testing of the feature. Also, use 0.0.0.0 for the subnet on the external management instance for testing feature.
- Make sure that your SNMP Collector's IP is configured in the SNMP Client groups in the SNMP fabric policies.
- Are you using SNMP v2 or v3? If v3 make sure all of the v3 parameters are configured.
- What does the following commands show on the APIC(s):
- netstat -aon | grep ":161"
- netstat -lp | grep "snmp"
- show snmp clientgroups ( you can xxxx out the IP addresses)
- moquery -c vzEntry | egrep -A 2 "161"
- moquery -c vzEntry | egrep -A 2 "162"
- moquery -c mgmtSubnet
- moquery -c mgmtRsOoBCons
- moquery -c vzOOBBrCP
- moquery -c snmpRsEpg | egrep "snmp.RsEpg|dn|tDn"
- ps aux | grep snmp
Also what version are you running?
Thanks
T.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-27-2017 06:15 AM
Tomas
Just so you know we are running v.2.1.1h.
I think I have it sorted now. It seems the oob EPG and external Management Network Instance does not like a custom contract. Once I replaced my own contract with the common/default I can now connect to the APICs form my snmp server.
thanks for the help
Roy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-27-2017 07:40 AM
Roy,
I am glad that it is working. For this question and for future questions, please mark your questions answered so that others can see the results for similar issues.
Thanks
T.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-29-2017 07:31 AM
Hey Roy,
I know you already solved the issue, just wanted to clarify the contract issue.
You can 100% create an oob contract and use it in the mgmt tenant to permit traffic. It sounds like when you were doing this though, you may have only added the filter for SNMP and not allowed any other traffic as you said you lost connectivity to your APICs. If you want to use a contract in the mgmt tenant, I would:
- Create the contract, naming it something other than default to avoid overlap with the common tenant
- Add an allow_all filter and the SNMP filter.
Hope that helps clear up the issue
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2023 04:48 PM
Hello, I am running into the same issue as the user above.
I have SNMP configured and polling successfully for my APIC's but I am unable to do so for my leaf/spine switches.
Per the above, I have 0.0.0.0/0 set as the subnet for the External Management Network Instance Profile and my OOB contract is permitting any.
My nodes are configured with static IP addresses in the OOB range.
My external polling servers are configured in the client groups.
In response to the additional troubleshooting requests from the APIC:
- netstat -aon | grep ":161"
- udp 0 0 0.0.0.0:161 0.0.0.0:* off (0.00/0/0)
udp6 0 0 :::161 :::* off (0.00/0/0)
- udp 0 0 0.0.0.0:161 0.0.0.0:* off (0.00/0/0)
- netstat -lp | grep "snmp"
- (No info could be read for "-p": geteuid()=15374 but you should be root.)
udp 0 0 0.0.0.0:snmp 0.0.0.0:* -
udp 0 0 0.0.0.0:snmptrap 0.0.0.0:* -
udp6 0 0 [::]:snmp [::]:* -
udp6 0 0 [::]:snmptrap [::]:* -
- (No info could be read for "-p": geteuid()=15374 but you should be root.)
- show snmp clientgroups ( you can xxxx out the IP addresses)
- SNMP Policy Name Description Client Entries Associated Management EPG
-------------------- -------------------- -------------------- -------------------- --------------------
default ERP-Solarwinds x.x.x.x default (Out-Of-Band)
default ERP-Solarwinds-P2 x.x.x.x default (Out-Of-Band)
ERP-SNMP ERP-SNMP-CG x.x.x.x,x.x.x.x default (Out-Of-Band)
- SNMP Policy Name Description Client Entries Associated Management EPG
- moquery -c vzEntry | egrep -A 2 "161"
- No output
- moquery -c vzEntry | egrep -A 2 "162"
- No output
- moquery -c mgmtSubnet
Total Objects shown: 1
# mgmt.Subnet
ip : 0.0.0.0/0
annotation :
childAction :
descr :
dn : uni/tn-mgmt/extmgmt-default/instp-OOB-MGMT/subnet-[0.0.0.0/0]
extMngdBy :
lcOwn : local
modTs : 2023-01-26T07:43:48.730-05:00
name :
nameAlias :
rn : subnet-[0.0.0.0/0]
status :
uid : 15374
- moquery -c mgmtRsOoBCons
Total Objects shown: 1
# mgmt.RsOoBCons
tnVzOOBBrCPName : OOB-default
annotation :
childAction :
deplInfo :
dn : uni/tn-mgmt/extmgmt-default/instp-OOB-MGMT/rsooBCons-OOB-default
extMngdBy :
forceResolve : yes
lcOwn : local
modTs : 2015-09-09T07:13:51.360-05:00
monPolDn : uni/tn-common/monepg-default
prio : unspecified
rType : mo
rn : rsooBCons-OOB-default
state : formed
stateQual : none
status :
tCl : vzOOBBrCP
tContextDn :
tDn : uni/tn-mgmt/oobbrc-OOB-default
tRn : oobbrc-OOB-default
tType : name
triggerSt : triggerable
uid : 15374
- moquery -c vzOOBBrCP
Total Objects shown: 1
# mgmt.RsOoBCons
tnVzOOBBrCPName : OOB-default
annotation :
childAction :
deplInfo :
dn : uni/tn-mgmt/extmgmt-default/instp-OOB-MGMT/rsooBCons-OOB-default
extMngdBy :
forceResolve : yes
lcOwn : local
modTs : 2015-09-09T07:13:51.360-05:00
monPolDn : uni/tn-common/monepg-default
prio : unspecified
rType : mo
rn : rsooBCons-OOB-default
state : formed
stateQual : none
status :
tCl : vzOOBBrCP
tContextDn :
tDn : uni/tn-mgmt/oobbrc-OOB-default
tRn : oobbrc-OOB-default
tType : name
triggerSt : triggerable
uid : 15374ERPTRAPIC1# moquery -c vzOOBBrCP
Total Objects shown: 3# vz.OOBBrCP
name : default
annotation :
childAction :
configIssues :
descr :
dn : uni/tn-mgmt/oobbrc-default
extMngdBy :
lcOwn : local
modTs : 2020-01-13T04:37:04.170-05:00
monPolDn : uni/tn-common/monepg-default
nameAlias :
ownerKey :
ownerTag :
prio : unspecified
reevaluateAll : no
rn : oobbrc-default
scope : global
status :
targetDscp : unspecified
uid : 16005# vz.OOBBrCP
name : OOB-default
annotation :
childAction :
configIssues :
descr :
dn : uni/tn-mgmt/oobbrc-OOB-default
extMngdBy :
lcOwn : local
modTs : 2020-01-13T04:31:09.015-05:00
monPolDn : uni/tn-common/monepg-default
nameAlias :
ownerKey :
ownerTag :
prio : unspecified
reevaluateAll : no
rn : oobbrc-OOB-default
scope : context
status :
targetDscp : unspecified
uid : 15374# vz.OOBBrCP
name : default
annotation :
childAction :
configIssues :
descr :
dn : uni/tn-common/oobbrc-default
extMngdBy :
lcOwn : local
modTs : 2015-08-25T04:23:57.118-05:00
monPolDn : uni/tn-common/monepg-default
nameAlias :
ownerKey :
ownerTag :
prio : unspecified
reevaluateAll : no
rn : oobbrc-default
scope : context
status :
targetDscp : unspecified
uid : 0
- moquery -c snmpRsEpg | egrep "snmp.RsEpg|dn|tDn"
- # snmp.RsEpg
dn : uni/fabric/snmppol-default/clgrp-ERP-Solarwinds-P2/rsepg
tDn : uni/tn-mgmt/mgmtp-default/oob-default
# snmp.RsEpg
dn : uni/fabric/snmppol-ERP-SNMP/clgrp-ERP-SNMP-CG/rsepg
tDn : uni/tn-mgmt/mgmtp-default/oob-default
# snmp.RsEpg
dn : uni/fabric/snmppol-default/clgrp-ERP-Solarwinds/rsepg
tDn : uni/tn-mgmt/mgmtp-default/oob-default
- # snmp.RsEpg
- ps aux | grep snmp
- ifc 4662 1.6 0.3 772016 213432 ? Ssl 2020 24971:11 /mgmt//bin/snmpd.bin -f -p /tmp/snmpd2.pid -a -A -Lf /var/run/mgmt/log/snmpd.log -c /data//snmp/snmpd.conf
ifc 11655 0.0 0.0 91848 9924 ? Ss 2020 0:00 /mgmt//bin/snmptrapd.bin -f -p /tmp/snmptrapd2.pid udp:162 udp6:162 -u ifc -a -c /data//snmp/snmptrapd.conf -C -d -f
admin 18843 0.0 0.0 9060 920 pts/0 S+ 19:46 0:00 grep snmp
- ifc 4662 1.6 0.3 772016 213432 ? Ssl 2020 24971:11 /mgmt//bin/snmpd.bin -f -p /tmp/snmpd2.pid -a -A -Lf /var/run/mgmt/log/snmpd.log -c /data//snmp/snmpd.conf
Again, SNMP works to the APICs but not to the leaf/spine switches. Additionally I am able to ping and on a packet capture from teh ASA firewall I can see the SNMP response coming back from the leaf/spine (although failing). Sorry to be so late to the party on this one, hopefully someone can provide assistance.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-26-2017 08:36 AM
also,
Ask the ACI Experts: SNMP in the ACI Fabric
https://supportforums.cisco.com/blog/13100731/ask-aci-experts-snmp-aci-fabric
