Problem with discovery

lukaszdewera — Fri, 01 Mar 2019 12:37:11 GMT

Hello

I'm facing problem with device discovery. I have 2 sites that has been discovered withou any issues. But when I tried to add another one today, I got "cli validation failure". There is no problem with routing nor ACL - snmp part works fine. I checked logs from TACACS+ server, and there was no attempt to log in from APIC-EM server. I logged in to linix console and tried with accoult used by APIC-EM and it worked. Then I captured tcpdump on physical interface and run discovery process. Noticed, that there was no ssh attempt. 10.10.10.10 is switch that does not work, 10.10.20.20 is APIC-EM server.

$ sudo tcpdump -n -i eth0 host 10.10.10.10

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

10:51:13.749932 IP 10.10.20.20.57388 > 10.10.10.10.161: F=apr U=apicem [!scoped PDU]5e_8e_9b_a2_d4_1f_2b_28_01_46_d2_2d_e3_84_8 0_6b_60_1d_b8_2f_77_64_20_a6_c2_6c_7b_e9_6f_ff_ce_e4_b4_c5_db_f0_b6_2f_08_6c_29_61_ce_27_fa_d4_bb_10_1f_97_b0_e8_c5_c1_e0_4a_2a_c4_ bc_f1_6e_4f_ed_a6_6a

10:51:13.752465 IP 10.10.10.10.161 > 10.10.20.20.57388: F=ap U=apicem [!scoped PDU]82_29_9f_f0_cc_f3_04_d8_f3_61_ab_b9_f3_70_23 _17_69_ca_56_41_71_f0_b2_eb_f3_82_57_15_fc_7e_15_33_2c_56_03_b5_97_11_3a_68_31_3d_8f_7d_1e_3d_ab_af_5f_1b_d9_08_84_0b_db_70_d0_09_f c

10:51:13.817258 IP 10.10.20.20 > 10.10.10.10: ICMP echo request, id 4135, seq 1, length 64

10:51:13.823884 IP 10.10.10.10 > 10.10.20.20: ICMP echo reply, id 4135, seq 1, length 64

10:51:14.818861 IP 10.10.20.20 > 10.10.10.10: ICMP echo request, id 4135, seq 2, length 64

10:51:14.819722 IP 10.10.10.10 > 10.10.20.20: ICMP echo reply, id 4135, seq 2, length 64

10:51:15.819891 IP 10.10.20.20 > 10.10.10.10: ICMP echo request, id 4135, seq 3, length 64

10:51:15.820750 IP 10.10.10.10 > 10.10.20.20: ICMP echo reply, id 4135, seq 3, length 64

10:51:15.873981 IP 10.10.20.20.57388 > 10.10.10.10.161: F=apr U=apicem [!scoped PDU]45_ae_c8_a3_ba_6c_e5_5d_9b_9c_a4_5c_83_51_e c_4c_d7_81_7f_95_d0_5d_98_0c_f9_2f_ae_b4_78_20_9b_05_bb_58_4e_5d_8c_d4_94_9e_55_ab_8b_05_fe_ea_8a_ac_64_3d_c4_60_cd_75_9a_4f_cd_d7_ cc_b9_ba_53_56_ce_1e

10:51:15.887281 IP 10.10.10.10.161 > 10.10.20.20.57388: F=ap U=apicem [!scoped PDU]8c_70_36_ef_43_9b_68_b2_ef_73_86_23_89_20_f9 _6e_2b_00_37_a0_c3_6b_a4_b6_ab_c9_b7_8f_88_2c_bc_ae_bf_70_fa_e3_52_5c_0e_7d_74_e4_91_50_6e_1e_b1_cc_09_ae_5e_21_a9_7f_25_3f_38_fc_f d

10:51:15.934290 IP 10.10.20.20.57388 > 10.10.10.10.161: F=apr U=apicem [!scoped PDU]51_34_29_d2_7d_e4_a7_7e_6c_30_a4_6a_6d_de_f 3_12_a0_33_c1_9e_5b_cb_ee_fc_0b_57_4f_5a_92_0e_f6_11_72_3e_b0_75_54_8f_e6_08_9d_b4_7b_99_bb_c9_af_c2_e6_cb_c1_eb_0c_17_62_a0_bb_f9_ cc_0e_79_06_14_34_37

10:51:15.936832 IP 10.10.10.10.161 > 10.10.20.20.57388: F=ap U=apicem [!scoped PDU]db_42_1f_b1_04_a8_9a_f8_b7_36_75_d3_f1_6c_e2 _48_fa_98_a7_e9_b6_c7_d5_d4_75_da_af_d0_39_14_0e_c0_2a_eb_a8_4b_80_02_b5_54_eb_49_70_60_d4_2c_9d_f2_15_c2_2a_81_45_b5_a1_92_1b_60_8 e

10:51:16.001203 IP 10.10.20.20 > 10.10.10.10: ICMP echo request, id 4136, seq 1, length 64

10:51:16.009239 IP 10.10.10.10 > 10.10.20.20: ICMP echo reply, id 4136, seq 1, length 64

10:51:17.002036 IP 10.10.20.20 > 10.10.10.10: ICMP echo request, id 4136, seq 2, length 64

10:51:17.007123 IP 10.10.10.10 > 10.10.20.20: ICMP echo reply, id 4136, seq 2, length 64

10:51:18.002886 IP 10.10.20.20 > 10.10.10.10: ICMP echo request, id 4136, seq 3, length 64

10:51:18.005456 IP 10.10.10.10 > 10.10.20.20: ICMP echo reply, id 4136, seq 3, length 64

Is there any log I can check what's wrong with this discovery? Any hints how to make discovery work again? I tried 2 different switch boxes and different model, but have the same result. Also rebooted APIC-EM server.

Re: Problem with discovery

lukaszdewera — Mon, 20 Mar 2017 14:58:52 GMT

Problem fixed, it appeared that there were issue in SNMP v3 configuration. What was misleading for me was, that APIC-EM was showing SNMP discovery as success and fail on CLI when in fact it was SNMP step that failed.

Re: Problem with discovery

aradford — Tue, 21 Mar 2017 12:09:25 GMT

Thanks for picking this up.

Can you confirm the version of APIC-EM you are using?

Adam

Re: Problem with discovery

lukaszdewera — Tue, 21 Mar 2017 13:56:48 GMT

I have APIC-EM Version 1.3.3.126.

Re: Problem with discovery

Rijo — Fri, 24 Mar 2017 03:38:18 GMT

Hi Lukasz,

We would like to resolve the issue you mentioned "What was misleading for me was, that APIC-EM was showing SNMP discovery as success and fail on CLI when in fact it was SNMP step that failed."

Can you let us know from which screen you tried adding the device and where you were seeing SNMP success and CLI failure message?

Thanks,

Rijo

Re: Problem with discovery

lukaszdewera — Fri, 24 Mar 2017 13:43:57 GMT

I tried to discovery using Discovery App. Devices with all green has both TACACS+ and correct SNMP settings. Devices with failed SNMP does not have SNMP but has TACACS+ configured. Device marked in red box has TACACS+ enabled (CLI access works, tried from apic-em server with credentials used by apic-em server) but has SNMPv3 not correctly configured.

SNMP on "red" box switch is:

p access-list standard ACL
permit 8.8.8.8

snmp-server user apicem_user apic-group v3 auth md5 AUTH priv aes 128 PRIV

snmp-server host 8.8.8.8 traps version 3 priv apicem_user

after adding

snmp-server group apic-group v3 priv access ACL

it works like a charm.

topic Re: Problem with discovery in Controllers

Problem with discovery

Re: Problem with discovery

Re: Problem with discovery

Re: Problem with discovery

Re: Problem with discovery

Re: Problem with discovery