ACI: Techsupport Analysis - 主要なログと解析例

[Yoshitaka Yanomori]

本ドキュメントでは、ACI Techsupport の主要なログについて説明します。

※ ACI Techsupportの取得方法について こちら　を参照ください。

※ ACI Version 2.x での確認結果をもとに掲載しております。

1. 生成される techsupport fileの種類

techsupport を取得すると APIC / Leaf / Spine において、それぞれ 以下のように 3種類のFileが生成されます。解析には 3種類全てのFileが必要です。

Apic：

dbgexp_tsod-techsupport_apic1_sysid-1_2017-07-04T17-14JST_1of3.tgz

dbgexp_tsod-techsupport_apic1_sysid-1_2017-07-04T17-14JST_db_2of3.tgz

dbgexp_tsod-techsupport_apic1_sysid-1_2017-07-04T17-14JST_logs_3of3.tgz

Leaf:

dbgexp_tsod-techsupport_leaf101_sysid-101_2017-01-29T16-15JST_1of3.tgz

dbgexp_tsod-techsupport_leaf101_sysid-101_2017-01-29T16-15JST_db_2of3.tgz

dbgexp_tsod-techsupport_leaf101_sysid-101_2017-01-29T16-15JST_logs_3of3.tgz

Spine:

dbgexp_tsod-techsupport_spine201_sysid-201_2017-07-04T15-55JST_1of3.tgz

dbgexp_tsod-techsupport_spine201_sysid-201_2017-07-04T15-55JST_db_2of3.tgz

dbgexp_tsod-techsupport_spine201_sysid-201_2017-07-04T15-55JST_logs_3of3.tgz

2. 主要なログと保存場所

APIC

Log	Log Description	file No.	Location in techsupport
access log	nginx access log  (GET/POST commands)	1of3	/var/run/mgmt/nginx/logs/access.log
audit log	login/logout、config change log	1of3	/var/log/dme/oldlog/clixxxx/audit_log_info
avread	Controller information ( Cluster size, chassis Id, etc)	1of3	/var/log/dme/oldlog/clixxxx/avread
fnvread	fabric node information ( ID, Serial, IP Adddress, State, etc)	1of3	/var/log/dme/oldlog/clixxxx/fnvread
rvread	Health status of shard /replica in each process of APIC Cluster	1of3	/var/log/dme/oldlog/clixxxx/rvread
fault_info	All Faults hisotry log in the system	1of3	/var/log/dme/oldlog/clixxxx/fault_info
event_info	All events history log in the system	1of3	/var/log/dme/oldlog/clixxxx/event_info
dmesg	System/kernel log	1of3	/var/log/dme/oldlog/clixxxx/dmesg
messages	System message log  (syslog message)	1of3	/var/log/messages
user_config	System configuraiton	1of3	/var/log/dme/oldlog/clixxxx/user_config
devicescript	Log for L4-L7 devices and device packages	1of3	/data/devicescript/<Device>/logs/
svc_ifc_*log*	Process log (eventmgr, policyelem, vmmmgr, etc)	3of3	/data/log/svc_ifc*log*
SWITCH (Leaf/Spine)
Log	Log Description	file No.	Location in techsupport
install log	Install log	1of3	/mnt/pss/installer.log
boot log	Boot log	1of3	/mnt/pss/bootlogs/*
dmesg	System/kernel log	1of3	/var/log/dme/oldlog/clixxxx/dmesg
messages	System message log  (syslog message)	1of3	/var/log/external/messages
access log	nginx access log  (GET/POST commands)	3of3	/var/sysmgr/tmp_logs/access.log
svc_ifc_*log*	Process log (eventmgr, policyelem, vmmmgr, etc)	3of3	/var/sysmgr/tmp_logs/svc_ifc_*log* /var/log/dme/oldlog/last_run/tmp_logs/svc_ifc_*log*.gz (archive)
epm-trace	Endpoint Manager log  (*1)	3of3	/var/sysmgr/tmp_logs/epm-trace.txt
epmc-trace	Endpoint Manager Client log (*1)	3of3	/var/sysmgr/tmp_logs/epmc-trace.txt
eltm_trace	Ethernet Logical Table Interface Manager log	3of3	/var/sysmgr/tmp_logs/eltm_trace_dump.txt
eltmc_trace	Ethernet Logical Table Interface Client log (*2)	3of3	/var/sysmgr/tmp_logs/eltmc_trace_dump.txt
nxos techsupport	NX-OS style tech-support (FEX, SUP TOR)	3of3	/var/sysmgr/tmp_logs/svc_ifc_techsup_nxos.tar

*1.  epm/epmc-trace file is not created on spine switch.

*2.  eltmc_trace_dump is not created on chassis type spine switch, eg. N9504/9508. The eltmc log is archived in nx-os techsupport file.

3. 解析例①  Messages file (syslog)

(1) messages file の locationを確認

(2) 1of3.tgz を解凍した後、var/log/external へ移動

     cd dbgexp_xxxx_1of3/var/log/external

(3) “messages” というファイルを を確認

     ~/dbgexp_xxxx_1of3/var/log/external$ ls

      messages

(4) 内容の確認

~/dbgexp_xxxx_1of3/var/log/external$ cat messages

<1027> Nov 13 00:59:58 spine-test1 %LOG_LOCAL0-3-SYSTEM_MSG[F1394][soaking][interface-physical-down][minor][sys/phys-[eth1/10]/phys/fault-F1394] Port is down, reason:notconnect(connected),used by:Fabric

<1030> Nov 13 00:59:58 spine-test %LOG_LOCAL0-6-SYSTEM_MSG[E4205126][port-down][info][sys/phys-[eth1/10]/phys] Port is down.Reason: notconnect

<1027> Nov 13 01:00:06 spine-test %LOG_LOCAL0-3-SYSTEM_MSG [F1394][soaking_clearing][interface-physical-down][minor][sys/phys-[eth1/10]/phys/fault-F1394] Port is down, reason:notconnect(connected), used by:Fabric

4. 解析例② NX-OS 形式の techsupport file

(1) techsupport file (nx-os) の location を確認

(2) logs_3of3.tgz を解凍した後、var/sysmgr/tmp_logs へ移動

~/dbgexp_xxxx_logs_3of3/var/sysmgr/tmp_logs$

(3) “svc_ifc_techsup_nxos_tar” というファイルを を確認

~/dbgexp_xxxx_logs_3of3/var/sysmgr/tmp_logs$ ls | grep tech

svc_ifc_techsup_nxos.tar

(4) tar を解凍すると show_tech_infoというディレクトリが生成

~/dbgexp_xxxx_logs_3of3/var/sysmgr/tmp_logs$ tar –xf svc_ifc_techsup_nxos.tar

(5) show_tech_info の中を参照すると 様々なファイルが保存
show-tech-sup-output.gz が NX-OS形式の tac-pac に相当

~/dbgexp_xxxx_logs_3of3/var/sysmgr/tmp_logs/show_tech_info$ ls -l

-rw-rw-r-- 1 techsupport techsupport   700161 Jan 19 03:20 CpuUsage.Log

-rw-rw-r-- 1 techsupport techsupport    55923 Jan 19 03:20 istack_debug.gz

-rw-rw-r-- 1 techsupport techsupport  3149472 Jan 19 03:20 MemoryLog.65%_usage

-rw-rw-r-- 1 techsupport techsupport       38 Jan 19 03:20 show-tech-fex-err.gz

-rw-rw-r-- 1 techsupport techsupport   724393 Jan 19 03:20 show-tech-fex.gz

-rw-rw-r-- 1 techsupport techsupport      143 Jan 19 03:20 show-tech-sup-err.gz

-rw-rw-r-- 1 techsupport techsupport 31307526 Jan 19 03:20 show-tech-sup-output.gz

-rw-rw-r-- 1 techsupport techsupport  5650129 Jan 19 03:20 show-tech-tor-lc.gz

(6) techsupport を解凍し、内容の確認や grep を実行

- 解凍

~/dbgexp_xxxx_logs_3of3/var/sysmgr/tmp_logs/show_tech_info$gunzip show-tech-sup-output.gz

- less show-tech-sup-output

~/dbgexp_xxxx_logs_3of3/var/sysmgr/tmp_logs/show_tech_info$less show-tech-sup-output

- grep xxx show-tech-sup-output

~/dbgexp_xxxx_logs_3of3/var/sysmgr/tmp_logs/show_tech_info$grep xxx show-tech-sup-output

※　techsupport(nx-os) は versionによって 保存場所が変更する場合もありますので、探せない場合は、find で “tech”  等を指定 して探してください。

  ~/tmp/node-101/dbgexp_tsod_xxx_logs_3of3$ find -name *tech*

  ./var/sysmgr/tmp_logs/svc_ifc_techsup_nxos.tar   <-- 検索の結果