Solved: Re: Firepower Management Center Virtual - Backup Failed - 6.2.3 - Cisco Community

5382

Views

4

Helpful

7

Replies

Hi Gents,

I'm facing a issue in my environment when I try execute a backup using FMC-virtual version 6.2.3. Looks like there is no resource enough. as you can see below:

Does anyone might help me on this?

---------------------------------------------------

/var/log/messages

Jun 5 08:31:20 FMC kernel: [<ffffffff815a6c78>] page_fault+0x28/0x30

Jun 5 08:31:20 FMC kernel: Task in /System killed as a result of limit of /System

Jun 5 08:31:20 FMC kernel: memory: usage 5005072kB, limit 5005072kB, failcnt 7414397

Jun 5 08:31:20 FMC kernel: memory+swap: usage 8372560kB, limit 8925368kB, failcnt 0

Jun 5 08:31:20 FMC kernel: kmem: usage 0kB, limit 9007199254740991kB, failcnt 0

Jun 5 08:31:20 FMC kernel: Memory cgroup stats for /System: cache:1444KB rss:5003620KB rss_huge:0KB mapped_file:204KB swap:3367488KB inactive_anon:1165056KB active_anon:3838784KB inactive_file:88KB active_file:4KB unevictable:0KB

----------------------------------------------------

admin@FMC:~$ free -m

total used free shared buffers cached

Mem: 6982 5602 1379 0 11 265

-/+ buffers/cache: 5326 1656

Swap: 5469 5368 100

------------------------------------------------------

Whole log:

admin@FMC:~$ tail -f /var/log/messages

Jun 5 08:20:24 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:20:24 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:20:24 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun 5 08:20:24 FMC process_msg : Received IPC message type : 12

Jun 5 08:20:24 FMC Response being sent to SAM : ▒

Jun 5 08:20:24 FMC , len(msg being sent) = 2576

Jun 5 08:20:24 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:20:24 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:20:24 FMC Connection closed...

Jun 5 08:20:24 FMC Waiting for someone to connect to me...

Jun 5 08:21:00 FMC SF-IMS[6307]: [6307] pm:process [INFO] Started store_whitelist_history (25397)

Jun 5 08:21:01 FMC SF-IMS[6307]: [6307] pm:log [INFO] Process 'store_whitelist_history' closed output.

Jun 5 08:21:02 FMC sudo: www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown www:www /var/log/CSMAgent.log

Jun 5 08:21:02 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:02 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:03 FMC sudo: www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/kill -s USR1 7440

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:03 FMC sudo: www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mv /var/tmp/backupeZIW/files_not_to_tar /var/tmp/backupfmoa/files_not_to_tar

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:03 FMC sudo: www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mv /var/tmp/backupeZIW/files_to_tar /var/tmp/backupfmoa/files_to_tar

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:03 FMC sudo: www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/sf/bin/sf-backup.pl /var/sf/backup/FMC-BACKUP-CONFIG.tar.gz

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:03 FMC sudo: www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:03 FMC sudo: www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:03 FMC sudo: www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:07 FMC sudo: www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun 5 08:21:07 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:07 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:07 FMC sudo: www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown www:www /var/log/CSMAgent.log

Jun 5 08:21:07 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:07 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:08 FMC sudo: www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown www:www /var/log/CSMAgent.log

Jun 5 08:21:08 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun 5 08:21:08 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:21:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 1 expired services (query time: 0.001 sec.)

Jun 5 08:21:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 4 expired client apps (query time: 0.007 sec.)

Jun 5 08:21:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] IPRep, time to check for updates

Jun 5 08:21:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for ip verify_peer=1 verifyhost=0

Jun 5 08:21:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for dns verifypeer=1 verifyhost=0

Jun 5 08:22:23 FMC syslog-ng[3574]: Log statistics; processed='source(msgs)=133688', processed='global(payload_reallocs)=137871', processed='src.none()=0', stamp='src.none()=0', processed='global(internal_queue_length)=0', processed='destination(CSCOmd_destination)=0', processed='global(msg_clones)=0', processed='destination(cron_destination)=212', processed='src.internal(msgs#2)=945', stamp='src.internal(msgs#2)=1528186343', processed='global(sdata_updates)=0', processed='destination(messages_destination)=92733', processed='center(received)=133688', processed='center(queued)=92945'

Jun 5 08:22:33 FMC SF-IMS[11943]: [11989] SFDataCorrelator:SSLEvents [INFO] Deleted 10 certificates from ssl_certificate_details

Jun 5 08:22:49 FMC Someone connected to me, receiving data...

Jun 5 08:22:49 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:22:49 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:22:49 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun 5 08:22:49 FMC process_msg : Received IPC message type : 12

Jun 5 08:22:49 FMC Response being sent to SAM : ▒

Jun 5 08:22:49 FMC , len(msg being sent) = 2576

Jun 5 08:22:49 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:22:49 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:22:49 FMC Connection closed...

Jun 5 08:22:49 FMC Waiting for someone to connect to me...

Jun 5 08:24:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 4 expired IP hosts (query time: 0.002 sec.)

Jun 5 08:24:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 4 expired client apps (query time: 0.007 sec.)

Jun 5 08:24:27 FMC SF-IMS[6854]: [6934] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:24:27 FMC SF-IMS[6854]: [28842] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:24:27 FMC SF-IMS[6854]: [17786] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:24:27 FMC SF-IMS[6854]: [22784] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:25:14 FMC Someone connected to me, receiving data...

Jun 5 08:25:14 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:25:14 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:25:14 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun 5 08:25:14 FMC process_msg : Received IPC message type : 12

Jun 5 08:25:14 FMC Response being sent to SAM : ▒

Jun 5 08:25:14 FMC , len(msg being sent) = 2575

Jun 5 08:25:14 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:25:14 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:25:14 FMC Connection closed...

Jun 5 08:25:14 FMC Waiting for someone to connect to me...

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): SQL Anywhere Personal Server Version 16.0.0.2271

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): OEM Authenticated Edition, licensed only for use with authenticated OEM applications.

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng):

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Copyright (c) 2015 SAP SE or an SAP affiliate company.

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): All rights reserved.

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Use of this software is governed by the Sybase License Agreement.

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Refer to http://www.sybase.com/softwarelicenses.

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng):

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Connection limit (Personal Server): 10

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Processors detected: 4 (containing 8 logical processors)

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Processor limit (Personal Server): 1

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Maximum number of processors the server will use: 1 physical processor(s), 4 core(s)

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): This server is licensed to:

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): cisco

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): cisco

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Running Linux 3.10.107sf.virtual-1 #1 SMP Thu Mar 8 18:37:12 UTC 2018 on X86_64

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Server built for X86_64 processor architecture

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): 177108K of memory used for caching

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Minimum cache size: 11512K, maximum cache size: 6431404K

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Using a maximum page size of 4096 bytes

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Multiprogramming level: 20

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Database server started at Tue Jun 05 2018 08:25

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Trying to start SharedMemory link ...

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): SharedMemory link started successfully

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Now accepting requests

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Starting database "dbvvms" (/var/tmp/backupfmoa/0/vms/database/vms.db) at Tue Jun 05 2018 08:25

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Database recovery in progress

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Last checkpoint at Tue Jun 05 2018 08:24

Jun 5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Checkpoint log...

Jun 5 08:25:28 FMC SQLAnywhere(dbvcsmeng): Transaction log: vms.log...

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Rollback log...

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Checkpointing...

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Starting checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:25

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Finished checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:25

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Recovery complete

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Database "dbvvms" (vms.db) started at Tue Jun 05 2018 08:25

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): This database is licensed for use with:

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Application: NMTG

Jun 5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Company: Cisco Systems

Jun 5 08:25:39 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 876464K

Jun 5 08:25:44 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1058868K

Jun 5 08:25:54 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1204476K

Jun 5 08:25:59 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1270452K

Jun 5 08:26:00 FMC SF-IMS[6307]: [6307] pm:process [INFO] Started store_whitelist_history (28151)

Jun 5 08:26:01 FMC SF-IMS[6307]: [6307] pm:log [INFO] Process 'store_whitelist_history' closed output.

Jun 5 08:26:04 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1296848K

Jun 5 08:26:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] IPRep, time to check for updates

Jun 5 08:26:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] ClamUpd, time to check for updates

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for ip verify_peer=1 verifyhost=0

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for dns verifypeer=1 verifyhost=0

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file .. from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/..

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file . from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/.

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] chown successful

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] The curl option for clam verify_peer=1 verify_host=2

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file .. from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/..

Jun 5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file . from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/.

Jun 5 08:26:55 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1504320K

Jun 5 08:27:02 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1646432K

Jun 5 08:27:07 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1696040K

Jun 5 08:27:12 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1708248K

Jun 5 08:27:17 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1738880K

Jun 5 08:27:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 2 expired services (query time: 0.001 sec.)

Jun 5 08:27:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 6 expired client apps (query time: 0.007 sec.)

Jun 5 08:27:22 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1745676K

Jun 5 08:27:27 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1747300K

Jun 5 08:27:32 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1748236K

Jun 5 08:27:44 FMC Someone connected to me, receiving data...

Jun 5 08:27:44 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:27:44 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:27:44 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun 5 08:27:44 FMC process_msg : Received IPC message type : 12

Jun 5 08:27:44 FMC Response being sent to SAM : ▒

Jun 5 08:27:44 FMC , len(msg being sent) = 2576

Jun 5 08:27:44 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:27:44 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:27:44 FMC Connection closed...

Jun 5 08:27:44 FMC Waiting for someone to connect to me...

Jun 5 08:28:26 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1805020K

Jun 5 08:28:31 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1848112K

Jun 5 08:28:36 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1858452K

Jun 5 08:28:41 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1859876K

Jun 5 08:28:46 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1845260K

Jun 5 08:28:51 FMC kernel: ffff8801f51d3c00 ffff8801eeef3c28 ffffffff815a1ecf ffff8801eeef3ca8

Jun 5 08:28:51 FMC kernel: [ 6867] 0 6867 6975 335 18 474 0 snmpd

Jun 5 08:28:51 FMC kernel: [ 6923] 0 6923 5574 46 14 87 0 stunnel

Jun 5 08:28:51 FMC kernel: [ 6924] 0 6924 5574 46 14 87 0 stunnel

Jun 5 08:28:51 FMC kernel: [ 7108] 0 7108 1036 8 7 12 0 erl_child_setup

Jun 5 08:28:51 FMC kernel: [ 7129] 0 7129 23559 5 18 206 0 diskmanager

Jun 5 08:28:51 FMC kernel: [ 7130] 0 7130 4386 0 11 3166 0 diskmgr_handler

Jun 5 08:28:51 FMC kernel: [ 7198] 0 7198 57353 6 22 230 0 diskmanager

Jun 5 08:28:51 FMC kernel: [ 7199] 0 7199 4386 1 11 3165 0 diskmgr_handler

Jun 5 08:28:51 FMC kernel: [ 7432] 0 7432 13800 237 29 8076 -600 perl

Jun 5 08:28:51 FMC kernel: [ 7435] 0 7435 101965 396 49 1506 0 sfestreamer

Jun 5 08:28:51 FMC kernel: [ 7436] 0 7436 13010 63 30 321 0 fpcollect

Jun 5 08:28:51 FMC kernel: [ 7438] 0 7438 18761 4176 39 6048 0 expire-session.

Jun 5 08:28:51 FMC kernel: [ 7439] 0 7439 66213 37421 133 19787 0 Pruner.pl

Jun 5 08:28:51 FMC kernel: [ 7440] 0 7440 11479 2049 25 5002 0 ActionQueueScra

Jun 5 08:28:51 FMC kernel: [ 7441] 0 7441 29056 3021 59 17030 0 PerlMessageHand

Jun 5 08:28:51 FMC kernel: [ 7442] 0 7442 16079 271 33 7993 0 snapshot_manage

Jun 5 08:28:51 FMC kernel: [ 7443] 67 7443 5585 1297 14 1470 0 SFTop10Cacher

Jun 5 08:28:51 FMC kernel: [ 7444] 0 7444 4683 291 13 3157 0 query_scheduler

Jun 5 08:28:51 FMC kernel: [ 7445] 0 7445 1238843 93886 452 63654 -600 java

Jun 5 08:28:51 FMC kernel: [ 7483] 0 7483 16364 1273 34 7225 0 uimp.pl

Jun 5 08:28:51 FMC kernel: [ 7484] 0 7484 3519 27 10 210 0 ui_archiver

Jun 5 08:28:51 FMC kernel: [ 7485] 0 7485 18332 1469 39 7417 0 TSS_Daemon.pl

Jun 5 08:28:51 FMC kernel: [ 7514] 67 7514 16746 3547 36 5364 0 SFTop10Cacher

Jun 5 08:28:51 FMC kernel: [ 7523] 0 7523 37569 9349 76 18837 -600 perl

Jun 5 08:28:51 FMC kernel: [ 7524] 0 7524 48997 1226 100 38753 -600 perl

Jun 5 08:28:51 FMC kernel: [ 7525] 0 7525 37754 3555 77 25263 -600 perl

Jun 5 08:28:51 FMC kernel: [ 7526] 0 7526 42423 5992 87 27488 -600 perl

Jun 5 08:28:51 FMC kernel: [ 7527] 0 7527 40560 6931 82 24203 -600 perl

Jun 5 08:28:51 FMC kernel: [ 7547] 0 7547 2857 6 11 21 0 inet_gethost

Jun 5 08:28:51 FMC kernel: [ 7548] 0 7548 2857 9 11 23 0 inet_gethost

Jun 5 08:29:28 FMC SF-IMS[7439]: Pruner.pl:Pruner.pl [WARN] Time change found. Resetting status in Pruner.pl

Jun 5 08:30:08 FMC Someone connected to me, receiving data...

Jun 5 08:30:08 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:30:08 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:30:08 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun 5 08:30:08 FMC process_msg : Received IPC message type : 12

Jun 5 08:30:09 FMC Response being sent to SAM : ▒

Jun 5 08:30:09 FMC , len(msg being sent) = 2576

Jun 5 08:30:09 FMC sla_worker : sizeof(msg) : 8192

Jun 5 08:30:09 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun 5 08:30:09 FMC Connection closed...

Jun 5 08:30:09 FMC Waiting for someone to connect to me...

Jun 5 08:30:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 1 expired IP hosts (query time: 0.004 sec.)

Jun 5 08:30:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 1 expired services (query time: 0.010 sec.)

Jun 5 08:30:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 5 expired client apps (query time: 0.008 sec.)

Jun 5 08:30:28 FMC SF-IMS[6854]: [6934] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:30:29 FMC SF-IMS[6854]: [28842] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:30:29 FMC SF-IMS[6854]: [17786] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:30:29 FMC SF-IMS[6854]: [22784] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): SQL Anywhere Personal Server Version 16.0.0.2271

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): OEM Authenticated Edition, licensed only for use with authenticated OEM applications.

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng):

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Copyright (c) 2015 SAP SE or an SAP affiliate company.

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): All rights reserved.

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Use of this software is governed by the Sybase License Agreement.

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Refer to http://www.sybase.com/softwarelicenses.

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng):

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Connection limit (Personal Server): 10

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Processors detected: 4 (containing 8 logical processors)

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Processor limit (Personal Server): 1

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Maximum number of processors the server will use: 1 physical processor(s), 4 core(s)

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): This server is licensed to:

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): cisco

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): cisco

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Running Linux 3.10.107sf.virtual-1 #1 SMP Thu Mar 8 18:37:12 UTC 2018 on X86_64

Jun 5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Server built for X86_64 processor architecture

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): 186440K of memory used for caching

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Minimum cache size: 11512K, maximum cache size: 6431404K

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Using a maximum page size of 4096 bytes

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Multiprogramming level: 20

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Database server started at Tue Jun 05 2018 08:30

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Trying to start SharedMemory link ...

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): SharedMemory link started successfully

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Now accepting requests

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Starting database "dbvvms" (/var/tmp/backupfmoa/0/vms/database/vms.db) at Tue Jun 05 2018 08:30

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Database recovery in progress

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Last checkpoint at Tue Jun 05 2018 08:29

Jun 5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Checkpoint log...

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Transaction log: vms.log...

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Rollback log...

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Checkpointing...

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Starting checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:30

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Finished checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:30

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Recovery complete

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Database "dbvvms" (vms.db) started at Tue Jun 05 2018 08:30

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): This database is licensed for use with:

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Application: NMTG

Jun 5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Company: Cisco Systems

Jun 5 08:30:50 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1434564K

Jun 5 08:31:00 FMC SF-IMS[6307]: [6307] pm:process [INFO] Started store_whitelist_history (29686)

Jun 5 08:31:00 FMC SF-IMS[6307]: [6307] pm:log [INFO] Process 'store_whitelist_history' closed output.

Jun 5 08:31:10 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1763520K

Jun 5 08:31:15 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1828840K

Jun 5 08:31:20 FMC kernel: 2_scheduler invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0

Jun 5 08:31:20 FMC kernel: 2_scheduler cpuset=/ mems_allowed=0

Jun 5 08:31:20 FMC kernel: CPU: 2 PID: 7110 Comm: 2_scheduler Not tainted 3.10.107sf.virtual-1 #1

Jun 5 08:31:20 FMC kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014

Jun 5 08:31:20 FMC kernel: ffff8801f51d3c00 ffff8801eee4bc28 ffffffff815a1ecf ffff8801eee4bca8

Jun 5 08:31:20 FMC kernel: ffffffff815a03f3 0000000400386900 00000000000000e2 ffff8801eee4bc58

Jun 5 08:31:20 FMC kernel: ffff8801eee4bc68 0000000000000282 ffffffff00000000 ffff8801eee4bca8

Jun 5 08:31:20 FMC kernel: Call Trace:

Jun 5 08:31:20 FMC kernel: [<ffffffff815a1ecf>] dump_stack+0x19/0x1b

Jun 5 08:31:20 FMC kernel: [<ffffffff815a03f3>] dump_header.isra.8+0x87/0x1d4

Jun 5 08:31:20 FMC kernel: [<ffffffff812466a7>] ? ___ratelimit+0xec/0x117

Jun 5 08:31:20 FMC kernel: [<ffffffff810a4a43>] oom_kill_process+0x6f/0x36f

Jun 5 08:31:20 FMC kernel: [<ffffffff81044107>] ? has_capability_noaudit+0x12/0x14

Jun 5 08:31:20 FMC kernel: [<ffffffff810d6bdb>] mem_cgroup_oom_synchronize+0x3df/0x4b8

Jun 5 08:31:20 FMC kernel: [<ffffffff810d631e>] ? mem_cgroup_can_attach+0x14b/0x14b

Jun 5 08:31:20 FMC kernel: [<ffffffff810a5167>] pagefault_out_of_memory+0xe/0x4a

Jun 5 08:31:20 FMC kernel: [<ffffffff8159f0e5>] mm_fault_error+0x5d/0xff

Jun 5 08:31:20 FMC kernel: [<ffffffff81032187>] __do_page_fault+0x2d5/0x47e

Jun 5 08:31:20 FMC kernel: [<ffffffff811100bf>] ? timerfd_get_remaining+0x11/0x21

Jun 5 08:31:20 FMC kernel: [<ffffffff811105f2>] ? do_timerfd_settime+0x292/0x2d1

Jun 5 08:31:20 FMC kernel: [<ffffffff81032358>] do_page_fault+0x9/0xb

Jun 5 08:31:20 FMC kernel: [<ffffffff815a6c78>] page_fault+0x28/0x30

Jun 5 08:31:20 FMC kernel: Task in /System killed as a result of limit of /System

Jun 5 08:31:20 FMC kernel: memory: usage 5005072kB, limit 5005072kB, failcnt 7414397

Jun 5 08:31:20 FMC kernel: memory+swap: usage 8372560kB, limit 8925368kB, failcnt 0

Jun 5 08:31:20 FMC kernel: kmem: usage 0kB, limit 9007199254740991kB, failcnt 0

Jun 5 08:31:20 FMC kernel: Memory cgroup stats for /System: cache:1444KB rss:5003620KB rss_huge:0KB mapped_file:204KB swap:3367488KB inactive_anon:1165056KB active_anon:3838784KB inactive_file:88KB active_file:4KB unevictable:0KB

Jun 5 08:31:20 FMC kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name

Jun 5 08:31:20 FMC kernel: [ 6713] 0 6713 84006 230 43 516 0 CloudAgent

Jun 5 08:31:20 FMC kernel: [ 6715] 0 6715 17758 2395 37 6938 0 SFDCNotificatio

Jun 5 08:31:20 FMC kernel: [ 6722] 101 6722 1965471 244039 1396 107727 -600 dbsrv16

Jun 5 08:31:20 FMC kernel: [ 6745] 101 6745 1426679 19778 240 32768 0 java

Jun 5 08:31:20 FMC kernel: [ 6746] 0 6746 3859 2 13 71 0 rabbitmq-server

Jun 5 08:31:20 FMC kernel: [ 6852] 0 6852 1282 18 7 39 0 sfmb

Jun 5 08:31:20 FMC kernel: [ 6853] 0 6853 124060 454 55 14393 -600 sftunnel

Jun 5 08:31:20 FMC kernel: [ 6854] 0 6854 5618 57 17 578 0 sfmgr

Jun 5 08:31:20 FMC kernel: [ 6855] 0 6855 2668 34 9 58 0 sfmbservice

Jun 5 08:31:20 FMC kernel: [ 6856] 0 6856 12325 8 26 876 0 SFRemediateD

Jun 5 08:31:20 FMC kernel: [ 6857] 0 6857 11244 71 24 6728 0 ntpd.pl

Jun 5 08:31:20 FMC kernel: [ 6858] 0 6858 21975 902 20 702 0 stunnel

Jun 5 08:31:20 FMC kernel: [ 6859] 67 6859 7049 44 17 428 0 hm_notifyd

Jun 5 08:31:20 FMC kernel: [ 6922] 0 6922 5574 46 14 87 0 stunnel

Jun 5 08:31:25 FMC msmtp: host=smtp2. tls=off auth=off from=

Jun 5 08:31:25 FMC SF-IMS[21190]: [21193] SFNotificationd:SFNotificationd [INFO] sfnotificationd.c:127:ReportStats(): EMAIL processing SUCCESS: 0.000934/sec for total: 2| FAILURE: 0.000000/sec for total: 0 in 2142 seconds

Jun 5 08:31:27 FMC SF-IMS[11943]: [11982] SFDataCorrelator:SSLEvents [INFO] Deleted 10 certificates from ssl_certificate_details

Jun 5 08:31:37 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun 5 08:31:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] IPRep, time to check for updates

Jun 5 08:31:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for ip verify_peer=1 verifyhost=0

Jun 5 08:31:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for dns verifypeer=1 verifyhost=0

Thanks in advanced,

Luiz Chagas.

1 Accepted Solution

Accepted Solutions

Hi Luiz,

No i do not have any documentation regarding this only my humble experience - 7 gb also seems fairly low to me considering the FMC basically should be considered a SQL server..

Have you tried rebooting the appliance .. Or use the cmds just to clear the swap:

>sudo swapoff -a

>sudo swapon -a

To clear the swap, this is not any good way of solving this issue, but it seems like you are having some issues with the swap.

View solution in original post

7 Replies 7

Hi Luiz,

Can you please make the cmd: vmstat 1

Leave it running for 10 secs and give me the output?

- Just a tip .. if you got enough memory, the FMC is running so much better with 16 or 32gb mem..

/Nikolaj

Hello Nikolaj,

I agree. So far, I didn't find a documentation about the recommendation of FMC in a virtual machine. Do you have this material?

Please follow below the vmstat 1 output.

admin@FMC:~$ vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

2 0 5009600 322728 10732 714704 7 4 103 112 1 4 10 4 86 1

2 0 5009600 322712 10732 714804 0 0 4 8 25484 79711 10 6 84 0

3 0 5009580 322712 10732 714880 64 0 96 40 23468 77174 10 6 83 1

2 0 5009580 322184 10732 714972 0 0 16 8 24956 81305 11 5 84 0

1 0 5009576 321876 10732 714812 32 0 36 12 24235 85176 9 6 85 0

1 0 5009576 322084 10732 714928 0 0 48 8 22826 76777 10 6 83 1

1 0 5009576 321824 10732 714984 0 0 4 2088 22770 79467 11 5 84 0

3 0 5009576 322132 10732 714868 0 0 48 32 23459 81441 10 6 84 0

1 0 5009572 322272 10732 714908 0 0 0 4216 21539 78084 11 6 83 0

1 0 5009568 322388 10732 715008 32 0 72 52 23193 80647 11 5 83 0

3 0 5009568 321700 10732 714844 0 0 0 8432 23903 76973 10 6 84 0

1 0 5009564 321768 10732 714908 0 0 16 8 25767 76423 10 6 84 0

1 0 5009564 322272 10732 714992 0 0 4 148 24570 82508 9 6 85 0

1 0 5009560 321788 10732 714808 32 0 32 12 22418 81604 10 6 83 0

1 0 5009560 321832 10732 714948 0 0 8 0 24307 79090 10 6 84 0

2 0 5009560 322072 10732 715004 0 0 4 8 23707 77960 9 5 85 0

1 0 5009560 321464 10732 714864 0 0 4 244 23971 81238 9 6 84 0

2 0 5009560 321584 10732 714996 0 0 84 40 24432 81103 10 6 84 0

1 0 5009560 322160 10732 714864 0 0 36 88 22738 81578 10 6 84 0

2 0 5009560 322048 10732 714932 0 0 0 8 20724 72593 10 6 84 0

1 0 5009516 322384 10732 714852 236 0 336 252 21027 77825 14 7 79 1

2 0 5009516 322100 10732 714684 0 0 12 20 21975 87591 10 6 84 0

4 0 5009512 322488 10732 714744 0 0 4 10116 21810 75971 9 6 85 1

2 0 5009508 322108 10732 714808 16 0 20 0 22867 75064 11 5 84 0

2 0 5009500 322680 10732 714896 0 0 0 0 23501 82838 10 10 80 0

2 1 5009500 321776 10732 714932 0 0 4 12 23650 72851 12 14 74 0

2 0 5009480 320080 10732 714968 32 0 44 28 19935 71271 26 6 67 0

4 0 5009480 320112 10732 714804 0 0 8 52 26372 82040 23 8 69 0

3 0 5009480 308504 10732 714848 0 0 8 3568 22534 80494 23 5 71 1

2 0 5009480 299948 10732 714884 0 0 0 8 23273 79672 22 6 72 0

3 0 5009480 319876 10732 714936 0 0 0 16 24975 76445 23 6 71 1

1 0 5009480 319968 10732 714972 0 0 0 2040 25489 77323 22 6 72 0

3 0 5009480 322296 10732 714816 0 0 0 32 23324 79974 18 9 73 0

2 0 5009480 321044 10732 714892 0 0 20 24 22498 76928 16 11 73 0

Many thanks.

Luiz Chagas

Hi Luiz,

No i do not have any documentation regarding this only my humble experience - 7 gb also seems fairly low to me considering the FMC basically should be considered a SQL server..

Have you tried rebooting the appliance .. Or use the cmds just to clear the swap:

>sudo swapoff -a

>sudo swapon -a

To clear the swap, this is not any good way of solving this issue, but it seems like you are having some issues with the swap.

Hi Nikolaj,

Thanks for your feedback. We're now proceeding to upgrade the memory to 32GB as recommended. I hope this kind of problems don't comes up again.

Best regards,

Luiz Chagas.

Great, you will experience a much more snappy FMC after the Memory upgrade

/Nikolaj

Just to let you know, Indeed after we upgrade the FMC's memory to 32GB, the backup has been executed successfully. No swap is seen anymore as you can see on item B.

------------------------------------------------

Before memory upgrade:

admin@FMC:~$ vmstat 1

A - procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b   swpd   free   buff cache   si   so    bi    bo   in   cs us sy id wa
2 0 5009600 322728 10732 714704    7    4   103   112    1    4 10 4 86 1
2 0 5009600 322712 10732 714804    0    0     4     8 25484 79711 10 6 84 0
3 0 5009580 322712 10732 714880   64    0    96    40 23468 77174 10 6 83 1
2 0 5009580 322184 10732 714972    0    0    16     8 24956 81305 11 5 84 0
1 0 5009576 321876 10732 714812   32    0    36    12 24235 85176 9 6 85 0

-------------------------------------------------

After memory upgrade:

admin@FMC:~$ vmstat 1

B - procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

1 0 0 15688528 29880 6644164 0 0 1256 1181 2046 253 18 6 73 3

2 0 0 15688044 29880 6644224 0 0 0 592 5928 10084 13 2 84 0

1 0 0 15687720 29880 6644276 0 0 0 260 6263 14496 19 2 79 0

1 0 0 15679176 29880 6644356 0 0 4 256 5515 8871 14 1 85 0

1 0 0 15672176 29880 6644404 0 0 0 292 5524 8787 15 1 84 0

1 0 0 15664992 29880 6644448 0 0 0 324 5428 7672 13 1 86 0

Great! Please remark one of the awnsers a correct if you like

Learn, share, save

Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.

New here? Get started with these tips. How to use Community New member guide

Log in to Community

Review Cisco Networking for a $25 gift card