cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
4866
Views
4
Helpful
7
Replies

Firepower Management Center Virtual - Backup Failed - 6.2.3

lfntchagas
Level 1
Level 1

Hi Gents,

I'm facing a issue in my environment when I try execute a backup using FMC-virtual version 6.2.3. Looks like there is no resource enough. as you can see below:

Does anyone might help me on this?

---------------------------------------------------

/var/log/messages

Jun  5 08:31:20 FMC kernel: [<ffffffff815a6c78>] page_fault+0x28/0x30

Jun  5 08:31:20 FMC kernel: Task in /System killed as a result of limit of /System

Jun  5 08:31:20 FMC kernel: memory: usage 5005072kB, limit 5005072kB, failcnt 7414397

Jun  5 08:31:20 FMC kernel: memory+swap: usage 8372560kB, limit 8925368kB, failcnt 0

Jun  5 08:31:20 FMC kernel: kmem: usage 0kB, limit 9007199254740991kB, failcnt 0

Jun  5 08:31:20 FMC kernel: Memory cgroup stats for /System: cache:1444KB rss:5003620KB rss_huge:0KB mapped_file:204KB swap:3367488KB inactive_anon:1165056KB active_anon:3838784KB inactive_file:88KB active_file:4KB unevictable:0KB

----------------------------------------------------

admin@FMC:~$ free -m

             total       used       free     shared    buffers     cached

Mem:          6982       5602       1379          0         11        265

-/+ buffers/cache:       5326       1656

Swap:         5469       5368        100

------------------------------------------------------

Whole log:

admin@FMC:~$ tail -f /var/log/messages

Jun  5 08:20:24 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:20:24 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:20:24 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun  5 08:20:24 FMC process_msg : Received IPC message type : 12

Jun  5 08:20:24 FMC Response being sent to SAM : ▒

Jun  5 08:20:24 FMC , len(msg being sent) = 2576

Jun  5 08:20:24 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:20:24 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:20:24 FMC Connection closed...

Jun  5 08:20:24 FMC Waiting for someone to connect to me...

Jun  5 08:21:00 FMC SF-IMS[6307]: [6307] pm:process [INFO] Started store_whitelist_history (25397)

Jun  5 08:21:01 FMC SF-IMS[6307]: [6307] pm:log [INFO] Process 'store_whitelist_history' closed output.

Jun  5 08:21:02 FMC sudo:      www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown www:www /var/log/CSMAgent.log

Jun  5 08:21:02 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:02 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:03 FMC sudo:      www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/kill -s USR1 7440

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:03 FMC sudo:      www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mv /var/tmp/backupeZIW/files_not_to_tar /var/tmp/backupfmoa/files_not_to_tar

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:03 FMC sudo:      www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/mv /var/tmp/backupeZIW/files_to_tar /var/tmp/backupfmoa/files_to_tar

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:03 FMC sudo:      www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/sf/bin/sf-backup.pl /var/sf/backup/FMC-BACKUP-CONFIG.tar.gz

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:03 FMC sudo:      www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:03 FMC sudo:      www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:03 FMC sudo:      www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:03 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:07 FMC sudo:      www : TTY=unknown ; PWD=/usr/local/sf/htdocs/admin/sensor ; USER=root ; COMMAND=/bin/chown -R www:www /var/tmp/ttc

Jun  5 08:21:07 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:07 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:07 FMC sudo:      www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown www:www /var/log/CSMAgent.log

Jun  5 08:21:07 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:07 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:08 FMC sudo:      www : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/chown www:www /var/log/CSMAgent.log

Jun  5 08:21:08 FMC sudo: pam_unix(sudo:session): session opened for user root by (uid=0)

Jun  5 08:21:08 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:21:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 1 expired services (query time: 0.001 sec.)

Jun  5 08:21:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 4 expired client apps (query time: 0.007 sec.)

Jun  5 08:21:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] IPRep, time to check for updates

Jun  5 08:21:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for ip  verify_peer=1  verifyhost=0

Jun  5 08:21:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for dns verifypeer=1    verifyhost=0

Jun  5 08:22:23 FMC syslog-ng[3574]: Log statistics; processed='source(msgs)=133688', processed='global(payload_reallocs)=137871', processed='src.none()=0', stamp='src.none()=0', processed='global(internal_queue_length)=0', processed='destination(CSCOmd_destination)=0', processed='global(msg_clones)=0', processed='destination(cron_destination)=212', processed='src.internal(msgs#2)=945', stamp='src.internal(msgs#2)=1528186343', processed='global(sdata_updates)=0', processed='destination(messages_destination)=92733', processed='center(received)=133688', processed='center(queued)=92945'

Jun  5 08:22:33 FMC SF-IMS[11943]: [11989] SFDataCorrelator:SSLEvents [INFO] Deleted 10 certificates from ssl_certificate_details

Jun  5 08:22:49 FMC Someone connected to me, receiving data...

Jun  5 08:22:49 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:22:49 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:22:49 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun  5 08:22:49 FMC process_msg : Received IPC message type : 12

Jun  5 08:22:49 FMC Response being sent to SAM : ▒

Jun  5 08:22:49 FMC , len(msg being sent) = 2576

Jun  5 08:22:49 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:22:49 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:22:49 FMC Connection closed...

Jun  5 08:22:49 FMC Waiting for someone to connect to me...

Jun  5 08:24:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 4 expired IP hosts (query time: 0.002 sec.)

Jun  5 08:24:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 4 expired client apps (query time: 0.007 sec.)

Jun  5 08:24:27 FMC SF-IMS[6854]: [6934] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:24:27 FMC SF-IMS[6854]: [28842] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:24:27 FMC SF-IMS[6854]: [17786] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:24:27 FMC SF-IMS[6854]: [22784] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:25:14 FMC Someone connected to me, receiving data...

Jun  5 08:25:14 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:25:14 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:25:14 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun  5 08:25:14 FMC process_msg : Received IPC message type : 12

Jun  5 08:25:14 FMC Response being sent to SAM : ▒

Jun  5 08:25:14 FMC , len(msg being sent) = 2575

Jun  5 08:25:14 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:25:14 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:25:14 FMC Connection closed...

Jun  5 08:25:14 FMC Waiting for someone to connect to me...

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): SQL Anywhere Personal Server Version 16.0.0.2271

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): OEM Authenticated Edition, licensed only for use with authenticated OEM applications.

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng):

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Copyright (c) 2015 SAP SE or an SAP affiliate company.

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): All rights reserved.

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Use of this software is governed by the Sybase License Agreement.

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Refer to http://www.sybase.com/softwarelicenses.

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng):

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Connection limit (Personal Server): 10

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Processors detected: 4 (containing 8 logical processors)

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Processor limit (Personal Server): 1

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Maximum number of processors the server will use: 1 physical processor(s), 4 core(s)

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): This server is licensed to:

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng):     cisco

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng):     cisco

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Running Linux 3.10.107sf.virtual-1 #1 SMP Thu Mar 8 18:37:12 UTC 2018 on X86_64

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Server built for X86_64 processor architecture

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): 177108K of memory used for caching

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Minimum cache size: 11512K, maximum cache size: 6431404K

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Using a maximum page size of 4096 bytes

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Multiprogramming level: 20

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Database server started at Tue Jun 05 2018 08:25

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Trying to start SharedMemory link ...

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng):     SharedMemory link started successfully

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Now accepting requests

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Starting database "dbvvms" (/var/tmp/backupfmoa/0/vms/database/vms.db) at Tue Jun 05 2018 08:25

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng): Database recovery in progress

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng):     Last checkpoint at Tue Jun 05 2018 08:24

Jun  5 08:25:27 FMC SQLAnywhere(dbvcsmeng):     Checkpoint log...

Jun  5 08:25:28 FMC SQLAnywhere(dbvcsmeng):     Transaction log: vms.log...

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng):     Rollback log...

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng):     Checkpointing...

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Starting checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:25

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Finished checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:25

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Recovery complete

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng): Database "dbvvms" (vms.db) started at Tue Jun 05 2018 08:25

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng): This database is licensed for use with:

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng):     Application: NMTG

Jun  5 08:25:29 FMC SQLAnywhere(dbvcsmeng):     Company: Cisco Systems

Jun  5 08:25:39 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 876464K

Jun  5 08:25:44 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1058868K

Jun  5 08:25:54 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1204476K

Jun  5 08:25:59 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1270452K

Jun  5 08:26:00 FMC SF-IMS[6307]: [6307] pm:process [INFO] Started store_whitelist_history (28151)

Jun  5 08:26:01 FMC SF-IMS[6307]: [6307] pm:log [INFO] Process 'store_whitelist_history' closed output.

Jun  5 08:26:04 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1296848K

Jun  5 08:26:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] IPRep, time to check for updates

Jun  5 08:26:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] ClamUpd, time to check for updates

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for ip  verify_peer=1  verifyhost=0

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for dns verifypeer=1    verifyhost=0

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file .. from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/..

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file . from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/.

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] chown successful

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] The curl option for clam verify_peer=1  verify_host=2

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file .. from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/..

Jun  5 08:26:54 FMC SF-IMS[6713]: [6740] CloudAgent:ClamUpdater [INFO] Removing file . from clamupd tmp dir. Full path is /var/sf/clamupd_download/tmp/.

Jun  5 08:26:55 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1504320K

Jun  5 08:27:02 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1646432K

Jun  5 08:27:07 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1696040K

Jun  5 08:27:12 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1708248K

Jun  5 08:27:17 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1738880K

Jun  5 08:27:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 2 expired services (query time: 0.001 sec.)

Jun  5 08:27:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 6 expired client apps (query time: 0.007 sec.)

Jun  5 08:27:22 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1745676K

Jun  5 08:27:27 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1747300K

Jun  5 08:27:32 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1748236K

Jun  5 08:27:44 FMC Someone connected to me, receiving data...

Jun  5 08:27:44 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:27:44 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:27:44 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun  5 08:27:44 FMC process_msg : Received IPC message type : 12

Jun  5 08:27:44 FMC Response being sent to SAM : ▒

Jun  5 08:27:44 FMC , len(msg being sent) = 2576

Jun  5 08:27:44 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:27:44 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:27:44 FMC Connection closed...

Jun  5 08:27:44 FMC Waiting for someone to connect to me...

Jun  5 08:28:26 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1805020K

Jun  5 08:28:31 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1848112K

Jun  5 08:28:36 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1858452K

Jun  5 08:28:41 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1859876K

Jun  5 08:28:46 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1845260K

Jun  5 08:28:51 FMC kernel: ffff8801f51d3c00 ffff8801eeef3c28 ffffffff815a1ecf ffff8801eeef3ca8

Jun  5 08:28:51 FMC kernel: [ 6867]     0  6867     6975      335      18      474             0 snmpd

Jun  5 08:28:51 FMC kernel: [ 6923]     0  6923     5574       46      14       87             0 stunnel

Jun  5 08:28:51 FMC kernel: [ 6924]     0  6924     5574       46      14       87             0 stunnel

Jun  5 08:28:51 FMC kernel: [ 7108]     0  7108     1036        8       7       12             0 erl_child_setup

Jun  5 08:28:51 FMC kernel: [ 7129]     0  7129    23559        5      18      206             0 diskmanager

Jun  5 08:28:51 FMC kernel: [ 7130]     0  7130     4386        0      11     3166             0 diskmgr_handler

Jun  5 08:28:51 FMC kernel: [ 7198]     0  7198    57353        6      22      230             0 diskmanager

Jun  5 08:28:51 FMC kernel: [ 7199]     0  7199     4386        1      11     3165             0 diskmgr_handler

Jun  5 08:28:51 FMC kernel: [ 7432]     0  7432    13800      237      29     8076          -600 perl

Jun  5 08:28:51 FMC kernel: [ 7435]     0  7435   101965      396      49     1506             0 sfestreamer

Jun  5 08:28:51 FMC kernel: [ 7436]     0  7436    13010       63      30      321             0 fpcollect

Jun  5 08:28:51 FMC kernel: [ 7438]     0  7438    18761     4176      39     6048             0 expire-session.

Jun  5 08:28:51 FMC kernel: [ 7439]     0  7439    66213    37421     133    19787             0 Pruner.pl

Jun  5 08:28:51 FMC kernel: [ 7440]     0  7440    11479     2049      25     5002             0 ActionQueueScra

Jun  5 08:28:51 FMC kernel: [ 7441]     0  7441    29056     3021      59    17030             0 PerlMessageHand

Jun  5 08:28:51 FMC kernel: [ 7442]     0  7442    16079      271      33     7993             0 snapshot_manage

Jun  5 08:28:51 FMC kernel: [ 7443]    67  7443     5585     1297      14     1470             0 SFTop10Cacher

Jun  5 08:28:51 FMC kernel: [ 7444]     0  7444     4683      291      13     3157             0 query_scheduler

Jun  5 08:28:51 FMC kernel: [ 7445]     0  7445  1238843    93886     452    63654          -600 java

Jun  5 08:28:51 FMC kernel: [ 7483]     0  7483    16364     1273      34     7225             0 uimp.pl

Jun  5 08:28:51 FMC kernel: [ 7484]     0  7484     3519       27      10      210             0 ui_archiver

Jun  5 08:28:51 FMC kernel: [ 7485]     0  7485    18332     1469      39     7417             0 TSS_Daemon.pl

Jun  5 08:28:51 FMC kernel: [ 7514]    67  7514    16746     3547      36     5364             0 SFTop10Cacher

Jun  5 08:28:51 FMC kernel: [ 7523]     0  7523    37569     9349      76    18837          -600 perl

Jun  5 08:28:51 FMC kernel: [ 7524]     0  7524    48997     1226     100    38753          -600 perl

Jun  5 08:28:51 FMC kernel: [ 7525]     0  7525    37754     3555      77    25263          -600 perl

Jun  5 08:28:51 FMC kernel: [ 7526]     0  7526    42423     5992      87    27488          -600 perl

Jun  5 08:28:51 FMC kernel: [ 7527]     0  7527    40560     6931      82    24203          -600 perl

Jun  5 08:28:51 FMC kernel: [ 7547]     0  7547     2857        6      11       21             0 inet_gethost

Jun  5 08:28:51 FMC kernel: [ 7548]     0  7548     2857        9      11       23             0 inet_gethost

Jun  5 08:29:28 FMC SF-IMS[7439]: Pruner.pl:Pruner.pl [WARN] Time change found.  Resetting status in Pruner.pl

Jun  5 08:30:08 FMC Someone connected to me, receiving data...

Jun  5 08:30:08 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:30:08 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:30:08 FMC before recv(), total_bytes_read = 8, msg_len = 10

Jun  5 08:30:08 FMC process_msg : Received IPC message type : 12

Jun  5 08:30:09 FMC Response being sent to SAM : ▒

Jun  5 08:30:09 FMC , len(msg being sent) = 2576

Jun  5 08:30:09 FMC sla_worker : sizeof(msg) : 8192

Jun  5 08:30:09 FMC before recv(), total_bytes_read = 0, hdr_len = 8

Jun  5 08:30:09 FMC Connection closed...

Jun  5 08:30:09 FMC Waiting for someone to connect to me...

Jun  5 08:30:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 1 expired IP hosts (query time: 0.004 sec.)

Jun  5 08:30:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 1 expired services (query time: 0.010 sec.)

Jun  5 08:30:21 FMC SF-IMS[11943]: [12231] SFDataCorrelator:Correlator [INFO] Purging 5 expired client apps (query time: 0.008 sec.)

Jun  5 08:30:28 FMC SF-IMS[6854]: [6934] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:30:29 FMC SF-IMS[6854]: [28842] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:30:29 FMC SF-IMS[6854]: [17786] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:30:29 FMC SF-IMS[6854]: [22784] sfmgr:sfmanager [INFO] Received Broadcast message route_size=132

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): SQL Anywhere Personal Server Version 16.0.0.2271

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): OEM Authenticated Edition, licensed only for use with authenticated OEM applications.

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng):

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Copyright (c) 2015 SAP SE or an SAP affiliate company.

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): All rights reserved.

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Use of this software is governed by the Sybase License Agreement.

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Refer to http://www.sybase.com/softwarelicenses.

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng):

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Connection limit (Personal Server): 10

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Processors detected: 4 (containing 8 logical processors)

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Processor limit (Personal Server): 1

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Maximum number of processors the server will use: 1 physical processor(s), 4 core(s)

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): This server is licensed to:

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng):     cisco

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng):     cisco

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Running Linux 3.10.107sf.virtual-1 #1 SMP Thu Mar 8 18:37:12 UTC 2018 on X86_64

Jun  5 08:30:43 FMC SQLAnywhere(dbvcsmeng): Server built for X86_64 processor architecture

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): 186440K of memory used for caching

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Minimum cache size: 11512K, maximum cache size: 6431404K

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Using a maximum page size of 4096 bytes

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Multiprogramming level: 20

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Database server started at Tue Jun 05 2018 08:30

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Trying to start SharedMemory link ...

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng):     SharedMemory link started successfully

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Now accepting requests

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Starting database "dbvvms" (/var/tmp/backupfmoa/0/vms/database/vms.db) at Tue Jun 05 2018 08:30

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng): Database recovery in progress

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng):     Last checkpoint at Tue Jun 05 2018 08:29

Jun  5 08:30:44 FMC SQLAnywhere(dbvcsmeng):     Checkpoint log...

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng):     Transaction log: vms.log...

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng):     Rollback log...

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng):     Checkpointing...

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Starting checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:30

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Finished checkpoint of "dbvvms" (vms.db) at Tue Jun 05 2018 08:30

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Recovery complete

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng): Database "dbvvms" (vms.db) started at Tue Jun 05 2018 08:30

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng): This database is licensed for use with:

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng):     Application: NMTG

Jun  5 08:30:45 FMC SQLAnywhere(dbvcsmeng):     Company: Cisco Systems

Jun  5 08:30:50 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1434564K

Jun  5 08:31:00 FMC SF-IMS[6307]: [6307] pm:process [INFO] Started store_whitelist_history (29686)

Jun  5 08:31:00 FMC SF-IMS[6307]: [6307] pm:log [INFO] Process 'store_whitelist_history' closed output.

Jun  5 08:31:10 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1763520K

Jun  5 08:31:15 FMC SQLAnywhere(dbvcsmeng): Cache size adjusted to 1828840K

Jun  5 08:31:20 FMC kernel: 2_scheduler invoked oom-killer: gfp_mask=0xd0, order=0, oom_score_adj=0

Jun  5 08:31:20 FMC kernel: 2_scheduler cpuset=/ mems_allowed=0

Jun  5 08:31:20 FMC kernel: CPU: 2 PID: 7110 Comm: 2_scheduler Not tainted 3.10.107sf.virtual-1 #1

Jun  5 08:31:20 FMC kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014

Jun  5 08:31:20 FMC kernel: ffff8801f51d3c00 ffff8801eee4bc28 ffffffff815a1ecf ffff8801eee4bca8

Jun  5 08:31:20 FMC kernel: ffffffff815a03f3 0000000400386900 00000000000000e2 ffff8801eee4bc58

Jun  5 08:31:20 FMC kernel: ffff8801eee4bc68 0000000000000282 ffffffff00000000 ffff8801eee4bca8

Jun  5 08:31:20 FMC kernel: Call Trace:

Jun  5 08:31:20 FMC kernel: [<ffffffff815a1ecf>] dump_stack+0x19/0x1b

Jun  5 08:31:20 FMC kernel: [<ffffffff815a03f3>] dump_header.isra.8+0x87/0x1d4

Jun  5 08:31:20 FMC kernel: [<ffffffff812466a7>] ? ___ratelimit+0xec/0x117

Jun  5 08:31:20 FMC kernel: [<ffffffff810a4a43>] oom_kill_process+0x6f/0x36f

Jun  5 08:31:20 FMC kernel: [<ffffffff81044107>] ? has_capability_noaudit+0x12/0x14

Jun  5 08:31:20 FMC kernel: [<ffffffff810d6bdb>] mem_cgroup_oom_synchronize+0x3df/0x4b8

Jun  5 08:31:20 FMC kernel: [<ffffffff810d631e>] ? mem_cgroup_can_attach+0x14b/0x14b

Jun  5 08:31:20 FMC kernel: [<ffffffff810a5167>] pagefault_out_of_memory+0xe/0x4a

Jun  5 08:31:20 FMC kernel: [<ffffffff8159f0e5>] mm_fault_error+0x5d/0xff

Jun  5 08:31:20 FMC kernel: [<ffffffff81032187>] __do_page_fault+0x2d5/0x47e

Jun  5 08:31:20 FMC kernel: [<ffffffff811100bf>] ? timerfd_get_remaining+0x11/0x21

Jun  5 08:31:20 FMC kernel: [<ffffffff811105f2>] ? do_timerfd_settime+0x292/0x2d1

Jun  5 08:31:20 FMC kernel: [<ffffffff81032358>] do_page_fault+0x9/0xb

Jun  5 08:31:20 FMC kernel: [<ffffffff815a6c78>] page_fault+0x28/0x30

Jun  5 08:31:20 FMC kernel: Task in /System killed as a result of limit of /System

Jun  5 08:31:20 FMC kernel: memory: usage 5005072kB, limit 5005072kB, failcnt 7414397

Jun  5 08:31:20 FMC kernel: memory+swap: usage 8372560kB, limit 8925368kB, failcnt 0

Jun  5 08:31:20 FMC kernel: kmem: usage 0kB, limit 9007199254740991kB, failcnt 0

Jun  5 08:31:20 FMC kernel: Memory cgroup stats for /System: cache:1444KB rss:5003620KB rss_huge:0KB mapped_file:204KB swap:3367488KB inactive_anon:1165056KB active_anon:3838784KB inactive_file:88KB active_file:4KB unevictable:0KB

Jun  5 08:31:20 FMC kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name

Jun  5 08:31:20 FMC kernel: [ 6713]     0  6713    84006      230      43      516             0 CloudAgent

Jun  5 08:31:20 FMC kernel: [ 6715]     0  6715    17758     2395      37     6938             0 SFDCNotificatio

Jun  5 08:31:20 FMC kernel: [ 6722]   101  6722  1965471   244039    1396   107727          -600 dbsrv16

Jun  5 08:31:20 FMC kernel: [ 6745]   101  6745  1426679    19778     240    32768             0 java

Jun  5 08:31:20 FMC kernel: [ 6746]     0  6746     3859        2      13       71             0 rabbitmq-server

Jun  5 08:31:20 FMC kernel: [ 6852]     0  6852     1282       18       7       39             0 sfmb

Jun  5 08:31:20 FMC kernel: [ 6853]     0  6853   124060      454      55    14393          -600 sftunnel

Jun  5 08:31:20 FMC kernel: [ 6854]     0  6854     5618       57      17      578             0 sfmgr

Jun  5 08:31:20 FMC kernel: [ 6855]     0  6855     2668       34       9       58             0 sfmbservice

Jun  5 08:31:20 FMC kernel: [ 6856]     0  6856    12325        8      26      876             0 SFRemediateD

Jun  5 08:31:20 FMC kernel: [ 6857]     0  6857    11244       71      24     6728             0 ntpd.pl

Jun  5 08:31:20 FMC kernel: [ 6858]     0  6858    21975      902      20      702             0 stunnel

Jun  5 08:31:20 FMC kernel: [ 6859]    67  6859     7049       44      17      428             0 hm_notifyd

Jun  5 08:31:20 FMC kernel: [ 6922]     0  6922     5574       46      14       87             0 stunnel

Jun  5 08:31:25 FMC msmtp: host=smtp2. tls=off auth=off from=

Jun  5 08:31:25 FMC SF-IMS[21190]: [21193] SFNotificationd:SFNotificationd [INFO] sfnotificationd.c:127:ReportStats(): EMAIL processing SUCCESS: 0.000934/sec for total: 2| FAILURE: 0.000000/sec for total: 0 in 2142 seconds

Jun  5 08:31:27 FMC SF-IMS[11943]: [11982] SFDataCorrelator:SSLEvents [INFO] Deleted 10 certificates from ssl_certificate_details

Jun  5 08:31:37 FMC sudo: pam_unix(sudo:session): session closed for user root

Jun  5 08:31:53 FMC SF-IMS[6713]: [6713] CloudAgent:CloudAgent [INFO] IPRep, time to check for updates

Jun  5 08:31:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for ip  verify_peer=1  verifyhost=0

Jun  5 08:31:54 FMC SF-IMS[6713]: [6740] CloudAgent:IPReputation [INFO] The curl option for dns verifypeer=1    verifyhost=0

Thanks in advanced,

Luiz Chagas.

1 Accepted Solution

Accepted Solutions

Hi Luiz,

No i do not have any documentation regarding this only my humble experience - 7 gb also seems fairly low to me considering the FMC basically should be considered a SQL server..

Have you tried rebooting the appliance .. Or use the cmds just to clear the swap:

>sudo swapoff -a

>sudo swapon -a

To clear the swap, this is not any good way of solving this issue, but it seems like you are having some issues with the swap.

View solution in original post

7 Replies 7

Nikolaj Pabst
Level 5
Level 5

Hi Luiz,

Can you please make the cmd: vmstat 1

Leave it running for 10 secs and give me the output?

- Just a tip .. if you got enough memory, the FMC is running so much better with 16 or 32gb mem..

/Nikolaj

Hello Nikolaj,

I agree. So far, I didn't find a documentation about the recommendation of FMC in a virtual machine. Do you have this material?

Please follow below the vmstat 1 output.

admin@FMC:~$ vmstat 1

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa

2  0 5009600 322728  10732 714704    7    4   103   112    1    4 10  4 86  1

2  0 5009600 322712  10732 714804    0    0     4     8 25484 79711 10  6 84  0

3  0 5009580 322712  10732 714880   64    0    96    40 23468 77174 10  6 83  1

2  0 5009580 322184  10732 714972    0    0    16     8 24956 81305 11  5 84  0

1  0 5009576 321876  10732 714812   32    0    36    12 24235 85176  9  6 85  0

1  0 5009576 322084  10732 714928    0    0    48     8 22826 76777 10  6 83  1

1  0 5009576 321824  10732 714984    0    0     4  2088 22770 79467 11  5 84  0

3  0 5009576 322132  10732 714868    0    0    48    32 23459 81441 10  6 84  0

1  0 5009572 322272  10732 714908    0    0     0  4216 21539 78084 11  6 83  0

1  0 5009568 322388  10732 715008   32    0    72    52 23193 80647 11  5 83  0

3  0 5009568 321700  10732 714844    0    0     0  8432 23903 76973 10  6 84  0

1  0 5009564 321768  10732 714908    0    0    16     8 25767 76423 10  6 84  0

1  0 5009564 322272  10732 714992    0    0     4   148 24570 82508  9  6 85  0

1  0 5009560 321788  10732 714808   32    0    32    12 22418 81604 10  6 83  0

1  0 5009560 321832  10732 714948    0    0     8     0 24307 79090 10  6 84  0

2  0 5009560 322072  10732 715004    0    0     4     8 23707 77960  9  5 85  0

1  0 5009560 321464  10732 714864    0    0     4   244 23971 81238  9  6 84  0

2  0 5009560 321584  10732 714996    0    0    84    40 24432 81103 10  6 84  0

1  0 5009560 322160  10732 714864    0    0    36    88 22738 81578 10  6 84  0

2  0 5009560 322048  10732 714932    0    0     0     8 20724 72593 10  6 84  0

1  0 5009516 322384  10732 714852  236    0   336   252 21027 77825 14  7 79  1

2  0 5009516 322100  10732 714684    0    0    12    20 21975 87591 10  6 84  0

4  0 5009512 322488  10732 714744    0    0     4 10116 21810 75971  9  6 85  1

2  0 5009508 322108  10732 714808   16    0    20     0 22867 75064 11  5 84  0

2  0 5009500 322680  10732 714896    0    0     0     0 23501 82838 10 10 80  0

2  1 5009500 321776  10732 714932    0    0     4    12 23650 72851 12 14 74  0

2  0 5009480 320080  10732 714968   32    0    44    28 19935 71271 26  6 67  0

4  0 5009480 320112  10732 714804    0    0     8    52 26372 82040 23  8 69  0

3  0 5009480 308504  10732 714848    0    0     8  3568 22534 80494 23  5 71  1

2  0 5009480 299948  10732 714884    0    0     0     8 23273 79672 22  6 72  0

3  0 5009480 319876  10732 714936    0    0     0    16 24975 76445 23  6 71  1

1  0 5009480 319968  10732 714972    0    0     0  2040 25489 77323 22  6 72  0

3  0 5009480 322296  10732 714816    0    0     0    32 23324 79974 18  9 73  0

2  0 5009480 321044  10732 714892    0    0    20    24 22498 76928 16 11 73  0

Many thanks.


Luiz Chagas

Hi Luiz,

No i do not have any documentation regarding this only my humble experience - 7 gb also seems fairly low to me considering the FMC basically should be considered a SQL server..

Have you tried rebooting the appliance .. Or use the cmds just to clear the swap:

>sudo swapoff -a

>sudo swapon -a

To clear the swap, this is not any good way of solving this issue, but it seems like you are having some issues with the swap.

Hi Nikolaj,

Thanks for your feedback. We're now proceeding to upgrade the memory to 32GB as recommended. I hope this kind of problems don't comes up again.

Best regards,

Luiz Chagas.

Great, you will experience a much more snappy FMC after the Memory upgrade

/Nikolaj

Just to let you know, Indeed after we upgrade the FMC's memory to 32GB, the backup has been executed successfully. No swap is seen anymore as you can see on item B.

------------------------------------------------

Before memory upgrade:

admin@FMC:~$ vmstat 1

A - procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
2  0 5009600 322728  10732 714704    7    4   103   112    1    4 10  4 86  1
2  0 5009600 322712  10732 714804    0    0     4     8 25484 79711 10  6 84  0
3  0 5009580 322712  10732 714880   64    0    96    40 23468 77174 10  6 83  1
2  0 5009580 322184  10732 714972    0    0    16     8 24956 81305 11  5 84  0
1  0 5009576 321876  10732 714812   32    0    36    12 24235 85176  9  6 85  0

-------------------------------------------------

After memory upgrade:

admin@FMC:~$ vmstat 1

B - procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa

1  0      0 15688528  29880 6644164    0    0  1256  1181 2046  253 18  6 73  3

2  0      0 15688044  29880 6644224    0    0     0   592 5928 10084 13  2 84  0

1  0      0 15687720  29880 6644276    0    0     0   260 6263 14496 19  2 79  0

1  0      0 15679176  29880 6644356    0    0     4   256 5515 8871 14  1 85  0

1  0      0 15672176  29880 6644404    0    0     0   292 5524 8787 15  1 84  0

1  0      0 15664992  29880 6644448    0    0     0   324 5428 7672 13  1 86  0

Great! Please remark one of the awnsers a correct if you like

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: