DRS failure

and.kuznetsov · ‎10-11-2021

Hello colleagues! I've got an issue with DRS system which doesn't want to work correctly. During the backup procedure the Master Agent sometimes get stuck at TFTP stage or can't even connect to subscriber showing

Status:  Local Agent is not responding. 
This may be due to Master or Local Agent being down.

This behaviour happens every time in various manner. For instance, sometimes it might be sub1 or sub2 or even pub.

I've already tried

Restart Master and Local Agents

Regenerate ipsec certificates, Tomcat certs are valid

Restart the whole cluster

Check connection between nodes in cluster with ping even with size 1500

Check time among all nodes

Here are traces from pub and sub which were involved in the last iteration.

I would strongly appreciate if someone could help with that problem.

Nithin Eluvathingal · ‎10-11-2021

IPsec certificates are valid on all servers ? After regenerating the certificates have you restarted the Local and Master ?

I can see [CUCM-SUB1] has disconnected during backup. Is there a network issue ? Is there a firewall in between the PUB and SUB ?

and.kuznetsov · ‎10-11-2021

After certificate regeneration I restarted all these services. IPsec cert are valid, all subs have correct pub-cert in their trust storages.

There is no Firewall between pub and sub. Be the way same problem sometimes happens between pub and sub which are located in the same vlan.

Vaijanath Sonvane · ‎10-12-2021

Hi,

Do you have enough disk space on common partition of all CUCM servers? You can check it by using show status command.

Please rate helpful posts and if applicable mark "Accept as a Solution".
Thanks, Vaijanath S.

and.kuznetsov · ‎10-13-2021

Hello! From my point of view, the storage is ok

Host Name : cucm-pub0

Date : Срд Окт 13, 2021 10:17:46

Time Zone : Moscow Standard Time (Europe/Moscow)

Locale : ru_RU.UTF-8

Product Ver : 11.0.1.10000-10

Unified OS Version : 6.0.0.0-2

Uptime:

10:17:47 up 18 days, 2:54, 1 user, load average: 0.86, 0.80, 0.62

CPU Idle: 0% System: 0% User: 0%

IOWAIT: 0% IRQ: 0% Soft: 0%

Memory Total: 8062528K

Free: 283928K

Used: 7778600K

Cached: 3141016K

Shared: 509848K

Buffers: 310064K

Total Free Used

Disk/active 19805456K 6738996K 12863916K (66%)

Disk/inactive 19805456K 18747728K 44992K (1%)

Disk/logging 69235192K 42232452K 23479108K (36%)

Host Name : cucm-sub0

Date : Срд Окт 13, 2021 10:19:23

Time Zone : Moscow Standard Time (Europe/Moscow)

Locale : ru_RU.UTF-8

Product Ver : 11.0.1.10000-10

Unified OS Version : 6.0.0.0-2

Uptime:

10:19:25 up 18 days, 2:32, 1 user, load average: 0.09, 0.11, 0.12

CPU Idle: 0% System: 0% User: 0%

IOWAIT: 0% IRQ: 0% Soft: 0%

Memory Total: 8062528K

Free: 174992K

Used: 7887536K

Cached: 4528612K

Shared: 320348K

Buffers: 295228K

Total Free Used

Disk/active 19805456K 6810928K 12791984K (66%)

Disk/inactive 19805456K 18747728K 44992K (1%)

Disk/logging 69235192K 49548264K 16163296K (25%)

Host Name : cucm-sub1

Date : Срд Окт 13, 2021 10:20:13

Time Zone : Moscow Standard Time (Europe/Moscow)

Locale : ru_RU.UTF-8

Product Ver : 11.0.1.10000-10

Unified OS Version : 6.0.0.0-2

Uptime:

10:20:15 up 18 days, 1:48, 1 user, load average: 0.35, 0.14, 0.06

CPU Idle: 0% System: 0% User: 0%

IOWAIT: 0% IRQ: 0% Soft: 0%

Memory Total: 8062596K

Free: 180408K

Used: 7882188K

Cached: 4384984K

Shared: 322900K

Buffers: 312276K

Total Free Used

Disk/active 19805456K 6787664K 12815248K (66%)

Disk/inactive 19805456K 18747728K 44992K (1%)

Disk/logging 69235192K 47094440K 18617120K (29%)

Host Name : cucm-sub2

Date : Срд Окт 13, 2021 10:21:12

Time Zone : Moscow Standard Time (Europe/Moscow)

Locale : ru_RU.UTF-8

Product Ver : 11.0.1.10000-10

Unified OS Version : 6.0.0.0-2

Uptime:

10:21:14 up 18 days, 2:08, 1 user, load average: 0.64, 0.40, 0.36

CPU Idle: 0% System: 0% User: 0%

IOWAIT: 0% IRQ: 0% Soft: 0%

Memory Total: 5994128K

Free: 143312K

Used: 5850816K

Cached: 2515340K

Shared: 283412K

Buffers: 181816K

Total Free Used

Disk/active 19805456K 6797404K 12805508K (66%)

Disk/inactive 19805456K 18747728K 44992K (1%)

Disk/logging 69235192K 48439996K 17271564K (27%

admin:utils network ping cucm-sub0

PING cucm-sub0.ytss.ru (172.16.160.162) 56(84) bytes of data.

64 bytes from cucm-sub0.ytss.ru (172.16.160.162): icmp_seq=1 ttl=64 time=0.257 ms

Error running command:

Control-C pressed

admin:utils network ping cucm-sub1

PING cucm-sub1.ytss.ru (172.17.160.151) 56(84) bytes of data.

64 bytes from cucm-sub1.ytss.ru (172.17.160.151): icmp_seq=1 ttl=60 time=58.7 ms

Error running command:

Control-C pressed

admin:utils network ping cucm-sub2

PING cucm-sub2.ytss.ru (172.16.35.253) 56(84) bytes of data.

64 bytes from cucm-sub2.ytss.ru (172.16.35.253): icmp_seq=1 ttl=61 time=2.17 ms

Error running command:

Control-C pressed