UCS Manager 3.1(2e) VIP address not accessible from web

Jelena Mitrovic · ‎12-26-2018

Hello,

We have UCS system with FI Cisco UCS 6332 16UP cluster. Both FI's mgmt addresses and VIP IP are in same subnet.

FI-A: x.x.x.12/24

FI-B: x.x.x.13/24

VIP-IP: x.x.x.14/24

We can successfully ping and ssh to all three IP addresses. We can access to the UCS Manager GUI usingx.x.x.12 address, but not using VIP-IP. When we try to access to the UCSM GUI from VIP-IP we don't get login screen, rather we get The connection has timed out. When we try to access to UCSM from x.x.x.13/24 address we get login screen but after login we get message "Login Error: UCSM is not available on secondary node".

UCS Manager version is 3.1(2e). Can you please help us understand why UCSM GUI is not accessible from VIP address? We know that this was working before.

Cluster output from VIP-IP CLI is:

FI-A# show cluster extended-state
Cluster Id: xxxxxxx

A: UP, PRIMARY
B: UP, SUBORDINATE

A: memb state UP, lead state PRIMARY, mgmt services state: UP
B: memb state UP, lead state SUBORDINATE, mgmt services state: UP
heartbeat state PRIMARY_OK

INTERNAL NETWORK INTERFACES:
eth1, UP
eth2, UP

HA READY
Detailed state of the device selected for HA storage:
Chassis 1, serial: xxxxx, state: active
Chassis 2, serial: xxxxx, state: active

FI-A(local-mgmt)# show pmon state

SERVICE NAME             STATE     RETRY(MAX)    EXITCODE    SIGNAL    CORE
------------             -----     ----------    --------    ------    ----
svc_sam_controller     running           0(4)           0         0      no
svc_sam_dme            running           0(4)           0         0      no
svc_sam_dcosAG         running           0(4)           0         0      no
svc_sam_bladeAG        running           0(4)           0         0      no
svc_sam_portAG         running           1(4)           0        15      no
svc_sam_statsAG        running           0(4)           0         0      no
svc_sam_hostagentAG    running           0(4)           0         0      no
svc_sam_nicAG          running           0(4)           0         0      no
svc_sam_licenseAG      running           0(4)           0         0      no
svc_sam_extvmmAG       running           0(4)           0         0      no
httpd.sh               running           0(4)           0         0      no
httpd_cimc.sh          running           0(4)           0         0      no
svc_sam_sessionmgrAG   running           0(4)           0         0      no
svc_sam_pamProxy       running           0(4)           0         0      no
dhcpd                  running           0(4)           0         0      no
sam_core_mon           running           0(4)           0         0      no
svc_sam_rsdAG          running           0(4)           0         0      no
svc_sam_svcmonAG       running           0(4)           0         0      no
FI-A(local-mgmt)#

Thanks in advance!

mojafri · ‎12-29-2018

Hi @Jelena Mitrovic,

Could you please provide below output?

connect local-mgmt a
show mgmt-ip-debug ip-tables | no-more
show mgmt-ip-debug ip-tables | wc -l
connect local-mgmt b
show mgmt-ip-debug ip-tables | no-more
show mgmt-ip-debug ip-tables | wc -l

Just to make sure VIP is not getting blocked in-between, what is the output from below:

1. telnet <vip> 443

2. From linux- curl -k <vip>

Also, you may wanna capture traffic on Primary FI, in your case its FI-A:

#connect nxos a

# ethanalyzer local interface mgmt capture-filter "host <source-ip> and (port 80 or port 443)" limit-captured-frames 0 | no-more

FYI, below is an expected behavior if you use subordinate FI IP to access UCSM.

When we try to access to UCSM from x.x.x.13/24 address we get login screen but after login we get message "Login Error: UCSM is not available on secondary node".

Jelena Mitrovic · ‎01-03-2019

Hi,

First of all I wish you Happy New Year and thank you for helping us!

I have attached file with show commands outputs.

I have tried telnet from my computer to the VIP IP on 443 but it is not working.

Connecting To X.X.X.14...Could not open connection to the host, on port 443: Connect failed

Ping, telnet on port 80 and ssh to the VIP IP from my computer are working but we are not getting UCSM GUI when we try to access to it on port 80.

Unfortunately I don't have any linux VM so I don't have curl -k <vip> output.

Ethanalizer was captured while trying to open https://VIP-IP from my computer.

mojafri · ‎01-04-2019

Happy new year to you too !

We can see the https connection hitting mgmt interface, however weird part is I don't see any packets in iptable for VIP as well as for FI-A ip.

0 0 samrules all -- * * 0.0.0.0/0 X.X.X.12

0 0 samrules all -- * * 0.0.0.0/0 X.X.X.14

Are you sure you are able to access UCSM using FI-A ip?

Do you have any jump-box in same subnet of FIs and try to access there. (for accessing GUI with port-80 it will re-direct it to 443).

What about using different PC/source?

How you landed into this situation?

Provide below output which accessing GUI using VIP:

1. ssh to VIP--> connect nx

A(nxos)#show system internal file /var/sysmgr/sam_logs/httpd.log | last 100

Quick workaround could be to change cluster-lead which will move VIP to FI-B, however I'd say to open TAC case and get the logs analyse.

Regards,

MJ

Jelena Mitrovic · ‎01-14-2019

Hi,

Sorry for my delayed response.

Please find my answers below.

Are you sure you are able to access UCSM using FI-A ip?

[JM] Yes we are able to access to X.X.X.12 GUI using HTTPS as well as using SSH.

Do you have any jump-box in same subnet of FIs and try to access there. (for accessing GUI with port-80 it will re-direct it to 443).

[JM] Unfortunately we don't have any jump host in the same subnet to try direct access to the UCSM VIP.

What about using different PC/source?

[JM] We have tried to access to the VIP from different PC but behavior was the same.

How you landed into this situation?

[JM] From what I know problem was noticed after reload of the mgmt switch was performed. We don't know if problem was present before this action.

Provide below output which accessing GUI using VIP:

1. ssh to VIP--> connect nx

A(nxos)#show system internal file /var/sysmgr/sam_logs/httpd.log | last 100

[JM] In the attach document is requested output.

Quick workaround could be to change cluster-lead which will move VIP to FI-B, however I'd say to open TAC case and get the logs analyse.

[JM] We will propose to change cluster lead fo FI-B to our customer and we will see when this operation can be performed and if needed we will open TAC case.

In the meantime is it safe to perform any configuration changes from GUI of FI-A address since UCSM is active on this FI and is accessible from GUI?

Regards,

Jelena