Enabling IPMI on B200-M3 blade

mwaldron104 · ‎03-03-2020

I've been trying without success to get IPMI working on B200-M3 blade servers. I created an InBand management IP address on the CIMC, by creating an IP Pool and associating it with the appropriate VLAN in a VLAN Group, then using that in the InBand Policy section of LAN Cloud global policies.

I can ping the InBand IP assigned to the CIMC, and the KVM console as well as Serial Over LAN works on it. I created an IPMI/Redfish Access Policy on the service profile, yet I can't get any response to ipmitool command:

ipmitool -I lanplus -H x.x.x.x -U xxxx chassis status

After about 10 seconds, I get: Error: Unable to establish IPMI v2 / RMCP+ session

Am I missing something?

Kirk J · ‎03-03-2020

What UCSM and blade bundle firmware version are you running?

I seem to remember an older bug that impacted inband connectivity issues (something where iptables didnt get correctly applied to the CIMC's bond interface for the inband config)...

Kirk...

mwaldron104 · ‎03-03-2020

Using UCSM 4.0(4e) and firmware bundle 4.0.4e.

It certainly behaves as if a firewall is blocking it. A tcpdump to other systems I have using ipmitool shows response packets to the command, whereas tcpdump to the UCS server shows the request packet going to the server, but no reply packet is received.

Kirk J · ‎03-03-2020

Greetings.

Connect to your UCSM via ssh.

#connect cimc x/y (x being chassis, y being blade)

cimc debug firmware utility

#network

The output should list the various CIMC internal interfaces followed by a netstat type output.

Curious to see if you see a 'bond' interface as well as entry for IPMI udp port

My lab (out of band) example:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:8192 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:4010 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:23000 0.0.0.0:* LISTEN
tcp 0 0 127.4.0.2:4010 127.4.0.254:44309 ESTABLISHED
tcp 0 0 127.4.0.2:4010 127.4.0.254:37194 ESTABLISHED
tcp 0 0 127.3.0.2:8192 127.3.0.254:51545 ESTABLISHED
tcp 0 0 127.3.0.2:4010 127.3.0.254:59053 ESTABLISHED
tcp 0 0 127.5.1.2:4010 127.5.254.1:43664 ESTABLISHED
tcp 0 0 127.3.0.2:4010 127.3.0.254:59044 ESTABLISHED
tcp 0 0 127.6.1.2:4010 127.6.254.1:52994 ESTABLISHED
tcp 0 0 127.4.0.2:4010 127.4.0.254:37203 ESTABLISHED
tcp 0 0 127.3.0.2:4010 127.3.0.254:52848 ESTABLISHED
tcp 0 0 :::2068 :::* LISTEN
tcp 0 0 :::8021 :::* LISTEN
tcp 0 0 :::22 :::* LISTEN
tcp 0 0 :::8022 :::* LISTEN
tcp 0 82 ::ffff:127.5.1.2:8021 ::ffff:127.5.254.1:38051 ESTABLISHED
udp 0 0 0.0.0.0:41530 0.0.0.0:*
udp 0 0 0.0.0.0:319 0.0.0.0:*
udp 0 0 0.0.0.0:320 0.0.0.0:*
udp 0 0 :::623 :::* <<<<<<<<<<<<<<<<<<

Kirk...

mwaldron104 · ‎03-03-2020

Output of the network command is pasted below. There is a bond.161 interface which is the VLAN being used for the inband management address. I redacted the actual addresses.

# network
bond0 Link encap:Ethernet HWaddr XXXXXXXXXXX
inet6 addr: XXXXXXXXXXX Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:5463100 errors:0 dropped:0 overruns:0 frame:0
TX packets:1464821 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:571878962 (545.3 MiB) TX bytes:316202263 (301.5 MiB)

bond0.161 Link encap:Ethernet HWaddr XXXXXXXX
inet addr:XXXXXXXX Bcast:0.0.0.0 Mask:XXXXXXXXX
inet6 addr: XXXXXXXXXX Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1655566 errors:0 dropped:0 overruns:0 frame:0
TX packets:355253 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:129353256 (123.3 MiB) TX bytes:34251089 (32.6 MiB)

eth0 Link encap:Ethernet HWaddr XXXXXXXXXX
inet6 addr: XXXXXXXXXXXXXXX Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1979528 errors:0 dropped:0 overruns:0 frame:0
TX packets:578026 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:178671458 (170.3 MiB) TX bytes:84368332 (80.4 MiB)

eth0.1 Link encap:Ethernet HWaddr XXXXXXXXXXXXXXX
inet addr:127.3.0.4 Bcast:127.3.255.255 Mask:255.255.0.0
inet6 addr: fe80::f872:eaff:fe8e:c32f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:300186 errors:0 dropped:0 overruns:0 frame:0
TX packets:200107 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:20772577 (19.8 MiB) TX bytes:39042520 (37.2 MiB)

eth0.4044 Link encap:Ethernet HWaddr XXXXXXXXXXXX
inet addr:127.5.1.4 Bcast:127.5.255.255 Mask:255.255.0.0
inet6 addr: fe80::f872:eaff:fe8e:c32f/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:135272 errors:0 dropped:0 overruns:0 frame:0
TX packets:135359 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:10543059 (10.0 MiB) TX bytes:20975264 (20.0 MiB)

eth1 Link encap:Ethernet HWaddr XXXXXXXXXXXXXXXX
inet6 addr: fe80::f872:eaff:fe8e:c32e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:3483572 errors:0 dropped:0 overruns:0 frame:0
TX packets:886795 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:393207504 (374.9 MiB) TX bytes:231833931 (221.0 MiB)
Interrupt:1

eth1.1 Link encap:Ethernet HWaddr XXXXXXXXXXXXX
inet addr:127.4.0.4 Bcast:127.4.255.255 Mask:255.255.0.0
inet6 addr: fe80::f872:eaff:fe8e:c32e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1011307 errors:0 dropped:0 overruns:0 frame:0
TX packets:559821 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:131156973 (125.0 MiB) TX bytes:134548155 (128.3 MiB)

eth1.4044 Link encap:Ethernet HWaddr XXXXXXXXXXXXXXX
inet addr:127.6.1.4 Bcast:127.6.255.255 Mask:255.255.0.0
inet6 addr: fe80::f872:eaff:fe8e:c32e/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1057746 errors:0 dropped:0 overruns:0 frame:0
TX packets:214233 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:104887903 (100.0 MiB) TX bytes:85960575 (81.9 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.255.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:58354 errors:0 dropped:0 overruns:0 frame:0
TX packets:58354 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5548855 (5.2 MiB) TX bytes:5548855 (5.2 MiB)

virt_eth0_0 Link encap:Ethernet HWaddr XXXXXXXXXXXXX
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:1979528 errors:0 dropped:0 overruns:0 frame:0
TX packets:578026 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:178671458 (170.3 MiB) TX bytes:84368332 (80.4 MiB)

virt_eth1_0 Link encap:Ethernet HWaddr XXXXXXXXXXXX
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:3483572 errors:0 dropped:0 overruns:0 frame:0
TX packets:886795 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:393207504 (374.9 MiB) TX bytes:231833931 (221.0 MiB)

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:8192 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:4010 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:23000 0.0.0.0:* LISTEN
tcp 0 0 127.4.0.4:4010 127.4.0.254:39271 ESTABLISHED
tcp 0 0 127.6.1.4:4010 127.6.254.1:39772 ESTABLISHED
tcp 0 0 127.4.0.4:4010 127.4.0.254:39279 ESTABLISHED
tcp 0 0 127.4.0.4:4010 127.4.0.254:55962 ESTABLISHED
tcp 0 0 127.4.0.4:8192 127.4.0.254:46561 ESTABLISHED
tcp 0 0 127.3.0.4:4010 127.3.0.254:57554 ESTABLISHED
tcp 0 0 127.3.0.4:4010 127.3.0.254:55523 ESTABLISHED
tcp 0 0 127.5.1.4:4010 127.5.254.1:44059 ESTABLISHED
tcp 0 0 127.3.0.4:4010 127.3.0.254:57551 ESTABLISHED
tcp 0 0 :::2068 :::* LISTEN
tcp 0 0 :::8021 :::* LISTEN
tcp 0 0 :::22 :::* LISTEN
tcp 0 0 :::8022 :::* LISTEN
tcp 0 0 ::ffff:127.5.1.4:8021 ::ffff:127.5.254.1:54948 ESTABLISHED
udp 0 0 0.0.0.0:319 0.0.0.0:*
udp 0 0 0.0.0.0:320 0.0.0.0:*
udp 0 0 0.0.0.0:57300 0.0.0.0:*
udp 0 0 :::623 :::*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags Type State I-Node Path
unix 2 [ ACC ] STREAM LISTENING 3121 /tmp/storage-socket-flex
unix 2 [ ] DGRAM 2644 /tmp/local-user-access.1339
unix 2 [ ACC ] STREAM LISTENING 7904601 /tmp/storage-socket
unix 2 [ ACC ] SEQPACKET LISTENING 7904603 /tmp/storage-events
unix 2 [ ACC ] STREAM LISTENING 3005 /tmp/rpSocket
unix 22 [ ] DGRAM 4555 /dev/log
unix 2 [ ] DGRAM 2549 /tmp/local-user-access
unix 2 [ ACC ] STREAM LISTENING 3066 /var/ipmi_vks_device
unix 2 [ ] DGRAM 8129602
unix 2 [ ] DGRAM 8129555
unix 2 [ ] DGRAM 8007412
unix 2 [ ] DGRAM 8007202
unix 2 [ ] DGRAM 7904574
unix 2 [ ] DGRAM 5653540
unix 2 [ ] DGRAM 2802132
unix 2 [ ] DGRAM 2802090
unix 2 [ ] DGRAM 1068297
unix 2 [ ] DGRAM 333397
unix 2 [ ] DGRAM 332497
unix 2 [ ] DGRAM 332492
unix 2 [ ] DGRAM 332486
unix 2 [ ] DGRAM 332478
unix 2 [ ] DGRAM 332467
unix 2 [ ] DGRAM 332465
unix 2 [ ] DGRAM 5773
unix 2 [ ] DGRAM 5618
unix 2 [ ] DGRAM 5305
unix 2 [ ] DGRAM 5302
unix 2 [ ] DGRAM 4237
unix 2 [ ] DGRAM 4109
unix 2 [ ] DGRAM 3953
unix 2 [ ] DGRAM 3743
unix 2 [ ] DGRAM 3596
unix 2 [ ] DGRAM 3593
unix 2 [ ] DGRAM 3519
unix 2 [ ] DGRAM 3403
unix 2 [ ] DGRAM 3229
unix 2 [ ] DGRAM 3161
unix 2 [ ] DGRAM 3136
unix 2 [ ] DGRAM 3120
unix 2 [ ] DGRAM 3087
unix 2 [ ] DGRAM 3065
unix 2 [ ] DGRAM 3047
unix 2 [ ] DGRAM 3037
unix 2 [ ] DGRAM 3023
unix 2 [ ] DGRAM 3013
unix 2 [ ] DGRAM 2904
unix 2 [ ] DGRAM 2625
unix 2 [ ] DGRAM 2546
unix 2 [ ] DGRAM 2398
unix 2 [ ] DGRAM 2367
unix 2 [ ] DGRAM 2357

Kirk J · ‎03-03-2020

Can you try telneting (or some kind of tcp port tester), to your CIMC inband IP address and check some of the other ports that should be listening such as 2068, 8021,22

Your output shows udp port 268 as open, but not sure why you don't get any response back from it.

Also, any chance you can throw an out-of-band IP address on it,,, and test to that IP ? Just looking to see if this is something going on with inband iptables related, which is where I've seen similar issues before.

If that is still under contract, then you may want to open a TAC case.

Normally, when tracking a TAC case for this type of issue, I'd need to go into debug mode on the CIMC, so we can look at iptables, route, check IPMI tool.

On M4s, there is a tcpdump utility built into the CIMC (at debug level), but not sure about M3s.

Kirk...

mwaldron104 · ‎03-03-2020

I can connect to 2068 and 22 using telnet, but can't connect to 8021. However, using nmap shows the ports open. My understanding is that IPMI uses udp port 623, which also seems to be open in nmap, but get a connection refused with telnet.

# nmap -p 2068 XXXXXX

Starting Nmap 5.51 ( http://nmap.org ) at 2020-03-03 16:14 EST
Nmap scan report for XXXXXXX
Host is up (0.00075s latency).
PORT STATE SERVICE
2068/tcp open advocentkvm

# nmap -p 8021 XXXXXXXX

Starting Nmap 5.51 ( http://nmap.org ) at 2020-03-03 16:14 EST
Nmap scan report for XXXXXXX
Host is up (0.00077s latency).
PORT STATE SERVICE
8021/tcp filtered ftp-proxy

# nmap -p 22 XXXXXXX

Starting Nmap 5.51 ( http://nmap.org ) at 2020-03-03 16:15 EST
Nmap scan report for XXXXXX
Host is up (0.00069s latency).
PORT STATE SERVICE
22/tcp open ssh

# nmap -p 623 -sU XXXXXX

Starting Nmap 5.51 ( http://nmap.org ) at 2020-03-03 16:17 EST
Nmap scan report for XXXXXX
Host is up (0.00067s latency).
PORT STATE SERVICE
623/udp open|filtered asf-rmcp

Kirk J · ‎03-04-2020

I'm working on setting something up in the lab to test.

Will post when I get it setup.

Kirk...

mwaldron104 · ‎03-04-2020

Awesome, thank you!

Kirk J · ‎03-13-2020

Finally got around to setting this up in the lab and it works.

The only thing I would note, is that I had added my IPMI related policy on an existing profile, and it didn't seem to really apply the config until after I selected the 'Reapply configuration' from the service profile context.

My config is B200M4, running 4.04g firmware.

Kirk...

mwaldron104 · ‎03-16-2020

Kirk,

Thanks for doing that. I tried to reapply configuration as well, but it has no effect. I have B200M3 blades, so perhaps something different there.

Do you think the fact that the blades have both an outband and inband management IP could be an issue? The server issuing the IPMI commands is on the vlan that's using the inband IP. I wonder perhaps if the CIMC is trying to talk back to it using the outband IP.