08-23-2013 11:15 AM - edited 03-01-2019 07:24 AM
Just tried upgrading 2 of the 4 chassis' I have and it went horribly wrong. Well not really, it just failed with almost no packet drops.
This is a OTV setup across two sites. I did a ISSU upgrade on 2 chassis simultanuously (1 in each site) and got exactly the same error on both.
2013 Aug 23 20:45:28 glsdswn7k001 %$ VDC-1 %$ %PLATFORM-2-MOD_REMOVE: Module 6 removed (Serial number JAFXXXX)
2013 Aug 23 20:45:28 glsdswcore001 %$ VDC-2 %$ last message repeated 1 time
2013 Aug 23 20:45:28 glsdswotv001 %$ VDC-3 %$ last message repeated 1 time
2013 Aug 23 20:49:23 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
2013 Aug 23 20:49:54 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
2013 Aug 23 20:50:25 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
2013 Aug 23 20:50:56 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
2013 Aug 23 20:51:27 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
2013 Aug 23 20:51:58 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
2013 Aug 23 20:52:29 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
2013 Aug 23 20:53:00 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
Resetting boot variables. Please wait.
2013 Aug 23 20:53:32 glsdswn7k001 %$ VDC-1 %$ %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id 0xFFFFFFFF).
[####################] 100%
2013 Aug 23 20:53:47 glsdswn7k001 %$ VDC-1 %$ %PLATFORM-2-MOD_REMOVE: Module 6 removed (Serial number JAFXXXX)
2013 Aug 23 20:53:47 glsdswcore001 %$ VDC-2 %$ last message repeated 1 time
2013 Aug 23 20:53:47 glsdswotv001 %$ VDC-3 %$ last message repeated 1 time
Failure recovery action::
"Standby will be rebooted to force netboot and image download".
Install has failed. Return code 0x4093001E (Standby failed to come online).
Please identify the cause of the failure, and try 'install all' again.
The logs show the following:
2013 Aug 23 20:47:10 glpdswn7k002 %PLATFORM-2-MOD_REMOVE: Module 6 removed (Serial number JAFXXXX)
2013 Aug 23 20:49:21 glpdswn7k002 %BOOTVAR-5-NEIGHBOR_UPDATE_AUTOCOPY: auto-copy supported by neighbor supervisor, starting...
2013 Aug 23 20:50:56 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:50:56 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 8979) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:51:27 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:51:27 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 9132) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:51:58 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:51:58 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 9349) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:52:29 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:52:29 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 9384) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:53:00 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:53:00 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 9402) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:53:31 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:53:31 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 9435) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:54:02 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:54:02 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 9623) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:54:33 glpdswn7k002 %SYSMGR-STANDBY-2-CONVERT_FUNC_FAILED: Conversion function failed for service "otv" (error-id
0xFFFFFFFF).
2013 Aug 23 20:54:33 glpdswn7k002 %SYSMGR-STANDBY-3-SERVICE_TERMINATED: Service "otv" (PID 9769) has finished with error code SY
SMGR_EXITCODE_SYSERR (1).
2013 Aug 23 20:55:00 glpdswn7k002 %SYSMGR-3-SERVICE_TERMINATED: Service "installer" (PID 26230) has finished with error code SYS
MGR_EXITCODE_FAILURE_NOCALLHOME (20).
2013 Aug 23 20:55:02 glpdswn7k002 %PLATFORM-2-MOD_REMOVE: Module 6 removed (Serial number JAFXXXX)
2013 Aug 23 20:57:21 glpdswn7k002 %BOOTVAR-5-NEIGHBOR_UPDATE_AUTOCOPY: auto-copy supported by neighbor supervisor, starting...
2013 Aug 23 20:59:03 glpdswn7k002 %MODULE-5-STANDBY_SUP_OK: Supervisor 6 is standby
Any idea anyone?
Solved! Go to Solution.
08-24-2013 05:18 AM
Did you read the release notes? It clearly states
OTV
Any upgrade from an image that is earlier than Cisco NX-OS Release 6.2(2) to an image that is Cisco NX-OS Release 6.2(2) or later in an OTV network is disruptive. When you upgrade from any previous release, the OTV overlay needs to be shut down for ISSU to operate.
For more details, see the "Preparing OTV for ISSU to Cisco NX-OS 5.2(1) or Later Releases in a Dual-Homed Site" section in the Cisco Nexus 7000 Series NX-OS OTV Configuration Guide.
Regards Pille
08-23-2013 12:29 PM
Thought I'd dump the config of the OTV VDC here as well...
version 6.1(4)
hostname lpdswotv002
feature tacacs+
cfs eth distribute
feature ospf
feature otv
feature udld
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature bfd
username admin password 5 kk role vdc-admin
ip domain-lookup
ip access-list ALL_IPs
10 permit ip any any
mac access-list ALL_MACs
10 permit any any
ip access-list HSRP_IP
10 permit udp any 224.0.0.2/32 eq 1985
20 permit udp any 224.0.0.102/32 eq 1985
mac access-list HSRP_VMAC
10 permit 0000.0c07.ac00 0000.0000.00ff any
20 permit 0000.0c9f.f000 0000.0000.0fff any
arp access-list HSRP_VMAC_ARP
10 deny ip any mac 0000.0c07.ac00 ffff.ffff.ff00
20 deny ip any mac 0000.0c9f.f000 ffff.ffff.f000
30 permit ip any mac any
vlan access-map HSRP_Localization 10
match mac address HSRP_VMAC
match ip address HSRP_IP
action drop
vlan access-map HSRP_Localization 20
match mac address ALL_MACs
match ip address ALL_IPs
action forward
vlan filter HSRP_Localization vlan-list 1500-1509,1600-1620
snmp-server user admin vdc-admin auth md5 0x4789e0334323ad58a117a4a94b priv 0x478934334338a117a4a94b localizedkey
rmon event 1 log trap public description FATAL(1) owner PMON@FATAL
rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 log trap public description ERROR(3) owner PMON@ERROR
rmon event 4 log trap public description WARNING(4) owner PMON@WARNING
rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO
ip routing event-history general size medium
ip route 0.0.0.0/0 10.236.0.13
vrf context management
ip route 0.0.0.0/0 10.236.16.1
vlan 1,14,1500-1509,1600-1620
vlan 14
name DC1_OTV_Site_VLAN
otv site-vlan 14
service dhcp
ip dhcp relay
interface Vlan1
interface port-channel6
description OTV Internal Interface
switchport
switchport mode trunk
switchport trunk allowed vlan 14,1500-1699
interface Overlay1
otv join-interface Ethernet7/1
otv extend-vlan 1500-1699
otv use-adjacency-server 10.236.0.1 10.237.0.10 unicast-only
no shutdown
interface Ethernet7/1
description OTV Join Interface
rate-mode dedicated force
mtu 9216
no ip redirects
ip address 10.236.0.14/30
no ipv6 redirects
ip ospf network point-to-point
no ip ospf passive-interface
ip router ospf 1 area 0.0.0.0
ip ospf bfd
no shutdown
interface Ethernet7/2
rate-mode dedicated force
switchport
switchport mode trunk
switchport trunk allowed vlan 14,1500-1699
channel-group 6 mode active
no shutdown
interface Ethernet7/3
interface Ethernet7/4
interface Ethernet7/5
interface Ethernet7/6
interface Ethernet7/7
interface Ethernet7/8
interface Ethernet7/9
rate-mode dedicated force
switchport
switchport mode trunk
switchport trunk allowed vlan 14,1500-1699
channel-group 6 mode active
no shutdown
interface Ethernet7/10
interface Ethernet7/11
interface Ethernet7/12
interface Ethernet7/13
interface Ethernet7/14
interface Ethernet7/15
interface Ethernet7/16
interface Ethernet7/17
interface Ethernet7/18
interface Ethernet7/19
interface Ethernet7/20
interface Ethernet7/21
interface Ethernet7/22
interface Ethernet7/23
interface Ethernet7/24
interface Ethernet7/25
interface Ethernet7/26
interface Ethernet7/27
interface Ethernet7/28
interface Ethernet7/29
interface Ethernet7/30
interface Ethernet7/31
interface Ethernet7/32
interface mgmt0
ip address 10.236.16.23/24
cli alias name wr copy run start
line console
terminal width 128
line vty
router ospf 1
router-id 10.236.16.23
log-adjacency-changes
passive-interface default
otv site-identifier 0x10
ip arp inspection filter HSRP_VMAC_ARP vlan 1500-1699
08-24-2013 05:18 AM
Did you read the release notes? It clearly states
OTV
Any upgrade from an image that is earlier than Cisco NX-OS Release 6.2(2) to an image that is Cisco NX-OS Release 6.2(2) or later in an OTV network is disruptive. When you upgrade from any previous release, the OTV overlay needs to be shut down for ISSU to operate.
For more details, see the "Preparing OTV for ISSU to Cisco NX-OS 5.2(1) or Later Releases in a Dual-Homed Site" section in the Cisco Nexus 7000 Series NX-OS OTV Configuration Guide.
Regards Pille
08-24-2013 05:30 AM
Hi Pille,
I actually did read it but apparently not thoroughly enough. Thanks for pointing this out.
I'll have another go.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide