IOx stops working on IR809

[gabrijels]

Hi,

I have a problem with IOx server failing after a certain time. The server works fine but then drops out the first time GI0 loses connection. After that GI2 can't be reached anymore and my docker application loses connection. I have provided logs from the router and the router configuration for reference. Any suggestion is welcome.

Thanks,

Gabrijel

----------------------------------------------------- LOG -----------------------------------------------------------------------------------------------------

*Jan 2 00:00:00.606: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = ir800 Next reboot level = ipbasek9 and License = ipbasek9
*Jan 2 00:00:00.678: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = ir800 Next reboot level = securityk9 and License = securityk9
*Jan 2 00:00:00.752: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = ir800 Next reboot level = datak9 and License = datak9
*Nov 13 20:00:02.581: CELLULAR: modem in slot 0 powered up

*Nov 13 20:00:08.775: %ACT2-5-STARTED: ACT2 process started.
*Nov 13 20:00:10.300: %LINK-3-UPDOWN: Interface GigabitEthernet2, changed state to up
*Nov 13 20:00:10.300: %LINK-3-UPDOWN: Interface GigabitEthernet1, changed state to down
*Nov 13 20:00:10.300: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 13 20:00:11.828: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2, changed state to up
*Nov 13 20:00:11.828: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1, changed state to down
*Nov 13 20:00:11.828: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 13 20:00:14.767: %SYS-5-CONFIG_I: Configured from memory by console
*Nov 13 20:00:14.907: %LINK-3-UPDOWN: Interface Async1, changed state to up
*Nov 13 20:00:14.907: %LINK-3-UPDOWN: Interface Async0, changed state to up
*Nov 13 20:00:15.087: %SYS-5-RESTART: System restarted --
Cisco IOS Software, ir800 Software (ir800-UNIVERSALK9-M), Version 15.8(3)M2a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Mon 06-May-19 01:49 by prod_rel_team
*Nov 13 20:00:15.193: %SYS-6-BOOTTIME: Time taken to reboot after reload = 162 seconds
*Nov 13 20:00:15.221: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Initialized
*Nov 13 20:00:15.221: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Enabled
*Nov 13 20:00:15.221: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Nov 13 20:00:15.225: %LINEPROTO-5-UPDOWN: Line protocol on Interface NVI0, changed state to up
*Nov 13 20:00:15.231: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 13 20:00:15.231: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 13 20:00:15.231: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 13 20:00:15.231: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 13 20:00:15.233: %QAT15_CRYPTO-6-SELF_TEST_RESULT: Self test info: (Self test started.)
*Nov 13 20:00:15.555: %QAT15_CRYPTO-6-SELF_TEST_RESULT: Self test info: (All Tests Passed)
*Nov 13 20:00:15.555: %SYS-3-HARIKARI: Process IR800 Test top-level routine exited
*Nov 13 20:00:15.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface Async0, changed state to up
*Nov 13 20:00:16.228: %LINK-5-CHANGED: Interface GigabitEthernet1, changed state to administratively down
*Nov 13 20:00:19.140: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 13 20:00:20.141: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 13 20:00:20.411: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)
*Nov 13 20:00:26.305: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 13 20:00:26.305: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 13 20:01:38.397: %IOX-6-SOCK_CONNECT: Received socket connection request from IOX Client
*Nov 13 20:01:38.399: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_REGISTER from IOX Client
*Nov 13 20:01:51.708: %IOX-6-SOCK_CONNECT: Received socket connection request from IOX Client
*Nov 13 20:01:54.746: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_REGISTER from IOX Client
*Nov 13 20:01:54.793: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_PROTECT_KEY from IOX Client
*Nov 13 20:01:55.001: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_PROTECT_KEY to IOX Client
*Nov 13 20:01:55.875: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_TIME from IOX Client
*Nov 13 20:01:58.914: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_PROTECT_KEY from IOX Client
*Nov 13 20:01:59.120: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_PROTECT_KEY to IOX Client
*Nov 13 20:02:23.438: %IOX-6-SOCK_CONNECT: Received socket connection request from IOX Client
*Nov 13 20:02:23.438: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_REGISTER from IOX Client
*Nov 13 20:02:38.414: %IOX-6-HEARTBEAT_TIMER_EXPIRED: IOX Heartbeat timer expired.
*Nov 13 20:02:43.874: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_UDI_INFO from IOX Client
*Nov 13 20:02:43.876: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_UDI_INFO to IOX Client
*Nov 13 20:02:43.884: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_UDI_KEY from IOX Client
*Nov 13 20:02:43.884: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_UDI_KEY to IOX Client
*Nov 13 20:02:49.836: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_AUTHENTICATION from IOX Client
*Nov 13 20:02:49.836: AAA/LOCAL/LOGIN(00000000): Failed to find interface
*Nov 13 20:03:06.933: %CELLULAR-2-MODEM_UP: Cellular0 modem is now UP.
*Nov 13 20:04:06.952: FOTA not supported on modems older than 74xx.
*Nov 13 20:04:07.112: %CELLWAN-2-MODEM_RADIO: Cellular0 Modem radio has been turned on
*Nov 14 20:02:00.381: %IOX-6-HEARTBEAT_TIMER_EXPIRED: IOX Heartbeat timer expired.
*Nov 14 20:02:05.665: %IOX-6-HEARTBEAT_TIMER_EXPIRED: IOX Heartbeat timer expired.
*Nov 16 06:00:40.743: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 16 06:00:41.743: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 16 06:00:57.748: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 16 06:00:58.748: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 16 06:01:10.752: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 16 06:01:11.752: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 16 06:01:24.756: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 16 06:01:25.757: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 16 06:04:16.810: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 16 06:04:17.811: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 16 06:04:38.817: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 16 06:04:39.818: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 16 15:43:55.736: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 16 15:43:56.737: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 16 16:46:46.921: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 16 16:46:47.921: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 16 16:58:09.135: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 16 16:58:10.136: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 19 00:33:52.011: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 19 00:33:53.011: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up

---------------------------------------------ROUTER CONFIGURATION -------------------------------------------------------------------------

Building configuration...

Current configuration : 4608 bytes
!
version 15.8
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname WeInt0
!
boot-start-marker
boot system flash:/ir800-universalk9-mz.SPA.158-3.M2a
boot-end-marker
!
!
enable password cisco
!
no aaa new-model
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!
!

!
ip dhcp excluded-address 192.168.10.1 192.168.10.5
!
ip dhcp pool gospool
network 192.168.10.0 255.255.255.0
default-router 192.168.10.1
domain-name cisco.com
dns-server 192.168.1.1
remember
!
!
!
ip name-server 192.168.1.1
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
!
!
chat-script lte "" "AT!CALL" TIMEOUT 20 "OK"
!
crypto pki trustpoint TP-self-signed-422713650
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-422713650
revocation-check none
rsakeypair TP-self-signed-422713650
!
!
crypto pki certificate chain TP-self-signed-422713650
certificate self-signed 01
30820229 30820192 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
30312E30 2C060355 04031325 494F532D 53656C66 2D536967 6E65642D 43657274
69666963 6174652D 34323237 31333635 30301E17 0D313930 37313832 30333631
345A170D 32303031 30313030 30303030 5A303031 2E302C06 03550403 1325494F
532D5365 6C662D53 69676E65 642D4365 72746966 69636174 652D3432 32373133
36353030 819F300D 06092A86 4886F70D 01010105 0003818D 00308189 02818100
AE128D3C D06AAFC5 A6C9D2F7 53C93883 5D4B5730 4F545DD4 59622984 E58A05D7
5A90FF47 A0CDEBF0 776B11D7 3B7F9D72 015DBA7B EBA7F880 FC98B997 2CD7A8D6
4F021260 9FF60694 58D5B3DA 5AD81BDF CF464A25 92523F79 FBBA69DC 373BA2C8
650DF2EE 9FB9F405 9D73BF18 5189E045 6EC1C471 2F9B099B D4478540 0E97D60B
02030100 01A35330 51300F06 03551D13 0101FF04 05300301 01FF301F 0603551D
23041830 1680147B 22C19923 C334610A C4ABC362 DBB94A9E A6AEC730 1D060355
1D0E0416 04147B22 C19923C3 34610AC4 ABC362DB B94A9EA6 AEC7300D 06092A86
4886F70D 01010505 00038181 000267C2 C4F85933 B4DF3CCC EC7727F6 512341ED
B8C6DA7B 48F94634 F300F472 2AF0B92A 079CFDA0 DA78B3D9 C477CDEC 3E67B551
C4D2D82C 31B1E983 11BCC12A 4979DE1C 1328F7C9 65F14593 FAEA0EEC B09BCDB2
E8B052E9 F4EA1C40 75748606 E57308B4 37DD7E5F DB571E61 9799DC23 1B938C09
47B2BDBA 36AF0B70 404BF42E 19
quit
license udi pid IR809G-LTE-GA-K9 sn FCW23030H60
!
!
username domagoj password 0 cisco123
username cisco privilege 15 password 0 cisco123
!
redundancy

!
!
!
!
!
controller Cellular 0
lte sim fast-switchover enable
lte failovertimer 5
!
!
!
!
!
!
!
!
!
!
!
interface GigabitEthernet0
ip address 192.168.1.99 255.255.255.0
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet1
no ip address
shutdown
duplex auto
speed auto
!
interface GigabitEthernet2
ip address 192.168.10.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
duplex auto
speed auto
ipv6 address autoconfig
ipv6 enable
!
interface Cellular0
no ip address
encapsulation slip
dialer in-band
dialer string lte
!
interface Cellular1
no ip address
encapsulation slip
!
interface Async0
no ip address
encapsulation relay-line
half-duplex
media-type rs485
!
interface Async1
no ip address
encapsulation scada
!
!
ip forward-protocol nd
!
ip http server
ip http authentication local
ip http secure-server
!
ip dns server
ip nat inside source list NAT_ACL interface GigabitEthernet0 overload
ip nat inside source static 192.168.10.6 192.168.1.106
ip nat inside source static 192.168.10.9 192.168.1.109
ip route 0.0.0.0 0.0.0.0 192.168.1.1
!
ip access-list standard NAT_ACL
permit 192.168.0.0 0.0.255.255
!
ipv6 ioam timestamp
!
!
!
control-plane
!
!
configuration mode exclusive
!
line con 0
stopbits 1
line 1
transport preferred none
transport input all
stopbits 1
speed 38400
line 2
transport preferred none
transport input all
stopbits 1
line 3
script dialer lte
no exec
transport preferred lat pad telnet rlogin lapb-ta mop udptn v120 ssh
transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh
line 8
no exec
transport preferred lat pad telnet rlogin lapb-ta mop udptn v120 ssh
transport output lat pad telnet rlogin lapb-ta mop udptn v120 ssh
line 1/3
transport preferred none
transport output none
stopbits 1
line 1/4
transport preferred none
transport input all
transport output all
stopbits 1
line 1/5 1/6
transport preferred none
transport output none
stopbits 1
speed 38400
line vty 0 4
login local
transport input all
transport output all
!
no scheduler max-task-time
relay line 1 1/5
no iox hdm-enable
iox client enable interface GigabitEthernet2
iox recovery-enable 5
!
!
!
!
!
!
!
end

[suressan]

Hi Gabrijel,

>>The server works fine but then drops out the first time GI0 loses connection. After that GI2 can't be reached anymore and my docker application loses connection

Could u pls describe your 809 network setup ? when you sever drops out, do you mean Local Manager is not reacheble ? Which version of IOS/IOx are you using ? Are you seeing the interfaces GigE0 and GigE2 going down without IOx in picture ?

What is the output of "show iox host list detail" ?

Thanks

Suresh S

[gabrijels]

Hi,

The IR809 is connected to my home router over GE0 port. When it goes down, I can't reach 192.168.1.106 any more, even though the application continued to work. Also, the docker application loses connection. GigE0 is still working and I can reach 192.168.1.99.

The docker application is set-up with the following activation:
{
"resources": {
"profile": "c1.small",
"network": [{"interface-name": "eth0", "network-name": "iox-nat0","port_map": {"mode": "1to1"},"ports":{"tcp":[8086,8080,9999,9000,1883]}}]
}
}

The output of sh iox host list details is as follows:

sh iox host list detail

IOX Server is running. Process ID: 326
Count of hosts registered: 1

Host registered:
===============
IOX Server Address: FE80::200:CFF:FE2A:11D9; Port: 22222

Link Local Address of Host: FE80::1FF:FE90:8B05
IPV4 Address of Host: 192.168.10.6
IPV6 Address of Host: fe80::1ff:fe90:8b05
Client Version: 0.4
Session ID: 2
OS Nodename: WeInt0-GOS-1
Host Hardware Vendor: Cisco Systems, Inc.
Host Hardware Version: 1.0
Host Card Type: not implemented
Host OS Version: 1.8.4.2
OS status: RUNNING

Interface Hardware Vendor: None
Interface Hardware Version: None
Interface Card Type: None

Services:
===============
Service Name: Secure Storage Service
Service Status: RUNNING
Session ID: 3

Service Name: Host Device Management Service
Service Status: DISABLED
Session ID: 0

----------------------------------WHEN IT IS NOT WORKING------------------------------------------------

IOX Server is running. Process ID: 326
Count of hosts registered: 1

Host registered:
===============
IOX Server Address: FE80::200:CFF:FE2A:11D9; Port: 22222

Link Local Address of Host: FE80::1FF:FE90:8B05
IPV4 Address of Host: 192.168.10.6
IPV6 Address of Host: fe80::1ff:fe90:8b05
Client Version: 0.4
Session ID: 2
OS Nodename: WeInt0-GOS-1
Host Hardware Vendor: Cisco Systems, Inc.
Host Hardware Version: 1.0
Host Card Type: not implemented
Host OS Version: 1.8.4.2
OS status: RUNNING

Interface Hardware Vendor: None
Interface Hardware Version: None
Interface Card Type: None

Services:
===============
Service Name: Secure Storage Service
Service Status: RUNNING
Session ID: 3

Service Name: Host Device Management Service
Service Status: DISABLED
Session ID: 0

[Emmanuel Tychon]

Hello Gabrijel -

I have read and re-read your question multiple time and we will need you to be a lot more precise and complete if you want us to provide any helpful support.

I understand the IR809 is connected to your home router over Ge0 port, therefore Ge0 is your WAN interface. When you say Ge0 is going down, do you mean the port goes down (like when physically pulling cable out)?

When Ge0 goes down you can't reach 192.168.1.106 any more. Since the interface connected to subnet 192.168.1.0/24 (Ge0) is down it make sense you can't reach 192.168.1.106 any more. You can check with "sh ip ro" that the route to that subnet is gone, or "show ip nat translations" to check which translations are currently active.

However when Ge0 is down you should be able to reach 192.168.10.6 as this is the internal IP, as well as 192.168.10.1 (Ge2).

When you say "After that GI2 can't be reached anymore and my docker application loses connection". I'm not sure what "after that" means, neither "the docker application loses connection". Connection to where? Then you say "the application continued to work", what are the reasons that make you believe that the application still runs since you also says that the docker application loses connection? How did you check?

Can you describe a very precise, step-by-step, of what is happening:

• For each step validate the routes are still there with "sh ip ro" and "show ip int br", "show ip nat translations".
• Tell us what you are trying to reach (IP) and from where (interface), what works and what does not.

Best Regards

Emmanuel

[gabrijels]

Hi guys,

I really appreciate your efforts. I have to say the whole setup behaves a bit flaky, and unpredictable. I am trying to get some meaning into all of this.

Application is a simple modbus master that polls 3 values from modbus sensors and sends them into cloud. When I say the docker application loses connection, it means that it can't send data into cloud. Every time the sensor is polled with data, its diode blinks, so I know the application is up and running.

Today I tried a little experiment where I turned off the home router and then turned it back on. During this time IR809 was turned on and connected to the home router over GE0. After this experiment, I was able to reach 192.168.1.99 and 192.168.1.106. Also the docker application got its connection after a while (approximately after sending second IOX_REQUEST_AUTHENTICATION on Nov 30 14:01:35.314). You can see the whole LOG at the end of this message.

Now what interests me is the difference between the two LOGs I have sent. In the first LOG and when HEARTBEAT_TIMER_EXPIRED event occurred (see Nov 14 20:02:00.381) there was no IOX_REQUEST_AUTHENTICATION event. In the second LOG with the IOX_REQUEST_AUTHENTICATION on Nov 30 14:01:35.314 my docker application was streaming the data to the cloud again!

The question is: what is the meaning of IOX-6-HEARTBEAT_TIMER_EXPIRED event, and could this be the reason for these problems?

Gabrijel

---------------------LOG---------------------------------------------
*Nov 30 12:37:19.206: %LINK-3-UPDOWN: Interface GigabitEthernet1, changed state to down
*Nov 30 12:37:19.206: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 30 12:37:20.734: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2, changed state to up
*Nov 30 12:37:20.734: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1, changed state to down
*Nov 30 12:37:20.736: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 30 12:37:23.689: %SYS-5-CONFIG_I: Configured from memory by console
*Nov 30 12:37:23.803: %LINK-3-UPDOWN: Interface Async0, changed state to up
*Nov 30 12:37:23.901: %SYS-5-RESTART: System restarted --
Cisco IOS Software, ir800 Software (ir800-UNIVERSALK9-M), Version 15.8(3)M2a, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Mon 06-May-19 01:49 by prod_rel_team
*Nov 30 12:37:24.029: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Initialized
*Nov 30 12:37:24.031: %VPN_HW-6-INFO_LOC: Crypto engine: onboard 0 State changed to: Enabled
*Nov 30 12:37:24.031: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Nov 30 12:37:24.031: %LINK-3-UPDOWN: Interface Async1, changed state to up
*Nov 30 12:37:24.037: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 30 12:37:24.037: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 30 12:37:24.037: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 30 12:37:24.037: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 30 12:37:24.039: %QAT15_CRYPTO-6-SELF_TEST_RESULT: Self test info: (Self test started.)
*Nov 30 12:37:24.113: %LINEPROTO-5-UPDOWN: Line protocol on Interface NVI0, changed state to up
*Nov 30 12:37:24.233: %QAT15_CRYPTO-6-SELF_TEST_RESULT: Self test info: (All Tests Passed)
*Nov 30 12:37:24.233: %SYS-3-HARIKARI: Process IR800 Test top-level routine exited
*Nov 30 12:37:24.803: %LINEPROTO-5-UPDOWN: Line protocol on Interface Async0, changed state to up
*Nov 30 12:37:25.154: %LINK-5-CHANGED: Interface GigabitEthernet1, changed state to administratively down
*Nov 30 12:37:28.220: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 30 12:37:29.221: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 30 12:37:29.221: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)
*Nov 30 12:37:35.229: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Nov 30 12:37:35.229: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Nov 30 12:38:45.799: %IOX-6-SOCK_CONNECT: Received socket connection request from IOX Client
*Nov 30 12:38:45.803: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_REGISTER from IOX Client
*Nov 30 12:38:59.371: %IOX-6-SOCK_CONNECT: Received socket connection request from IOX Client
*Nov 30 12:39:02.190: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_PROTECT_KEY from IOX Client
*Nov 30 12:39:02.396: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_PROTECT_KEY to IOX Client
*Nov 30 12:39:02.398: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_REGISTER from IOX Client
*Nov 30 12:39:03.394: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_TIME from IOX Client
*Nov 30 12:39:06.367: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_PROTECT_KEY from IOX Client
*Nov 30 12:39:06.573: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_PROTECT_KEY to IOX Client
*Nov 30 12:39:30.887: %IOX-6-SOCK_CONNECT: Received socket connection request from IOX Client
*Nov 30 12:39:30.891: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_REGISTER from IOX Client
*Nov 30 12:39:45.816: %IOX-6-HEARTBEAT_TIMER_EXPIRED: IOX Heartbeat timer expired.
*Nov 30 12:39:51.177: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_UDI_INFO from IOX Client
*Nov 30 12:39:51.177: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_UDI_INFO to IOX Client
*Nov 30 12:39:51.185: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_GET_UDI_KEY from IOX Client
*Nov 30 12:39:51.185: %IOX-6-SOCK_SEND: Sending IOX_REQUEST_RESPONSE resp with opcode IOX_REQUEST_GET_UDI_KEY to IOX Client
*Nov 30 12:40:00.246: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_AUTHENTICATION from IOX Client
*Nov 30 12:40:00.246: AAA/LOCAL/LOGIN(00000000): Failed to find interface
*Nov 30 12:40:15.563: %CELLULAR-2-MODEM_UP: Cellular0 modem is now UP.
*Nov 30 12:41:15.582: FOTA not supported on modems older than 74xx.
*Nov 30 12:41:15.742: %CELLWAN-2-MODEM_RADIO: Cellular0 Modem radio has been turned on
*Nov 30 12:48:14.253: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 30 12:48:15.254: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 30 12:48:49.264: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 30 12:48:50.265: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 30 12:54:51.378: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 30 12:54:52.379: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 30 12:55:13.385: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 30 12:55:14.385: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 30 13:07:29.058: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_AUTHENTICATION from IOX Client
*Nov 30 13:07:29.058: AAA/LOCAL/LOGIN(00000000): Failed to find interface
*Nov 30 13:12:41.584: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_AUTHENTICATION from IOX Client
*Nov 30 13:12:41.584: AAA/LOCAL/LOGIN(00000000): Failed to find interface
*Nov 30 13:15:10.665: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_AUTHENTICATION from IOX Client
*Nov 30 13:15:10.665: AAA/LOCAL/LOGIN(00000000): Failed to find interface
*Nov 30 13:46:53.359: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 30 13:46:54.359: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 30 13:47:10.364: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 30 13:47:11.365: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 30 13:48:14.384: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 30 13:48:15.385: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 30 13:48:36.391: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 30 13:48:37.392: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 30 13:48:49.395: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 30 13:48:50.396: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 30 13:49:03.400: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 30 13:49:04.400: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 30 13:52:06.457: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Nov 30 13:52:07.458: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Nov 30 13:52:27.464: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to up
*Nov 30 13:52:28.464: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to up
*Nov 30 13:53:07.036: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_AUTHENTICATION from IOX Client
*Nov 30 13:53:07.036: AAA/LOCAL/LOGIN(00000000): Failed to find interface
*Nov 30 14:01:35.314: %IOX-6-SOCK_MESSAGE: Received IOX_REQUEST message with opcode IOX_REQUEST_AUTHENTICATION from IOX Client
*Nov 30 14:01:35.314: AAA/LOCAL/LOGIN(00000000): Failed to find interface

[gabrijels]

Hi,

So after approximately one day, the docker application lost its connection again. My laptop is connected over wifi to my home network, and the router is connected to the router by a cable. I am sending you the screen snapshot and the output of the sh iox list detail. I hope it helps.

gabrijels@gabrijels-HP-ProBook-640-G3:~$ ssh cisco@192.168.1.99
Password:
Password:

WeInt0#ping ip 192.168.1.106
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.106, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
WeInt0#ping ip 192.168.10.6
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.6, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
WeInt0#ping ip 192.168.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
WeInt0#sh ip ro
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 192.168.1.1 to network 0.0.0.0

S* 0.0.0.0/0 [1/0] via 192.168.1.1
192.168.1.0/24 is variably subnetted, 4 subnets, 2 masks
C 192.168.1.0/24 is directly connected, GigabitEthernet0
L 192.168.1.99/32 is directly connected, GigabitEthernet0
L 192.168.1.106/32 is directly connected, GigabitEthernet0
L 192.168.1.109/32 is directly connected, GigabitEthernet0
192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.10.0/24 is directly connected, GigabitEthernet2
L 192.168.10.1/32 is directly connected, GigabitEthernet2
WeInt0#show ip int br
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0 192.168.1.99 YES NVRAM up up
GigabitEthernet1 unassigned YES NVRAM administratively down down
Async0 unassigned YES unset up up
Async1 unassigned YES unset up down
GigabitEthernet2 192.168.10.1 YES NVRAM up up
Cellular0 unassigned YES NVRAM up up
Cellular1 unassigned YES NVRAM down down
NVI0 127.1.3.2 YES unset up up
WeInt0#show ip nat translations
Pro Inside global Inside local Outside local Outside global
tcp 192.168.1.106:52229 192.168.10.6:52229 34.214.22.26:443 34.214.22.26:443
--- 192.168.1.106 192.168.10.6 --- ---
--- 192.168.1.109 192.168.10.9 --- ---
WeInt0#

--------------------------------------------------------------------------------------------------------------------------------

The output of sh iox list detail:

IOX Server is running. Process ID: 326
Count of hosts registered: 1

Host registered:
===============
IOX Server Address: FE80::200:CFF:FEC6:3859; Port: 22222

Link Local Address of Host: FE80::1FF:FE90:8B05
IPV4 Address of Host: 192.168.10.6
IPV6 Address of Host: fe80::1ff:fe90:8b05
Client Version: 0.4
Session ID: 2
OS Nodename: WeInt0-GOS-1
Host Hardware Vendor: Cisco Systems, Inc.
Host Hardware Version: 1.0
Host Card Type: not implemented
Host OS Version: 1.8.4.2
OS status: RUNNING

Interface Hardware Vendor: None
Interface Hardware Version: None
Interface Card Type: None

Services:
===============
Service Name: Secure Storage Service
Service Status: RUNNING
Session ID: 3

Service Name: Host Device Management Service
Service Status: DISABLED
Session ID: 0

[gabrijels]

Also check out the result of sh ip arp before and after the reload....

WeInt0#Show ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.1.1 0 749d.7995.ff20 ARPA GigabitEthernet0
Internet 192.168.1.5 6 9061.ae81.6f46 ARPA GigabitEthernet0
Internet 192.168.1.99 - d4e8.80ac.8b1f ARPA GigabitEthernet0
Internet 192.168.1.106 - d4e8.80ac.8b1f ARPA GigabitEthernet0
Internet 192.168.1.109 - d4e8.80ac.8b1f ARPA GigabitEthernet0
Internet 192.168.10.1 - 0000.0cc6.3859 ARPA GigabitEthernet2
WeInt0#reload
Proceed with reload? [confirm]c
WeInt0#confirm
Translating "confirm"...domain server (192.168.1.1)
(192.168.1.1)
Translating "confirm"...domain server (192.168.1.1)

Translating "confirm"...domain server (192.168.1.1)

% Bad IP address or host name
% Unknown command or computer name, or unable to find computer address
WeInt0#reload
Proceed with reload? [confirm]c
WeInt0#onfirm
Translating "onfirm"...domain server (192.168.1.1)
(192.168.1.1)
Translating "onfirm"...domain server (192.168.1.1)

Translating "onfirm"...domain server (192.168.1.1)

% Bad IP address or host name
% Unknown command or computer name, or unable to find computer address
WeInt0#packet_write_wait: Connection to 192.168.1.99 port 22: Broken pipe
gabrijels@gabrijels-HP-ProBook-640-G3:~$
gabrijels@gabrijels-HP-ProBook-640-G3:~$
gabrijels@gabrijels-HP-ProBook-640-G3:~$ ssh cisco@192.168.1.99
Password:

WeInt0#Show ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.1.1 0 749d.7995.ff20 ARPA GigabitEthernet0
Internet 192.168.1.5 0 9061.ae81.6f46 ARPA GigabitEthernet0
Internet 192.168.1.99 - d4e8.80ac.8b1f ARPA GigabitEthernet0
Internet 192.168.1.106 - d4e8.80ac.8b1f ARPA GigabitEthernet0
Internet 192.168.1.109 - d4e8.80ac.8b1f ARPA GigabitEthernet0
Internet 192.168.10.1 - 0000.0cef.7869 ARPA GigabitEthernet2
Internet 192.168.10.6 0 0200.0190.8b05 ARPA GigabitEthernet2
WeInt0#

[gabrijels]

@Emmanuel Tychon one more curious thing. This happens repeatedly 24 hours after the docker application was started. So it doesn't have anything to do with dropping the internet connection... Any clue about this curious issue?

[gabrijels]

@suressan maybe you missed my post. Do you have some hint about what could be wrong?