I have a problem with my SG-500 switch: it seems to have problems at the login part. No user is allowed to log in anymore. I have 2 of them (cisco and reserve), and none of them work. Tried http/https and telnet
The switch reports to an external syslog server, so I gained some information from there:
---
2019-03-28T15:42:13.395772+01:00 fp-sw2.flenpharma.local %AAA-I-CONNECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 ACCEPTED
2019-04-01T11:29:21.122903+02:00 fp-sw2.flenpharma.local %AAA-I-CONNECT: New http connection for user cisco, source 10.0.0.222 destination 10.0.0.52 ACCEPTED
---
2019-06-05T10:22:13.721458+02:00 fp-sw2.flenpharma.local %AAA-I-CONNECT: New https connection for user reserve, source 10.0.0.222 destination 10.0.0.52 ACCEPTED
2019-06-05T10:40:55.412059+02:00 fp-sw2.flenpharma.local %AAA-I-CONNECT: New https connection for user reserve, source 10.0.0.222 destination 10.0.0.52 ACCEPTED
2019-06-21T09:53:01.991250+02:00 fp-sw2.flenpharma.local %AAA-I-CONNECT: New https connection for user reserve, source 10.0.0.222 destination 10.0.0.52 ACCEPTED
---
However, none of my logins after 21/6 are logged. This is very strange, as I visit the switch almost weekly to check its status.
We also have a rotation schedule of 4 passwords, tried all of them on both accounts, but none of them work.
I guess there must be an issue: all accounts work on other switches (SG-200 series), and while this is not a proof, it somewhat makes me think a part of the Cisco OS has problems. (installed firmware is 1.4.9.4)
a summary of what happened today ("cat *.log | grep 2019-07" on syslog server):
---
2019-07-22T11:39:06.782768+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New telnet connection, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T09:25:40.457009+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New http connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T09:26:12.069834+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New http connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (2)
2019-07-23T09:26:12.071262+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New http connection for user reserve, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T09:27:19.191128+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New http connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (3)
2019-07-23T09:27:19.192215+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New http connection for user reserve, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (2)
2019-07-23T09:28:22.630188+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New http connection for user reserve, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (1)
2019-07-23T09:28:22.631150+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New http connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (1)
2019-07-23T09:28:22.632023+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T09:28:33.720252+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user reserve, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T09:33:22.223066+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user reserve, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (2)
2019-07-23T09:33:22.224045+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (6)
2019-07-23T09:33:24.218799+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T09:36:53.372529+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (3)
2019-07-23T09:40:08.219243+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (1)
2019-07-23T09:43:12.250640+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T10:05:01.622641+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T10:20:21.964062+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T10:25:21.960484+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user cisco, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (3)
2019-07-23T10:27:18.168898+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user reserve, source 10.0.0.222 destination 10.0.0.52 REJECTED
2019-07-23T10:32:18.157598+02:00 fp-sw2.flenpharma.local %AAA-W-REJECT: New https connection for user reserve, source 10.0.0.222 destination 10.0.0.52 REJECTED, aggregated (4)
2019-07-23T08:32:53.731600+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/5
2019-07-23T08:33:02.486670+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/5, aggregated (1)
2019-07-23T08:41:22.769242+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/5
2019-07-23T08:41:31.564408+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/5, aggregated (1)
2019-07-23T09:01:00.442889+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/1
2019-07-23T09:01:00.662360+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/4
2019-07-23T09:01:00.686184+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/16
2019-07-23T09:19:02.237545+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/6
2019-07-23T09:24:02.232626+02:00 fp-sw2.flenpharma.local %LINK-I-Up: gi1/6, aggregated (1)
2019-07-23T08:32:52.106429+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/5
2019-07-23T08:33:02.485652+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/5, aggregated (1)
2019-07-23T08:41:21.134215+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/5
2019-07-23T08:41:31.563353+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/5, aggregated (1)
2019-07-23T09:00:57.068285+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/1
2019-07-23T09:00:57.275321+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/16
2019-07-23T09:00:57.633118+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/4
2019-07-23T09:00:58.549408+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/10
2019-07-23T09:22:45.934794+02:00 fp-sw2.flenpharma.local %LINK-W-Down: gi1/6
2019-07-23T08:33:02.487537+02:00 fp-sw2.flenpharma.local %STP-W-PORTSTATUS: gi1/5: STP status Forwarding
2019-07-23T08:41:31.565307+02:00 fp-sw2.flenpharma.local %STP-W-PORTSTATUS: gi1/5: STP status Forwarding
2019-07-23T09:01:04.937386+02:00 fp-sw2.flenpharma.local %STP-W-PORTSTATUS: gi1/1: STP status Forwarding
2019-07-23T09:01:05.157402+02:00 fp-sw2.flenpharma.local %STP-W-PORTSTATUS: gi1/4: STP status Forwarding
2019-07-23T09:01:05.181580+02:00 fp-sw2.flenpharma.local %STP-W-PORTSTATUS: gi1/16: STP status Forwarding
2019-07-23T09:19:06.731870+02:00 fp-sw2.flenpharma.local %STP-W-PORTSTATUS: gi1/6: STP status Forwarding
2019-07-23T09:24:06.232553+02:00 fp-sw2.flenpharma.local %STP-W-PORTSTATUS: gi1/6: STP status Forwarding, aggregated (1)
----
... ok, this looks fine (.222 is my pc, .52 is the switch). So no clue at all...
I cannot simply restart the switch: it hosts many critical servers (DNS, file sharing, iSCSI, DBs etc) in the network, and a restart would imply shutting down all those services.
Anyone who can point me to a (possible) clue of what has happened? And if I can recover from it?
Thank you
*EDIT*: tried with serial cable, using 115200/8/1 no Parity / Flow control, but I do not get a login prompt asking for a username, so that's also not an option. The connection works on other switches, though
