I can't copy bin files remotly to Catlayst 9300 Series switches

O.K. · ‎10-21-2024

Hello everyone,

We have Catalyst 3650 series switches in remote locations, and we are in the middle of a refresh project at the moment.

The following scenario is a bit strange, and I can't understand what the problem can be here.

In the same location, there are 2 switches, one of them C3650 and other one C9300.

Both switches mgmt IPs are in the same subnet.

When I try to copy an image file from a Debian to C9k it throws an error but C3650 not.

I assume it is the config, since I only can't copy the images to C9k series.

Error is as follows:

Authorization denied.
user@debian:~$ Connection to c9k-switch closed by remote host.

In the switch logs, I can see the following:

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user] [Source: 1.1.1.1] [localport: 22] at 11:57:40 CET(S) Mon Oct 21 2024

I've turned on the debug, but it is not really something to see, in my opinion.

ab-cde-fgh-s01#
Oct  21 11:18:26.921: AAA/BIND(00000060): Bind i/f
Oct  21 11:18:26.921: AAA/AUTHEN/LOGIN (00000060): Pick method list 'default'
Oct  21 11:18:26.921: TPLUS: Queuing AAA Authentication request 96 for processing
Oct  21 11:18:26.921: TPLUS(00000060) login timer started 1020 sec timeout
Oct  21 11:18:26.921: TPLUS: processing authentication start request id 96
Oct  21 11:18:26.921: TPLUS: Authentication start packet created for 96(user)
Oct  21 11:18:26.921: TPLUS: Using server 1.1.1.1
Oct  21 11:18:26.922: TPLUS(00000060)/0/NB_WAIT/7FBCA2A4D588: Started 5 sec timeout
Oct  21 11:18:26.950: TPLUS(00000060)/0/NB_WAIT: socket event 2
Oct  21 11:18:26.950:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
Oct  21 11:18:26.950: TPLUS(00000060)/0/NB_WAIT: wrote entire 44 bytes request
Oct  21 11:18:26.950: TPLUS(00000060)/0/READ: socket event 1
Oct  21 11:18:26.950: TPLUS(00000060)/0/READ: Would block while reading
Oct  21 11:18:27.379: TPLUS(00000060)/0/READ: socket event 1
Oct  21 11:18:27.379: TPLUS(00000060)/0/READ: read entire 12 header bytes (expect 16 bytes data)
Oct  21 11:18:27.379: TPLUS(00000060)/0/READ: socket event 1
Oct  21 11:18:27.379: TPLUS(00000060)/0/READ: read entire 28 bytes response
Oct  21 11:18:27.379:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXX fetched
Oct  21 11:18:27.379: TPLUS(00000060)/0/7FBCA2A4D588: Processing the reply packet
Oct  21 11:18:27.379: TPLUS: Received authen response status GET_PASSWORD (8)
Oct  21 11:18:27.408: TPLUS: Queuing AAA Authentication request 96 for processing
Oct  21 11:18:27.409: TPLUS(00000060) login timer started 1020 sec timeout
Oct  21 11:18:27.409: TPLUS: processing authentication continue request id 96
Oct  21 11:18:27.409: TPLUS: Authentication continue packet generated for 96
Oct  21 11:18:27.409: TPLUS(00000060)/0/WRITE/7FBCA28C7CA8: Started 5 sec timeout

Do you have any idea what else I can check?
Thanks in advance for the answers!

Regards!

Leo Laohoo · ‎10-21-2024

If this is using TFTP, try the following:

conf t
 ip tftp source interface <MANAGMENT VLAN>
end

O.K. · ‎10-21-2024

Hi,
thank you for your answer, in this case I'm trying to copy files using scp.

Elliot Dierksen · ‎10-21-2024

The ciphers and key exchange methods in IOS is sometimes well behind what is supported in current OS distributions. I suspect that part of the negotiation is failing. You might have to enable something on the Linux side that it considers insecure.

O.K. · ‎10-21-2024

Hello @Elliot Dierksen ,

Thank you for your answer!

Can you please explain to me what do you exactly mean?
I assume if the case cipher or key exchange would be, that should be handled by ssh and in the loggs mentioned that they (client and server) couldn't handle the negotiation.

balaji.bandi · ‎10-21-2024

how is your Cat 9300 config looks like - have you enabled SCP Server and bulk-mode

what is the command you used to connect

Authorization denied.

as per information denied

show run also help us here for us to identify the issue.

run debug - debug ip scp

check below guide :

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst9300/software/release/17-1/configuration_guide/sys_mgmt/b_171_sys_mgmt_9300_cg/secure_copy.html

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

O.K. · ‎10-21-2024

Hello @balaji.bandi ,

Thank you for your answer.

the command

ip scp server enable

is already in the running config, but I don't know what you mean by "bulk-mode"

Which part of the config you mean exactly? It is actually pretty same configuration as the 3650 series (as I wrote, this is a refresh). Config more or less the same (some commands are expecting different, but it is just syntax, functionality same.)
I'm using the following command:

scp /path/to/cisco_catalyst/C93xx/cat9k_iosxe.17.09.05.SPA.bin user@switch:/flash/

balaji.bandi · ‎10-23-2024

we need to look complete errors from ISE (or AAA side)

have you tried from Switch to SCP Server pulling the file from Linux ? (is that works ?)

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

amikat · ‎10-22-2024

Hi,

Can you please post the complete tacacs+ debug (I understand that authorization is performed via tacacs server as well).

Thanks & Regards,

Antonin

O.K. · ‎10-23-2024

Hi,
here is tacacs debug and below that aaa config.

switch10#
Oct 23 09:06:04.825: TAC+/AUTHOR: (2763585750): user=user
Oct 23 09:06:04.825: TAC+/AUTHOR: (2763585750): send AV service=shell
Oct 23 09:06:04.825: TAC+/AUTHOR: (2763585750): send AV cmd=debug
Oct 23 09:06:04.826: TAC+/AUTHOR: (2763585750): send AV cmd-arg=tacacs
Oct 23 09:06:04.826: TAC+/AUTHOR: (2763585750): send AV cmd-arg=<cr>
Oct 23 09:06:05.676: TPLUS: Queuing AAA Accounting request 60 for processing
Oct 23 09:06:05.676: TPLUS: processing accounting request id 60
Oct 23 09:06:05.676: TPLUS: Sending AV task_id=4197
Oct 23 09:06:05.676: TPLUS: Sending AV timezone=CET(S)
Oct 23 09:06:05.676: TPLUS: Sending AV service=shell
Oct 23 09:06:05.676: TPLUS: Sending AV start_time=1729667165
Oct 23 09:06:05.676: TPLUS: Sending AV priv-lvl=15
Oct 23 09:06:05.676: TPLUS: Sending AV cmd=debug tacacs <cr>
Oct 23 09:06:05.676: TPLUS: Accounting request created for 60(user)
Oct 23 09:06:05.676: TPLUS: using previously set server 1.1.1.1 from group tacacs+
Oct 23 09:06:05.676: TPLUS(0000003C)/0/NB_WAIT/7F93477AD9F0: Started 5 sec timeout
Oct 23 09:06:05.726: TPLUS(0000003C)/0/NB_WAIT: socket event 2
Oct 23 09:06:05.726:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
switch10#
Oct 23 09:06:05.726: TPLUS(0000003C)/0/NB_WAIT: wrote entire 144 bytes request
Oct 23 09:06:05.726: TPLUS(0000003C)/0/READ: socket event 1
Oct 23 09:06:05.726: TPLUS(0000003C)/0/READ: Would block while reading
Oct 23 09:06:06.132: TPLUS(0000003C)/0/READ: socket event 1
Oct 23 09:06:06.132: TPLUS(0000003C)/0/READ: read entire 12 header bytes (expect 5 bytes data)
Oct 23 09:06:06.132: TPLUS(0000003C)/0/READ: socket event 1
Oct 23 09:06:06.132: TPLUS(0000003C)/0/READ: read entire 17 bytes response
Oct 23 09:06:06.132:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
switch10#
Oct 23 09:06:06.132: TPLUS(0000003C)/0/7F93477AD9F0: Processing the reply packet
Oct 23 09:06:06.132: TPLUS: Received accounting response with status PASS
switch10#
Oct 23 09:06:16.757: TPLUS: Queuing AAA Authentication request 62 for processing
Oct 23 09:06:16.757: TPLUS(0000003E) login timer started 1020 sec timeout
Oct 23 09:06:16.757: TPLUS: processing authentication start request id 62
Oct 23 09:06:16.757: TPLUS: Authentication start packet created for 62(user)
Oct 23 09:06:16.757: TPLUS: Using server 1.1.1.1
Oct 23 09:06:16.757: TPLUS(0000003E)/0/NB_WAIT/7F9344EF4270: Started 5 sec timeout
Oct 23 09:06:16.808: TPLUS(0000003E)/0/NB_WAIT: socket event 2
Oct 23 09:06:16.808:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
switch10#
Oct 23 09:06:16.808: TPLUS(0000003E)/0/NB_WAIT: wrote entire 44 bytes request
Oct 23 09:06:16.808: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:16.808: TPLUS(0000003E)/0/READ: Would block while reading
Oct 23 09:06:17.212: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:17.212: TPLUS(0000003E)/0/READ: read entire 12 header bytes (expect 16 bytes data)
Oct 23 09:06:17.212: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:17.212: TPLUS(0000003E)/0/READ: read entire 28 bytes response
Oct 23 09:06:17.212:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
switch10#
Oct 23 09:06:17.212: TPLUS(0000003E)/0/7F9344EF4270: Processing the reply packet
Oct 23 09:06:17.212: TPLUS: Received authen response status GET_PASSWORD (8)
Oct 23 09:06:18.662: TPLUS: Queuing AAA Authentication request 62 for processing
Oct 23 09:06:18.662: TPLUS(0000003E) login timer started 1020 sec timeout
Oct 23 09:06:18.662: TPLUS: processing authentication continue request id 62
Oct 23 09:06:18.662: TPLUS: Authentication continue packet generated for 62
Oct 23 09:06:18.662: TPLUS(0000003E)/0/WRITE/7F9344EF4270: Started 5 sec timeout
Oct 23 09:06:18.662:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
Oct 23 09:06:18.662: TPLUS(0000003E)/0/WRITE: wrote entire 33 bytes request
Oct 23 09:06:18.783: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:18.783: TPLUS(0000003E)/0/READ: read entire 12 header bytes (expect 6 bytes data)
Oct 23 09:06:18.783: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:18.783: TPLUS(0000003E)/0/READ: read entire 18 bytes response
Oct 23 09:06:18.783:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
Oct 23 09:06:18.783: TPLUS(0000003E)/0/7F9344EF4270: Processing the reply packet
Oct 23 09:06:18.783: TPLUS: Received authen response status PASS (2)
Oct 23 09:06:18.783: %SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: user] [Source: 1.1.1.1] [localport: 22] at 09:06:18 CET(S) Wed Oct 23 2024
Oct 23 09:06:18.880: TPLUS: Queuing AAA Authorization request 62 for processing
Oct 23 09:06:18.880: TPLUS(0000003E) login timer started 1020 sec timeout
Oct 23 09:06:18.880: TPLUS: processing authorization request id 62
Oct 23 09:06:18.880: TPLUS: Protocol set to None .....Skipping
Oct 23 09:06:18.880: TPLUS: Sending AV service=shell
Oct 23 09:06:18.880: TPLUS: Sending AV cmd*
Oct 23 09:06:18.880: TPLUS: Authorization request created for 62(user)
Oct 23 09:06:18.880: TPLUS: using previously set server 1.1.1.1 from group tacacs+
Oct 23 09:06:18.881: TPLUS(0000003E)/0/NB_WAIT/7F9348EB3480: Started 5 sec timeout
switch10#
Oct 23 09:06:18.946: TPLUS(0000003E)/0/NB_WAIT: socket event 2
Oct 23 09:06:18.946:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
Oct 23 09:06:18.946: TPLUS(0000003E)/0/NB_WAIT: wrote entire 63 bytes request
Oct 23 09:06:18.946: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:18.946: TPLUS(0000003E)/0/READ: Would block while reading
Oct 23 09:06:19.461: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:19.461: TPLUS(0000003E)/0/READ: read entire 12 header bytes (expect 18 bytes data)
Oct 23 09:06:19.461: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:19.461: TPLUS(0000003E)/0/READ: read entire 30 bytes response
Oct 23 09:06:19.461:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXXXXX fetched
Oct 23 09:06:19.461: TPLUS(0000003E)/0/7F9348EB3480: Processing the reply packet
Oct 23 09:06:19.461: TPLUS: Processed AV priv-lvl=15
Oct 23 09:06:19.461: TPLUS: received authorization response for 62: PASS
Oct 23 09:06:19.462: TPLUS: Queuing AAA Authorization request 62 for processing
Oct 23 09:06:19.462: TPLUS: processing authorization request id 62
Oct 23 09:06:19.462: TPLUS: Sending AV service=shell
Oct 23 09:06:19.462: TPLUS: Sending AV protocol=ssh
Oct 23 09:06:19.462: TPLUS: Sending AV cmd=scp
Oct 23 09:06:19.462: TPLUS: Sending AV cmd-arg=-t
Oct 23 09:06:19.462: TPLUS: Sending AV cmd-arg=/flash/
Oct 23 09:06:19.462: TPLUS: Authorization request created for 62(user)
Oct 23 09:06:19.462: TPLUS: using previously set server 1.1.1.1 from group tacacs+
Oct 23 09:06:19.462: TPLUS(0000003E)/0/NB_WAIT/7F9348EB3480: Started 5 sec timeout
Oct 23 09:06:19.512: TPLUS(0000003E)/0/NB_WAIT: socket event 2
Oct 23 09:06:19.512:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXX fetched
Oct 23 09:06:19.512: TPLUS(0000003E)/0/NB_WAIT: wrote entire 106 bytes request
Oct 23 09:06:19.512: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:19.512: TPLUS(0000003E)/0/READ: Would block while reading
Oct 23 09:06:19.922: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:19.922: TPLUS(0000003E)/0/READ: read entire 12 header bytes (expect 12 bytes data)
Oct 23 09:06:19.922: TPLUS(0000003E)/0/READ: socket event 1
Oct 23 09:06:19.922: TPLUS(0000003E)/0/READ: read entire 24 bytes response
Oct 23 09:06:19.922:  %TAC+: Could not fetch shared secret from tacacs server. Global shared secret is XXXXXXXXXXXXXXX fetched
Oct 23 09:06:19.922: TPLUS(0000003E)/0/7F9348EB3480: Processing the reply packet
switch10#
Oct 23 09:06:19.922: TPLUS: received authorization response for 62: FAIL
Oct 23 09:06:19.923: SCP: [22 -> 1.1.1.1:46242] send Authorization denied.
Oct 23 09:06:19.924:  Socket I/O cleanup message sent to TACACS
TPLUS Proc:SOCKET IO CLEANUP EVENT

switch10#sh run | s aaa
aaa new-model
aaa authentication login default group tacacs+ local-case
aaa authentication enable default group tacacs+ enable
aaa authentication dot1x default group radius
aaa authorization exec default group tacacs+ none
aaa authorization commands 15 default group tacacs+ local
aaa authorization network default group radius
aaa authorization auth-proxy default group radius
aaa accounting exec default start-stop group tacacs+
aaa accounting commands 1 default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
aaa session-id common

amikat · ‎10-23-2024

Hi,

Thanks for the information supplied. It appears that the failing authorization MAY be due to a discrepancy between the tacacs server parameters settings (AV pairs) and your command used parameters (scp /path/to/cisco_catalyst/C93xx/cat9k_iosxe.17.09.05.SPA.bin user@switch:/flash/).

Is the same tacacs server used for both Cat3650 and Cat9300 boxes with identical user configuration?

As an interim test can you please modify (temporalily) the "aaa authorization commands 15 default group tacacs+ local" command as per beneath and check if there is any improvement:

aaa authorization commands 15 default group tacacs+ local if-authenticated

Best regards,

Antonin

O.K. · ‎10-23-2024

Hi,

Thank you for your suggestion.

@amikat wrote:
Is the same tacacs server used for both Cat3650 and Cat9300 boxes with identical user configuration?

Yes, same tacacs for both of them, actually we have not changed anything, there are still installed cat3650 switches, and it just works, and they have the same aaa as cat9300 boxes. (Posted above)

@amikat wrote:
aaa authorization commands 15 default group tacacs+ local if-authenticated

I've tried this and the result is the same, it didn't work.

I think AV pairs is a good point, but I can't think of any possible location to check, because there isn't anything configured on the switch related to AV pairs.

amikat · ‎10-23-2024

Hi,

Thanks for the reply. The AV pairs are specific to a user profile stored at the TACACS+ daemon. So place to check is the TACACS+ server.

Can you please answer my previous post. Also please check the syntax of your scp command in respect of the Cat3560 and the Cat9300 versions.

Best regards,

Antonin

Richard Burts · ‎10-23-2024

I agree with @amikat that the issue is related to the Tacacs server, especially given this message

"Oct 23 09:06:19.922: TPLUS: received authorization response for 62: FAIL"

Can you check the logs of the server and see if there is information in the logs about this attempt?

HTH

Rick

amikat · ‎10-23-2024

Hi,

Further to my previous post:

TPLUS: Sending AV cmd-arg=-t

Are you using "-t" option/parameter with your scp command?

Best regards,

Antonin