vstack and tftp traffic issue - WS-C3560E-48TD

[maso Kamali]

Hello, 

I have a strange problem with what appears to be vstack/tftp related. client reported udp traffic flooding their port, after a wire capture on the port I noticed that our switch was broadcasting tftp requests out of their port, see attached pic. 

the vstack feature was disabled after the first wire capture, but the tftp broadcast didn't stop. I ran a vstack debug that show the process is still active despite "smartinstall" being disabled. Also, the command "ip tftp source-interface x/y" keeps appearing in the configuration and cycles through different ports every 10-15 minutes. static configuration or removal of the command didn't stop this behavior. There are no EEM applets or external sources that could be responsible for this. 

A copy of "debug vstack all" is attached.

Any insight / info on this issue would be greatly appreciated. 

Thank you,

C3560E-SW# sh vstack config
Role: Client (SmartInstall disabled)
Vstack Director IP address: 0.0.0.0

*** Following configurations will be effective only on director ***
Vstack default management vlan: 1
Vstack management Vlans: none
Join Window Details:
Window: Closed
Operation Mode: manual
Vstack Backup Details:
Mode: On (default)
Repository:

C3560E-SW#

[Leo Laohoo]

Post the complete output to the command " sh run | i vstack" & "sh run int vlan 1".

[maso Kamali]

Here is the full output and some logs for vstack. 

#sh run all | i vstack
no vstack join-window mode auto
vstack join-window close
no vstack

Feb 24 10:46:16 104.250.156.90 32518: Feb 24 11:46:15 MST: %SYS-5-CONFIG_NV_I: Nonvolatile storage configured from tftp://154-222-63-74.static.reverse.lstn.net/random_file by console
Feb 24 10:46:25 104.250.156.90 32519: Feb 24 11:46:24 MST: %SYS-5-CONFIG_NV_I: Nonvolatile storage configured from tftp://154-222-63-74.static.reverse.lstn.net/random_file by console
Feb 24 10:46:29 104.250.156.90 32520: Feb 24 11:46:28 MST: %SM-4-BADEVENT: Event 'ibcs_e_download_msg_req_recv' is invalid for the current state 'ibcs_s_accept': smi_ibc_serv SMI IBCS sm
Feb 24 10:46:29 104.250.156.90 32521: -Traceback= 10D2968 10DA104 10DB100 10E2AF0 10E3AA0 10E3C70 10E140C 10E160C 10E17A4 10E1AA0 10DB6AC 1F19D8C 1F10560
Feb 24 10:46:29 104.250.156.90 32522: Feb 24 11:46:28 MST: %SM-4-BADEVENT: Event 'ibcs_e_download_msg_resp_send' is invalid for the current state 'ibcs_s_accept': smi_ibc_serv SMI IBCS sm
Feb 24 10:46:29 104.250.156.90 32523: -Traceback= 10D2968 10DA12C 10DB22C 10E2AF0 10E3AA0 10E3C70 10E140C 10E160C 10E17A4 10E1AA0 10DB6AC 1F19D8C 1F10560

[Leo Laohoo]

It sounds like there is a VStack director configured somewhere in the network.  it's not this 3560E.

And I also suspect VLAN 1 is enabled all throughout this network (which makes tracking down the VStack Director more difficult).

[maso Kamali]

It's possible but we don't use the feature and I am the only person that would configure such feature, to my knowledge there is no director, at least not in our network. 

how would you track down the director IP with little to no configs on the "client" switch?

thanks

[Leo Laohoo]

You will need to log into every switch and router and find out where the director is using the command "sh run | i vstack".  Disable the director using the command "no vstack director <IP address>". 

There is a possibility that there could be more than one VStack Director in the network.

[maso Kamali]

I'll search for the director. I am curious if the director IP is not configured on the client, can the director influence the client in any way? 

Thanks

[Leo Laohoo]

I am curious if the director IP is not configured on the client, can the director influence the client in any way? 

SmartInstall works with CDP and VLAN 1 enabled.  

I don't know who enabled SmartInstall and what your network is like.  

But if someone has enabled SmartInstall, I don't know what configuration(s) are enabled.  The most important thing to do is find the Directors and disable them.  

What would happen if this was left alone?  If a new switch is turned on and is connected to the network, the Director might push a configuration to this new switch and may, potentially, cause network issue(s) or, worst, create a security vulnerability. 

[J. H. · Accepted solution]

We got the exact same thing on our 4500E's two days ago!

It looks like instrusion attempts that actually was successful, but since we have full CVS of configs we can see that the only thing that was changed in running-config was different TFTP source-interfaces!

It began at about 19:00 and continued until 23:00 something local time. Then it just stopped.

This was changed several times in the running-config:
ip tftp source-interface vlan <xxx>

This is the message in logs (via external syslog-server):
%SYS-5-CONFIG_NV_I: Nonvolatile storage configured from tftp://154-237-63-74.static.reverse.lstn.net/random_file by console

It says "by console" here, but no user was logged in at the time from what we can see.... Seems like running-configuration was changed severals times, but not the startup-configurations.

Ofcourse we are denying all traffic to our switches magement interfaces, except from our own management computers..... So this is highly weird and can't really be explained so far. I will go deeper into logs but will probably make a Cisco case about this.

[Leo Laohoo · Accepted solution]

Cisco PSIRT – Mitigating and Detecting Potential Abuse of Cisco Smart Install Feature

[simonmanchester]

Our attacker tried on the 24th - and came from a very similar ip address. The logs show the following address (remarkably similar to yours)

Feb 24 18:54:56 UTC: %SM-4-BADEVENT: Event 'ibcs_e_download_msg_req_recv' is invalid for the current state 'ibcs_s_accept': smi_ibc_serv SMI IBCS sm
-Traceback= 665BA0z 2242B8Cz 2282414z 22819E4z 228959Cz 2288094z 2290B24z 2290B9Cz 228202Cz 2FF8720z 2FF4AA0z

Feb 24 18:57:42 UTC: %SYS-5-CONFIG_NV_I: Nonvolatile storage configured from tftp://154-222-63-74.static.reverse.lstn.net/random_file by console

Feb 24 18:59:02 UTC: VSTACK_ERR:
!! smi_ibc_dl_handle_events: download not successful

I have a case open via our support contract who have opened a case with cisco, hopefully they get to the bottom of this.

[simonmanchester]

We have had this on one of our site routers (a WS-C3560X-24T-S acting as a site router). An attacker enabled tftp and attempted to upload a new startup-config, but luckily it corrupted and we caught it in time. I wonder if there is a vulnerability in the 3560's that allows them to get in and change stuff?

[maso Kamali]

In the beginning it appeared very much like an attack but port 4786 was closed and CDP disabled. Netflow data didn't show any tftp upload attempts to the switch but the switch itself was/is broadcasting tftp request out all ports looking for a file called  "random_file" via tftp.

I have not been successful in stopping the tftp requests from going out and the vstack debug revealed the process is still active and it's responsible for the tftp traffic.

I searched through our entire network with directors in sight and vstack is disabled on that switch , so I am at a loss at this point. I hope one of the cisco tac folks sees this. 

Thank you,

[Leo Laohoo]

Cisco recently released a document about the mis-use of the SmartInstall feature:  Cisco Smart Install Protocol Misuse

This is a major issue if VLAN 1 is enabled on the network and someone has started configuring SmartInstall and didn't clean up the configuration properly.  

[maso Kamali]

JH, 

Please let me know how that TAC ticket goes, your experience is identical to our. I wonder if the attacker would have been able to download a copy of the config in addition to manipulating the running config. 

Thanks