cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
47482
Views
70
Helpful
14
Replies

MAC Address Flapping and SCCM Wake Up Proxy

ds6123
Level 1
Level 1

Just wondering if anyone has run into this problem.  I ran into it the other day at a customer site.  We had a hard time tracking down the issue.

Apparently Microsoft's SCCM has a new feature called "wake up proxy" whereby host a pretends to be host b (host a spoof's the mac address of host b) when host b goes to sleep.  Needless to say, we were seeing tons of mac address flapping and port-security issues.  I can only imagine what I would have seen if there was also dynamic arp inspection configured.  Not sure if this it the default behavior of the latest SCCM or not, but it seems like a crazy feature.

The wake up proxy feature is explained here in more detail.  This is probably the most "amazing" feature since MS network load balancing.

Configuration Manager supports two wake on local area network (LAN) technologies to wake up computers in sleep mode when you want to install required software, such as software updates and applications: traditional wake-up packets and AMT power-on commands.

If you have Configuration Manager SP1, you can supplement the traditional wake-up packet method by using the wake-up proxy client settings. Wake-up proxy uses a peer-to-peer protocol and elected computers to check whether other computers on the subnet are awake, and to wake them if necessary. When the site is configured for Wake On LAN and clients are configured for wake-up proxy, the process works as follows:

Computers that have the Configuration Manager SP1 client installed and that are not asleep on the subnet check whether other computers on the subnet are awake. They do this by sending each other a TCP/IP ping command every 5 seconds.

If there is no response from other computers, they are assumed to be asleep. The computers that are awake become manager computers for the subnet.

Because it is possible that a computer might not respond because of a reason other than it is asleep (for example, it is turned off, removed from the network, or the proxy wake-up client setting is no longer applied), the computers are sent a wake-up packet every day at 2 P.M. local time. Computers that do not respond will no longer be assumed to be asleep and will not be woken up by wake-up proxy.

To support wake-up proxy, at least three computers must be awake for each subnet. To achieve this, three computers are non-deterministically chosen to be guardian computers for the subnet. This means that they stay awake, despite any configured power policy to sleep or hibernate after a period of inactivity. Guardian computers honor shutdown or restart commands, for example, as a result of maintenance tasks. If this happens, the remaining guardian computers wake up another computer on the subnet so that the subnet continues to have three guardian computers.

Manager computers ask the network switch to redirect network traffic for the sleeping computers to themselves.

The redirection is achieved by the manager computer broadcasting an Ethernet frame that uses the sleeping computer’s MAC address as the source address. This makes the network switch behave as if the sleeping computer has moved to the same port that the manager computer is on. The manager computer also sends ARP packets for the sleeping computers to keep the entry fresh in the ARP cache. The manager computer will also respond to ARP requests on behalf of the sleeping computer and reply with the MAC address of the sleeping computer.

Warning

During this process, the IP-to-MAC mapping for the sleeping computer remains the same. Wake-up proxy works by informing the network switch that a different network adapter is using the port that was registered by another network adapter. However, this behavior is known as a MAC flap and is unusual for standard network operation. Some network monitoring tools look for this behavior and can assume that something is wrong. Consequently, these monitoring tools can generate alerts or shut down ports when you use wake-up proxy.

Do not use wake-up proxy if your network monitoring tools and services do not allow MAC flaps.

When a manager computer sees a new TCP connection request for a sleeping machine and the request is to a port that the sleeping machine was listening on before it went to sleep, the manager computer sends a wake-up packet to the sleeping computer, and then stops redirecting traffic for this computer.

The sleeping computer receives the wake-up packet and wakes up. The sending computer automatically retries the connection and this time, the computer is awake and can respond.

Wake-up proxy has the following prerequisites and limitations:

Important

If you have a separate team that is responsible for the network infrastructure and network services, notify and include this team during your evaluation and testing period. For example, on a network that uses 802.1X network access control, wake-up proxy will not work and can disrupt the network service. In addition, wake-up proxy could cause some network monitoring tools to generate alerts when the tools detect the traffic to wake-up other computers.

The supported clients are Windows 7, Windows 8, Windows Server 2008 R2, Windows Server 2012.

Guest operating systems that run on a virtual machine are not supported.

Clients must run Configuration Manager SP1 and be enabled for wake-up proxy by using client settings. Although wake-up proxy operation does not depend on hardware inventory, clients do not report the installation of the wake-up proxy service unless they are enabled for hardware inventory and submitted at least one hardware inventory.

Network adapters (and possibly the BIOS) must be enabled and configured for wake-up packets. If the network adapter is not configured for wake-up packets or this setting is disabled, Configuration Manager will automatically configure and enable it for a computer when it receives the client setting to enable wake-up proxy.

If a computer has more than one network adapter, you cannot configure which adapter to use for wake-up proxy; the choice is non-deterministic. However, the adapter chosen is recorded in the SleepAgent_<DOMAIN>@SYSTEM_0.log file.

The network must allow ICMP echo requests (at least within the subnet). You cannot configure the 5 second interval that is used to send the ICMP ping commands.

Communication is unencrypted and unauthenticated, and IPsec is not supported.

The following network configurations are not supported:

802.1X with port authentication

Wireless networks

Network switches that bind MAC addresses to specific ports

IPv6-only networks

DHCP lease durations less than 24 hours


1 Accepted Solution

Accepted Solutions

schwa
Level 1
Level 1

I don't know enough four letter words to describe how I feel about this issue.

View solution in original post

14 Replies 14

mvann
Level 1
Level 1

Thanks for posting this up. I noticed this on our switches a few weeks ago and opened a TAC case when I couldn't sort it out. TAC couldn't pin it down but I ran across this yesterday. Kudos

Daryl Eagle
Level 1
Level 1

I want to thank you as well, we have arp inspection on and this issue took down our network. Microsoft creating "features" that mimic malicious code and worms **forehead palm**

dhavens
Level 1
Level 1

Been tracking this problem for 2 weeks we run 802.1x authentication and this "Feature" blew up the network.

Finally discovered the the "Enhancement" and disabled it. I still can not belive that this is on by default.

Alejandro.Song
Level 1
Level 1

Thank you very much for your sharing.  We met the same problem about 1 month ago and i noticed the problem have some related with SCCM.  But i can't figure out why SCCM could cause this problem. Finally i found this page. Thank you very much!!! 

lamond2004
Level 1
Level 1

Thank you for posting this. :D

We have an Enterasys, not Cisco network, and one of our sites has been in increasing turmoil for about a week due to this.

Whoever at Microsoft invented this "malware" clearly does not understand how LAN switches work.

It might work fine on $50 Netgear or D-Link, but it's going to slaughter an enterprise grade switch. And where is SCCM most often deployed - large enterprises!

In our situation, once a MAC had been borrowed and hopped to another port, the network card on the PC which is the true owner of the MAC address stops working. It constantly flaps the link up and down, the link light going on and off every few seconds. The only way to make the PC communicate again with anything, even another switch is to reboot the PC. Disabling then re-enabling the network card made no difference, it needs a hard reboot.

We started Wiresharking the problem this afternoon and were immediately drawn to lots of unusual client-to-client traffic. Lots of TCP requests were being sent to specific PC's on port TCP 25536 which Microsoft's sleepagentservice.exe is running on. That pointed the finger at SCCM into the frame and here we are.

Who are Microsoft employing to come up with this stuff? Do they not read RFC's?

The funny thing is when this started happening it had the smell of Malware - I didn't expect it to be true.

Thanks again for posting,

Mark

Kevin Dorrell
Level 10
Level 10

I have just spent a couple of days trying to track this down, thinking we had a worm or a virus.  In fact, from the network point of view it is indistinguishable from a worm.

Thank you for this great posting.

Kevin Dorrell

Luxembourg

 

nazirahmadd10
Level 1
Level 1

Has anyone got any solution for this issue?

schwa
Level 1
Level 1

I don't know enough four letter words to describe how I feel about this issue.

Not applicable

My network environment become intermittent since last 2 week after SCCM active.Any solutions for this issue ?

Thanks for posting this! Your information lead to resolving an issue which was beginning to make our entire network team fill with dread, as more and more client machines around our site were beginning to have inexplicable connectivity issues.

 

On our cisco network, we have port-security enabled to allow only 2 mac addresses on switch ports. We were seeing various switch stacks (2960s, 3850s) experiencing port-security violations.

 

When inspecting these port security violations, we were seeing client mac addresses appearing on multiple switch ports on the same switch at the same time. We were starting to think either a IOS bug was affecting us, our mac address tables were becoming corrupted, or we had a network loop somewhere.


After spending about an hour on google looking for help, we found this page(!), and quickly established that an SCCM administrator had pushed out 'Wake-up proxy client'=YES to 100 machines on site as a test, and it was that setting interacting with sleeping clients that was causing our switches to see spoofed mac addresses on incorrect switch ports. 

 

We quickly rolled back those settings, and now all is good in the world again.

 

louis0001
Level 3
Level 3

Wow. Been on this issue for a few weeks too. Thank you very much for this as it would have took me another few weeks (at least) to have got to the bottom of it. You've saved me loads of time and stress! Thank you.

bhum11
Level 1
Level 1

We also have been tracking down 802.1x, switching, virus/work/malware for almost a week now.  One of the server guys apparently implemented this and didn't put in a change and denied doing anything when asked.....  What a pain!!!

Marc Luethi
Level 1
Level 1

Hi all!

 

Even in 2019 with a fresh SCCM deployment based on very late to latest releses, this is still a problem. We're running MAB with an ISE, and port security violations started to pop up soon after SCCM clients were added to the clients. It appeared that random PCs would start forging MAC source addresses, interestingly though only with MAC addresses already known and registered on the ISE

 

We reconfigured the switch ports from single-host to multi-auth and we could show how these "additional" MAC addresses actually were stable on the ports they appeared on, and how the issue wandered around across the switches involved. 

 

Combined intuition and Google-Fu of a network and a client systems admin eventually found this posting and also the dark corner of of Microsoft's SCCM documentation, where they describe the malwareish behaviour of the WoL Proxy. 

 

Ever since WoL proxy's been turned off, things have calmed down a lot. 

 

Thanks for posting this!

Marc

 

Thank you very much for this post.

Review Cisco Networking for a $25 gift card