unicast on all switch ports

nibauramos · ‎09-01-2011

Hello,

I'm using a CISCO 2960 in my network... I have somewhere in the network (not connected to this switch) one VMWARE that as a mail server with ip address a.b.c.d and make address in the virtual network interface of 00:0c:29:26:99:5d.

The thing is...I notice that all ports were blinking too much considering none of the hosts connected to it are in use...there should be very little traffic, I connected my laptop to port 14 in this switch and launch a packet sniffer... I receive packets that are addressed to the mail server running on the vmware, even though my mail server is not connected to this switch, nor is this switch in the path between the source and the origin of the packet I captured. I don't receive just one packet, I receive tons, enough to make a follow tcp strem in my wireshark and see the entire SMTP conversation.

I think this would be an expected behavior if for some reason the switch didn't know behind what port lies the mac 00:0c:29:26:99:5d (or if the mac was a broadcast, which it is not) so I connected to the switch and issued the following command:

#show mac address-table | include 000c.2926.995d

1 000c.2926.995d DYNAMIC Po1

I see that my switch has only one entry for this mac, and it is a port-channel, it's correct, this port-channel connects this switch to the rest of the network.

The port where I'm testing and capturing packets (port 14) doesn't belong to the port-channel:

#show running-config interface gi 0/14

interface GigabitEthernet0/14

switchport trunk encapsulation dot1q

switchport trunk native vlan 900

switchport mode trunk

no cdp enable

end

The configuration of the portchannel is perfectly simple:

#show running-config interface po1

interface Port-channel1

switchport trunk encapsulation dot1q

switchport mode trunk

end

lets see how manny interfaces are configured in this port-channel:

#show running-config | include channel

interface Port-channel1

channel-group 1 mode on

Only two interfaces....

The first one:

#show running-config interface gigabitEthernet 0/24

Building configuration...

Current configuration : 155 bytes

!

interface GigabitEthernet0/24

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 1 mode on

end

..and the second one:

#show running-config interface gigabitEthernet 0/23

Building configuration...

Current configuration : 155 bytes

!

interface GigabitEthernet0/23

switchport trunk encapsulation dot1q

switchport mode trunk

channel-group 1 mode on

end

So... any idea what could cause this behavior? the switch knows where the mac is so why is he forwarding the packets to all ports?

Thank you for your help

Alexander Maroukian · ‎09-01-2011

Hi,

Are the packets received from the server? Can you tell where is the source of the packets to the server?

Regards,

Alex

boss.silva · ‎09-01-2011

Hello,

Some time ago I've faced the same exact situation, but with a catalyst 4006 acting as an access-switch. A reboot solved the issue.

You might be hitting a bug. I suggest you open a TAC case for it.

Regards,

Bruno Silva.

nibauramos · ‎09-01-2011

Hello,

I tryied rebooting it but it didn't solve, here is the beggining of show version for the ios version in case it helps:

#show version

Cisco IOS Software, C3560 Software (C3560-IPBASE-M), Version 12.2(25)SEB2, RELEASE SOFTWARE (fc1)

Compiled Tue 07-Jun-05 23:34 by yenanh

Regarding the question of if the packets are receiver from the server, yes... the packets are received with a source mac/ip from the server and a destination mac/ip of the client that is connecting to it. And also the other way around, source ip/mac of the client and destination ip/mac of the server.

To describe how thing are connected, clients are connected to switch A, Switch A connects with a port-channel to Switch B (the "problematic" switch, the 2960) and switch A also connects to switch C that is where the mail server is connected.

thank you

Carlos Ramos

Alexander Maroukian · ‎09-01-2011

Carlos, are the server and the client in different vlans? Does the server and the client have different gateways? Asymmetric routing could cause this behavior.

Forwarding Table overflow is another cause as this is not likely to be the case but you should check it.

Regards,

Alex

nibauramos · ‎09-01-2011

Yes, server is in one vlan, client is in another vlan, clients have one gateway (router A), that has in its routing table a route to router B (that is the gateway of the network segment where the server is).

Just that, but the problem in here is at layer 2 right? no matter what tipe of routing process is happening, in the end I am receiving a unicast packet layer 2 packet, that has one mac address in its header representing the destination of the packet. Teh switch receives it, he knows to wich port the packet should be forwarded,...but... he forwards to all. Am i wrong?

How could routing cause this behavior?

Thank you

Carlos Ramos

Alexander Maroukian · ‎09-01-2011

Ok Carlos, here the idea is explained pretty nicely:

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml

Hope this will help you.

Regards,

Alex

nibauramos · ‎09-01-2011

Hello, nice documento however there is one diference... they say:

"The very cause of flooding is that destination MAC address of the packet is not in the L2 forwarding table of the switch. In this case the packet will be flooded out of all forwarding ports in its VLAN (except the port it was received on). Below case studies display most common reasons for destination MAC address not being known to the switch."

but that is the main diference, I have it in de mac address table, the switch has everything to do it right...but it doesnt...

Alexander Maroukian · ‎09-01-2011

I don't know why the mac addresses are the same as the server's and the client's when you are doing inter-vlan routing they should be different as the frame header rewrites when routed and get the router's mac address.

Regards,

Alex

nibauramos · ‎09-02-2011

I'm sorry, you are right, and after a closer look I found a few things more that had escaped me.

- Contrary to what I had indicated I am only receiving packets from the clients to the server, has i made a follow tcp stream it seemed to have parts of the conversation from the server to the client but no, I'm sorry, my mistake.

- Just like you said, you are right, the destination mac is the mac of my server, however the source mac is not from my clients but from the gateway of the mail server;

- I have also found something strange, has I said before, clients are connected to switch A, they send a packet, switch A is connected to switch B and switch C;

Switch B is where I identified the problem, and the mail server is on switch C, so: Switch A uppon receiving a ethernet packet directed to the mail server (on switch C that is directly connected) should know (assuming it is not the first packet in either direction) that the mac address of my mail server is accessible on port X I issue in that switch the following:

#show mac address-table | include 995d

1 000c.2926.995d DYNAMIC Gi2/0/18

wich is correct, however, for some reason....during the time the mac address shows just like I'm showing you...the problem persists in switch B, I continue to receive unicast trafic to the mail server in all the ports of my switch... but then something strange happens.... I stopped receiving traffic on my sniffer, things worked ok for a while...during that time I discovered the following:

#show mac address-table | include 995d

1 000c.2926.995d DYNAMIC Gi2/0/18

160 000c.2926.995d DYNAMIC Gi1/0/1

He "finds" a diferent path to the mail server...the strangest thing is that hidden behind port Gi1/0/1 is a traffic shaper that has nothing to do with the mail server...but the switch somehow discoveres tha mail server mac through there and during that time everything works ok...after a while...the new mac entry disappears and the problem returns....

I'm starting to think there is some problem with my vlan's configurations, some broadcast domain is not 100% correctly configured... there are tons of vlans here, and many trunks, many of them don't correctly speccify wich vlans can pass... I'll try to review all the configuration...but it will take a while...

But no matter what, If I forget everything and concentrate in switch B only, I still have one switch, that has my mail server mac address accessible through port-channel 1 (and he knows it cause I check its mac address table), it receives a packet through that very same port-channel 1 with a destination mac of my mail server, and for some reason it just forwards the packet to all ports....

Thank you all for the help!

Alexander Maroukian · ‎09-02-2011

Hi Carlos,

Nice analysis. I have to think over again with the new info. Something comes to my mind - there is very small chance that you may have duplicated mac addresses in you infrastructure on your virtual machines. You said that this mac address appears on other interface which have nothing to do with the mail server. Have you checked your virtual infrastructure? For me the strange thing is that the normal working order is exactly in this very moment where you have this duplicated mac address. Do you use some kind of stp on this (the mail server)vlan? On which switch you have issued these commands A or B?

Looking at this:

#show mac address-table | include 995d

1 000c.2926.995d DYNAMIC Gi2/0/18

160 000c.2926.995d DYNAMIC Gi1/0/1

Why this mac address is in these two vlans 1 and 160? You should check them - where this mac address (of the mail server) should belong.

Regards,

Alex

nibauramos · ‎09-05-2011

The problem seems to be in there.... I have no idea why the mac appears in both vlans... it should only appear in vlan 1 not 160.

The command I issued was in the first switch (switch A).

I have just discovered something more vlan 160 and vlan 1, both carry traffic for the same layer 3 network segment, why? because:

Clients connect to network segment, "throug" vlan 160, but the gateway is in vlan 1, why? because there is a passive network shaper, we have an appliance that has one interface in vlan 160 and another in vlan 1, and shapes all traffic from one side dto the other. well I thought this could be the thing generating the problem....but no... I replaced this appliance by a simple network cable and problem persists, there is something in the network that causes the switch to learn that mac on the "wrong" interface...

However and I say again, no matter what, If I forget everything and concentrate in switch B only, I still have one switch, that has my mail server mac address accessible through port-channel 1 (and he knows it cause I check its mac address table), it receives a packet through that very same port-channel 1 with a destination mac of my mail server, and for some reason it just forwards the packet to all ports.... and THAT IS WRONG

this is killing me...must solve it

thanks for all the help so far!

PS: I checked all my virtual hosts...no duplicate mac found, thanks for the tip

nibauramos · ‎09-05-2011

Something to add:

I checked STP debug messages of congfiguration changes and I am receiving every second the following message:

5w3d: STP CFG: found port cfg GigabitEthernet1/0/1 (44B6138)

That is the por that connects from one vlan to another...

nibauramos · ‎09-05-2011

Hello, so I found the problem, I came here to write new developments 3 times today, and has I was typing in every single time, I stopped cause I remember something new I have to come here more often

Sorry for the long post, but it isn't simple to explain, and I think I found (understood) the problem....

This is our scenario:

So to my understanding this is what is happening:

The server is in one network (Network A), the client is in another (Network B), however, for the server to reach the client it has to reach it through one gateway on its network (client GW), both the server and the client GW are in the same IP network, but....for shaping purposes it was decided to break the layer 2 broadcast domains that support that network, bridging them together through the shaper (bridging), this causes that when the server sends a packet to the network it is registered in the switch that his mac address is accessible through that port on vlan 1, and then, when the destination (even though it is on the same layer 3 domain) is connected to the other vlan (160) the shaper bridges it causing the switch to register the mac address of the mail server (the shaper is completely passive and does not alter the source mac) again, now in a diferent port on vlan 160. So the normal state contrary to what I said is to have always the mac in both fisical ports, but on different vlans each, just like I was finding it when everything works correctly.

The problem now is....so why does sometimes the mac address of the mail server disappears from the mac address table of vlan 160? That is "the most interesting", the problem is that the next hop when trying to reach the client ip from within the server, isn't the client gw, there is the gateway of the server....and.....the gateway of the server is no more no less than the switch itself! This switch has one interface, vlan1 interface that has an ip address, that ip address is in the same network of client GW, and Server, however that interface is "only" on vlan1, and not on vlan160...so when th router has to forward on packet that came in through vlan 1 to an ip address that actually is on the same ip network but is connected on vlan 160, well...in that case...I don't know what happens but I can take a guess... the packet enters vlan 160 because it is the layer 2 functionality of the switch working, however...has the packet is routed it's mac address is changed by the switch layer 3 functionality, causing that packet to enter vlan 160 with the mac of the router and not the mac of the server, when the client replies.... well, its true, there is no entry on the mac address table that says where that mac is connected in vlan 160, only on vlan 1, so the switch does what it should do, it floods all vlan 160 ports in order to try and reach the server.

Am I wrong? What I conclude? Amntaining a layer 3 domain supported in multiple layer 2 domains, should be done, right?

It's not my config, I don't like it, and when I saw it I didn't like it in the first place however never thought it could generate such a problem.

Solution: We are creating a new server zone and will start to move servers to this zone, that will be a different one from all this, in the end there will remain no-one in vlan 160 and problem will be overcome. Meanwhile we will live with this.

Thank you all for your help, and if any of my "deductions" up there are wrong, please correct me

Carlos

Alexander Maroukian · ‎09-05-2011

From design POV I would prefer different both the vlan(not vlan1 if native) and the ip subnet for the server if possible.

Just look how these vlans are bridged. How do the traffic is passing between them only through one single way or once using the path through the shaper and another time not using the shaper path over some other path. If this is done with ACLs look at them. Is there another transparent l2 device between these two vlans?

About although not recommended practice it is possible to have normally working network with common ip subnet and multiple vlans.

Move the servers as soon as possible to the safe specially designed zone - not only in OSI layer 1 but also in upper layers

Regards,

Alex