Re: Switches and SUN boxes - problem

pokwan · ‎06-17-2004

Hi,

We have a SUN box on Catalyst 4003 and another SUN box on Catalyst 3750 on the same segment. Recently we had a message on both the boxes saying that "IP Hardware 'mac address' trying to be our address..." (as below) The 'mac address' is actually its own IP address! Can a switch/router cause a problem like this? These SUN boxes have static IP address rather than DHCP. Also, these are the ONLY 2 machine that have these messages or the like on both the switches. If a switch/router can cause a problem like this, can someone explain it to me.

TIA.

PF

Jun 14 12:56:38 sun2 ip: [ID 903730 kern.warning] WARNING: IP: Hardware address '00:03:ba:08:d2:99' trying to be our address 01!

Jun 14 12:56:16 sun1 ip: [ID 903730 kern.warning] WARNING: IP: Hardware address '00:03:ba:08:d3:75' trying to be our address 01!

s.uslay · ‎06-17-2004

How many NICs in each SUN ? just one ? Do you get the same message when you unplug one of the SUN boxes ?

Is there ant "smart" software on SUn like Checkpoint or Stonebeat that would use virtual IP addresses on top of real addresses ?

Serhat

pokwan · ‎06-17-2004

Serhat,

Many thanks for your reply. Here's the answers to your questions.

There is only one nic per box. The sun box on the catalyst 4003 has gb connection and the sun box on the catalyst 3570 has a 100mb connection.

The messages only appeared on Monday. (This incident may have happened months before). They did not try unplugging one of the sun boxes when it happened. The messages stopped after a while without intervention.

There is no "smart" software installed on the sun boxes.

Thanks.

PF

s.uslay · ‎06-17-2004

some more questions ;

1) Was this one-off incident that does not happen now ?

2) Would you (or did you ) try connecting Sun boxes to the same switch?

3) Are the boxes in the same subnet ?

I have seen some ARP abnormalities before on Sun boxes before, but you have to make sure that nothing is wrong on Layer2 ...

pokwan · ‎06-17-2004

Serhat,

1. I was told it did happen months before. It is not happening at this stage (only Monday at the time specified in the message)

2. We did not try connecting the Sun boxes to the same switch. They are on different building.

3. Yes, they are on the same subnet.

How do I make sure nothing is wrong on layer 2? What is strange is it only happened to these 2 boxes not any other client PCs and servers on the same subnet or on the same switches. If there is a problem with later 2, the problem would have occured with other divices on the network too, wouldn't it?

Thanks.

PF

s.uslay · ‎06-17-2004

It is more clear now, it did happen months before and again on Monday morning June 14th...

I could only suggest that run snoop on the sun boxes and pipe it to grep while running continuous PING to sun's IP address and wait for another occurence.

The other options is to span the Sun port to a PC running Ethereal and filter for ARP so that the buffer won't get full.

Serhat

pokwan · ‎06-17-2004

Serhat,

Thanks very much for your help. I will convey your suggestion to the sun administrators. Will do the span port as well.

PF

eric_voisard · ‎07-12-2004

HI,

I hope I'm not participating too late...

I'm also experiencing this issue with Sun boxes and Catalyst Switches. Each of the Sun having an unique ethernet interface with its own, known and legitimate MAC address.

After having discussed with a dozen of Sun admins who have experienced this problem, it clearly appears not being a Sun problem.

But is seems more likely to be a Cisco one. There should either be something wrong with the ethernet packets "routing" through the switches (spanning tree?) and/or some packet reflection to the sender.

Btw this message not only occurs on Sun systems: Netware in example has a same one which occurs in same network configuration.

What I've found so far is that in the site here, it started as soon as the Catalyst to which the Suns are connected has been linked to the inhouse network through a Gigabits fiber (to another Catalyst).

By coincidence, it appears some Cisco devices (i.e. IDS-4235) have a bug (CSCdy24260) in the Gigabits interface driver which causes this message. It's quite possible the same Gigabits driver is used for the Catalyst's Gigabits fiber modules...

I still don't have a definitive answer, but you also, have Gigabits links...

Yours, Eric

s.uslay · ‎07-12-2004

Eric,

I have only seen this sort of problem with SUNs when they have a "smart" applications like Checkpoint or Stonebeat on them.

Without running trace/snoop it is difficult to judge as Cisco problem.It happened here on 6509s on 100BaseT interfaces.

Serhat

eric_voisard · ‎07-13-2004

Hi Sherhat,

From what the other Sun admins said to me is that it can occur when a Sun host is sharing the same MAC addresss between two ethernet interfaces (IEEE802 compliant), and doing IPMP in example. But it's not my case, the only "smart" application on these servers is a database.

A hint is that another Sun sysadmin noticed that when it began to occur on his Suns it also began on some other OSes and some "connected" devices. Though I haven't seen it elsewhere here.

And there is the fact it also occurs on Netware with no reason and that they too look after the network to fix the problem.

As you said, it's difficult to judge and it's a long hunt. We've set a sniffer here, with luck we'll be able to catch the culprit...

If I find something, I'll update this thread...

Yours, Eric

smif101 · ‎07-14-2004

I have looked around and it looks it is one of two problems. The first one is that people have had this problem if they select IPv6 and IPv4 during setup. The other one is it could be a flaky port on the switch the NIC. I would recommend swapping these SUN machines with the ports of two other computers and see where the problem follows.

eric_voisard · ‎07-15-2004

Update:

First, it shouldn't be related to the Gigabits interface driver since it just occured twice yesterday, but with a 100Mb/s uplink...

However the sniffer was running. So we tried to find some reflected packets: we decided a reflected packet should look an exact image of a packet sent by the Sun, with the Sun's mac address as source. Then like if the Sun was indeed sending a second packet.

I think we found something:

1) a process on the Sun was trying to connect to a hosts which was actually down. Each 30 seconds it was trying to connect (what was normal behaviour).

In the capture we saw at this time that the Sun was doing ARP queries for this host. Each 30 seconds the Sun was sending a burst of 6 ARP queries, one about each second (time is approximative in the following):

Sun -> ARP Query @ time = time

Sun -> ARP Query @ time = time+1

Sun -> ARP Query @ time = time+2

Sun -> ARP Query @ time = time+3

Sun -> ARP Query @ time = time+4

Sun -> ARP Query @ time = time+5

Of course, no answer from the host.

Sun -> ARP Query @ time = time+30

Sun -> ARP Query @ time = time+31

Sun -> ARP Query @ time = time+32

Sun -> ARP Query @ time = time+33

Sun -> ARP Query @ time = time+34

Sun -> ARP Query @ time = time+35

Still no answer. And suddently:

Sun -> ARP Query @ time = time+60

Sun -> ARP Query @ time = time+60.300 (an extra packet in between!)

Sun -> ARP Query @ time = time+61

Sun -> ARP Query @ time = time+62

Sun -> ARP Query @ time = time+63

Sun -> ARP Query @ time = time+64

Sun -> ARP Query @ time = time+65

This at the time of the IP conflict!

There is an extra packet just after the first one and which looks like if sent by the server, but which is not counted with its burst of 6 packet. It looks like a reflected packet...

2) During the second IP conflict was another ARP query, but to a working host:

Sun -> ARP Query @ time=time

Sun <- Answer from host @ time+0.060

Sun -> ARP Query @ time=time+0.200

Sun <- Answer from host @ time+0.250

Now it looks like the Sun got an immediate answer from the host. So it had not to send a second query. But there is one, which occured a very short time after the first one (too short imho), and the host answered to both.

Again it looks like some reflected packet... If so, then this packet wasn't only reflected to the Sun's port on the switch, but was a real broadcast that every host saw, what I think explains why the host answered to both queries...

I've the feeling these extra packets are not sent by the Sun. Since it occurs only when the switch is linked to the inhouse network, it can also be a host somewhere on the LAN, moreover since every host should have seen the Sun's broadcasts. But on both cases here, it was with a query for hosts directly connected to the switch.

Still investigating...

Eric

PS: Smif101,tomorrow I'll check if both IPv4 and v6 are running. Any clue on what can be the cause in this case?

s.uslay · ‎07-15-2004

Can you run snoop on SUN (or HP protocol analyzer or sniffer) between Sun's NIC and network and try to match the data seen on the network to data sent by Sun. This will remove the possibility of "reflected" packet.Then you can concentrate on Sun's configuration, just a suggestion, I don't know if it is easy to set it up at your side.

Serhat

eric_voisard · ‎07-16-2004

Hi,

mmh, but will snoop behave differently than any other sniffer,I mean it will sniff what's arriving to the ethernet interface (incl. reflected packets), unless when there are outgoing packets it takes them from the driver's output buffer (not from the network)...

But I think I should put some emphasis to the fact that the problem ONLY occurs when the local switch is interconnected to the inhouse network. This fact alone should discriminate the Sun...

For the rest, I forgot to say that we've tried to replace the local switch with another same Catalyst, what didn't solve anything and what should also discriminate the local switch...

And btw, I've checked if both IPv4 and IPv6 were running, but only IPv4 is up.

Yours, Eric

s.uslay · ‎07-18-2004

Hi Eric,

YOu have to find out if the extra packet is really sent by SUN or "reflected" packet right ?

Then snoop on SUN should show this if it is leaving NIC or not? If it was a reflected one that should not be on the outbound of NIC. (but still visible in network Sniff ?_

Correct me if I am wrong?

Sun -> ARP Query @ time = time+60

Sun -> ARP Query @ time = time+60.300 (an extra packet in between!)

Sun -> ARP Query @ time = time+61

Sun -> ARP Query @ time = time+62

Sun -> ARP Query @ time = time+63

Sun -> ARP Query @ time = time+64

Sun -> ARP Query @ time = time+65

Serhat