12-25-2014 10:39 PM - edited 03-07-2019 10:00 PM
Team,
Last evening (Christmas eve) we setup a pair of Cisco 3850 with IP Base version 3.3.35SE (recommended) and 3.7.0E (very latest).
We got these to replace a very old switch that had died. Attached to this network are windows 7 PC's with all the standard patches, service packs, etc.
with standard port configs - no PC would work - and in fact on each screen we got the windows 7 IP Conflict pop up box.
This seemed very odd to us, as we know these IP's are all static (no dhcp on this segment at all)
we went with a very vanilla config on each port
interface g1/0/1
switchport host
that is it - nothing special at all.
well, after hours of research we found the 3850 has a problem where its "ip device tracking" (even though disabled, by way of NOT being enabled on any interface) will effect the windows 7 PC's ip address in use detection port start up phase!
This is a very big problem. I am frankly SHOCKED Cisco would release a major switch that is going to not work when connected to the average network with windows 7 PC's.
we tried 3+ hours of prescribed work-arounds found when researching this issue -
ip device tracking probe delay 10 (global config)
ip device tracking max 0 (disabed, on interface)
finally,
nmsp attach suppress (interface, however this appears to be a default command in all IOS-XE versions we tried, as the command did NOT show in the show run) . this effected many different nic card vendors (laptops, desktops) and nic card drivers levels from old to very recent.
Finally,
we compared a 3850 in another location to this one - and we never got HIT by this problem before because that 3850 only as TRUNK ports and no windows 7 hosts directly attached.
Doing more research, I found out this also can effect vmware guests running windows SERVER.
this is now a huge issue as we have a scheduled deployment of 3850's throughout our network which is going to be put on hold.
the work-around I came up with which is not great is -
Make ALL the "access" ports connected to PC TRUNK ports and leave the NATIVE vlan (untagged) as the vlan you want the PC's to be in
interface g1/0/1
switchport mode trunk
switchport trunk native vlan 1
this is NOT an acceptable workaround as this presents security issues even with
switchport trunk allowed vlan 1, etc. as the only allowed vlan.
Note: this issue manifested itself and windows 7 PC's were UNABLE to use the network. if you do "ipconfig /all | more" you would see
192.168.0.140(duplicate) and the interface would actually use 169.254.0.239(duplicate) so the duplicate message appeared twice in the output.
1) With and without an SVI interface on each 3850 for the vlan where the windows 7 machines had a duplicate
2) when we had an SVI and the command ip device tracking probe use-svi (or whatever the hidden command is I forget now, but it took it)
3) when we had aaa new-model configured - and not configured - thinking this was some artifact of having aaa turn on something like 802.1x port state
4) when could confirm NO DHCP SNOOPING
5) when we DID not use static IP's - and had the switch assign DHCP addresses - the Windows 7 PC's STILL had duplicates and didnt work for their "Just leased" ip's.
6) when we could confirm ios-xe ip device tracking = disabled with show ip device tracking status, etc.
This is a major problem for this 3850 and unless we get a definitive answer on why this is happening and how we can rectify we are going to have to return our 3850's and get HP Procurve's something I would rather avoid doing. There is NO REASON I can imagine other than older switches who's ports default to ROUTED ports (i.e.. no ip switchport) where a switch should not at least function as a bare switch with essentially a default configuration out of the box.
Any ideas? I'm working well now with the ports ALL in trunking mode with vlan 1 native, but this is not a scalable workaround we can live with as we have security risks of a port not blocking certain vlans from going out ports to pc's, etc. that attackers could send tags on at that point, etc.
thanks,
Joe Brunner
#19366
12-25-2014 11:54 PM
interface g1/0/1 switchport host
Where's STP portfast?
12-26-2014 12:00 AM
that command is what i type - portfast is added and we write mem
"switchport host" is a maco
once done you get this -
int g1/0/1
switchport mode access
spanning-tree portfast
we tried all variants i should add of having a "default port" - default switch vs. anything we could think to throw on the interfaces or in global config.
this all started with "the switch just not working" once it was powered on and devices connected
thanks
12-26-2014 03:37 AM
Hello
This all stems from a switch replacement correct?
Are these 3850's in a stack?
Does it have a managment ip address -If so, is it using the old switch ip address
What are they connecting to? (a router/L3 switch/anohter switch- cisco-HP etc..)
How are they connected( L3 interface/L2 trunk/access port)
Are thse switches performing inter-vlan routing or just acting as host switches?
Is ip routing enabled?
Do you have multiple vlans in your network and if so ar ethe being propergated to these new switches?
Your 7 pcs = are they just client pcs not servers?
can you confirm something like ICS isnt enabled (Internet connection sharing) on any of them?
Are the just using one NIC each?
Can you post the running config of theses two switches
Also
sh switch
sh int trunk
sh vlan brief
sh vtp status
sh cdp neighbours
sh ip route
res
Paul
12-26-2014 05:06 AM
thanks for replying - i'm not onsite (its a standalone network) - but here is what it is -
Answers in line -
This all stems from a switch replacement correct?
yes a 10 year old Allied Telesyn switch was replaced that had no config - like a hub, just used for connectivity.
Are these 3850's in a stack?
>yes, tested all aspects of the stack many times.
Does it have a managment ip address -If so, is it using the old switch ip address
>old switch had no ip - i made a "management interface" on vlan 1 - BUT no ip on the built-in management interface on the switch.
What are they connecting to? (a router/L3 switch/anohter switch- cisco-HP etc..)
>various other devices - only 1 link back to a single 3750x stack. that switch is "hardened" so to speak to reveal or propagate very little by design.
How are they connected( L3 interface/L2 trunk/access port)
>all ports are left in trunk mode with vlan 1 as the active and untagged port. this was the workaround done to ever get the switch going. in "out of the box" or default mode as we initially wanted (no config) links to windows 7 PC's didnt work. links to linux or other devices non-windows did work!
Are thse switches performing inter-vlan routing or just acting as host switches?
>dumb flat network, no routing.
Is ip routing enabled?
>not unless enabled on 3850 by default. I didnt type "ip routing"
Do you have multiple vlans in your network and if so ar ethe being propergated to these new switches?
Your 7 pcs = are they just client pcs not servers?
client PC's - no servers OS per say.
can you confirm something like ICS isnt enabled (Internet connection sharing) on any of them?
>yes not enabled.
Are the just using one NIC each?
> one machine is dual homed - but we know where its "second nic" goes - to another cisco network which is NOT connected back to this one. we traced all our ports a few times thinking even perhaps some small hub was "reflecting" traffic back to us - like a blackbox. Strangest thing -
default config out of the box - with ALL ports SHUTDOWN EXCEPT the single windows 7 facing port - the windows 7 machine STILL registered an IP CONFLICT when connected to the 3850 - even when it had NO SVI's!!! (i know mind numbing). if you disconnected the pc and connected it to an old cisco switch - it worked fine!!! wow.
sh switch
2 identical 3850's in working stack. power and network stacked. both at same version, etc - upgraded each time with "software install file flash:<long ios name>.bin
tested all power and general 3850 stacking. saw no issues.
sh int trunk
>all ports are now trunks (hence the workaround used to get it up).
has 20 trunks to PC's and some single connected switches (far away on fiber) - all allow only vlan 1 - no other vlans were created - very very simple network. vlan 1 is native
sh vlan brief
>just vlan 1 - no vlans created, checked this many times - had vlan 100 at one point - made sure it was gone over a period of hours.
sh vtp status
not setup - left complete default; no vtp domain set - connected to all switches in transparent model if a switch connection exists.
sh cdp neighbours
cant post (for god and country LOL) but there is one link back to our "core" so to speak - that switch is hardened not to allow any settings to slip over to new switches so hence no vtp, cdp is one to help troubleshooting.
sh ip route
just the L and C routes for the vlan 1 ip address 192.168.17.1/24
no static routes
no vlan interfaces other than int vlan 1
no ip address on g0/0/0 -> the default 3850 management interface hard assigned to the 3850 VRF you cant remove.
int g0/0/0
ip vrf forwarding Switch_Mgmt
i can get over there if you think of anything else key to show the group.
thanks,
Joe
12-26-2014 09:15 AM
Are you saying in your original post that the problem manifests with both IOS-XE 3.3.5 and 3.7.0?
We have recently deployed a large number of 3850s (~400 scattered across single switch and stack deployments), all running IOS-XE 3.3.5 and have not encountered this problem. We have held off on IOS-XE 3.7.0 since it is a very new major release.
The client ports are a broad mix of devices, including many Windows 7 and Windows 8 PCs with both static and (mostly) DHCP-assigned addresses. The interfaces don't have any special setup or workaround - mostly just "switchport mode access", "switchport access vlan nn" and spanning-tree portfast".
The last time I saw a problem anything like yours, it was due to some newer ASA code needing me to turn off proxy ARP for the interface connecting towards the clients ("sysopt noproxyarp <interface>" in the ASA config).
12-26-2014 09:16 AM
they came out of the box brand new Wednesday with 3.2.3SE (Sept 2013)
we tried 3.3.35SE and then 3.7.0 to (desperately) try and resolve this issue.
our main facilities have 3.2.2 SE (June 2013) but no windows 7 access ports.
this is very strange - we are opening a tac case and we will post that # here shortly.
12-27-2014 06:56 AM
I'd be very curious to see how far you get with this. I was getting hit by random duplicate IP errors for machines with static IPs over a year ago. Back then I found a thread on here that said to use nmsp attach suppress on all the physical interfaces and things have been better since. I'm on 3.3.3 right now and when I do a show run, I do see the nmsp attach suppress in the config.
There's been a 4500X tossed into the mix and the nmsp attach suppress doesn't work on there, i had to put device tracking max to 0 on all ports.
Right now my biggest problem is VMs reporting false duplicate IPs (not often but it happens occasionally) but the physical machines seem fine.
03-30-2015 06:16 AM
Thanks for the insight.
we voted to hold off our customer 3850 upgrades for a year and explore other switch vendors...
there should not need to be a "work around" to simply unboxing the switching and using it at its simplest level - a layer 2 switch with 1 vlan.
Cisco has done a "Pete carroll passing on the 1 yard line with Marshawn Lynch" and out thinked itself with this ip device tracking. i hope that feature was worth upsetting a loyal client base.
not cool
02-04-2015 10:28 AM
Here is the official Cisco doc on this issue:
In environments I've seen, the following global commands (from the article above) resolved the issue:
ip device tracking probe delay 10
ip device tracking probe use-svi
Of course, totally disabling ip device tracking--by disabling every single feature that uses it--also resolves. This is pretty hard to do, though, as there is not a simple 'no ip device tracking' global command available, and it is difficult to predict what future features in IOS/IOS-XE would re-enable ip device tracking.
Hope this helps!
Luke Walcher, CISSP
P.S. Microsoft perspective on this issue:
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide