weird behavior on a 3750 Stack with IPTV devices

[igor3771]

Hi guys,

i am facing a serious issue in a hotel that was recently equipped with a somewhat old 3750 Stack to enable multicast IPTV solution at a cheap rate (refurbished hardware, everything new was not deliverable)

So here's the problem:

The stack is created properly and is working fine so far. What I experience is that after some time (varies from hours to days) some TVs are unreachable and have no connection to the network anymore although I can see them connected on their ports.

They used to have a auto standby setting that let them go to sleep after x minutes and after y minutes into a deep standby where the network interface also went down. this has been changed so the network part of the TV is always on, just the display is turned off. So network wise they should always be reachable wich they are for mentioned varying time. There is no absolute number, seems to be random.

A TV management system is connected to the same network and there i can see all TVs either in standby or tuned on. Some go from standby to offline although they are not. The Ethernet ports on the switches where these TVs are connected to are up, the Mac address is also present but no ping to the device is possible and the device itself has no connectivity to the network. Even a hard reboot of the TV does not help. Only when the whole stack is rebooted, connectivity is restored. And the same behavior repeats itself on random other TV´s then.

I can see this on both switches, the master and the member.

here's the relevant config:

cisco WS-C3750-48TS (PowerPC405) processor (revision Q0) with 131072K bytes of memory.
Processor board ID FDO1320Z1UZ
Last reset from power-on
1 Virtual Ethernet interface
96 FastEthernet interfaces
8 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:25:83:FE:EC:80
Motherboard assembly number : 73-9680-12
Power supply part number : 341-0028-02
Motherboard serial number : FDO13200H5D
Power supply serial number : DTN131310H3
Model revision number : Q0
Motherboard revision number : A0
Model number : WS-C3750-48TS-S
System serial number : FDO1320Z1UZ
SFP Module assembly part number : 73-7757-03
SFP Module revision Number : A0
SFP Module serial number : FDO13200CV9
Top Assembly Part Number : 800-25854-04
Top Assembly Revision Number : A0
Version ID : V07
CLEI Code Number : COMD200BRA
Hardware Board Revision Number : 0x01

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
1 52 WS-C3750-48TS 12.2(55)SE12 C3750-IPBASEK9-M
* 2 52 WS-C3750-48TS 12.2(55)SE12 C3750-IPBASEK9-M

Switch 01
---------
Switch Uptime : 1 hour, 37 minutes
Base ethernet MAC Address : 00:14:69:CC:86:80
Motherboard assembly number : 73-9680-08
Power supply part number : 341-0028-02
Motherboard serial number : CAT09220GD6
Power supply serial number : DTH091819T3
Model revision number : L0
Motherboard revision number : B0
Model number : WS-C3750-48TS-E
System serial number : CAT0945R0B6
SFP Module assembly part number : 73-7757-03
SFP Module revision number : A0
SFP Module serial number : CAT09220PN8
Top assembly part number : 800-26384-02
Top assembly revision number : A0
Version ID : V05
CLEI Code Number : CNMV400CRE

Configuration register is 0xF

IPTV1#

IPTV1#show mac address-table
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
All 0100.0ccc.cccc STATIC CPU
All 0100.0ccc.cccd STATIC CPU
All 0180.c200.0000 STATIC CPU
All 0180.c200.0001 STATIC CPU
All 0180.c200.0002 STATIC CPU
All 0180.c200.0003 STATIC CPU
All 0180.c200.0004 STATIC CPU
All 0180.c200.0005 STATIC CPU
All 0180.c200.0006 STATIC CPU
All 0180.c200.0007 STATIC CPU
All 0180.c200.0008 STATIC CPU
All 0180.c200.0009 STATIC CPU
All 0180.c200.000a STATIC CPU
All 0180.c200.000b STATIC CPU
All 0180.c200.000c STATIC CPU
All 0180.c200.000d STATIC CPU
All 0180.c200.000e STATIC CPU
All 0180.c200.000f STATIC CPU
All 0180.c200.0010 STATIC CPU
All ffff.ffff.ffff STATIC CPU
1 70af.24ab.68c3 DYNAMIC Fa2/0/13
1 70af.24ab.68c9 DYNAMIC Fa1/0/3
1 70af.24ab.919e DYNAMIC Fa2/0/5
1 70af.24ab.91ad DYNAMIC Fa2/0/10
1 70af.24ab.91b8 DYNAMIC Fa1/0/23
1 70af.24ab.91d7 DYNAMIC Fa2/0/18
1 70af.24ab.91f7 DYNAMIC Fa2/0/7
1 70af.24ab.91fb DYNAMIC Fa1/0/25
1 70af.24ab.91fd DYNAMIC Fa2/0/12
1 70af.24ab.91fe DYNAMIC Fa2/0/45
1 70af.24ac.d7ba DYNAMIC Fa2/0/43
1 70af.24ac.d7c6 DYNAMIC Fa2/0/15
1 70af.24ac.d7c8 DYNAMIC Fa2/0/23
1 70af.24ac.d7cb DYNAMIC Fa2/0/39
1 70af.24ac.d7d4 DYNAMIC Fa2/0/35
1 70af.24ac.d7d9 DYNAMIC Fa2/0/48
1 70af.24ac.d90d DYNAMIC Fa2/0/41
1 70af.24ac.d92c DYNAMIC Fa2/0/27
1 70af.24ac.d933 DYNAMIC Fa2/0/26
1 70af.24ac.d934 DYNAMIC Fa2/0/32
1 70af.24ac.d936 DYNAMIC Fa2/0/47
1 70af.24ac.d9cf DYNAMIC Fa1/0/18
1 70af.24ac.d9da DYNAMIC Fa1/0/11
1 70af.24ac.d9db DYNAMIC Fa2/0/28
1 70af.24ac.e23f DYNAMIC Fa2/0/17
1 70af.24ac.e25a DYNAMIC Fa2/0/24
1 70af.24ac.e2b7 DYNAMIC Fa2/0/42
1 70af.24ac.e2f9 DYNAMIC Fa2/0/33
1 70af.24ac.e323 DYNAMIC Fa2/0/44
1 70af.24ac.e324 DYNAMIC Fa2/0/34
1 70af.24ac.e340 DYNAMIC Fa2/0/21
1 70af.24ac.e346 DYNAMIC Fa2/0/2
1 70af.24ac.e362 DYNAMIC Fa2/0/22
1 70af.24ac.e367 DYNAMIC Fa2/0/8
1 70af.24ac.e3a1 DYNAMIC Fa2/0/14
1 70af.24ae.1bab DYNAMIC Fa2/0/6
1 70af.24ae.1bb2 DYNAMIC Fa2/0/37
1 70af.24ae.1bf4 DYNAMIC Fa2/0/16
1 70af.24ae.1cb4 DYNAMIC Fa2/0/25
1 70af.24ae.1cb6 DYNAMIC Fa2/0/38
1 70af.24ae.1cba DYNAMIC Fa2/0/30
1 70af.24ae.1cc0 DYNAMIC Fa1/0/8
1 70af.24ae.1ce3 DYNAMIC Fa1/0/20
1 70af.24ae.1d34 DYNAMIC Fa2/0/3
1 70af.24ae.1d3b DYNAMIC Fa2/0/46
1 70af.24ae.1d7d DYNAMIC Fa2/0/40
1 70af.24ae.1d95 DYNAMIC Fa2/0/20
1 70af.24ae.1e95 DYNAMIC Fa1/0/24
1 70af.24ae.1eac DYNAMIC Fa2/0/9
1 70af.24ae.24e1 DYNAMIC Fa1/0/22
1 70af.24ae.29af DYNAMIC Fa1/0/10
1 70af.24ae.2a3e DYNAMIC Fa2/0/31
1 70af.24ae.2a50 DYNAMIC Fa2/0/29
1 70b3.d584.99af DYNAMIC Gi2/0/1
1 d4cf.f943.504c DYNAMIC Fa2/0/19
1 f09f.c263.726d DYNAMIC Gi1/0/2
1 f492.bf8f.1763 DYNAMIC Gi1/0/2
1 fc3f.db0e.7c21 DYNAMIC Gi1/0/2
Total Mac Addresses for this criterion: 78
IPTV1#

IPTV1#show switch stack-ports
Switch # Port 1 Port 2
-------- ------ ------
1 Ok Down
2 Ok Down

IPTV1#show switch stack-ring activity

Sw Frames sent to stack ring (approximate)
------------------------------------------------
1 466411
2 349344925

Total frames sent to stack ring : 349811336

IPTV1#show switch
Switch/Stack Mac Address : 0025.83fe.ec80
H/W Current
Switch# Role Mac Address Priority Version State
----------------------------------------------------------
1 Member 0014.69cc.8680 1 0 Ready
*2 Master 0025.83fe.ec80 9 0 Ready

snippets from the config:

switch 1 provision ws-c3750-48ts
switch 2 provision ws-c3750-48ts

.

.

.

interface FastEthernet1/0/1
switchport mode access
!
interface FastEthernet1/0/2
switchport mode access
!
interface FastEthernet1/0/3
switchport mode access
!

.

.

spanning-tree mode rapid-pvst
spanning-tree portfast default

.

.

I tried to set mac aging timer to max and see what happens.

What I saw before is that after some time learn Mac addresses on the ports aged out and after pinging the devices from the switch itself the Mac address on that port from the respective device was learnt but no reply. And the device when turned on has no network access. 

It looks like either the TV has an issue after some time or the switch does not handle the learned Mac addresses properly.

Any idea?

[balaji.bandi]

Not sure how we can fix this issue, but try some testing here :

Do you have any VTP configured, remove VTP to transparent.

other bits :

1. by default arp will be 4 hours try to reduce and see if that fix issue

arp timeout seconds

2. try one of port keepalive settings keeping the port up.

what TV is this ?

Also what is the DHCP IP reservation time. ?

otehr part why this is down down ?

IPTV1#show switch stack-ports
Switch # Port 1 Port 2
-------- ------ ------
1 Ok Down
2 Ok Down

[igor3771]

Hi, thanks for the ideas.

VTP is set to transparent right now. I will try all suggestions one after the other to see if it has any effect.

DHCP lease time is 86400 seconds

The TVs are Philips professional series on latest firmware.

The exact Model is 50HFL5014/12

They do have issues with Multicast handling (dont send proper leave group messages and the likes), their network interfaces are only Fast Ethernet and according to what the vendor said, their autoneg behavior is quite poor, thats why I thought going with a Fast Ethernet switch might be worth a try simply to avoid more issues as faced on previously used switches from a different vendor with GE-ports.

Multicast works just fine after tweaking igmp settings, the IPTV source is connected to a GigE - Port on that stack and delivers roughly 40  multicast groups to the connected TVs with a total upstream bandwidth of ~600MBit/s. The devices join and leave properly now so the FE-Ports are not in trouble with too many multicast groups bound to them.

[Leo Laohoo]

---

@igor3771 wrote:

They used to have a auto standby setting that let them go to sleep after x minutes and after y minutes into a deep standby where the network interface also went down. this has been changed so the network part of the TV is always on, just the display is turned off.

---

Wait.  Step back.

To some people, they are just "TVs" but to others, they are "codec box", i.  e.  The network cable plugs into these boxes and then an HDMI cable goes to the TV.  The "remote control" talks to these codec boxes and that is where the channels are being switched.  

If there is no "codec box" involved (and the back of the TV is directly connected to the switch) then I have also seen this problem and this are ALWAYS the case with the first two (maybe three) generations of so-called "smart" TVs with WiFi &/or wired NIC.  For the wired NIC, it is mostly solved by updating the firmware of the TV.  

Question:  Are these "things" all living in one (big), (fat) VLAN?

[igor3771]

Well, you are right.

These TVs have an integrated media box with an androidish OS running on them. 75 TVs all in one vlan (192.168.2.0/24).

Thats what I cold figure out. There is no external HDMI connection, this box is part of the TV.

Management Computer and Multicast Source in the same subnet and thats it.

I wanted to avoid multicast routing, thats why I put them all into one network. 

[Leo Laohoo]

---

@igor3771 wrote:

androidish OS

---

Ok, it is a "hybrid" (of sort).  

That is going to be a nightmare because there will be no documentation, vendor/manufacturer support.  Is this correct?

[Georg Pauwen]

Hello,

can you post the running config of the switch (stack) so we can see how exactly the multicast is configured ?

[paul driver]

Hello

---

@igor3771 wrote:

. this has been changed so the network part of the TV is always on, just the display is turned off. So network wise they should always be reachable wich they are for mentioned varying time. There is no absolute number, seems to be random.

t looks like either the TV has an issue after some time

---

I would say first thought its the changed setting on the IPTV devices, but you state this is a new setup with these 3750's?
I am assuming this has worked in the past, if so what hardware did the 3750's replace?

[igor3771]

The hotel network was set up with cheap ubiquity stuff. Turned out these switches cant be configured properly with IGMP settings so I picked these switches. 

!
ip igmp snooping querier address 192.168.2.3
ip igmp snooping vlan 1 querier version 2
ip igmp snooping vlan 1 querier query-interval 10
ip igmp snooping vlan 1 querier max-response-time 10
ip igmp snooping vlan 1 immediate-leave
ip igmp snooping vlan 1 last-member-query-count 1
cluster enable IPTV 0
!

!
interface Vlan1
ip address 192.168.2.3 255.255.255.0
ip pim passive
ip igmp last-member-query-count 1

arp timeout 3600
!

here's what happens:

I can see TVs in the management system (when they are booted up they register there so I know what ip and MAC address they have)

They appear with a red or green icon depending on the state they are (standby or turned on)

If they are greyed out they are unreachable.

192.168.2.226 was greyed out (MAC:70:AF:24:AC:D9:36)

192.168.2.147 was in standby (MAC:70:AF:24:AC:E2:5A)

MAC address table before I ping both:

IPTV1#show mac address-table
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
All 0100.0ccc.cccc STATIC CPU
All 0100.0ccc.cccd STATIC CPU
All 0180.c200.0000 STATIC CPU
All 0180.c200.0001 STATIC CPU
All 0180.c200.0002 STATIC CPU
All 0180.c200.0003 STATIC CPU
All 0180.c200.0004 STATIC CPU
All 0180.c200.0005 STATIC CPU
All 0180.c200.0006 STATIC CPU
All 0180.c200.0007 STATIC CPU
All 0180.c200.0008 STATIC CPU
All 0180.c200.0009 STATIC CPU
All 0180.c200.000a STATIC CPU
All 0180.c200.000b STATIC CPU
All 0180.c200.000c STATIC CPU
All 0180.c200.000d STATIC CPU
All 0180.c200.000e STATIC CPU
All 0180.c200.000f STATIC CPU
All 0180.c200.0010 STATIC CPU
All ffff.ffff.ffff STATIC CPU
1 70af.24ab.68c3 DYNAMIC Fa2/0/13
1 70af.24ab.68c9 DYNAMIC Fa1/0/3
1 70af.24ab.919e DYNAMIC Fa2/0/5
1 70af.24ab.91ad DYNAMIC Fa2/0/10
1 70af.24ab.91b8 DYNAMIC Fa1/0/23
1 70af.24ab.91d7 DYNAMIC Fa2/0/18
1 70af.24ab.91f7 DYNAMIC Fa2/0/7
1 70af.24ab.91fb DYNAMIC Fa1/0/25
1 70af.24ab.91fd DYNAMIC Fa2/0/12
1 70af.24ab.91fe DYNAMIC Fa2/0/45
1 70af.24ac.d7ba DYNAMIC Fa2/0/43
1 70af.24ac.d7c6 DYNAMIC Fa2/0/15
1 70af.24ac.d7c8 DYNAMIC Fa2/0/23
1 70af.24ac.d7cb DYNAMIC Fa2/0/39
1 70af.24ac.d7d4 DYNAMIC Fa2/0/35
1 70af.24ac.d7d9 DYNAMIC Fa2/0/48
1 70af.24ac.d90d DYNAMIC Fa2/0/41
1 70af.24ac.d92c DYNAMIC Fa2/0/27
1 70af.24ac.d933 DYNAMIC Fa2/0/26
1 70af.24ac.d934 DYNAMIC Fa2/0/32
1 70af.24ac.d936 DYNAMIC Fa2/0/47
1 70af.24ac.d9cf DYNAMIC Fa1/0/18
1 70af.24ac.d9da DYNAMIC Fa1/0/11
1 70af.24ac.d9db DYNAMIC Fa2/0/28
1 70af.24ac.e23f DYNAMIC Fa2/0/17
1 70af.24ac.e25a DYNAMIC Fa2/0/24
1 70af.24ac.e2b7 DYNAMIC Fa2/0/42
1 70af.24ac.e2f9 DYNAMIC Fa2/0/33
1 70af.24ac.e323 DYNAMIC Fa2/0/44
1 70af.24ac.e324 DYNAMIC Fa2/0/34
1 70af.24ac.e340 DYNAMIC Fa2/0/21
1 70af.24ac.e346 DYNAMIC Fa2/0/2
1 70af.24ac.e362 DYNAMIC Fa2/0/22
1 70af.24ac.e367 DYNAMIC Fa2/0/8
1 70af.24ac.e3a1 DYNAMIC Fa2/0/14
1 70af.24ae.1bab DYNAMIC Fa2/0/6
1 70af.24ae.1bb2 DYNAMIC Fa2/0/37
1 70af.24ae.1bf4 DYNAMIC Fa2/0/16
1 70af.24ae.1cb4 DYNAMIC Fa2/0/25
1 70af.24ae.1cb6 DYNAMIC Fa2/0/38
1 70af.24ae.1cba DYNAMIC Fa2/0/30
1 70af.24ae.1cc0 DYNAMIC Fa1/0/8
1 70af.24ae.1ce3 DYNAMIC Fa1/0/20
1 70af.24ae.1d34 DYNAMIC Fa2/0/3
1 70af.24ae.1d3b DYNAMIC Fa2/0/46
1 70af.24ae.1d7d DYNAMIC Fa2/0/40
1 70af.24ae.1d95 DYNAMIC Fa2/0/20
1 70af.24ae.1e95 DYNAMIC Fa1/0/24
1 70af.24ae.1eac DYNAMIC Fa2/0/9
1 70af.24ae.24e1 DYNAMIC Fa1/0/22
1 70af.24ae.29af DYNAMIC Fa1/0/10
1 70af.24ae.2a3e DYNAMIC Fa2/0/31
1 70af.24ae.2a50 DYNAMIC Fa2/0/29
1 70af.24ae.2acd DYNAMIC Fa1/0/27
1 70b3.d584.99af DYNAMIC Gi2/0/1
1 d4cf.f943.504c DYNAMIC Fa2/0/19
1 f09f.c263.726d DYNAMIC Gi1/0/2
1 f492.bf8f.1763 DYNAMIC Gi1/0/2
1 fc3f.db0e.7c21 DYNAMIC Gi1/0/2
Total Mac Addresses for this criterion: 79

After pinging :

IPTV1#ping 192.168.2.226

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.226, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
IPTV1#show mac address-table
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
All 0100.0ccc.cccc STATIC CPU
All 0100.0ccc.cccd STATIC CPU
All 0180.c200.0000 STATIC CPU
All 0180.c200.0001 STATIC CPU
All 0180.c200.0002 STATIC CPU
All 0180.c200.0003 STATIC CPU
All 0180.c200.0004 STATIC CPU
All 0180.c200.0005 STATIC CPU
All 0180.c200.0006 STATIC CPU
All 0180.c200.0007 STATIC CPU
All 0180.c200.0008 STATIC CPU
All 0180.c200.0009 STATIC CPU
All 0180.c200.000a STATIC CPU
All 0180.c200.000b STATIC CPU
All 0180.c200.000c STATIC CPU
All 0180.c200.000d STATIC CPU
All 0180.c200.000e STATIC CPU
All 0180.c200.000f STATIC CPU
All 0180.c200.0010 STATIC CPU
All ffff.ffff.ffff STATIC CPU
1 70af.24ab.68c3 DYNAMIC Fa2/0/13
1 70af.24ab.68c9 DYNAMIC Fa1/0/3
1 70af.24ab.919e DYNAMIC Fa2/0/5
1 70af.24ab.91ad DYNAMIC Fa2/0/10
1 70af.24ab.91b8 DYNAMIC Fa1/0/23
1 70af.24ab.91d7 DYNAMIC Fa2/0/18
1 70af.24ab.91f7 DYNAMIC Fa2/0/7
1 70af.24ab.91fb DYNAMIC Fa1/0/25
1 70af.24ab.91fd DYNAMIC Fa2/0/12
1 70af.24ab.91fe DYNAMIC Fa2/0/45
1 70af.24ac.d7ba DYNAMIC Fa2/0/43
1 70af.24ac.d7c6 DYNAMIC Fa2/0/15
1 70af.24ac.d7c8 DYNAMIC Fa2/0/23
1 70af.24ac.d7cb DYNAMIC Fa2/0/39
1 70af.24ac.d7d4 DYNAMIC Fa2/0/35
1 70af.24ac.d7d9 DYNAMIC Fa2/0/48
1 70af.24ac.d90d DYNAMIC Fa2/0/41
1 70af.24ac.d92c DYNAMIC Fa2/0/27
1 70af.24ac.d933 DYNAMIC Fa2/0/26
1 70af.24ac.d934 DYNAMIC Fa2/0/32
1 70af.24ac.d936 DYNAMIC Fa2/0/47
1 70af.24ac.d9cf DYNAMIC Fa1/0/18
1 70af.24ac.d9da DYNAMIC Fa1/0/11

IPTV1#ping 192.168.2.147

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.147, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms
IPTV1#

The MAC of the greyed out TV appears but no response. Now when I reload the switch .226 will respond.

IPTV1#show mac address-table
Mac Address Table
-------------------------------------------

Vlan Mac Address Type Ports
---- ----------- -------- -----
All 0100.0ccc.cccc STATIC CPU
All 0100.0ccc.cccd STATIC CPU
All 0180.c200.0000 STATIC CPU
All 0180.c200.0001 STATIC CPU
All 0180.c200.0002 STATIC CPU
All 0180.c200.0003 STATIC CPU
All 0180.c200.0004 STATIC CPU
All 0180.c200.0005 STATIC CPU
All 0180.c200.0006 STATIC CPU
All 0180.c200.0007 STATIC CPU
All 0180.c200.0008 STATIC CPU
All 0180.c200.0009 STATIC CPU
All 0180.c200.000a STATIC CPU
All 0180.c200.000b STATIC CPU
All 0180.c200.000c STATIC CPU
All 0180.c200.000d STATIC CPU
All 0180.c200.000e STATIC CPU
All 0180.c200.000f STATIC CPU
All 0180.c200.0010 STATIC CPU
All ffff.ffff.ffff STATIC CPU
1 70af.24ab.68c3 DYNAMIC Fa2/0/13
1 70af.24ab.68c9 DYNAMIC Fa1/0/3
1 70af.24ab.919e DYNAMIC Fa2/0/5
1 70af.24ab.91ad DYNAMIC Fa2/0/10
1 70af.24ab.91b8 DYNAMIC Fa1/0/23
1 70af.24ab.91d7 DYNAMIC Fa2/0/18
1 70af.24ab.91f7 DYNAMIC Fa2/0/7
1 70af.24ab.91fb DYNAMIC Fa1/0/25
1 70af.24ab.91fd DYNAMIC Fa2/0/12
1 70af.24ab.91fe DYNAMIC Fa2/0/45
1 70af.24ac.d7ba DYNAMIC Fa2/0/43
1 70af.24ac.d7c6 DYNAMIC Fa2/0/15
1 70af.24ac.d7c8 DYNAMIC Fa2/0/23
1 70af.24ac.d7cb DYNAMIC Fa2/0/39
1 70af.24ac.d7d4 DYNAMIC Fa2/0/35
1 70af.24ac.d7d9 DYNAMIC Fa2/0/48
1 70af.24ac.d90d DYNAMIC Fa2/0/41
1 70af.24ac.d92c DYNAMIC Fa2/0/27
1 70af.24ac.d933 DYNAMIC Fa2/0/26
1 70af.24ac.d934 DYNAMIC Fa2/0/32
1 70af.24ac.d936 DYNAMIC Fa2/0/47
1 70af.24ac.d9cf DYNAMIC Fa1/0/18
1 70af.24ac.d9da DYNAMIC Fa1/0/11
1 70af.24ac.d9db DYNAMIC Fa2/0/28
1 70af.24ac.e23f DYNAMIC Fa2/0/17
1 70af.24ac.e25a DYNAMIC Fa2/0/24
1 70af.24ac.e2b7 DYNAMIC Fa2/0/42
1 70af.24ac.e2f9 DYNAMIC Fa2/0/33
1 70af.24ac.e323 DYNAMIC Fa2/0/44
1 70af.24ac.e324 DYNAMIC Fa2/0/34
1 70af.24ac.e340 DYNAMIC Fa2/0/21
1 70af.24ac.e346 DYNAMIC Fa2/0/2
1 70af.24ac.e362 DYNAMIC Fa2/0/22
1 70af.24ac.e367 DYNAMIC Fa2/0/8
1 70af.24ac.e3a1 DYNAMIC Fa2/0/14
1 70af.24ae.1bab DYNAMIC Fa2/0/6
1 70af.24ae.1bb2 DYNAMIC Fa2/0/37
1 70af.24ae.1bf4 DYNAMIC Fa2/0/16
1 70af.24ae.1cb4 DYNAMIC Fa2/0/25
1 70af.24ae.1cb6 DYNAMIC Fa2/0/38
1 70af.24ae.1cba DYNAMIC Fa2/0/30
1 70af.24ae.1cc0 DYNAMIC Fa1/0/8
1 70af.24ae.1ce3 DYNAMIC Fa1/0/20
1 70af.24ae.1d34 DYNAMIC Fa2/0/3
1 70af.24ae.1d3b DYNAMIC Fa2/0/46
1 70af.24ae.1d7d DYNAMIC Fa2/0/40
1 70af.24ae.1d95 DYNAMIC Fa2/0/20
1 70af.24ae.1e95 DYNAMIC Fa1/0/24
1 70af.24ae.1eac DYNAMIC Fa2/0/9
1 70af.24ae.24e1 DYNAMIC Fa1/0/22
1 70af.24ae.29af DYNAMIC Fa1/0/10
1 70af.24ae.2a3e DYNAMIC Fa2/0/31
1 70af.24ae.2a50 DYNAMIC Fa2/0/29
1 70af.24ae.2acd DYNAMIC Fa1/0/27
1 70b3.d584.99af DYNAMIC Gi2/0/1
1 d4cf.f943.504c DYNAMIC Fa2/0/19
1 f09f.c263.726d DYNAMIC Gi1/0/2
1 f492.bf8f.1763 DYNAMIC Gi1/0/2
1 fc3f.db0e.7c21 DYNAMIC Gi1/0/2
Total Mac Addresses for this criterion: 79
IPTV1#ping 192.168.2.226

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.2.226, timeout is 2 seconds:
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/2/8 ms
IPTV1#

[igor3771]

Whats also interesting is that these boxes behave identical over both switches. So its not only happening on the member or master, it can be seen on both chassis.

[igor3771]

After some checking I guess I was looking at the wrong places. Seems the ports are getting way too much traffic and started dropping packets due to misconfiguration or better, missing config. I remember I had it up and running properly and most likely forgot to save the config. After a power outage old config was loaded and since then these problems occur.

So what I found out so far:

reducing the multicast source interface traffic to less than 100Mbit/s brings all TVs properly back to communicate with the management server and all is fine. Of course just a fraction of the programs is being distributed now and I cant seem to find the proper setting anymore.

The multicast source is connected to a Gbit-Port on the cluster and to my understanding only ports that join a certain group get the traffic copied to them but when I raise the traffic to more than 100Mbit/s I get typical congestion behavior like bad sound, artifacts in the stream and going higher with traffic complete loss of signal. In parallel my pings to the devices get packet loss, rising in the same way. Clear sign of multicast not set up properly. The config I saved and restored was the final one (at least I thought I saved it..) but its not working properly.

So here's the setup:

Multicast source connected to Gi2/0/1

TV´s are all connected to Fa-Ports

the management Server is connected to Gi2/0/3

IPTV1#show interfaces Gi2/0/1
GigabitEthernet2/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0025.83fe.ec81 (bia 0025.83fe.ec81)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 27/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 106272000 bits/sec, 9678 packets/sec
5 minute output rate 1000 bits/sec, 1 packets/sec
656994677 packets input, 902696814852 bytes, 0 no buffer
Received 656994623 broadcasts (656994623 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 656994623 multicast, 0 pause input
0 input packets with dribble condition detected
117357 packets output, 11279926 bytes, 0 underruns
0 output errors, 0 collisions, 2 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
IPTV1#

IPTV1#show interfaces Gi2/0/3
GigabitEthernet2/0/3 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 0025.83fe.ec9b (bia 0025.83fe.ec9b)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 10/100/1000BaseTX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:02:50, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 7000 bits/sec, 5 packets/sec
5 minute output rate 19000 bits/sec, 7 packets/sec
6928241 packets input, 9720346089 bytes, 0 no buffer
Received 8485 broadcasts (8198 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 8198 multicast, 0 pause input
0 input packets with dribble condition detected
154003091 packets output, 206062364432 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
IPTV1#

looks good so far, no?

So here's my config, maybe someone sees what I am obviously too blind to see

!
no aaa new-model
clock timezone Europe/Berlin 2
switch 1 provision ws-c3750-48ts
switch 2 provision ws-c3750-48ts
system mtu routing 1500
vtp mode transparent
ip routing
ip name-server 8.8.8.8
!
!
ip igmp snooping querier address 192.168.2.3
ip igmp snooping vlan 1 querier version 2
ip igmp snooping vlan 1 querier query-interval 10
ip igmp snooping vlan 1 querier max-response-time 10
ip igmp snooping vlan 1 immediate-leave
ip igmp snooping vlan 1 last-member-query-count 1
ip igmp profile 239
permit
range 239.0.0.0 239.0.0.254
cluster enable IPTV 0
!

!
spanning-tree mode rapid-pvst
spanning-tree portfast default
spanning-tree extend system-id
!
vlan internal allocation policy ascending
lldp run
!
!
!
interface FastEthernet1/0/1
switchport mode access
ip igmp max-groups 1
ip igmp max-groups action replace
ip igmp filter 239
!
interface FastEthernet1/0/2
switchport mode access
ip igmp max-groups 1
ip igmp max-groups action replace
ip igmp filter 239
!

.

.

.

!
interface GigabitEthernet1/0/1
switchport mode access
!
interface GigabitEthernet1/0/2
switchport mode access
speed nonegotiate
!
interface GigabitEthernet1/0/3
switchport mode access
!
interface GigabitEthernet1/0/4
switchport mode access
!

!
interface FastEthernet2/0/1
description TV_505
switchport mode access
ip igmp max-groups 1
ip igmp max-groups action replace
ip igmp filter 239
!
interface FastEthernet2/0/2
description TV_214
switchport mode access
ip igmp max-groups 1
ip igmp max-groups action replace
ip igmp filter 239
!

.

.

.

!
interface GigabitEthernet2/0/1
switchport mode access
!
interface GigabitEthernet2/0/2
switchport mode access
!
interface GigabitEthernet2/0/3
switchport mode access
speed 1000
!
interface GigabitEthernet2/0/4
description uplink_network
switchport mode access
!

!
interface Vlan1
ip address 192.168.2.3 255.255.255.0
ip helper-address 192.168.2.1
ip pim passive
ip igmp last-member-query-count 1
ip cgmp proxy
!
ip classless
ip route 0.0.0.0 0.0.0.0 192.168.2.1
no ip http server
no ip http secure-server

IPTV1#show ip igmp groups
IGMP Connected Group Membership
Group Address Interface Uptime Expires Last Reporter Group Accounted
233.89.188.1 Vlan1 16:09:26 00:02:55 192.168.2.1
239.255.255.250 Vlan1 17:33:22 00:02:39 192.168.2.88
239.255.102.18 Vlan1 17:26:02 00:02:35 192.168.2.88
239.0.0.2 Vlan1 07:06:43 00:02:36 192.168.2.129
IPTV1#

IPTV1#show ip igmp groups detail

Flags: L - Local, U - User, SG - Static Group, VG - Virtual Group,

       SS - Static Source, VS - Virtual Source,

       Ac - Group accounted towards access control limit

Interface: Vlan1

Group: 233.89.188.1

Flags:

Uptime: 16:11:02

Group mode: EXCLUDE (Expires: 00:02:56)

Last reporter: 192.168.2.1

Source list is empty

Interface: Vlan1

Group: 239.255.255.250

Flags:

Uptime: 17:34:59

Group mode: EXCLUDE (Expires: 00:02:54)

Last reporter: 192.168.2.88

Source list is empty

Interface: Vlan1

Group: 239.255.102.18

Flags:

Uptime: 17:27:38

Group mode: EXCLUDE (Expires: 00:02:02)

Last reporter: 192.168.2.88

Source list is empty

Interface: Vlan1

Group: 239.0.0.2

Flags:

Uptime: 07:08:20

Group mode: EXCLUDE (Expires: 00:02:54)

Last reporter: 192.168.2.129

Source list is empty

IPTV1#

the last one is a TV actually watching channel 2 connected to this port:

IPTV1#show interfaces Fa2/0/13
FastEthernet2/0/13 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 0025.83fe.ec8f (bia 0025.83fe.ec8f)
Description: TV_511
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 28/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 72274480
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 11053000 bits/sec, 1004 packets/sec
1476850 packets input, 119983507 bytes, 0 no buffer
Received 6021 broadcasts (5881 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 5881 multicast, 0 pause input
0 input packets with dribble condition detected
133697664 packets output, 183778421844 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
IPTV1#

[Jukka-Pekka]

Hi,

Were these with half-duplex nics?

I have seem this behavior long time ago also.