network down for 3 minutes irregular ...

[wowhsieh]

It last about half years that network will go down for just about 3 minutes irregular every few days, and then back to normal. I setup some monitor software tools to monitor cloud/ISP/wan load-balance device/firewall/L3 switch (refer attachment), it seems that L3 switch could be the problem?!

When network down, the monitor software tools lost ping to Cisco 3750 L3 switch vlan99 & all the way to the internet for 3 minutes, but lost ping to Cisco 3750 local vlan16 interface for only 1 minutes.

I wonder that network down seems should be blamed to Cisco 3750?!

[Julio E. Moisa]

Hi

What about the logs and CPU utilization during the downtime? I don't know your work load or type of traffic but you could consider upgrade the switch to other recent model in a future if it is required. During a normal day how is the CPU and memory utilization on the 3750?

[wowhsieh]

I had checked SystemMessage and find ...

1. Message has no timestamp? (the system time had not been setup properly in the beginning, I already setup just now)

After I checked, by default, log messages are not time stamped, it need enabled timestamp of log messages,

2. The Event Description display port down, how to find out the actual reason? (maybe it's the reason why network down for 3 minutes irregular)

[Georg Pauwen]

Hello,

in addition to Julio's remarks, can you post the output of:

sh interfaces GigabiEthernet1/0/19

[wowhsieh]

 the output of interfaces GigabiEthernet1/0/19 ...

I found that from system log, it seems some interface port flapping down and up, looks like there's looping?!

[Leo Laohoo]

Well, the time-and-date of those interface going down/up is of no use.

[wowhsieh]

I check cpu and memory utilization seems ok last 72 hrs except at last 65 hrs, will check that again when network down next time.

[Leo Laohoo]

It sounds like a network loop of some sort as a device reboots (network loop or storm) and then recovers (another network loop or storm).

[paul driver]

Hello

Can you post the config of the 3750 please and also inc:
sh int trunks
sh etherchannel summary
sh interface | in line|\/sec
sh spanning-tree detail | in transitions|sent|received
sh spanning-tree vlan 16 detail
sh spanning-tree vlan 99 detail

res
Paul

[wowhsieh]

Hi Paul, please refer attachment about the output of these commands, thanks ...

sh int trunks
sh etherchannel summary
sh interface | in line|\/sec
sh spanning-tree detail | in transitions|sent|received
sh spanning-tree vlan 16 detail 
sh spanning-tree vlan 99 detail

[paul driver]

Hello

Referring to your OP -  stating connectivity issues with vlan 16 and 99.

You current topology regards the trunks and ether-channeling show:

1. no vlan 99 allowed on any trunks ( only access port gig1/0/22)
2. vlan 16 only allowed across Port-channel 4,
3. Only 3 vlans(13, 16,100) showing any generated traffic again only pc4 is allowing all three of these
4. vlans, gig1/0/5 - 7 allowing vlan 13
5. I do see high stp tcn and transitions referring to Vlan 16 PC4 & Vlan 99- access-port gig1/0/22 - With gig1/0/22 recording the highest generated traffic rate of all active interfaces but its related vlan 99 this port is assigned to shows no active traffic?
   
   

4      Po4(SU)         LACP      Gi1/0/13(P) Gi1/0/14(P)
Po4         1-2,13,16,61,100,102-103
GigabitEthernet1/0/13 is up, line protocol is up (connected)
  5 minute input rate 1708000 bits/sec, 231 packets/sec
  5 minute output rate 2347000 bits/sec, 300 packets/sec

GigabitEthernet1/0/14 is up, line protocol is up (connected)
  5 minute input rate 207000 bits/sec, 84 packets/sec
  5 minute output rate 350000 bits/sec, 40 packets/sec

Port-channel4 is up, line protocol is up (connected)
  5 minute input rate 1904000 bits/sec, 322 packets/sec
  5 minute output rate 2684000 bits/sec, 346 packets/sec

VLAN0016
  Number of topology changes 1 last change occurred 6d01h ago
          from Port-channel4
  Times:  hold 1, topology change 35, notification 2

GigabitEthernet1/0/22 is up, line protocol is up (connected)
  5 minute input rate 3032000 bits/sec, 430 packets/sec
  5 minute output rate 3899000 bits/sec, 539 packets/se

 Vlan99 is up, line protocol is up
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec

VLAN0099
  Number of topology changes 1 last change occurred 6d01h ago
          from GigabitEthernet1/0/22
  Times:  hold 1, topology change 35,

Seeing has gig1/0/22 is a access-port can you confirm if you have stp portfast enabled for this interface to negate unnecessary stp tcns on interface resets or flapping which could result in Pc4 re-converging stp

res
Paul

[wowhsieh]

Hi Paul,

Thanks for your analysis, I don't know what's vlan99 purpose(it's not my installation, I just help troubleshooting).The 3750's gateway is on vlan99 and default route to 172.22.99.254, and port 22 & 19 both up-link to ASA port 1 & 3 respectively, since there were lots of stp on port 22 and very possibly the reason why network down for 3 minutes, what do you think maybe should remove port 22 link and then change default route to vlan100 interface ip 172.22.100.254?!

[paul driver]

Hello

Port 18 is a trunk

GigabitEthernet1/0/18 is up, line protocol is up (connected)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 2000 bits/sec, 3 packets/sec

Port 22 isnt - but shows most traffic

GigabitEthernet1/0/22 is up, line protocol is up (connected)
  5 minute input rate 3032000 bits/sec, 430 packets/sec
  5 minute output rate 3899000 bits/sec, 539 packets/sec

Its not clear what the trunk port 18 is for, so it that case I d would try disconnecting port 18 first and apply stp portfast on port 22, and probably change the cabling and see if that helps?

res
Paul

[wowhsieh]

I think perhaps due to security reason, vlan 108-126 via port 19 using vlan tag goto ASA port 3 172.22.100.254 for routing and security management, and other vlan(1,2,4,5,13,16,17,61,99,100) which has vlan interface ip then via 3750's port 22 and default route goto ASA port 1 172.22.99.254. thus such structure cause looping?!

I will suggest user disconnecting port 18 first( port 18 is connecting to a WiFi authentication gateway lan port and then wan port back to 3750 port 17, it's rarely used so this seems the easiest way to do first) and see if help.