ACL stop blocking packets

[netadmincsm]

Hi All

I have a master ACL called by around 6 or 7 Vlan Interface. 

About 15 or 20 minutes after applying the ACL, the traffic (that was blocked at the beginning) start to be permitted.

As soon as I start to modify the ACL the packets are block again.  Then, without changing anything the packet start to be permitted 15 or 20 minutes later.

Anyone have an idea ?

Thank you very much

[Jon Marshall]

switch model and what IOS version ?

config example ie. a vlan interface configuration and the access-list ?

details of where you are pinging from ?

Jon

[netadmincsm]

IOS version : c3750-ipservicesk9-mz.122-50.SE5

Switch model : WS-C3750G-24Config

Vlan Interface configuration :

interface Vlan10

ip address 10.1.184.17 255.255.255.248

ip access-group ACLMaster in

ip access-group ACLServices out

end

ip access-list extended ACLMaster

permit ip 10.1.128.0 0.0.66.127 10.1.129.0 0.0.66.255

permit ip 10.1.128.0 0.0.66.127 10.1.185.0 0.0.66.255

permit ip 10.1.184.0 0.0.66.127 10.1.129.0 0.0.66.255

permit ip 10.1.184.0 0.0.66.127 10.1.185.0 0.0.66.255

permit ip 10.1.128.0 0.0.66.63 10.1.128.0 0.0.66.63

permit ip 10.1.128.0 0.0.66.63 10.1.184.0 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.128.0 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.184.0 0.0.66.63

permit ip 10.1.128.64 0.0.66.63 10.1.128.64 0.0.66.63

permit ip 10.1.128.64 0.0.66.63 10.1.184.64 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.128.64 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.184.64 0.0.66.63

permit ip 10.1.128.0 0.0.66.63 10.1.128.128 0.0.66.63

permit ip 10.1.128.0 0.0.66.63 10.1.184.128 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.128.128 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.184.128 0.0.66.63

permit ip 10.1.128.64 0.0.66.63 10.1.128.192 0.0.66.63

permit ip 10.1.128.64 0.0.66.63 10.1.184.192 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.128.192 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.184.192 0.0.66.63

permit ip 10.1.128.0 0.0.66.127 10.1.135.192 0.0.64.63

permit ip 10.1.184.0 0.0.66.127 10.1.191.192 0.0.64.63

deny ip 10.1.128.0 0.0.66.127 10.1.128.0 0.0.71.255

deny ip 10.1.128.0 0.0.66.127 10.1.184.0 0.0.71.255

deny ip 10.1.184.0 0.0.66.127 10.1.128.0 0.0.71.255

deny ip 10.1.184.0 0.0.66.127 10.1.184.0 0.0.71.255

permit ip 10.1.128.0 0.0.66.127 10.0.0.0 0.255.255.255

permit ip 10.1.184.0 0.0.66.127 10.0.0.0 0.255.255.255

permit ip 10.1.128.0 0.0.66.127 172.16.0.0 0.15.255.255

permit ip 10.1.184.0 0.0.66.127 172.16.0.0 0.15.255.255

permit ip 10.1.128.0 0.0.66.127 192.168.0.0 0.0.255.255

permit ip 10.1.184.0 0.0.66.127 192.168.0.0 0.0.255.255

deny ip any any

!

I'm pinging google from 10.1.184.18.  It must be blocked with the deny ip any any at the end.

Thanks for your help !

[Jon Marshall]

Apologies for delay in replying.

Well your acl should certainly do the trick. So in effect you restricting this vlan to only talk internally to the rest of your network ?

Jon

[netadmincsm]

Right.

I permit a few subnet with each other then block the rest of our services subnet.

Then I permit any other privates addresses that we could have, then block Internet connectivity

Thanks

[Jon Marshall]

There is nothing wrong with your acl config. So it isn't obvious what's happening.

Just a few thoughts -

1) could be a bug. I'll have a check in bug tool to see if anything resembles this problem

2) is that the full L3 vlan interface config ie. you aren't running HSRP or have other config

3) there isn't another path that the clients could take is there ie. maybe an ICMP redirect or something going on. When the client can ping through did you do a traceroute to make sure it was using vlan 10 ip address as it's next-hop ?

Obviously if this is the only way out of vlan 10 ie. the L3 SVI then ignore point 3).

Edit - one other thing. Can you post the output of "sh ip access-list ACLMaster". It may be that the switch is interpreting a wildcard mask incorrectly and putting something else into your config.

Edit2 - sorry, one final thing. When you ping google what IP address does this resolve to.

Jon

[netadmincsm]

Hi Jon

No I'm not running HSRP on this interface

I look for a different path ping could have taken but there isn't any.

sh ip access ACLMaster

Extended IP access list ACLMaster

permit ip 10.1.128.0 0.0.66.127 10.1.129.0 0.0.66.255

permit ip 10.1.128.0 0.0.66.127 10.1.185.0 0.0.66.255

permit ip 10.1.184.0 0.0.66.127 10.1.129.0 0.0.66.255 (271 matches)

permit ip 10.1.184.0 0.0.66.127 10.1.185.0 0.0.66.255 (6 matches)

permit ip 10.1.128.0 0.0.66.63 10.1.128.0 0.0.66.63

permit ip 10.1.128.0 0.0.66.63 10.1.184.0 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.128.0 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.184.0 0.0.66.63 (393476 matches)

permit ip 10.1.128.64 0.0.66.63 10.1.128.64 0.0.66.63

permit ip 10.1.128.64 0.0.66.63 10.1.184.64 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.128.64 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.184.64 0.0.66.63

permit ip 10.1.128.0 0.0.66.63 10.1.128.128 0.0.66.63

permit ip 10.1.128.0 0.0.66.63 10.1.184.128 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.128.128 0.0.66.63

permit ip 10.1.184.0 0.0.66.63 10.1.184.128 0.0.66.63 (101 matches)

permit ip 10.1.128.64 0.0.66.63 10.1.128.192 0.0.66.63

permit ip 10.1.128.64 0.0.66.63 10.1.184.192 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.128.192 0.0.66.63

permit ip 10.1.184.64 0.0.66.63 10.1.184.192 0.0.66.63

permit ip 10.1.128.0 0.0.66.127 10.1.135.192 0.0.64.63

permit ip 10.1.184.0 0.0.66.127 10.1.191.192 0.0.64.63

deny ip 10.1.128.0 0.0.66.127 10.1.128.0 0.0.71.255

deny ip 10.1.128.0 0.0.66.127 10.1.184.0 0.0.71.255

deny ip 10.1.184.0 0.0.66.127 10.1.128.0 0.0.71.255 (3 matches)

deny ip 10.1.184.0 0.0.66.127 10.1.184.0 0.0.71.255

permit ip 10.1.128.0 0.0.66.127 10.0.0.0 0.255.255.255

permit ip 10.1.184.0 0.0.66.127 10.0.0.0 0.255.255.255 (12 matches)

permit ip 10.1.128.0 0.0.66.127 172.16.0.0 0.15.255.255

permit ip 10.1.184.0 0.0.66.127 172.16.0.0 0.15.255.255

permit ip 10.1.128.0 0.0.66.127 192.168.0.0 0.0.255.255

permit ip 10.1.184.0 0.0.66.127 192.168.0.0 0.0.255.255

deny ip any any (1194892 matches)

When I ping google it's resolve 74.125.91.147

Thank you veru much

[Jon Marshall]

Can you clear the counters off the acl ie. "clear access-list counters " and run the ping again. We need to see which acl line is allowing the traffic.

Jon

[netadmincsm]

I restarted the switch a couple of hours ago, and I haven't seen the issue yet.

But as soon as I have the issue again I'll clear the counter.

I also look at the CPU and mem usage.  Here's what I have.

    1111111          11111     1111111111                    1

    7700000888889999900000999995555511111777779999988888999997

100

90

80

70

60

50

40

30

20 **                         *****                         *

10 **********************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               CPU% per second (last 60 seconds)

    1121111111111111111111111111111111111111111111111111111111

    7595778259477478645466785785996677667877706574247887568768

100

90

80

70

60

50

40

30   *

20 ******* ** ** *** * ********************* ***   **********

10 ##########################################################

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

    2225424259

    7136924889

100          *

90          *

80          *

70          *

60    *    **

50    **   **

40    ** * **

30 *  ** ****

20 **********

10 ##########

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

------------------ History of Processor Mempool ------------------

    4444444444444444444444444444444444444444444444444444444444

    6666666666666666666666666666666666666666666666666666666666

100

90

80

70

60

50 **********************************************************

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               Free memory per second (last 60 seconds)

    4444444444444444444444444444444444444444444444444444444444

    6666666666666666666666666666666666666666666666666666666666

100

90

80

70

60

50 ##########################################################

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               Free memory per minute (last 60 minutes)

              * = maximum # = average

    44444444445

    66666666663

100

90

80

70

60

50 ###########

40 ###########

30 ###########

20 ###########

10 ###########

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   Free memory per hour (last 72 hours)

                  * = maximum # = average

------------------ History of I/O Mempool ------------------

    3333333333333333333333333333333333333333333333333333333333

    2222222222222222222222222222222222222222222222222222222222

100

90

80

70

60

50

40

30 **********************************************************

20 **********************************************************

10 **********************************************************

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               Free memory per second (last 60 seconds)

    3333333333333333333333333333333333333333333333333333333333

    2222222222222222222222222222222222222222222222222222222222

100

90

80

70

60

50

40

30 ##########################################################

20 ##########################################################

10 ##########################################################

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               Free memory per minute (last 60 minutes)

              * = maximum # = average

    33333333333

    22222222201

100

90

80

70

60

50

40

30 ###########

20 ###########

10 ###########

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   Free memory per hour (last 72 hours)

                  * = maximum # = average

------------------ History of Driver text Mempool ------------------

100

90

80

70

60

50

40

30

20

10

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               Free memory per second (last 60 seconds)

100

90

80

70

60

50

40

30

20

10

   0....5....1....1....2....2....3....3....4....4....5....5....

             0    5    0    5    0    5    0    5    0    5

               Free memory per minute (last 60 minutes)

              * = maximum # = average

100

90

80

70

60

50

40

30

20

10

   0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

             0    5    0    5    0    5    0    5    0    5    0    5    0

                   Free memory per hour (last 72 hours)

                  * = maximum # = average

                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)

Processor    3F8F174    72006968    38386828    33620140    31850700    16768896

      I/O    6400000    12582912     8532852     4050060     3558332     4047808

Driver te    2C00000     1048576          44     1048532     1048532     1048532

Do you think I'm running ot of mem or CPU proccess ?

[grinch182]

hello there,

Could some body please explain me how such wildcard is possible?

permit ip 10.1.128.0 0.0.66.63 10.1.184.0 0.0.66.63

[Jon Marshall]

Wildcard masks when used in acls do not to be contiguous in the same way subnet masks do so as long as you can derive the value from the binary it will be accepted ie. an octet in binary

128 64 32 16 8 4 2 1

  0   1   0   0  0 0 1 0  = 66

Edit - the above doesn't quite line up but you get the idea.

Jon

[grinch182]

Thanks a lot. I didn't know that you can use wildcard this way. Maybe you could provide me some useful link to read about it?

GRinch

[Jon Marshall]

Note that it only applies to acls and not the wildcard masks using in EIGRP/OSPF router config.

Don't really have any links as such but i did a post a while back about non-contiguous wildcard masks comparing 3.255.255.255. with 252.255.255.255 and what each of them matched ie.

0.0.0.0 3.255.255.255. would match 0.0.0.0 /  1.0.0.0 / 2.0.0.0 and 3.0.0.0

0.0.0.0 252.255.255.255 would match all networks divisible by 4

I've attached a file containing the full explanation and it helps if you actually test on a router so you can see it in action.

Please come back with any other questions if needed.

Jon

[grinch182]

Thanks a lot! What kind of extension does this file have? =)

[Jon Marshall]

Should be a normal text file - can you not open it ? If not i can post the contents here.

Jon