Extreme Packet Loss after network Maintenance

[Mitchell Tuckness]

Hi all,

I am having an issue that I cannot seem to figure out. Yesterday I had to replace a battery in one of my UPS units so I had to shut a bunch of stuff down. Before I did that I went into each unit and did a wr mem and a copy running to startup to make sure that I had the configs I thought I was running. Before this, I had had no packet loss and the connection was fine. After doing the battery switch, I plugged everything back in and turned them on and was back up and running, except I kept getting dropped from places and web page was slow loading and that is when I noticed the packet loss.

It is very regular and very annoying. I think that at some point I was in testing stuff and messing with the config and made a change but never wrote it and then forgot.

I was hoping you all could look over the configs for the ASA and the 2811 to see if anything looks off. Also, and I don't know if this is relevant, but I can no longer get replies back from traceroutes. I recall I used to have no issue with this, but now, everything is blocked in between the ASA and the destination even though I have it set to inspect ICMP packets.

I do not get packet loss on the inside interface or anywhere within the inside.

I know just enough to be dangerous!

Here is the ASA config:

ASA5510# sh run
: Saved
:
ASA Version 9.1(4)
!
hostname ASA5510
domain-name m.int
enable password encrypted
xlate per-session deny tcp any4 any4
xlate per-session deny tcp any4 any6
xlate per-session deny tcp any6 any4
xlate per-session deny tcp any6 any6
xlate per-session deny udp any4 any4 eq domain
xlate per-session deny udp any4 any6 eq domain
xlate per-session deny udp any6 any4 eq domain
xlate per-session deny udp any6 any6 eq domain
passwd  encrypted
names
dns-guard
!
interface Ethernet0/0
 description LAN Interface
 nameif Inside
 security-level 100
 ip address 10.10.1.1 255.255.255.252
!
interface Ethernet0/1
 description WAN Interface
 nameif Outside
 security-level 0
 ip address 68.233.x.x 255.255.255.128
!
interface Ethernet0/2
 description DMZ
 nameif DMZ
 security-level 100
 ip address 10.10.0.1 255.255.255.252
!
interface Ethernet0/3
 description VOIP
 nameif VOIP
 security-level 100
 ip address 10.10.2.1 255.255.255.252
!
interface Management0/0
 management-only
 shutdown
 nameif management
 security-level 0
 no ip address
!
boot system disk0:/asa914-k8.bin
ftp mode passive
dns domain-lookup Inside
dns domain-lookup Outside
dns server-group DefaultDNS
 name-server 8.8.8.8
 name-server 8.8.4.4
 name-server 68.233.xx.x
 name-server 68.233.xx.x
 domain-name m.int
same-security-traffic permit inter-interface
object network ROUTER-2811
 host 10.10.1.2
object network ROUTER-2821
 host 10.10.0.2
object network WEBCAM-01
 host 192.168.1.5
object network DNS-SERVER
 host 192.168.1.2
object network ROUTER-3745
 host 10.10.2.2
object network RDP-DC1
 host 192.168.1.2
object network BLUE
 host 192.168.1.6
 description Blue Iris Server
object network M_LAP_LEA
 host 192.168.1.20
 description Laptop from LEA
object-group network PAT-SOURCE
 network-object 10.10.1.0 255.255.255.252
 network-object 10.10.0.0 255.255.255.252
 network-object 10.10.2.0 255.255.255.252
 network-object 192.168.0.0 255.255.255.0
 network-object 172.16.10.0 255.255.255.0
 network-object 172.16.20.0 255.255.255.0
 network-object 128.162.1.0 255.255.255.0
 network-object 128.162.10.0 255.255.255.0
 network-object 128.162.20.0 255.255.255.0
 network-object 192.168.1.0 255.255.255.0
 network-object 192.168.10.0 255.255.255.0
 network-object 192.168.20.0 255.255.255.0
 network-object 172.16.1.0 255.255.255.0
 network-object 162.128.1.0 255.255.255.0
 network-object 162.128.10.0 255.255.255.0
 network-object 162.128.20.0 255.255.255.0
 network-object 142.16.1.0 255.255.255.0
 network-object 142.16.10.0 255.255.255.0
 network-object 142.16.20.0 255.255.255.0
object-group network DM_INLINE_NETWORK_2
 network-object host 98.22.xxx
object-group network Outside_access_in
object-group protocol DM_INLINE_PROTOCOL_1
 protocol-object gre
access-list USERS standard permit 10.10.1.0 255.255.255.0
access-list Outside_access_in extended permit tcp host 98.22.xxx object ROUTER-2811 eq ssh
access-list Outside_access_in extended permit tcp host 98.22.xxx object ROUTER-2821 eq ssh
access-list Outside_access_in extended permit tcp host 98.22.xxx interface Outside eq https
access-list Outside_access_in extended permit tcp host 98.22.xxx object WEBCAM-01 eq www inactive
access-list Outside_access_in extended permit tcp host 98.22.xxx object RDP-DC1 eq xxxx
access-list Outside_access_in extended permit tcp host 98.22.xxx object BLUE eq xxxx
access-list Outside_access_in extended permit tcp host 98.22.xxx object ROUTER-3745 eq ssh
access-list Outside_access_in extended permit tcp any object BLUE eq xxxx
access-list dmz-access-vlan1 extended permit ip 128.162.1.0 255.255.255.0 any
access-list dmz-access remark Permit all traffic to DC1
access-list dmz-access extended permit ip 128.162.1.0 255.255.255.0 host 192.168.1.2
access-list dmz-access remark Permit only DNS traffic to DNS server
access-list dmz-access extended permit udp 128.162.1.0 255.255.255.0 host 192.168.1.2 eq domain
access-list dmz-access remark Permit ICMP to all devices in DC
access-list dmz-access extended permit icmp 128.162.1.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list dmz-access remark Permit all traffic to DC1
access-list dmz-access remark Permit only DNS traffic to DNS server
access-list dmz-access remark Permit ICMP to all devices in DC
pager lines 24
logging enable
logging asdm informational
mtu Inside 1500
mtu Outside 1500
mtu DMZ 1500
mtu VOIP 1500
mtu management 1500
icmp unreachable rate-limit 1 burst-size 1
icmp deny any Outside
asdm image disk0:/asdm-715.bin
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
!
object network ROUTER-2811
 nat (Inside,Outside) static interface service tcp ssh x
object network ROUTER-2821
 nat (DMZ,Outside) static interface service tcp ssh x
object network WEBCAM-01
 nat (Inside,Outside) static interface service tcp www x
object network ROUTER-3745
 nat (VOIP,Outside) static interface service tcp ssh x
object network RDP-DC1
 nat (Inside,Outside) static interface service tcp xxxx xxxx
object network BLUE
 nat (Inside,Outside) static interface service tcp xxxx xxxx
!
nat (any,Outside) after-auto source dynamic any interface
access-group Outside_access_in in interface Outside
route Outside 0.0.0.0 0.0.0.0 68.233.151.1 1
route DMZ 128.162.1.0 255.255.255.0 10.10.0.2 1
route DMZ 128.162.10.0 255.255.255.0 10.10.0.2 1
route DMZ 128.162.20.0 255.255.255.0 10.10.0.2 1
route VOIP 142.16.1.0 255.255.255.0 10.10.2.2 1
route VOIP 142.16.10.0 255.255.255.0 10.10.2.2 1
route VOIP 142.16.20.0 255.255.255.0 10.10.2.2 1
route Inside 172.16.10.0 255.255.255.0 10.10.1.2 1
route Inside 172.16.20.0 255.255.255.0 10.10.1.2 1
route Inside 192.168.1.0 255.255.255.0 10.10.1.2 1
route Inside 192.168.10.0 255.255.255.0 10.10.1.2 1
route Inside 192.168.20.0 255.255.255.0 10.10.1.2 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
aaa-server PNL-RADIUS protocol radius
aaa-server PNL-RADIUS (Inside) host 192.168.1.2
 key *****
 radius-common-pw *****
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL
http server enable
http 0.0.0.0 0.0.0.0 Inside
http 98.22.xxx 255.255.255.255 Outside
snmp-server host Inside 192.168.1.2 community ***** version 2c udp-port 161
snmp-server location x
snmp-server contact x
snmp-server community *****
snmp-server enable traps snmp authentication linkup linkdown coldstart warmstart
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 Inside
ssh 98.22.xxx 255.255.255.255 Outside
ssh timeout 60
ssh version 2
ssh key-exchange group dh-group1-sha1
console timeout 0
threat-detection basic-threat
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
ntp server 24.56.178.140 source Outside prefer
username xxxx password x encrypted privilege 15
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns migrated_dns_map_1
 parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns migrated_dns_map_1
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect esmtp
  inspect sqlnet
  inspect skinny
  inspect sunrpc
  inspect xdmcp
  inspect sip
  inspect netbios
  inspect tftp
  inspect ip-options
  inspect icmp
  inspect icmp error
  inspect pptp
 class class-default
  user-statistics accounting
!
service-policy global_policy global
prompt hostname context
no call-home reporting anonymous
call-home
 profile CiscoTAC-1
  no active
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination address email callhome@cisco.com
  destination transport-method http
  subscribe-to-alert-group diagnostic
  subscribe-to-alert-group environment
  subscribe-to-alert-group inventory periodic monthly
  subscribe-to-alert-group configuration periodic monthly
  subscribe-to-alert-group telemetry periodic daily
password encryption aes
hpm topN enable
Cryptochecksum:949189d67866f6c09450769d41649992
: end

Here is the 2811:

C2811#sh run
Building configuration...

Current configuration : 3925 bytes
!
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname C2811
!
boot-start-marker
boot system flash
boot-end-marker
!
!
enable secret 4 x
!
aaa new-model
!
!
!
!
!
!
!
aaa session-id common
!
!
dot11 syslog
no ip source-route
!
!
ip cef
no ip dhcp use vrf connected
!
!
!
ip domain name m.int
ip name-server 192.168.1.2
ip name-server 8.8.8.8
ip name-server 68.233.xxx.x
ip name-server 68.233.xxx.x
no vlan accounting input
!
multilink bundle-name authenticated
!
!
password encryption aes
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-1290569776
 enrollment selfsigned
 subject-name cn=IOS-Self-Signed-Certificate-1290569776
 revocation-check none
 rsakeypair TP-self-signed-1290569776
!
!
crypto pki certificate chain TP-self-signed-1290569776
 certificate self-signed 01
  3082022B 30820194 A0030201 02020101 300D0609 2A864886 F70D0101 05050030
  31312F30 2D060355 04031326 494F532D 53656C66 2D536967 6E65642D 43657274
  69666963 6174652D 31323930 35363937 3736301E 170D3134 30313035 30363130
  33395A17 0D323030 31303130 30303030 305A3031 312F302D 06035504 03132649
  4F532D53 656C662D 5369676E 65642D43 65727469 66696361 74652D31 32393035
  36393737 3630819F 300D0609 2A864886 F70D0101 01050003 818D0030 81890281
  8100B18F F63C5121 00785DE0 854601BA EE77DAA3 21286D8C 6E700C37 237CC1BE
  611023AF FBE04BBE 7B4B3233 E4E129DD A74604E5 62AA39BF 77F98D5D D63944E9
  2345AE37 D93C5753 E425E85A EB22C2C9 CFC5D1A0 F800449B 0419A5C8 A0A101EC
  02928172 7B30A609 71ADA3D4 68F4F484 AF2B3249 0E225DB2 C72C136A E670D761
  DDE30203 010001A3 53305130 0F060355 1D130101 FF040530 030101FF 301F0603
  551D2304 18301680 1461F6DE 8EF50F7B 0E46359F 421EA106 9375F65F 30301D06
  03551D0E 04160414 61F6DE8E F50F7B0E 46359F42 1EA10693 75F65F30 300D0609
  2A864886 F70D0101 05050003 81810049 BA55F695 8525265F ED2D77EE 8706BF10
  63A7E644 202F6663 9EA5551F 47F7FC50 D4021EDD E3DC5A80 39FD161A C337D20D
  71B98875 0F1FE887 649E81D3 F93F7A1B A1E18B99 A77B1A59 84DB4711 867913FD
  044084FB 651ECA6E C6EDF35C E43A2946 8C01781E 26DB9484 C8740A82 4A7CA266
  A0655526 CBCB4982 F30D68E9 D70753
        quit
!
!
license udi pid CISCO2811 sn FTX1041A07T
username x secret 5 x
username xxxx privilege 15 secret 4 x
!
redundancy
!
!
ip ssh time-out 60
ip ssh authentication-retries 5
ip ssh version 2
!
!
!
!
!
!
!
interface FastEthernet0/0
 description CONNECTION TO INSIDE INT. OF ASA
 ip address 10.10.1.2 255.255.255.252
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface FastEthernet0/1
 no ip address
 ip virtual-reassembly in
 duplex auto
 speed auto
!
interface FastEthernet0/1.1
 description VLAN 10
 encapsulation dot1Q 10
 ip address 192.168.10.1 255.255.255.0
 ip helper-address 192.168.1.2
 ip virtual-reassembly in
!
interface FastEthernet0/1.2
 description VLAN 20
 encapsulation dot1Q 20
 ip address 192.168.20.1 255.255.255.0
 ip helper-address 192.168.1.2
 ip virtual-reassembly in
!
interface FastEthernet0/1.3
 description Trunk Interface VLAN 1
 encapsulation dot1Q 1 native
 ip address 192.168.1.1 255.255.255.0
 ip helper-address 192.168.1.2
 ip virtual-reassembly in
!
interface Dialer0
 no ip address
!
ip default-gateway 10.10.1.1
ip forward-protocol nd
no ip http server
ip http authentication local
ip http secure-server
!
!
ip dns server
ip route 0.0.0.0 0.0.0.0 10.10.1.1
ip ospf name-lookup
!
access-list 1 permit any
dialer-list 1 protocol ip permit
!
!
!
snmp-server community Maladomini-RW RW
!
tftp-server system:running-config 1
!
!
!
control-plane
!
!
!
line con 0
 exec-timeout 0 0
 password 7 x
line aux 0
line vty 0 4
 exec-timeout 0 0
 password 7 0527031B2C49470758
 transport input ssh
!
scheduler allocate 20000 1000
end

[Vibhor Amrodia]

Hi,

As per your description of the issue. This all started after the ASA device reloaded.

I don't see any misconfiguration on the ASA device. What traffic are you testing with. You have not observed any traffic issues on the inside interface correct ?

Is the traffic being dropped when going through the ASA device ? If yes , have you applied some captures on the ASA device interface and see if you see the traffic traversing through the ASA device.

I think this would be the best approach.

Use a specific Source and type of test traffic. Capture it on the ASA device and see if you get any missing packets.

Other than that check the ASA device counters:-

show asp drop

show service-policy

show interface

Thanks and Regards,

Vibhor Amrodia