10-14-2018 11:10 PM - edited 03-12-2019 07:01 AM
Hi all
I am observing my firewall 5506-x since a month that after 2 or 3 days, firewall suddenly stop resolving DNS queries and all internet traffic stop. while in this situation IP communication remain OK which means i can ping/browse across the firewall but if i will reboot firewall then it will start working normally.
any idea what is happening with firewall ?
regards
Solved! Go to Solution.
10-15-2018 11:54 PM
Hi,
Normally wouldn't configuration issues cause working functions to stop working as they wouldn't work from the beginning. But I see that you have a lot of functions enabled like netflow, route maps, threat inspection etc enabled. Normally you are more exposed to hitting bugs with the more functions you enable. You are also running a code that no longer is downloadable.
My suggestion would be to upgrade the firewall to 9.6.4 or 9.8.2 (both last interim release) that have fixes for various bugs like memory leaks (which could explain your issues) alon with monitoring the logs on your syslog server
Best regards,
Martin
10-15-2018 12:38 AM
Hi
What code do you use? ASA, ASA + Firepower or FTD? What version?
10-15-2018 01:57 AM - edited 10-15-2018 01:59 AM
Hi Martin ...
Thanks for reply..
this is ASA 5506-x FirePOWER
# sh ver
Cisco Adaptive Security Appliance Software Version 9.6(2)23
Device Manager Version 7.6(1)
Compiled on Thu 28-Sep-17 07:50 PDT by builders
System image file is "disk0:/asa962-23-lfbff-k8.SPA"
Config file at boot was "startup-config"
ProjectFW up 3 days 23 hours
Hardware: ASA5506, 4096 MB RAM, CPU Atom C2000 series 1250 MHz, 1 CPU (4 cores)
Internal ATA Compact Flash, 7168MB
BIOS Flash N25P64 @ 0xfed01000, 16384KB
Encryption hardware device : Cisco ASA Crypto on-board accelerator (revision 0x1)
Number of accelerators: 1
1: Ext: GigabitEthernet1/1 : address is 2c5a.0f79.d225, irq 255
2: Ext: GigabitEthernet1/2 : address is 2c5a.0f79.d226, irq 255
3: Ext: GigabitEthernet1/3 : address is 2c5a.0f79.d227, irq 255
4: Ext: GigabitEthernet1/4 : address is 2c5a.0f79.d228, irq 255
5: Ext: GigabitEthernet1/5 : address is 2c5a.0f79.d229, irq 255
6: Ext: GigabitEthernet1/6 : address is 2c5a.0f79.d22a, irq 255
7: Ext: GigabitEthernet1/7 : address is 2c5a.0f79.d22b, irq 255
8: Ext: GigabitEthernet1/8 : address is 2c5a.0f79.d22c, irq 255
10-15-2018 02:04 AM
Hi
Have configured Firepower or are you just using ASA? Do you get any logs when the issue occurs?
10-15-2018 02:14 AM
Hi Martin........
I am just using Firewall services not FirePOWER....unfortunately i didn't copy logs. (Just configured syslog today.. ) right now firewall is working fine. One thing i want to share is....I think due to some strange activity ASA skips all ACL rules for DNS query and all dns queries fall in global deny list. because that specific time i checked packet trace which was denied by global acl list. but i am not able to find the reason why it happening like this. need your expert judgement here.
10-15-2018 02:41 AM
Can you mask you config and paste it here?
10-15-2018 04:26 AM
Ok Martin...here is config file info
!!
!!
!!
: Saved
:
: Serial Number: XXXXXXXXXXX
: Hardware: ASA5506, 4096 MB RAM, CPU Atom C2000 series 1250 MHz, 1 CPU (4 cores)
:
ASA Version 9.6(2)23
!
hostname ProjectFW
domain-name abc.com
enable password 7sI3Z.cdfer2iY encrypted
passwd 7sI3Z.xcvfgt2iY encrypted
names
!
interface GigabitEthernet1/1
nameif outside
security-level 0
ip address x.x.x.x 255.255.255.0
!
interface GigabitEthernet1/2
nameif inside
security-level 100
ip address 192.168.81.7 255.255.255.0
dhcprelay server 192.168.81.25
!
interface GigabitEthernet1/3
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/4
nameif Winside
security-level 100
ip address 192.168.83.7 255.255.255.0
!
interface GigabitEthernet1/5
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/6
shutdown
nameif xxx_Outside
security-level 0
ip address x.x.x.x 255.255.255.252
!
interface GigabitEthernet1/7
shutdown
no nameif
no security-level
no ip address
!
interface GigabitEthernet1/8
shutdown
no nameif
no security-level
no ip address
!
interface Management1/1
management-only
no nameif
no security-level
no ip address
!
boot system disk0:/asa962-23-lfbff-k8.SPA
ftp mode passive
dns domain-lookup inside
dns server-group DefaultDNS
name-server 192.168.81.25 inside
domain-name abc.com
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network Printer_192.168.81.31_Plotter
host 192.168.81.31
description Plotter
object network Printer_192.168.81.36_xxx
host 192.168.81.36
description xxxx
object network Printer_192.168.81.47_xxx
host 192.168.81.47
description xxx Printer
object network Printer_192.168.81.41_xxx
host 192.168.81.41
description xxxx
object network Printer_192.168.81.45_xxxx
host 192.168.81.45
description xxxx
object network Printer_192.168.81.48_xxxx
host 192.168.81.48
description xxxx
object network Printer_192.168.81.42_xxxx
host 192.168.81.42
description xxxx
object network Printer_192.168.81.43_xxxx
host 192.168.81.43
description xxxx
object-group service Syslog udp
port-object eq syslog
access-list inside_access_in extended permit udp object-group DM_INLINE_NETWORK_2 any object-group DM_INLINE_UDP_1 log disable
access-list inside_access_in extended permit object-group DM_INLINE_SERVICE_1 192.168.81.0 255.255.255.0 object-group ApprovedDNSServer
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCP_16 log disable
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 object sftp.norc.org eq ssh
access-list inside_access_in extended permit object-group TCP-UDP 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCPUDP_1
access-list inside_access_in extended permit ip object Server_xxx any
access-list inside_access_in extended permit ip object-group NoRestrictionSource_aaa any log disable
access-list inside_access_in extended permit ip object-group NoRestrictionSource_IT_Team any log disable
access-list inside_access_in extended permit ip object-group NoRestrictionSource_aaaa any log disable
access-list inside_access_in extended permit ip object-group Server_Live any
access-list inside_access_in extended deny ip object-group Restricted_IPs any log disable
access-list inside_access_in extended permit ip object-group NoRestrictionSources any
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCP_14
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCP_8
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group ApplePushNotificationService
access-list inside_access_in extended permit tcp any object-group Printers object-group Printer_TCP
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCP_15
access-list inside_access_in extended permit ip 192.168.81.0 255.255.255.0 192.168.83.0 255.255.255.0 inactive
access-list inside_access_in extended permit icmp 192.168.81.0 255.255.255.0 any
access-list inside_access_in extended permit udp 192.168.81.0 255.255.255.0 any eq ntp
access-list inside_access_in extended permit tcp object Server_MigrationManager_81.199 any object-group DM_INLINE_TCP_18 inactive
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCP_9
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCP_5
access-list inside_access_in extended permit udp any any object-group WhatsApp_UDP
access-list inside_access_in extended permit object-group TCP-UDP any any object-group XMPP
access-list inside_access_in extended permit tcp 192.168.81.0 255.255.255.0 any object-group DM_INLINE_TCP_7 inactive
access-list inside_access_in extended permit ip 192.168.81.0 255.255.255.0 object-group GoodServers
access-list inside_access_in extended deny ip any object-group Blocked_Addresses
access-list inside_access_in extended deny object-group TCP-UDP any any object-group Torrent
access-list inside_access_in extended deny ip object-group Blocked_Sources any
access-list inside_access_in extended deny object-group TCP-UDP 192.168.81.0 255.255.255.0 any object-group HotspotShield
access-list inside_access_in extended deny ip any any
access-list Winside_access_in extended permit udp 192.168.83.0 255.255.255.0 any object-group DM_INLINE_UDP_2
access-list Winside_access_in extended permit object-group DM_INLINE_SERVICE_6 192.168.83.0 255.255.255.0 object Server_xxx
access-list Winside_access_in extended permit udp 192.168.83.0 255.255.255.0 object-group Printers object-group Printer_HP_Ports
access-list Winside_access_in extended permit tcp 192.168.83.0 255.255.255.0 192.168.81.0 255.255.255.0 object-group DM_INLINE_TCP_13
access-list Winside_access_in extended permit tcp object-group xxx object-group DM_INLINE_NETWORK_4 object-group xxx
access-list Winside_access_in extended permit tcp 192.168.83.0 255.255.255.0 object-group ApprovedDNSServer eq domain
access-list Winside_access_in extended permit ip object-group NoRestrictionSources any
access-list Winside_access_in extended permit tcp 192.168.83.0 255.255.255.0 object-group Printers object-group Printer_TCP
access-list Winside_access_in extended permit icmp any any
access-list Winside_access_in extended permit object-group DM_INLINE_SERVICE_5 any object Server_xxx inactive
access-list Winside_access_in extended permit object-group DM_INLINE_SERVICE_4 192.168.83.0 255.255.255.0 any inactive
access-list outside_access_in extended permit object-group ICMP any any
access-list outside_access_in extended permit tcp any interface outside object-group DM_INLINE_TCP_1
access-list outside_access_in extended permit tcp any object aaa_x.x.x.x_ object-group DM_INLINE_TCP_2
access-list outside_access_in extended permit tcp any object aab_x.x.x.x object-group DM_INLINE_TCP_3
access-list outside_access_in extended permit tcp any object aac_x.x.x.x object-group DM_INLINE_TCP_4
access-list inboundSurvey extended permit icmp any any object-group DM_INLINE_ICMP_1
access-list inboundSurvey extended permit tcp any object xxx_Interface_Outside object-group DM_INLINE_TCP_12
access-list OUTSIDE-IN extended permit icmp any any
access-list outside_access_Out extended permit tcp any4 object xxx object-group DM_INLINE_TCP_10
access-list outside_access_Out extended permit tcp any4 object Server_MigrationManager_81.199 object-group DM_INLINE_TCP_0
access-list outside_access_Out extended permit ip object-group xxx object xxx_81.29
access-list outside_access_Out extended permit object-group DM_INLINE_SERVICE_7 any4 object xxx
access-list outside_access_Out extended permit object-group DM_INLINE_SERVICE_0 any4 object xxx
access-list outside_access_Out extended permit icmp object-group DM_INLINE_NETWORK_3 any inactive
access-list outside_access_Out extended permit icmp any any
access-list outside_access_Out extended deny ip any any
access-list x_Outsite_access_in extended permit icmp any any
access-list xxx-THROTTLE extended permit ip object Server_xxx any
access-list xxxx-THROTTLE extended permit ip any object Server_xxx inactive
access-list xxx extended permit tcp 192.168.83.0 255.255.255.0 any object-group DM_INLINE_TCP_17
pager lines 24
logging enable
logging timestamp
logging trap critical
logging asdm informational
logging host inside 192.168.81.x
no logging message 106015
no logging message 313001
no logging message 313008
no logging message 106023
no logging message 710003
no logging message 106100
no logging message 302015
no logging message 302014
no logging message 302013
no logging message 302018
no logging message 302017
no logging message 302016
no logging message 302021
no logging message 302020
flow-export destination inside 192.168.81.17 9996
flow-export delay flow-create 15
mtu outside 1500
mtu inside 1500
mtu Winside 1500
mtu NTL_Outside 1500
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
arp rate-limit 16384
nat (inside,outside) source dynamic any interface dns
nat (Winside,inside) source dynamic any interface dns
!
object network aaa
nat (inside,outside) static interface service tcp 3389 3389
object network aab
nat (inside,outside) static xxx
object network aac
nat (inside,outside) static xxx
object network aad
nat (inside,outside) static xxx
object network aae
nat (inside,outside) static xxx
access-group outside_access_Out in interface outside
access-group inside_access_in in interface inside
access-group Winside_access_in in interface Winside
access-group x Outsite_access_in in interface xxx Outside
!
route-map xxx permit 10
match ip address xxx
set ip next-hop x.x.x.x
!
route-map xxx permit 20
!
route outside 0.0.0.0 0.0.0.0 x.x.x.x 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
timeout conn-holddown 0:00:15
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL
http server enable
http 192.168.1.0 255.255.255.0 inside
http 192.168.81.0 255.255.255.0 inside
snmp-server host inside 192.168.81.x community *****
no snmp-server location
no snmp-server contact
snmp-server community *****
service sw-reset-button
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet 192.168.81.0 255.255.255.0 inside
telnet timeout 30
ssh stricthostkeycheck
ssh timeout 5
ssh key-exchange group dh-group1-sha1
console timeout 0
dhcpd domain abc.com
!
dhcpd address 192.168.83.30-192.168.83.245 Winside
dhcpd dns 192.168.81.25 interface Winside
dhcpd option 3 ip 192.168.83.253 interface Winside
dhcpd option 6 ip 192.168.81.25 interface Winside
!
dhcprelay server 192.168.81.25 inside
dhcprelay enable Winside
dhcprelay timeout 160
threat-detection basic-threat
threat-detection scanning-threat shun except object-group NoRestrictionSource_IT_Team
threat-detection scanning-threat shun except object-group NoRestrictionSource_xx
threat-detection scanning-threat shun except object-group NoRestrictionSource_xx
threat-detection scanning-threat shun except object-group NoRestrictionSources
threat-detection scanning-threat shun except object-group NoShunnGroup
threat-detection scanning-threat shun duration 1800
threat-detection statistics host number-of-rate 3
threat-detection statistics port number-of-rate 3
threat-detection statistics protocol number-of-rate 3
threat-detection statistics access-list
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
dynamic-access-policy-record DfltAccessPolicy
username xxx password 2dMuEBodaRTg/ojQ encrypted privilege 15
username xxx password rFMCRvdj4RRRNLzF encrypted privilege 15
username xxx password cmyrcWm5arRxckSs encrypted privilege 15
!
class-map global-class-NetFlow
match any
class-map CM-xxx-THROTTLE
match access-list xxx-THROTTLE
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum client auto
message-length maximum 512
no tcp-inspection
policy-map PM-xxx-THROTTLE
class CM-xxx-THROTTLE
police input 4000000 4000
police output 4000000 4000
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect esmtp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect sip
inspect netbios
inspect tftp
inspect ip-options
class global-class-NetFlow
flow-export event-type all destination 192.168.81.17
class class-default
user-statistics accounting
policy-map global-policy
class inspection_default
inspect icmp
inspect icmp error
!
service-policy global_policy global
service-policy PM-xxx-THROTTLE interface inside
prompt hostname context
no call-home reporting anonymous
hpm topN enable
Cryptochecksum:0d74e8c37dxxxxxxxe6d6166105
: end
no asdm history enable
10-15-2018 11:54 PM
Hi,
Normally wouldn't configuration issues cause working functions to stop working as they wouldn't work from the beginning. But I see that you have a lot of functions enabled like netflow, route maps, threat inspection etc enabled. Normally you are more exposed to hitting bugs with the more functions you enable. You are also running a code that no longer is downloadable.
My suggestion would be to upgrade the firewall to 9.6.4 or 9.8.2 (both last interim release) that have fixes for various bugs like memory leaks (which could explain your issues) alon with monitoring the logs on your syslog server
Best regards,
Martin
10-16-2018 04:18 AM
Hi Martin,
Thanks for your reply....
Actually you are right.... this was Memory leak bug. i found the solution. this is Cisco Bug: CSCvd71473. it relates to DNS memory leak "slow memory leak when using many DNS queries". This issue is seen whenever a DNS query gets resolved on ASA. We could see a small amount of memory leak (around 64 Bytes with each DNS query getting resolved) on ASA.
for detailed information please read this Bug info:
https://quickview.cloudapps.cisco.com/quickview/bug/CSCvd71473
Now i must look for latest codes as you said.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide