Cisco ASA logging issue

[engineer467]

Hello,

There is a strange issue, this cisco asa firewall is configured to send syslogs to an external server.

The firewall sends syslogs for few days and then suddenly there are no messages received on syslog server.

I have checked on the network level, everything is allowed and working.

Not sure what else to check now. Appreciate if someone can help.

Below is the configuration for logging-

logging enable
logging timestamp
logging standby
logging buffered informational
logging trap informational
logging asdm informational
logging device-id ipaddress outside
logging host outside xx.xx.xx.xx 17/10121
no logging message 313005
no logging message 607001
no logging message 302021
no logging message 302020

Output of show logging -

Syslog logging: enabled
Facility: 20
Timestamp logging: enabled
Hide Username logging: enabled
Standby logging: enabled
Debug-trace logging: disabled
Console logging: disabled
Monitor logging: disabled
Buffer logging: level informational, 242780794 messages logged
Trap logging: level informational, facility 20, 243580494 messages logged
Logging to outside xx.xx.xx.xx  udp/10121, UDP TX:12274
Global TCP syslog stats::
NOT_PUTABLE: 0, ALL_CHANNEL_DOWN: 0
CHANNEL_FLAP_CNT: 0, SYSLOG_PKT_LOSS: 0
PARTIAL_REWRITE_CNT: 0
Permit-hostdown logging: disabled
History logging: disabled
Device ID: 'outside' interface IP address "xx.xx.xx.xx"
Mail logging: disabled
ASDM logging: level informational, 242780794 messages logged

show logging queue

Logging Queue length limit : 512 msg(s)
0 msg(s) discarded due to queue overflow
0 msg(s) discarded due to memory allocation failure
Current 0 msg on queue, 251 msgs most on queue

One thing i have noticed, the UDP TX: count in the show logging output stays same for few minutes and is increased only by 1 or 2. 

Thanks.

[ngkin2010]

Hi,

Check if your ACE (acl entry) are configured with 'log' keyword.

e.g. access-list ACL-IN extended permit ip 192.168.1.0 255.255.255.0 host 8.8.8.8 log <---

Confirm if UDP 514 are sending to remote logging server by 'packet capture'

e.g.

capture TEMP buff 2048 interface <EGRESS-INTERFAC-TO-SYSLOG-SVR> match udp host <ASA's IP> host <SYSLOG-SVR> eq 514
show capture TEMP
show capture TEMP
<.. check any packet fired out from ASA ..>
show capture TEMP
<...>
no capture TEMP

[engineer467]

Hello ngkin,

Yes logging is enabled on ACEs.

I entered the below command, replaced 514 with 10121-

capture TEMP buff 2048 interface <EGRESS-INTERFAC-TO-SYSLOG-SVR> match udp host <ASA's IP> host <SYSLOG-SVR> eq 10121

Below is the output of packet capture-

8 packets captured

1: 11:22:44.980509 ASA's IP.514 > Syslog Server IP.10121: udp 199
2: 11:22:44.980524 ASA's IP.514 > Syslog Server IP..10121: udp 225
3: 11:22:46.909010 ASA's IP.514 > Syslog Server IP..10121: udp 200
4: 11:22:47.194127 ASA's IP.514 > Syslog Server IP..10121: udp 201
5: 11:22:47.269822 ASA's IP.514 > Syslog Server IP..10121: udp 202
6: 11:22:47.271729 ASA's IP.514 > Syslog Server IP..10121: udp 202
7: 11:22:47.363796 ASA's IP.514 > Syslog Server IP..10121: udp 159
8: 11:22:47.363811 ASA's IP.514 > Syslog Server IP..10121: udp 180
8 packets shown

[ngkin2010]

Clearly, your ASA are sending SYSLOG message.

You have to check if there is any routing issue in between ASA and SYSLOG server.

Or if the packets were dropped by firewall.

Also you are advised to confirm it's not an application layer issue. (e.g. check on SYSLOG server, confirm there is no incoming UDP:10121 message from the ASA) If it's a linux server, use "tcpdump -vvv -n 'host ASA-IP and port 10121' "

[engineer467]

Great help man, thank you.

There is another firewall in between our firewall and syslog server.

firewall 1(outside interface)-->core sw-->(inside interface)firewall 2(outside interface)-->syslog server

Hope the above explains the network setup.

There is a firewall rule in firewall 2 allowing syslog traffic towards syslog server.

I ran a capture on firewall 2 on its inside interface, and I can see the syslog packets coming from firewall 1.

Then I ran another capture on its outside interface, but this time there are no packets captured.

Since the firewall rule is already in place allowing syslog traffic, what else could be the reason?

[ngkin2010]

Hi,

Is it a Cisco ASA? You may try to run 'packet tracer input <INGRESS-INTERFACE> udp <ASA-IP> 514 <SYSLOG-IP> 10121' to check if any reason that dropped by this firewall.

You also could add "asp-drop" to view the dropped packet + dropped reason.

capture TEMP type asp-drop all

Event viewer of ASDM also provide enough detail as well.

There is a number of reason that would dropped by a firewall, hopefully the above commands could give you a proper reason to fix.

Usually, it may be:
1. No policy allowed the connection
2. dropped due to Unicast Reverse Path Forwarding (uRPF)

[engineer467]

Its allowed as shown in packet tracer.

[ngkin2010]

Try to confirm the packets are allowed with 'capture' command on egress interface.

Try to confirm the connection is allowed according to log message on ASDM.

Try to confirm the routing is correct (e.g. show route to check which egress interface used to reach SYSLOG server)

Try to confirm if the packets are received by SYSLOG server or not (e.g. tcpdump as mentioned previously).

[engineer467]

did a packet capture with asp drop-

capture temp type asp-drop all match udp host fw ip host sys ip eq 10121

got the below drop reason-

Drop-reason: (flow-expired) Expired flow

[ngkin2010 · Accepted solution]

Oh...It become more complicated to find out the root cause.

You may try to verify the existing flow by
'show conn protocol udp address <ASA-IP> port 514 address <SYSLOG-IP> port 10121 detail'

Then try to clear the existing flow on ASA:
'clear conn protocol udp address <ASA-IP> port 514 address <SYSLOG-IP> port 10121'

Afterward, do again with packet tracer
'packet tracer input <INGRESS-INTERFACE> udp <ASA-IP> 514 <SYSLOG-IP> 10121'

And show it again, you should see a new session if there is new packet sent from your ASA:
'show conn protocol udp address <ASA-IP> port 514 address <SYSLOG-IP> port 10121 detail'


I am not quite sure the root cause, but I hope resetting the existing connection could help to address the issue. Kindly let us know your result.

[engineer467 · Accepted solution]

hello ngkin,

i ran captures, there was zero udp 10121 traffic on the egress interface. so i cleared the connections, also increased the udp timeout value.

The logs are sent to the syslog, now I will monitor if for few days to confirm if it was the udp timeout issue.

Not sure but lets see. I will update you anyways.

Thanks a lot for your help.