Re: CLI grep command in AsyncOS 4.5

mark_ironport · ‎10-19-2005

A late addition to AsyncOS 4.5 was the CLI "grep" command. Yes, it is a "hidden" command and we would like input from IronPort Nation on whether this is useful and what additions they would like to see to it. Here are a brief set of examples on its use:

boxster.mfg> help grep

grep -e regexp [-e regexp] [-i] [-p] [-t] log_name

Run the Unix grep command over the named log file.
Normally the grep will be run over all files in the
named log subscription. Using the -t option will run
the grep over a tail of the log file. Using the -i
option will ignore case sensitivities. Using the -p
option will paginate the output.

regexp - an extended grep expression
log_name - the name of the log file

Manually using grep will look like this:

boxster.mfg> grep

Currently configured logs:
1. "antivirus" Type: "Anti-Virus Logs" Retrieval: FTP Poll
2. "asarchive" Type: "Anti-Spam Archive" Retrieval: FTP Poll
3. "avarchive" Type: "Anti-Virus Archive" Retrieval: FTP Poll
4. "bounces" Type: "Bounce Logs" Retrieval: FTP Poll
5. "brightmail" Type: "Symantec Brightmail Anti-Spam Logs" Retrieval: FTP Poll
6. "case" Type: "Context Adaptive Scanning Engine" Retrieval: FTP Poll
7. "cli_logs" Type: "CLI Audit Logs" Retrieval: FTP Poll
8. "error_logs" Type: "IronPort Text Mail Logs" Retrieval: FTP Poll
9. "ftpd_logs" Type: "FTP Server Logs" Retrieval: FTP Poll
10. "gui_logs" Type: "HTTP Logs" Retrieval: FTP Poll
11. "mail_logs" Type: "IronPort Text Mail Logs" Retrieval: FTP Poll
12. "scanning" Type: "Scanning Logs" Retrieval: FTP Poll
13. "sntpd_logs" Type: "NTP logs" Retrieval: FTP Poll
14. "status" Type: "Status Logs" Retrieval: FTP Poll
15. "system_logs" Type: "System Logs" Retrieval: FTP Poll
Enter the number of the log you wish to grep.
[]> 11

Enter the regular expression to grep.
[]> MID

Do you want this search to be case insensitive? [Y]> n

Do you want to tail the logs? [N]>

Do you want to paginate the output? [N]>

For a single regular expression:

boxster.mfg> grep -e MID mail_logs

For multiple regular expressions:

boxster.mfg> grep -e MID -e RID mail_logs

For multiple regular expressions in the current log:

boxster.mfg> grep -e MID -e RID mail_logs -t

For multiple regular expressions in the current log with a case
insensitive match and paginate:

boxster.mfg> grep -i -e MID -e RID mail_logs -t -p

Grab a specific ICID line:

boxster.mfg> grep -e "[[:space:]]ICID[[:space:]]1238" mail_logs

Tue Oct 18 07:58:19 2005 Info: New SMTP ICID 1238 interface main (172.18.0.40) address 172.18.0.40 reverse dns host porky3.mfg verified yes
Tue Oct 18 07:58:19 2005 Info: ICID 1238 ACCEPT SG None match ALL SBRS rfc1918
Tue Oct 18 07:58:19 2005 Info: Start MID 211476 ICID 1238
Tue Oct 18 07:58:19 2005 Info: MID 211476 ICID 1238 From: <m@mfg02.mfg>
Tue Oct 18 07:58:19 2005 Info: MID 211476 ICID 1238 RID 0 To: <m@mfg02.mfg>
Tue Oct 18 07:58:19 2005 Info: ICID 1238 close

Look for DHAP connections:

boxster.mfg> grep -t -e "Dropping connection" mail_logs

Press Ctrl-C to stop.
Wed Oct 19 18:05:28 2005 Warning: LDAP: Dropping connection due to potential Directory Harvest Attack from host=('172.18.0.40', None), dhap_limit=5, sender_group=Default
Wed Oct 19 18:05:30 2005 Warning: LDAP: Dropping connection due to potential Directory Harvest Attack from host=('172.18.0.40', None), dhap_limit=5, sender_group=Default
Wed Oct 19 18:05:32 2005 Warning: LDAP: Dropping connection due to potential Directory Harvest Attack from host=('172.18.0.40', None), dhap_limit=5, sender_group=Default
Wed Oct 19 18:06:15 2005 Warning: LDAP: Dropping connection due to potential Directory Harvest Attack from host=('172.18.0.40', None), dhap_limit=5, sender_group=Default
Wed Oct 19 18:06:17 2005 Warning: LDAP: Dropping connection due to potential Directory Harvest Attack from host=('172.18.0.40', None), dhap_limit=5, sender_group=Default

tminchin_ironport · ‎10-20-2005

Good effort. We had this in our list of feature requests.

Being a client (ie never satisfied) "it would be nice" if you also had the feature to track the MID/ICID/DCIDs related to a particular string.

eg whenever a string matches say tom@w3c.com then show me all the MID/ICID/DCID lines associated with that string match.

Tom

PS I have no idea how you would achieve that, but it would be handy in a monitoring/debug state

Corey_ironport · ‎10-20-2005

We had this on our short list of feature requests as well.

I'm not sure I like the implementation, though. How hard would it be to just implement the standard command-line grep and tail (among other useful commands)? The menu stuff is nice, but not nearly as powerful and flexible. A lot of times, on our old Sendmail servers, I'd tail a log file and pipe it to grep so that I would only see what I wanted (in real-time) instead of the entire log.
There's sort of a trick way to do this on the IronPorts, today, using Cygwin, but I haven't had a chance to play with it yet.

mark_ironport · ‎10-20-2005

There is a batch command mode (giving the arguments on the command line instead of going through the menus) and there is a '-t' flag that does the 'tail -F' operation on the current log file before piping it through grep to give the real-time display. Do these address your concerns?

Corey_ironport · ‎10-21-2005

It does, but just not in a way that I'm used to.
Maybe if I understood IronPort's reasoning for not giving us access to *some* of the good old, tried and true 'nix commands that many of us are already used to...

kisanak_ironport · ‎10-24-2005

the persistent usage of this grep command, does it will impact the performance?

Corey_ironport · ‎10-24-2005

the persistent usage of this grep command, does it will impact the performance?

I haven't seen any negative impacts on performance. It's obviously using some CPU cycles, but I doubt that it would be enough to make a difference unless your box is already sweating from the load.

klieber_ironport · ‎10-27-2005

I agree that the menu-driven interface is less than desirable, but the batch interface seems to allow at least some flexibility.

I understand the reasoning for locking the box down as tightly as it is, but I'll echo Corey's comments by saying it would be nice to have access to some of the traditional unix commands that many of us use on a day to day basis. tail, (e|f)grep, cat, wc, less/more, etc. are all non-destructive tools that are used for reading, rather than writing, files. That, combined with a proper application of *nix file permissions should keep us user types from shooting ourselves in the foot.

In the mean time, this is certainly a nice start. I assume this version of grep supports standard POSIX regexes? And is it grep or egrep? (or is that what the -e means?)

--kurt

Donald Nash · ‎11-07-2005

(Been away from IPN for a while, catching up).

As a long time Unix sysadmin, I'm also a little frustrated by the locked down nature of AsyncOS, especially since I know that under the hood it's my particular Unix flavor of choice. For example, when I see the CPU stuck at or near 100%, I really want to fire up "top" and see what's soaking up the CPU. I had that problem recently and needed to make a tech support call for what turned out to be something I could have diagnosed myself had I just been able to use "top". I've also had times when direct access to "netstat" and "ifconfig -a" would have saved me a lot of trouble (like when someone changed the duplex setting on the switch port).

That said, I know that IronPort has some valuable intellectual property to protect from prying eyes. But it seems to me that a compromise could be reached. We don't need to be able to peek under every rock in order to have much improved sysadmin abilities.

Oh yeah, I like the "grep" command. :-) I've been wishing for that for quite a while.

Jim243_ironport · ‎12-21-2005

Why do you have to put in the -e when using grep? Since it is always required why don't you just asume the -e option. Is there some other flag that I can use there? I know it is a little thing, but it just doesn't make sense to me. Also if you type the command without the -e the error message returned:

You must specify a log subscription

is not correct. I did specify the log, I just left off the -e.

I know this is a nit, but...

mark_ironport · ‎12-21-2005

The main reason is to allow for multiple grep patterns to be used by specifying multiple "-e pattern" options. Also, other options like -t and -p can be used as well. Having said that, I could look into modifying it to allow a single pattern to be specified without the "-e" along with the mail subscription name.

mark_ironport · ‎01-12-2006

Responding to feedback, I changed the grep command to not require a -e to specify the regex pattern. Thus, these are equivalent:

grep -e foo mail_logs
grep foo mail_logs

and these are equivalent:

grep -e foo -e bar mail_logs
grep -e foo bar mail_logs

Also, I added a -C option to allow for a "count" number of lines of context around the found pattern.

These changes will be in AsyncOS 4.6.

tminchin_ironport · ‎01-15-2006

would it possible for some magic like this?

grep for a string - then be able to show all the MID/ICID/DCID lines associated with that string? Greedy on memory - but it has value in a debugging/troubleshooting situation.