|
Log Analysis: More Than Just Web Logs Using Log Analysis Tools for programs other than web server logs isn’t widely practiced. Perhaps people don’t know that you can produce very similar reports for firewalls, proxies, mail, and FTP servers. FTP and mail servers secrete similar information to web servers – IP addresses, files that were accessed, and things that were done. In this article we wish to talk briefly about what type of information you can gain from looking at more than just web server logs.
When dealing with mail servers, one might like to know how many email messages were delivered. Equally important, one might be interested in finding out how many emails were rejected due to them being flagged as spam, virus, or non-existent users. One of the first things people might find surprising when looking at mail logs, is the number of rejected emails due to non-existent users. This is normally a result of spammers trying common names, hoping to get lucky. These dictionary attacks can be fun to graph, along side statistics about how many viruses and spam has been blocked. With mail logs, there aren’t very many tools that produce the nice web pages we witnessed with web logs. There are, however, countless tools that generate a text summary of information. These can be just as useful, though not quite as pleasing to the upper management when trying to justify expensive antivirus software. Qmail, Postfix and Sendmail all have 3 rd party programs that make short work of log files, usually displaying really pertinent summaries. Awstats also has the capability to parse mail logs, creating neat little summaries of data transferred, sender addresses, recipient addresses, messages processed, and error codes. FTP logs can be just as important as web server logs to some people. Most log parsing tools work on the xferlog format produced by wu-ftpd, and others. Awstats, Analog, and Webalizer all have the capability to tell you who connected to your FTP server and accessed which files, as well as the important summaries of how much data was really transferred. The xferlog produced by many FTP servers is the most common log format, but these tools can all be configured to work with most arbitrary log formats. The configuration requires that you correctly specify how your server logs to the file, so that the parsing program can successfully make sense of the data. This is often times non-trivial, so focusing on log analysis tools that work with specific servers out of the box makes everyone’s life easier. Squid, a popular caching web proxy, has some very interesting things to say in the log files. Everything from what sites your users are accessing to how much web traffic is being used can be found here. The most popular tool, Calamaris, was recently shut down due to European software patents, but can still be obtained on the Internet. Calamaris generates text or html reports including traffic, websites visited and TCP statistics. Squidalyzer and Webalizer both display similar information, but squidalyzer is more focused on singling out users. When admins need to gather details about a specific user’s web browsing habits on the job, squidalyzer is the go-to program. IDS (Intrusion Detection System) programs, like snort, can produce vast amounts of data. Even expensive commercial IDS programs and devices can produce massive amounts of data, which becomes useless until someone can spend the time to make sense of it all. There exist quite a few analysis tools that parse the logs and generate text-based summaries for snort and popular firewall programs. Snort’s logs can be quite daunting, if snort is configured incorrectly. Log parsing tools can quickly help you figure out why a particular “problem” is showing up too much. The quick and easy way to view snort logs is with an html page, complete with pie charts. Snortalog is the most wide-used and highly regarded. Snortalog makes large amounts of snort data much more manageable, and even provides links to some reports about specific attacks that it detected. The essence of logs dictates that people look at them. Many valuable pearls of information can be gleamed from logs. When you suddenly realize that your Internet connection is being clogged, this is not the time to start thinking about implementing a usable system for log viewing. Web, firewall, ftp and proxy logs can all (very clearly) show an administrator what is happening. Plagued with unexplainable data transfers? No problem, we can easily identify what server is hosting unauthorized content and clogging the network by checking web and ftp logs. In the more common cases, when a computer is infected and running an unauthorized web server, we may have been able to notice a priori that the machine was attacked by regularly looking at snort log reports. It all comes down to spending the time to watch the logs, and unfortunately this can consume the greater part of an administrator’s day. With fancy reporting tools, singling out the problems can become much easier, freeing up time to actually investigate and fix them. |