Using Awk to do Quick and Dirty Analysis of Apache Logs

Published December 13th, 2008

Problem Statement

Often, we need to quickly analyze the Apache log files for certain sites without running extensive log analysis program that take a long time to run or runs on a schedule. For quick and dirty probing of Apache logs, you are better of with simple command-line tools. Here we will show you how to perform a few quick and dirty log analysis using standard Linux commands and a scripting language called Awk.

Finding unique IP addresses for a given day

Say you want to find out which unique IP addresses visited your site for a given day, you can run the following command from your shell prompt:

$ grep '[date string]' /path/to/access.log | awk '{print $1}' | sort | uniq

For example, to find out the list of unique IP addresses that have visited the ApacheHacker.com blog yesterday (12/Dec/2008), we can run:

grep '12/Dec/2008' /logs/apachehacker/access.log | awk '{print $1}' | sort | uniq

Finding which IP address visited how many times for a given day

To find out which IP address visited your site how many times a day, run:

$ grep '[date string]' /path/to/access.log |  \
   awk '{cnt[$1]++;} END{for (ip in cnt){printf("%-15s visited: %04d time(s).\n", ip, cnt[ip])}}'

For example, to find out which IP address visited the ApacheHacker.com blog on Dec 12, 2008, we can run:

$ grep '12/Dec/2008' /logs/apachehacker/access.log | \
  awk '{cnt[$1]++;} END{for (ip in cnt){printf("%-15s visited: %04d time(s).\n", ip, cnt[ip])}}'

Here is a sample output:

74.6.8.116      visited: 0001 time(s).
74.6.18.246     visited: 0001 time(s).
93.126.3.33     visited: 0016 time(s).
80.48.192.249   visited: 0044 time(s).
69.62.207.163   visited: 0058 time(s).
89.45.49.247    visited: 0022 time(s).
82.159.52.211   visited: 0015 time(s).
71.226.202.64   visited: 0017 time(s).
203.83.248.74   visited: 0002 time(s).
206.196.125.113 visited: 0001 time(s).
76.188.138.185  visited: 0016 time(s).
203.112.77.18   visited: 0006 time(s).
142.179.135.57  visited: 0016 time(s).
91.121.201.145  visited: 0016 time(s).
74.6.18.215     visited: 0002 time(s).
66.249.73.21    visited: 0001 time(s).
66.150.96.121   visited: 0027 time(s).
75.147.236.233  visited: 0002 time(s).
75.119.230.207  visited: 0020 time(s).
203.129.155.4   visited: 0022 time(s).
142.166.3.122   visited: 0002 time(s).
64.1.215.163    visited: 0003 time(s).

Category: Access Control | Tags: