Apache server log inferer

Problem Statement

• Apache HTTP server generates access log for each request mentioning details like..
  • ip
  • date-time
  • full page path
  • size
  • ... etc
• The author of this project uses the log file to grasp certain properties related to web site management like
  • user web request characteristic
  • favorite pages
  • user's view about the web site
  • etc
• The author of the project have also tried to make this project easily extendable for more features

Input

• Input data taken by the project
  • Apache HTTP server generated access log file
• The access log file is available in various format
• This peoject will read log file having format below
  %requestor_ip - - %datetime "%request_string" %status_code %bytes_sent %referrer %user_agent
• Owner of the server is allowed to customize the datetime format. But this project will take belowed datetime format
  [day/month/year:hour:minute:second zone]

Output

• Usefull information extraction and visualization:
  • Get the top hit pages and genration of bar plot for comparison
  • Get potential list of broken links and the source page on the host site. This can also be used get popular older page names
  • Get the keywords for which your site is famouns on search engines
• Extenstion Framework:
  • Provides architectural framework so that user can add his functionality

• Screenshot

  

Motivation

• Boom of web:
  • cheap internet
  • rise in blogs
  • forums
  • web advertisement
• Web minig tools
  • Finds vistitor's intersts
  • And at la$t make money.. $$$

Successfully implemented work

• Initially proposed system
  • Show popular pages based on number of hits
  • Increasing popularity of the page in last ten weeks
  • Get potentially broken links on the site: uses intelligent approach
  • Show graphical representation of information whenever possible
  • HTML report generation(optional)
  • Provide framework for easy extension(optional)
  • Detect attacks(optonal)
• Implemented work:
  • Show popular pages based on number of hits
  • Get potentially broken links on the site: uses intelligent approach
  • Provide framework for easy extension
  • Increasing popularity of the page in last ten weeks(in text mode)
  • Show visitor's view about web site on popular search engines

Timewise efforts required

• Parsing
  • 4 hrs
• sqlite interfacing
  • 3 hrs
• Data extraction utilities
  • 3 hrs
• Proper wxPython inteface
  • 10 hrs
• matplotlib and pylab image generation
  • 2 hours
• wxPython-matplotlib interfacing
  • 4 hrs
• Code refinement and refactoring
  • 5 hrs
• Small module or method level tests
  • 5 hrs
• Other
  • 4 hrs
• Total
  • ~40 hrs

Solution Design

• Layeres in solution

Foundations of this solution

• MVC architecture
• Independent Layers
• Power of database at the heart
• Exploiting usability by easy customization

Implementation Details

• Programming Language : Python
• GUI library : wxPython
• Database library : sqlite3
• Plotting and charting library : matplotlib

Implementation Details

• Code written by the author:
  • parser
  • sqlite
  • gui
• Code borrowed from internet:
  • plotting
• Lines of code written by author:
  • ~650 lines

Testing

• Layer wise testing
  • at Parsing level : Checking if parsing works correctly
  • at Information level: Verifying sqlite query output
  • at Presentation level: Proper alignment of compoments, resizing and representation

Challenges

• Learning power of python
• Not static nature of python: testing is harder
• wxPython : proper alignment, also faced some bugs
• matplotlib
• Lots of small tests

Special features

Thank you