Web log mining and WWW mining architecture

Web log mining (the main part of web usage mining) is the discovery of useful patterns of user behaviour from web server logs: files that record each request made to a web site (IP address, time stamp, requested URL, status code, bytes transferred, referrer, user agent). It helps in site improvement, personalisation and understanding of customers.

WWW mining architecture

Web server logs / content / link data
            |
   [1. Data collection & pre-processing]
      cleaning, user & session identification,
      page-view & path completion
            |
   [2. Pattern discovery]
      association rules, sequential patterns,
      clustering, classification, statistics
            |
   [3. Pattern analysis]
      filtering, OLAP, visualisation
            |
   Useful knowledge: personalisation,
   recommendations, site design, caching

1. Data sources: server logs, proxy server logs, client-side data (cookies), site content and its hyperlink structure, user profiles and registration data.
2. Pre-processing: data cleaning (remove image requests, failed requests, robot visits), user identification (IP + agent, cookies), session identification (a time-out, for example 30 minutes), path completion and formatting into a form for mining.
3. Pattern discovery: apply mining techniques: association rules (pages visited together), sequential patterns (navigation order), clustering of users or pages, classification, and statistical analysis (most visited pages).
4. Pattern analysis: remove uninteresting patterns; use queries, OLAP and visualisation to interpret.
5. Applications: recommendation and personalisation, web site restructuring, targeted ads, caching and prefetching, fraud and attack detection.