Web mining with structure mining

Web mining is the use of data mining techniques to automatically discover and extract useful information from web documents and services. It is divided into content mining, structure mining and usage mining.

Web structure mining

Web structure mining analyses the hyperlink structure of the web (and the structure within a page) to discover useful knowledge. The web is viewed as a directed graph where pages are nodes and hyperlinks are edges.

Two kinds

• Hyperlink (inter-page) structure: relations between pages through hyperlinks.
• Document (intra-page) structure: the tree structure of tags in HTML/XML within a page (DOM tree), used to extract data.

Main tasks and algorithms

• Page ranking: PageRank scores each page from its in-links and the importance of the linking pages; HITS gives each page an authority score (from in-links) and a hub score (from out-links).
• Community discovery: finding groups of pages on the same topic through dense links.
• Web crawling: deciding which links to follow first.
• Classifying and clustering pages using link patterns.
• Detecting spam / link farms.
• Website structure analysis: improving navigation and site design.

   [A]---->[B]
    ^       |
    |       v
   [D]<----[C]

Example

Search engines use structure mining: a page on "data mining" that is linked from many respected university pages gets a high rank for that topic.

Advantage and issue

It makes search results more relevant. Limitations: link spam and the large size of the web graph.