Hypertext retrieval and mining
(Graduate elective)
CS-610, Spring 2001

Staff

• Instructor: Soumen Chakrabarti
• TA's: TBA

Timing

Evening lectures 6:30--8pm W,Th F12 CSE. No designated office hours; appointments by email.

Scribe notes

Scribe notes are to be typed in LaTeX2e with diagrams originally in EPS or PNG format. It should be possible to compile the source into PDF using pdflatex and into HTML using TtH. Style/class files will be provided. A list of volunteers follows in chronological order.

• Avinash and Ashu
• Satyen
• Mits and Pradeep Kumar

Lecture calendar

2001-01-05

• Administrative details
• Course overview

2001-01-09

• Basics of HTTP, HTML, URL
• Search Engine Watch
• Threading, lock and IO contentions
• Spider traps

2001-01-10

• Writing a crawler, continued
• Caching DNS and asynch resolver client
• Checking for past visits
• Frontier workflow management
• Per-host queues, system load monitor
• The Mercator Web crawler
• The anatomy of a large-scale hypertextual Web search engine

2001-01-12

• Finding if two documents are near-duplicates using shingles
• Mirror site detection

2001-01-17

• Inverted index
• Boolean and proximity search
• TFIDF and the vector space model
• Recall and precision
• Ranking techniques
• A thoughtful paper on the politics of ranking
• Index compression and updating techniques
• (Read Managing Gigabytes for additional info on this section of the course.)

2001-01-18

• Strategies to prioritize the crawl frontier

2001-01-24

• Bottom-up hierarchical agglomerative clustering
• k-Means clustering
• Speeding up by sampling and projection

2001-01-31

• Singular value decomposition
• Latent semantic indexing (LSI)

2001-02-01

• LSI: computational details
• Connections between LSI and other eigen properties
• Query processing in LSI
• The need for feature selection for unsupervised learning

2001-02-07

• Feature selection for unsupervised learning, continued
• The problem with hidden variables
• Mixture models and EM, another tutorial

2001-02-08

• Formulation and analysis of EM, continued
• Application of EM to clustering documents

Assignments

Groups will be formed and group-to-assignment allotment will be declared on the course newsgroup on a FCFS basis. The list of projects will be available from an IITB internal Website to be specified in class.