Web Science/Part2: Emerging Web Properties/Modelling Similarity of Text/Comparing Results of Similarity Merasures

23

Web Science/Part2: Emerging Web Properties

Web Science/Part2: Emerging Web Properties/MoocIndex

Web Science/Part2: Emerging Web Properties/Modelling Similarity of Text/Comparing Results of Similarity Merasures

• edit MOOC index

Comparing Results of Similarity Merasures

Learning goals

Video

Script

Quiz

Further reading

Discussion

Learning goals

1. Understand that different modeling choices can produce very different results.
2. Have a feeling how you could statistically compare the differences of the models.
3. Know how you could extract keywords from documents with the tf-idf approach.
4. Try to argue which model you like best in a certain scenario.

Video

Script

The script can be Found at File:Comparing-Results-of-Similarity-Merasures.pdf

Quiz

1 which method can be used best to find characteristic words of a text?

	jaccard
	TF-IDF
	TF
	Language Model
	Smoothed Language Model

2 Which method works well in an information retrieval setting

	jaccard
	TF-IDF
	Language Model
	Smoothed Language Model

3 Which method should be used when you don't have several occurences of the same elements?

	jaccard
	TF-IDF
	Language Model
	Smoothed Language Model

Further reading

1. tba

Discussion

Navigation

1. How big is the World Wide Web
   1. Problems with the question about the size of the Web
   2. 3 ways to study the Web
   3. A simplistic descriptive model
   4. An unrealistic, simplistic generative model
   5. Summary, further reading, homework
2. Simple statistical descriptive Models for the Web
   1. Counting Words And Documents
   2. Typical length of a document
   3. How to formulate a research hypothesis
   4. Number of words needed to understand most of Wikipedia
   5. Linguists way of checking simplicity of text
3. Advanced statistical descriptive models for the Web
   1. The Zipf law for text
   2. Visually straight lines on log log plots
   3. Fitting a curve on a log log plot
   4. Zipf law powerlaw or pareto law.webm
4. Modelling Similarity of Text
   1. Similarity Measures and their Applications
   2. Jaccard Similarity for Sets
   3. Cosine Similarity For Vectorspaces
   4. Probabilistic Similarity Measures Kullback Leibler Divergence
   5. Comparing Results of Similarity Merasures
5. Generative Models for the Web
   1. Introduction to generative modelling.webm
   2. Sampling from a probability distribution
   3. Evaluating a generative model
   4. Pittfalls when increasing the number of model parameters
6. Modeling the Web as a graph
   1. Reviewing terms from graph theory
   2. The standard web graph model
   3. Descriptive statistics of the web graph
   4. Topology of the web graph
   5. Modelling-graphs-with-linear-algebra

Probabilistic Similarity Measures Kullback Leibler Divergence