Web Science/Part2: Emerging Web Properties/Advanced statistical descriptive models for the Web/Zipf law powerlaw or pareto law.webm

17

Web Science/Part2: Emerging Web Properties

Web Science/Part2: Emerging Web Properties/MoocIndex

Web Science/Part2: Emerging Web Properties/Advanced statistical descriptive models for the Web/Zipf law powerlaw or pareto law.webm

• edit MOOC index

Zipf law powerlaw or pareto law.webm

Learning goals

Video

Script

Quiz

Further reading

Discussion

Learning goals

1. Know how to transform a rank frequency diagram to a powerlaw plot.
2. Understand how powerlaw and pareto plots relate to each other.
3. Be able to explain why a pareto plot is just and inverted rank frequency diagram
4. Be able to transform the zipf coefficient to the powerlaw and pareto coefficient and vice versa.
5. Understand that building the CDF is basically like building the integral.

Video

Script

Find the slide deck at File:Zipf_law_powerlaw_or_pareto_law.pdf

Quiz

1 What is true about the power law plot for words

	The x axis depicts the frequncy
	The x axis depicts how many words occur exactly y times
	The x axis depicts how many words occur exactly x times
	The y-axis depicts the frequency
	The y-axis depicts how many words occur exactly x times
	The y-axis depicts how many words occur exactly y times

2 What is the connection between the zipf parameter and power law coefficient (if the zipf parameter is called $k$ and the powerlaw coefficient is called $\alpha$)?

	There is no connection
	$k={\frac {1}{(\alpha -1)}}$
	$k={\frac {1}{1-\alpha }}$
	$k={\frac {1}{1+\alpha }}$

Further reading

1. tba

Discussion

Navigation

1. How big is the World Wide Web
   1. Problems with the question about the size of the Web
   2. 3 ways to study the Web
   3. A simplistic descriptive model
   4. An unrealistic, simplistic generative model
   5. Summary, further reading, homework
2. Simple statistical descriptive Models for the Web
   1. Counting Words And Documents
   2. Typical length of a document
   3. How to formulate a research hypothesis
   4. Number of words needed to understand most of Wikipedia
   5. Linguists way of checking simplicity of text
3. Advanced statistical descriptive models for the Web
   1. The Zipf law for text
   2. Visually straight lines on log log plots
   3. Fitting a curve on a log log plot
   4. Zipf law powerlaw or pareto law.webm
4. Modelling Similarity of Text
   1. Similarity Measures and their Applications
   2. Jaccard Similarity for Sets
   3. Cosine Similarity For Vectorspaces
   4. Probabilistic Similarity Measures Kullback Leibler Divergence
   5. Comparing Results of Similarity Merasures
5. Generative Models for the Web
   1. Introduction to generative modelling.webm
   2. Sampling from a probability distribution
   3. Evaluating a generative model
   4. Pittfalls when increasing the number of model parameters
6. Modeling the Web as a graph
   1. Reviewing terms from graph theory
   2. The standard web graph model
   3. Descriptive statistics of the web graph
   4. Topology of the web graph
   5. Modelling-graphs-with-linear-algebra

Fitting a curve on a log log plot