Web Science/Part2: Emerging Web Properties/Simple statistical descriptive Models for the Web/Number of words needed to understand most of Wikipedia

11

Web Science/Part2: Emerging Web Properties

Web Science/Part2: Emerging Web Properties/MoocIndex

Web Science/Part2: Emerging Web Properties/Simple statistical descriptive Models for the Web/Number of words needed to understand most of Wikipedia

• edit MOOC index

Number of words needed to understand most of Wikipedia

Learning goals

Video

Script

Quiz

Further reading

Discussion

Learning goals

1. Understand what a log-log plot is
2. Improve your skills in reading and interpreting diagrams
3. Know about the word rank / frequency plot
4. Should be able to transfer a histogram or curve into a cumulative distribution function

Video

Script

Find the slide deck at File:Number_of_words_needed_to_understand_most_of_Wikipedia.pdf

Quiz

We saw that more than half of the unique word tokens on Simple English Wikipedia occured only once. Which of the following statements are true?

	picking a random word from the simple english wikipedia the chance is higher than 50% that it occured only once
	picking 100 random words from wikipedia we expect more than 50 of them to occure only once
	picking a random word the chance for getting a word that occurs only once is less than 10%

Further reading

1. http://www.courses.vcu.edu/PHY-rhg/astron/html/mod/006/index.html
2. Falsifiability

Discussion

Navigation

1. How big is the World Wide Web
   1. Problems with the question about the size of the Web
   2. 3 ways to study the Web
   3. A simplistic descriptive model
   4. An unrealistic, simplistic generative model
   5. Summary, further reading, homework
2. Simple statistical descriptive Models for the Web
   1. Counting Words And Documents
   2. Typical length of a document
   3. How to formulate a research hypothesis
   4. Number of words needed to understand most of Wikipedia
   5. Linguists way of checking simplicity of text
3. Advanced statistical descriptive models for the Web
   1. The Zipf law for text
   2. Visually straight lines on log log plots
   3. Fitting a curve on a log log plot
   4. Zipf law powerlaw or pareto law.webm
4. Modelling Similarity of Text
   1. Similarity Measures and their Applications
   2. Jaccard Similarity for Sets
   3. Cosine Similarity For Vectorspaces
   4. Probabilistic Similarity Measures Kullback Leibler Divergence
   5. Comparing Results of Similarity Merasures
5. Generative Models for the Web
   1. Introduction to generative modelling.webm
   2. Sampling from a probability distribution
   3. Evaluating a generative model
   4. Pittfalls when increasing the number of model parameters
6. Modeling the Web as a graph
   1. Reviewing terms from graph theory
   2. The standard web graph model
   3. Descriptive statistics of the web graph
   4. Topology of the web graph
   5. Modelling-graphs-with-linear-algebra

How to formulate a research hypothesis

Linguists way of checking simplicity of text