Web Science/Part2: Emerging Web Properties/Simple statistical descriptive Models for the Web

7

Web Science/Part2: Emerging Web Properties

Web Science/Part2: Emerging Web Properties/Simple statistical descriptive Models for the Web

Web Science/Part2: Emerging Web Properties/MoocIndex

• edit MOOC index

Simple statistical descriptive Models for the Web

Learning goals

Associated units

Further reading

Discussion

Learning goals

1. Formulating a research hypothesis and test it by means of simple descriptive statistics
2. Reading diagrams

Associated units

• 
• 
• 
• 

0

open

0

questions

Counting Words And Documents

1. Understand why we selected simple English Wikipedia as a toy example for modeling the web
2. Understand that a task already as simple as counting words includes modeling choices
3. Be familiar with the term “unique word token”
4. Know some basic tools to count words and documents

• 
• 
• 
• 

2

open

3

questions

Typical length of a document

Be familiar with some basic statistical objects like Median, Mean, and Histograms

Should be able to relate a histogram to its cumulative distribution function

• 
• 
• 
• 

1

open

1

question

How to formulate a research hypothesis

Understand the ongoing, cyclic process of research

Know what falsifiable means and why every research hypothesis needs to be falsifiable

Be able to formulate your own research hypothesis

• 
• 
• 
• 

1

open

2

questions

Number of words needed to understand most of Wikipedia

Understand what a log-log plot is

Improve your skills in reading and interpreting diagrams

Know about the word rank / frequency plot

Should be able to transfer a histogram or curve into a cumulative distribution function

• 
• 
• 
• 

0

open

0

questions

Linguists way of checking simplicity of text

Get a feeling for interdisciplinary research

Know the Automated Readability Index

Have a strong sense of support for our research hypothesis

Be able to critically discuss the limits of our models

new unit

Further reading

no further reading defined

You can define further reading here.
In general you can use the edit button in the upper right corner of a section to edit its content.

Discussion

Navigation

1. How big is the World Wide Web
   1. Problems with the question about the size of the Web
   2. 3 ways to study the Web
   3. A simplistic descriptive model
   4. An unrealistic, simplistic generative model
   5. Summary, further reading, homework
2. Simple statistical descriptive Models for the Web
   1. Counting Words And Documents
   2. Typical length of a document
   3. How to formulate a research hypothesis
   4. Number of words needed to understand most of Wikipedia
   5. Linguists way of checking simplicity of text
3. Advanced statistical descriptive models for the Web
   1. The Zipf law for text
   2. Visually straight lines on log log plots
   3. Fitting a curve on a log log plot
   4. Zipf law powerlaw or pareto law.webm
4. Modelling Similarity of Text
   1. Similarity Measures and their Applications
   2. Jaccard Similarity for Sets
   3. Cosine Similarity For Vectorspaces
   4. Probabilistic Similarity Measures Kullback Leibler Divergence
   5. Comparing Results of Similarity Merasures
5. Generative Models for the Web
   1. Introduction to generative modelling.webm
   2. Sampling from a probability distribution
   3. Evaluating a generative model
   4. Pittfalls when increasing the number of model parameters
6. Modeling the Web as a graph
   1. Reviewing terms from graph theory
   2. The standard web graph model
   3. Descriptive statistics of the web graph
   4. Topology of the web graph
   5. Modelling-graphs-with-linear-algebra