PML-Homework 3 Solved

Exercise 1
1.        Write the factorization of the joint distribution 𝑃(𝐴, 𝐵, 𝐶, 𝐷, 𝐸, 𝐹) corresponding to the Bayesian network:

 

2.        Indicate whether the following statements on conditional indepence are True or False and motivate your answer. a. A ⊥⊥ B

b.  A ⊥⊥ F

c.   A ⊥⊥ C | {B,E}

d.  F ⊥⊥ D | B

e.   B ⊥⊥ D | C

Exercise 2
Topic models are statistical models that learn the distribution of the abstract topics occurring in a collection of documents. In this context, documents are collections of topics and topics are collections of words.

We consider a dictionary of 𝑁 words, an ordered collection of 𝐷 documents and an ordered collection of 𝑇 possible topics appearing in each document. We build our model using the indexes representing each quantity in its collection (e.g. 𝑖 is the 𝑖-th word in the dictionary). For simplicity, we assume that all documents contain the same number of words. Each document can contain multiple topics; specifically, we associate a topic to each word appearing in the document.

We make the following assumptions:

 𝐷 is the total number of documents

𝑇 is the total number of topics

𝑊 < 𝑁 is the number of words per document

For the 𝑗-th word and the 𝑖-th document, we sample a topic 𝑡𝑖,𝑗 from a Categorical distribution on 𝜃𝑖

The distribution 𝜃𝑖 of topics in the 𝑖-th document is a Dirichlet distribution with concentration parameters 𝛼 (the length of 𝛼 is 𝑇)

The 𝑗-th word in the 𝑖-th document, namely 𝑤𝑖,𝑗, is sampled from a Categorical distribution on 𝜑𝑡𝑖,𝑗

The distribution 𝜑𝑘 of words per topic 𝑘 is a Dirichlet distribution with concentration parameters 𝛽 (the length of 𝛽 is 𝑁)

1.           Write the generative process of the following graphical model. Be aware of plate notation!

 

2.           Set 𝑇 = 5, 𝐷 = 10, 𝑊 = 50, 𝑁 = 100 and use pyro to implement this graphical model as a function of the hyperparameters model(alpha, beta) , that outputs theta, phi, t, w . 3. Evaluate your function on the hyperparameters

alpha = torch.tensor([0.5, 0.3, 0.1, 0.4, 0.2])

beta = dist.Gamma(1./T, 1.).sample((W,)) and print the shape of the output tensors theta, phi, t, w .