EEL5840 - Homework 1 Solved

Your programs must be written in Python. All code must be able to compile and run for full credit. Comment all code following proper coding conventions. Remember, if we can’t read it, we can’t grade it! (For more information on python coding standards, refer to: https://www.python.org/dev/peps/pep-0008/)


Question 1 - 3.5 points
Consider the polynomial curve fitting example discussed in class. As discussed, when the model order is too small, the training data is generally underfit and when the model order is too high, the result can overfit the training data. Write a small script of code that mimics our polynomial curve fitting function. The code should generate simulated data from the true function with added zero-mean Gaussian noise (with the true function assumed to be sinc function). The code should also generate a separate validation test data set generated in the same way. Then, after fitting the polynomial to the training data across a range of model orders and evaluated on both the training and testing data, your code should generate a plot similar to the one shown in Figure 1. Also, provide a discussion based on your plot about which model order, M, should be used to avoid over-training.

 

 

Figure 1: Figure 1.5 from the Bishop textbook. The y-axis corresponds to the root-meansquare error between the predicted and the true value (on either the training data or test data sets). The x-axis corresponds to the model order.

Question 2 - 4.5 points
Recall:

Assuming a univariate Gaussian data likelihood given N i.i.d. data points:

                                                                     )                                                              (1)

and a Gaussian prior distribution on the mean:

                                                                        )                                                                  (2)

with fixed variances ( , and ), the posterior distribution is given by:

p(µ|X) = N(µ|µN,σN2 )

where
(3)
                                                                                                            (4)

                                                                                                                                                         (5)

where µML is the maximum likelihood solution for µ given the N data points.

•    In or Binomial/Beta example in class, we computed the ML and MAP solutions for the µ parameter of the Binomial distribution iteratively with an increasing number of trials/random draws. Recall, the parameter µ represented the probability of heads.

•    In this homework question, you will do the same sort of experiment for random draws from a Gaussian distribution (i.e., a Gaussian data likelihood) with a Gaussian prior distribution on the mean parameter (assume a fixed, known variance for both the Gaussian likelihood and Gaussian prior).

•    Write a script that iteratively draws one data point from the true Gaussian distribution (with known mean). Each iteration, compute and ML solution and the MAP solution for the Gaussian mean. After each draw, update the prior distribution to be replaced with the posterior distribution from the previous draw (just like the Binomal/Beta example in class).

•    In your solution, provide:

– Display multiple sample runs of your code and include a description of what the code shows you about ML vs MAP solutions. Your discussion should illustrate that you understand ML and MAP concepts and their differences. Your discussion should answer, at a minimum, the following questions:

∗ What happens when the prior mean is initialized to the wrong value? To the correct value?

∗ What happens as you vary the prior variance from small to large?

∗ What happens when the likelihood variance is varied from small to large?

∗ How do the initial values of the prior mean, prior variance, and likelihood variance interact to effect the final estimate of the mean?

Question 3 - 0.5 points
Consider f(x) = 3xTx + 4yTx − 1 where x,y ∈ Rd.

1. What is  ? Show your work.

Question 4 - 0.5 points
Consider f(x) = −10xTQx + 4yTx + 2 where x,y ∈ Rd, Q ∈ Rd×d and Q is symmetric.

1. What is  ? Show your work.

Question 5 - 0.5 points
Consider f(x) = 8xTQx − 2yTQTx + 6 where x,y ∈ Rd, Q ∈ Rd×d and Q is symmetric.

1.    What is  ? Show your work.

2.    What is  ? Show your work.

Question 6 - 0.5 points
Consider   where x ∈ Rd.