PML-Homework 6 Solved

 Let’s consider a binary classification problem on Half Moons dataset, which consists of two interleaving half circles. The input is two-dimensional and the response is binary (0,1).

We observe 100 points x from this dataset and their labels y:

 

 

              1.0.1         scikit-learn GaussianProcessClassifier

1.   GaussianProcessClassifier from scikit-learn library [1] approximates the non-Gaussian posterior by a Gaussian using Laplace approximation. Define an RBF kernel gp_sklearn.kernels.RBF with lenghtscale parameter = 1 and fit a Gaussian Process classifier to the observed data (x,y).

[ ]: 

2.   Use plot_sklearn_predictions function defined below to plot the posterior predictive mean function over a finite grid of points. You should pass as inputs the learned GP classifier sklearn_gp_classifier, the observed points x and their labels y.

 [25]: def meshgrid(x, n, eps=0.1): x0, x1 = np.meshgrid(np.linspace(x[:, 0].min()-eps, x[:, 0].max()+eps, n),

np.linspace(x[:, 1].min()-eps, x[:, 1].max()+eps, n))

x_grid = np.stack([x0.ravel(), x1.ravel()], axis=-1) return x0, x1, x_grid

def plot_sklearn_predictions(sklearn_gp_classifier, x, y): x0, x1, x_grid = meshgrid(x, 30)

preds = sklearn_gp_classifier.predict_proba(x_grid) preds_0 = preds[:,0].reshape(x0.shape)

 

1.0.2 Pyro classification with HMC inference Consider the following generative model

                                                                                   yn|pn ∼ Bernoulli(pn)               n = 1,...,N

logit 

 LogNormal(0,1)

LogNormal(0,1)

We model the binary response variable with a Bernoulli likelihood. The logit of the probability is a Gaussian Process with predictors xn and kernel matrix Kσ,l, parametrized by variance ρ and lengthscale l.

We want to solve this binary classification problem by means of HMC inference, so we need to reparametrize the multivariate Gaussian GP(µ,Kσ,l(xn)) in order to ensure computational efficiency. Specifically, we model the logit probability as

logit(p) = µ · 1N + η · L,

where L is the Cholesky factor of Kσ,l and ηn ∼ N(0,1). This relationship is implemented by the get_logits function below.

 [6]: def get_logits(x, mu, sigma, l, eta): kernel = gp.kernels.RBF(input_dim=2, variance=torch.tensor(sigma),␣

,→lengthscale=torch.tensor(l))

K  = kernel.forward(x, x) + torch.eye(x.shape[0]) * 1e-6

L  = K.cholesky() return mu+torch.mv(L,eta)

3.   Write a pyro model gp_classifier(x,y) that implements the reparametrized generative model, using get_logits function and pyro.plate on independent observations.

[ ]: 

4.   Use pyro NUTS on the gp_classifier model to infer the posterior distribution of its parameters. Set num_samples=10 and warmup_steps=50. Then extract the posterior samples using pyro .get_samples() and print the keys of this dictionary using .keys() method.

[ ]: 

The posterior_predictive function below outputs the prediction corresponding to the i-th sample from the posterior distribution. plot_pyro_predictions calls this method to compute the average prediction on each input point and plots the posterior predictive mean function over a finite grid of points.

 

5.   Pass the learned posterior samples obtained from NUTS inference to plot_pyro_predictions and plot the posterior predictive mean.

[ ]: