Machine Learning Exercise 5 Solution

Machine Learning Course
Goals. The goal of this exercise is to
• Do classification using linear regression.
EPFL

• Implement and debug logistic regression using gradient descent.
• Compare it to linear regression.
• Implement Newton’s method for logistic regression.
Setup, data and sample code. Obtain the folder labs/ex05 of the course github repository
github.com/epfml/ML course
We will use the dataset height weight genders.csv in this exercise, and we have provided sample code templates that already contain useful snippets of code required for this exercise.
• The logistic function (a.k.a. sigmoid): σ(x) := (1 + e−x)−1
• The derivative of the logistic function is:
• The logistic regression predicts y ∈ {0,1} with the following probability
p(y = 1|w,x) = σ(x⊤w) p(y = 0|w,x) = 1 − σ(x⊤w).
• The negative log-likelihood loss for the binary classification problem (a.k.a. cross entropy loss) is:

where yi ∈ {0,1} for all i.
1 Classification Using Linear Regression
We will try to use linear regression to do classification. Although this is not a good idea in general (as discussed in the class), it will work for simple data. Concretely, we will use the height-weight data from the first exercise to predict the binary valued gender (sex). For better visualization, we will randomly sample 200 data points from the data.
Exercise 1:
Classification using linear regression.
• Use least squares(y, tx) from the previous exercise to compute the weights w on the height-weight data. Please COPY your previous implementation to the template file of this exercise least squares.py.
• Visualize the data points and the decision boundary with visualization() as in Figure 1.

Figure 1: classification by least square.
2 Logistic Regression
Exercise 2:
Implement logistic regression using gradient descent.
• Fill in the notebook function sigmoid().
• Fill in the two notebook functions calculate loss() and calculate gradient(). The first function should return the negative log-likelihood loss, while the second function should return the gradient of this loss w.r.t. the parameters w.
• Implement gradient descent learning by gradient descent() for logistic regression. You should calculate the loss and its gradient w.r.t. w. Now, you can update the weights w, and the function should return the loss and these updated weights w.
• Plot the predictions to get a visualization similar to the right part of Figure 1. Check if you get similar or different results.
Now that we have gradient descent, we can easily implement Newton’s method.
Exercise 3:
Newton’s method for logistic regression
wt+1 = wt − γ∇2L(wt)−1∇L(wt).
• Fill in the notebook function calculate hessian(). Now, complete logistic regression() by using calculate loss(), calculate gradient() and calculate hessian(). This function logistic regression() should return the loss, gradient, and Hessian. Note that it should not update nor return the parameters w. • Your gradient descent code can now be turned into a Newton’s method algorithm to find the optimal parameters w. Please fill in the notebook function learning by newton method(), which does a single parameter update. The function should return the loss and the updated parameters. hint: Use np.linalg.solve instead of directly computing the inverse ∇2L(wt)−1.
2
Exercise 4:
Penalized logistic regression.
• Fill in the notebook function penalized logistic regression(). This requires you to add the regularization term λ∥w∥2. As a sanity check, set λ to a very small value and see if it gives the same result as before. Once complete, please fill in the notebook function learning by penalized gradient(), and increase the value of λ and check whether (the norm of) w is shrinking or not.
• Check if this gives the same answer as gradient descent. To debug, print the function value and the norm of the gradient in every iteration. All these values should decrease in every iteration.
3