EE5904_ME5404-Neural Networks: Homework 1 Solved

Q1.Consider the signal-flow graph of the perceptron shown in the above figure. The activation function, ( )v , where v is the induced local field, can be designed by the user. If the activation function is chosen as hard limiter (i.e. step function), then it becomes the classical perceptron, and the decision boundary is shown to be a hyperplane.  In this problem, let’s explore other choices of the activation function, and its effect on the decision boundary.  Let’s assume that the classification decision made by the perceptron is simply a threshold rule defined as follows:

Observation vector x x x[ 1      2                      x ] belongs to class C1 if the output y >ξ, whereξis a user-defined threshold; otherwise, x belongs to class C2. 

Consider the following three choices of activation function:

1) The activation function is a quadratic function: ( )v   (v a)2    c;

1ev 2) The activation function is the hyperbolic tangent function: ( )v 1ev ;

(v m )2

3) The activation function is the Bell-shaped Gaussian function: ( )v e 2 . For each case, investigate whether the resulting decision boundary is a hyper-plane or not.

Q2. Consider the logic function, EXCLUSIVE OR (XOR).  Truth Table of XOR

x1
0
1
0
1
x2
0
0
1
1
y 
0
1
1
0
 

It is well known that the XOR problem is not linearly separable. It seems obvious by visually checking, which however cannot be accepted as mathematical proof. Therefore, please supply a rigorous mathematical proof for this statement.

Q3. The perceptron could be used to perform numerous logic functions, such as AND, OR, COMPLEMENT and NAND function, whose truth tables are tabulated as follows respectively.

 

x1
0
0
1
1
 
x1
0
0
1
1
x2
0
1
0
1
x2
0
1
0
1
y
0
0
0
1
y
0
1
1
1
 

                           AND                                                                                OR
 

 

x
0
1
y
1
0
 

x1
0
0
1
1
x2
0
1
0
1
y
1
1
1
0
                     COMPLEMENT                                                                    NAND
 

a). Demonstrate the implementation of the logic functions AND, OR, COMPLEMENT and NAND with selection of weights by off-line calculations.

Marks)
b). Demonstrate the implementation of the logic functions AND, OR, COMPLEMENT and NAND with selection of weights by learning procedure. Suppose initial weights are chosen randomly and learning rate  is 1. Plot out the trajectories of the weights for each case. Compare the results with those obtained in (a). Try other learning rates, and report your observations with different learning rates.

Marks)
c). What would happen if the perceptron is applied to implement the EXCLUSIVE OR function with selection of weights by learning procedure? Suppose initial weight is chosen randomly and learning rate  is 1.0. Do the computer experiment and explain your finding.                                                  )

Q4. Single layer perceptron with pure linear activation function can be used to fit a linear model to a set of input-output pairs. Suppose that we are given the following pairs:

{(0.5,8.0), (1.5, 6.0), (3, 5), (4.0, 2), (5.0, 0.5)} and a single linear neuron as shown in the following figure.

a). Find the solution of w and b using the standard linear least-squares (LLS) method. Plot out the fitting result.

b). Suppose that initial weight is chosen randomly and learning rate  is 0.02. Find the solution of w and b using the least-mean-square (LMS) algorithm for 100 epochs. Plot out the fitting result and the trajectories of the weights versus learning steps. Will the weights converge

c). Compare the results obtained by LLS and the LMS methods.

d) Repeat the simulation study in b) with different learning rates , and explain your findings.
Q5. In a variant of the LMS algorithm called the leaky LMS algorithm, the cost function to be minimized is defined by

1 2                    1                2

E n( )  e ( )n  || w n( ) ||

2             2

where w(n) is the weight vector, e(n) is the estimation error, and λ is a positive constant. As in the ordinary LMS algorithm, we have the estimation error,

e n( )  d n( )  wT( ) ( )n x n

where d(n) is the desired response corresponding to the input vector x(n).

Following the similar procedure to derive the learning algorithm for LMS, show that the time update for the weight vector of the leaky LMS algorithm is defined by

w(n1)(1)w(n)x(n)e(n)

which includes the ordinary LMS algorithm as a special case.