a) Discuss about quadratic cost / mean squared error. (2 Marks) Quadratic cost, or Mean Squared Error (MSE), is one of the simplest cost functions primarily used for regression tasks. It measures the performance of a model by calculating the squared difference between the true label ($y_i$) and the predicted output (${\hat{y}}_i$). Formula: $C=\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$ Squaring the difference serves two purposes:

1. It ensures the difference always remains positive.

2. It severely penalizes large errors compared to smaller ones.

b) Point out the benefit or importance of using cross entropy cost function. (3 Marks) Benefits of the Cross Entropy cost function include:

1. It helps to minimize the impact of saturated neurons on learning speed, allowing the network to learn faster.

2. It is relatively easy to compute and optimize.

3. It is robust to outliers in the data.

4. It is well-suited and highly effective for multi-class classification tasks.

c) Using this one-dimensional quadratic function $f(x)=2x^2+4x+2$ find the second step of gradient. [$\eta =0.1,x_0=5$], [$P_{n+1}=P_n-\eta \mathrm{\nabla }f(P_n)$] (5 Marks)

Solution: Given, Function, $f(x)=2x^2+4x+2$ Learning rate, $\eta =0.1$ Starting point, $x_0=5$

First, we need to find the derivative (gradient) of the function: $f^{\prime }(x)=\frac{d}{dx}(2x^2+4x+2)=4x+4$

The gradient descent update rule is: $x_{n+1}=x_n-\eta f^{\prime }(x_n)$

Step 1 - Finding $x_1$: Gradient at $x_0=5$ is $f^{\prime }(x_0)=4(5)+4=20+4=24$ So, $x_1=x_0-\eta f^{\prime }(x_0)$ $\Rightarrow x_1=5-(0.1\times 24)$ $\Rightarrow x_1=5-2.4$ $\Rightarrow x_1=2.6$

Step 2 - Finding $x_2$: Gradient at $x_1=2.6$ is $f^{\prime }(x_1)=4(2.6)+4=10.4+4=14.4$ So, $x_2=x_1-\eta f^{\prime }(x_1)$ $\Rightarrow x_2=2.6-(0.1\times 14.4)$ $\Rightarrow x_2=2.6-1.44$ $\Rightarrow x_2=1.16$

Answer: The second step of gradient is $1.16$.