How to update the weights in backpropagation algorithm when activation function in not linear?

Mon, 10 Aug 2020 21:55:19 +0000

The goal of backpropagation is to optimize the weights so that the neural network can learn how to correctly map arbitrary inputs to outputs.

Assume for the following neural network, inputs = [$i_1,i_2$] = [0.05, 0.10], we want the neural network to output = [$o_1$,$o_2$] = [0.01, 0.99], and for learning rate, $\alpha=0.5$.
In addition, the activation function for the hidden layer (both $h_1$ and $h_2$) is sigmoid (logistic):

$S(x)=\frac{1}{1+e^{-x}}$

https://i.imgur.com/cnY5feu.png

Hint:
$w_{new} = w_{old} - \alpha \frac{\partial E}{\partial w}$

$E_{\text {total}}=\sum \frac{1}{2}(\text {target}-\text {output})^{2}$

a) Show step by step solution to calculate weights $w_1$ to $w_8$ after one update in table below.
b) Calculate initial error and error after one update (assume biases $[b_1,b_2]$ are not changing during the updates).

Updating weights in backpropagation algorithm
Weights	Initialization	New weights after one step
$w1$	0.15	?
$w2$	0.20	?
$w3$	0.25	?
$w4$	0.30	?
$w5$	0.40	?
$w6$	0.45	?
$w7$	0.50	?
$w8$	0.55	?

How to update weights in backpropagation algorithm (a numerical example)?

Thu, 11 Apr 2019 17:02:04 +0000

Assume we have the following neural network and all activation functions are $f(z)=z$. If the weights are initialized with the values you see in table below, what will be new updated weights after one step if learning rate, $\alpha = 0.05$?

Assume the input values are [$i_1$,$i_2$] = [2,3] and target value $out = 1$.

Hint:
$w_{new} = w_{old} - \alpha \frac{\partial E}{\partial w}$

$E_{\text {total}}=\sum \frac{1}{2}(\text {target}-\text {output})^{2}$

Updating weights in backpropagation algorithm
Weights	Initialization	New weights after one step
$w1$	0.11	?
$w2$	0.21	?
$w3$	0.12	?
$w4$	0.08	?
$w5$	0.14	?
$w6$	0.15	?

https://i.imgur.com/v0RMeOQ.png

Ask Ghassem - Recent questions tagged back-propagation

How to update the weights in backpropagation algorithm when activation function in not linear?

How to update weights in backpropagation algorithm (a numerical example)?