Models of Neuron

Perceptron

Developed by Frank Rosenblatt by using McCulloch and Pitts model, perceptron is the basic operational unit of artificial neural networks. It employs supervised learning rule and is able to classify the data into two classes.

Operational characteristics of the perceptron: It consists of a single neuron with an arbitrary number of inputs along with adjustable weights, but the output of the neuron is 1 or 0 depending upon the threshold. It also consists of a bias whose weight is always 1. Following figure gives a schematic representation of the perceptron.

Perceptron thus has the following three basic elements −

•  Links − It would have a set of connection links, which carries a weight including a bias always having weight 1.
•  Adder − It adds the input after they are multiplied with their respective weights.
•  Activation function − It limits the output of neuron. The most basic activation function is a Heaviside step function that has two possible outputs. This function returns 1, if the input is positive, and 0 for any negative input.

Training Algorithm

Perceptron network can be trained for single output unit as well as multiple output units.

Training Algorithm for Single Output Unit

Step 1 − Initialize the following to start the training −

•     Weight
•     Bias
•     Learning rate α

For easy calculation and simplicity, weights and bias must be set equal to 0 and the learning rate must be set equal to 1.

Step 2 − Continue step 3-8 when the stopping condition is not true.

Step 3 − Continue step 4-6 for every training vector x.

Step 4 − Activate each input unit as follows −

x_i=s_i  (i=1 to n)

Step 5 − Now obtain the net input with the following relation −

Here ‘b’ is bias and ‘n’ is the total number of input neurons.

Step 6 − Apply the following activation function to obtain the final output.

Step 7 − Adjust the weight and bias as follows −

Case 1 − if y ≠ t then,

 w_i(new)=w_i(old)+αtx_i

b(new)=b(old)+αt

Case 2 − if y = t then,

w_i(new) = w_i(old)

b(new)=b(old)

Here ‘y’ is the actual output and ‘t’ is the desired/target output.

Step 8 − Test for the stopping condition, which would happen when there is no change in weight.

Training Algorithm for Multiple Output Units

The following diagram is the architecture of perceptron for multiple output classes.

Step 1 − Initialize the following to start the training −

•     Weights
•     Bias
•     Learning rate α

For easy calculation and simplicity, weights and bias must be set equal to 0 and the learning rate must be set equal to 1.

Step 2 − Continue step 3-8 when the stopping condition is not true.

Step 3 − Continue step 4-6 for every training vector x.

Step 4 − Activate each input unit as follows −

x_i=s_i  (i=1ton)

Step 5 − Obtain the net input with the following relation –

Here ‘b’ is bias and ‘n’ is the total number of input neurons.

Step 6 − Apply the following activation function to obtain the final output for each output unit j = 1 to m −

Step 7 − Adjust the weight and bias for x = 1 to n and j = 1 to m as follows −

Case 1 − if y_j ≠ t_j then,

w_ij(new)=w_ij(old)+αt_jx_i

b_j(new)=b_j(old)+αt_j

Case 2 − if y_j = t_j then,

w_ij(new)=w_ij(old)

b_j(new)=b_j(old)

Here ‘y’ is the actual output and ‘t’ is the desired/target output.

Step 8 − Test for the stopping condition, which will happen when there is no change in weight.

Adaptive Linear Neuron Adaline

Adaline which stands for Adaptive Linear Neuron, is a network having a single linear unit. It was developed by Widrow and Hoff in 1960. Some important points about Adaline are as follows −

•   It uses bipolar activation function.
• It uses delta rule for training to minimize the Mean-Squared Error MSE between the actual output and the desired/target output.
• The weights and the bias are adjustable.

Architecture

The basic structure of Adaline is similar to perceptron having an extra feedback loop with the help of which the actual output is compared with the desired/target output. After comparison on the basis of training algorithm, the weights and bias will be updated.

Step 1 − Initialize the following to start the training −

• Weights
• Bias
  
• Learning rate α

For easy calculation and simplicity, weights and bias must be set equal to 0 and the learning rate must be set equal to 1.

Step 2 − Continue step 3-8 when the stopping condition is not true.

Step 3 − Continue step 4-6 for every bipolar training pair s:t.

Step 4 − Activate each input unit as follows −

x_i=s_i ( i= 1 to n)

Step 5 − Obtain the net input with the following relation −

Here ‘b’ is bias and ‘n’ is the total number of input neurons.

Step 6 − Apply the following activation function to obtain the final output −

Step 7 − Adjust the weight and bias as follows −

Case 1 − if y ≠ t then,

w_i(new)=w_i(old)+α(t−y_in)x_i

b(new)=b(old)+α(t−y_in)

Case 2 − if y = t then,

w_i(new)=w_i(old)

b(new)=b(old)

Here ‘y’ is the actual output and ‘t’ is the desired/target output.

(t−y_in)  is the computed error.

Step 8 − Test for the stopping condition, which will happen when there is no change in weight or the highest weight change occurred during training is smaller than the specified tolerance.

Multiple Adaptive Linear Neuron Madaline

Madaline which stands for Multiple Adaptive Linear Neuron, is a network which consists of many Adalines in parallel. It will have a single output unit. Some important points about Madaline are as follows −

 It is just like a multilayer perceptron, where Adaline will act as a hidden unit between the input and the Madaline layer.

  The weights and the bias between the input and Adaline layers, as in we see in the Adaline architecture, are adjustable.

    The Adaline and Madaline layers have fixed weights and bias of 1.  Training can be done with the help of Delta rule.

Architecture

The architecture of Madaline consists of “n” neurons of the input layer, “m” neurons of the Adaline layer, and 1 neuron of the Madaline layer. The Adaline layer can be considered as the hidden layer as it is between the input layer and the output layer, i.e. the Madaline layer.

Training Algorithm

By now we know that only the weights and bias between the input and the Adaline layer are to be adjusted, and the weights and bias between the Adaline and the Madaline layer are fixed.

Step 1 − Initialize the following to start the training −

    Weights

    Bias

    Learning rate α

For easy calculation and simplicity, weights and bias must be set equal to 0 and the learning rate must be set equal to 1.

Step 2 − Continue step 3-8 when the stopping condition is not true.

Step 3 − Continue step 4-6 for every bipolar training pair s:t.

Step 4 − Activate each input unit as follows −

x_i=s_i (i=1 to n)

Step 5 − Obtain the net input at each hidden layer, i.e. the Adaline layer with the following relation −

Here ‘b’ is bias and ‘n’ is the total number of input neurons.

Step 6 − Apply the following activation function to obtain the final output at the Adaline and the Madaline layer −

     Output at the hidden Adaline unit

Q_j= f(Q_inj)

Final output of the network

y=f(y_in)

Step 7 − Calculate the error and adjust the weights as follows −

Case 1 − if y ≠ t and t = 1 then,

w_ij(new)=w_ij(old)+α(1−Q_inj)x_i

b_j(new)=b_j(old)+α(1−Q_inj)

In this case, the weights would be updated on Qj where the net input is close to 0 because t = 1.

Case 2 − if y ≠ t and t = -1 then,

w_ik(new)=w_ik(old)+α(−1−Q_ink)x_i

b_k(new)=b_k(old)+α(−1−Q_ink)

In this case, the weights would be updated on Qk where the net input is positive because t = -1.

Here ‘y’ is the actual output and ‘t’ is the desired/target output.

Case 3 − if y = t then

There would be no change in weights.

Step 8 − Test for the stopping condition, which will happen when there is no change in weight or the highest weight change occurred during training is smaller than the specified tolerance.

Credits: https://www.tutorialspoint.com/