Artificial Neural network basics

 

Introduction to Artificial Neural Networks

Neural Networks are a subset of machine learning models inspired by the structure and function of the human brain. They are designed to recognize patterns and learn from data, making them powerful tools for tasks such as image recognition, natural language processing, and predictive analytics.

1. Basic Structure

• Neurons: The fundamental building block of a neural network is the neuron (or node). Each neuron receives input, processes it, and sends output to other neurons.

• Layers: Neural networks are composed of layers of neurons:

  • Input Layer: This is where the network receives the initial data.

  • Hidden Layers: These layers process the inputs received from the input layer. A network can have multiple hidden layers.

  • Output Layer: This layer produces the final output of the network.

2. Types of Neural Networks

• Feedforward Neural Networks: The simplest type, where connections move only in one direction—from input to output.

• Convolutional Neural Networks (CNNs): Primarily used for image and video recognition. They apply convolutional layers to detect patterns.

• Recurrent Neural Networks (RNNs): Suitable for sequential data like time series or natural language. They have connections that loop back, allowing them to maintain memory of previous inputs.

3. Key Concepts

• Activation Functions: These functions determine whether a neuron should be activated. Common activation functions include Sigmoid, ReLU (Rectified Linear Unit), and Tanh.

• Weights and Biases: Each connection between neurons has an associated weight, and each neuron has a bias. These parameters are adjusted during training to minimize the error.

• Forward Propagation: The process of passing the input data through the network to generate an output.

• Backpropagation: The training algorithm that adjusts the weights and biases by minimizing the error. It involves calculating the gradient of the loss function with respect to each weight by using the chain rule.

4. Training Neural Networks

• Data Preparation: Split the dataset into training, validation, and test sets. Normalize the data to ensure that it has a mean of zero and a standard deviation of one.

• Loss Function: A function that measures the difference between the network's predicted output and the actual output. Common loss functions include Mean Squared Error (MSE) and Cross-Entropy Loss.

• Optimization Algorithm: Methods like Stochastic Gradient Descent (SGD), Adam, or RMSprop are used to minimize the loss function.

• Epochs and Batches: Training is done in epochs, where the entire dataset is passed through the network multiple times. During each epoch, the data is divided into smaller batches for efficient computation.

5. Applications of Neural Networks

• Image and Speech Recognition: Detecting objects in images or converting speech to text.

• Natural Language Processing: Tasks like language translation, sentiment analysis, and chatbots.

• Healthcare: Diagnosing diseases from medical images or predicting patient outcomes.

• Finance: Fraud detection, stock price prediction, and algorithmic trading.

Example in Python

Here's a simple example of a feedforward neural network using the Keras library:

python

import numpy as np
from keras.models import Sequential
from keras.layers import Dense

# Generate some data
X = np.random.rand(1000, 20)  # 1000 samples, 20 features
y = np.random.randint(2, size=(1000, 1))  # Binary labels

# Create the model
model = Sequential()
model.add(Dense(64, input_dim=20, activation='relu'))  # Hidden layer
model.add(Dense(1, activation='sigmoid'))  # Output layer

# Compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Train the model
model.fit(X, y, epochs=10, batch_size=32)

# Evaluate the model
loss, accuracy = model.evaluate(X, y)
print(f"Loss: {loss}, Accuracy: {accuracy}")

This example demonstrates:

• Model Creation: Building a simple neural network with one hidden layer.

• Compilation: Setting the optimizer, loss function, and metrics.

• Training: Training the model with the input data.

• Evaluation: Evaluating the model's performance on the same data.

What is a Perceptron?

A perceptron is a type of artificial neuron used in machine learning. It's the building block of larger neural network architectures. The perceptron algorithm was invented in 1958 by Frank Rosenblatt and is primarily used for binary classification tasks.

Structure of a Perceptron

A perceptron consists of:

1. Input Features (x₁, x₂, ..., xₙ): These are the input values or features of the data.

2. Weights (w₁, w₂, ..., wₙ): Each input feature is associated with a weight.

3. Bias (b): An additional parameter added to the weighted sum of inputs to adjust the output.

4. Activation Function: Determines whether the neuron should activate or not, based on the weighted sum of inputs.

Perceptron Equation

The perceptron computes a weighted sum of the input features and passes it through an activation function to produce an output:

z = Σ (from i=1 to n) wᵢxᵢ + b

The output is determined by applying the activation function (usually a step function):

Training a Perceptron

Training a perceptron involves adjusting the weights and bias to minimize the classification error. The steps are:

1. Initialize Weights and Bias: Set initial values for weights and bias, typically to small random numbers.

2. Compute the Output: For each input sample, compute the weighted sum and apply the activation function.

3. Update Weights and Bias: Adjust the weights and bias based on the difference between the predicted output and the actual output. This is done using the following update rule:

• For Weights: wᵢ = wᵢ + Δwᵢ 

• For Bias:Δwᵢ = η * (y - ŷ) * xᵢ

• Here, η is the learning rate, y is the actual output, and ŷ is the predicted output. 

Inputs of a Neural Network:

• Features (X): The raw data or features provided to the network (e.g., pixel values for an image, words in a text).

Outputs of a Neural Network:

• Predictions (Y_hat): The network's prediction or classification based on the input data (e.g., identifying the object in an image, predicting the next word in a sentence).

The input data passes through the network layers, undergoes transformations via weights and activation functions, and finally results in an output prediction.

Weights and Bias in Neural Networks

Weights and Bias are fundamental components in the architecture of neural networks. Here's a brief overview:

1. Weights

• Purpose: Weights determine the importance of each input feature. They are the parameters that the network learns during training to make accurate predictions.

• Function: Each input to a neuron is multiplied by its corresponding weight. This helps in amplifying or diminishing the input signal.

• Update: Weights are updated through a process called backpropagation, where the network learns by minimizing the error (loss) using gradient descent.

2. Bias

• Purpose: The bias allows the activation function to be shifted left or right, which helps the model in capturing the true patterns in the data.

• Function: It's added to the weighted sum of the inputs. This helps in fitting the data better by allowing the activation function to be adjusted.

• Update: Like weights, the bias is also adjusted during the training process to minimize the error.

Mathematical Representation

z = Σᵢ wᵢxᵢ + b

z = wᵀx + b

Example

In a simple neural network with three input features and one neuron:

• Inputs: $x_1,x_2,x_{3}x\_1,\; x\_2,\; x\_3$

• Weights: $w_1, w_2,w_{\mathrm{3, }}w\_1,\; w\_2,\; w\_3$

• Bias: $bb$

The neuron's output before applying the activation function would be:

$$z=w_1{x}_1+w_2{x}_2+w_3{x}_3+bz\; =\; w\_1x\_1\; +\; w\_2x\_2\; +\; w\_3x\_3\; +\; b$$

This weighted sum $zz$ is then passed through an activation function (like ReLU or Sigmoid) to get the final output.

Understanding weights and biases is crucial as they are the parameters that a neural network learns and optimizes during the training process. This optimization allows the model to make accurate predictions and learn from the data effectively.

Working of a Neuron

1. Inputs: The neuron receives input signals (features) x₁, x₂, ..., xₙ

2. Weights: Each input is multiplied by a corresponding weight $w_1,w_2,\dots ,w\_n$.

3. Weighted Sum: The neuron calculates the weighted sum of the inputs:

$$\mathrm{<span> \; </span><span> \; </span><span> \; </span>}$$z = Σᵢ wᵢxᵢ + b

Here, $bb$ is the bias term.

4. Activation Function: The weighted sum $zz$ is passed through an activation function (like ReLU, Sigmoid, or Tanh) to produce the neuron's output $aa$:

5. Output: The output $aa$ is then passed to the next layer of neurons or becomes the final output if it is the output layer.

This process enables the neuron to learn and make decisions based on the input data. Each neuron's output serves as the input for the next layer in a multi-layer network.

Types of Activation Functions

Activation functions play a crucial role in determining whether a neuron should be activated, thereby influencing the model's learning and output. Here are some common types of activation functions:

1. Sigmoid (Logistic) Function

• Formula: σ(x) = 1 / (1 + e⁻ˣ)

• Range: 0 to 1

• Usage: Often used in binary classification problems.

• Pros: Smooth gradient, outputs probabilities.

• Cons: Can cause vanishing gradient problems.

2. Hyperbolic Tangent (Tanh) Function

• Formula: tanh(x) = 2/(1 + e⁻²ˣ) - 1

• Range: -1 to 1

• Usage: Commonly used in hidden layers of neural networks.

• Pros: Zero-centered, smooth gradient.

• Cons: Can also suffer from vanishing gradients.

3. Rectified Linear Unit (ReLU)

• Formula: ReLU(x) = max(0, x)

• Range: 0 to infinity

• Usage: Widely used in hidden layers of deep neural networks.

• Pros: Efficient computation, mitigates vanishing gradient problem.

• Cons: Can cause "dying ReLU" problem where neurons get stuck at 0.

4. Leaky ReLU

• Formula: Leaky ReLU(x) = max(αx, x)

• Range: Negative infinity to infinity

• Usage: Variant of ReLU to address the "dying ReLU" problem.

• Pros: Prevents neurons from getting stuck at 0.

5. Parametric ReLU (PReLU)

• Formula: eLU(x) = max(αx, x)

•  where $\alpha$ is a learnable parameter.

• Range: Negative infinity to infinity

• Usage: Adaptive version of Leaky ReLU.

• Pros: Allows learning the value of $\alpha \backslash alpha$.

6. Exponential Linear Unit (ELU)

• Formula:

            ELU(x) = { x, if x > 0; α(eˣ - 1), if x ≤ 0 }

• Range: Negative infinity to infinity

• Usage: Used in deep networks for faster convergence.

• Pros: Alleviates vanishing gradient, outputs can be negative.

7. Softmax

• Formula:

            Softmax(xᵢ) = eˣⁱ / Σⱼ eˣʲ

• Range: 0 to 1 (sum to 1)

• Usage: Often used in the output layer of multi-class classification problems.

• Pros: Outputs probabilities, suitable for multi-class tasks.

Summary Table

Activation Function	Formula	Range	Common Use
Sigmoid	1 / (1 + e⁻ˣ)	0 to 1	Binary classification
Tanh	2/(1 + e⁻²ˣ) - 1	-1 to 1	Hidden layers
ReLU	max(0, x)	0 to ∞	Hidden layers in deep networks
Leaky ReLU	$\mathrm{<span> \; </span><span> \; </span><span> \; </span>max}(\alpha x,x)$	-∞ to ∞	Hidden layers
PReLU	$\mathrm{<span> \; </span><span> \; </span><span> \; </span>max}(\alpha x,x)$	-∞ to ∞	Adaptive hidden layers
ELU

{ x, if x > 0; α(eˣ - 1), if x ≤ 0 }

| -∞ to ∞ | | 0 to 1 | Output layer for multi-class classification |

Each activation function has its advantages and is suited for specific tasks. Choosing the right one can greatly impact the performance and efficiency of your neural network.

Parameters and Hyperparameters of Neural Networks

Understanding the difference between parameters and hyperparameters is crucial for building and training neural networks effectively.

Parameters

Parameters are the variables that the model learns from the training data. They are internal to the model and are updated during training. Key parameters include:

• Weights: The multipliers applied to each input feature or the connections between neurons.

• Biases: The offsets added to the weighted sum before applying the activation function.

These parameters are optimized through the training process using backpropagation and gradient descent to minimize the loss function.

Hyperparameters

Hyperparameters are the settings that need to be defined before the training process begins. They govern the overall behavior and structure of the model. Key hyperparameters include:

• Learning Rate: The step size used by the optimization algorithm to update the model parameters. A higher learning rate can speed up training but may overshoot the optimal solution.

• Number of Epochs: The number of times the entire training dataset passes through the network during training.

• Batch Size: The number of training samples used in one forward/backward pass. Smaller batch sizes lead to more updates per epoch, while larger batches provide more stable updates.

• Number of Layers and Neurons: The architecture of the neural network, including the number of hidden layers and neurons per layer.

• Activation Functions: The functions applied to the outputs of neurons to introduce non-linearity (e.g., ReLU, Sigmoid).

• Optimization Algorithm: The method used to minimize the loss function (e.g., Stochastic Gradient Descent, Adam).

• Dropout Rate: The fraction of neurons randomly dropped during training to prevent overfitting.

• Regularization Parameter: Parameters like L1 or L2 regularization to penalize large weights and reduce overfitting.

Common Notations in Neural Networks

1. Input Variables

   • $x$: Input feature vector.

   • $xi$: The ith feature in the input vector.

2. Output Variables

   • $y$: Actual output or target value.

   • $\widehat{y}$: Predicted output.

3. Weights and Biases

   • $w$: Weight matrix.

   • wᵢⱼ: Weight connecting the ith neuron in the previous layer to the jth neuron in the current layer.

   • $b$: Bias term.

   • $bⱼ$: Bias for the jth neuron.

4. Layers

   • $L$: Total number of layers in the neural network.

   • $l$: Layer index (1 through L).

   • $n⁽ˡ⁾$: Number of neurons in the lth layer.

   • a⁽ˡ⁾: Activation output of the lth layer.

5. Activation Functions

   • $\sigma$: Sigmoid activation function.

   • $\text{ReLU}$: Rectified Linear Unit activation function.

   • $\tanh$: Hyperbolic tangent activation function.

   • $\text{softmax}$: Softmax activation function.

6. Feedforward and Backpropagation

   • z⁽ˡ⁾: Weighted sum (linear transformation) at the lth layer.

   • a⁽ˡ⁾ = activation(z⁽ˡ⁾) Activation output at the lth layer.

   • δ⁽ˡ⁾: Error term (delta) at the lth layer during backpropagation.

   • $\alpha$: Learning rate.

7. Loss Functions

   • $\mathcal{L}$: Loss function.

   • $J$: Cost function.

Example

Here's an example notation for a simple two-layer neural network:

• Input layer: $x=[x_1,x_2,\dots ,x_n]x\; =\; [x\_1,\; x\_2,....,\; x\_n]$

• Weights and Biases:

  • First layer: w⁽¹⁾, b⁽¹⁾

  • Second layer: w⁽²⁾, b⁽²⁾

• Activations:

  • First layer output: a⁽¹⁾ = σ(w⁽¹⁾x + b⁽¹⁾)

  • Second layer output: ŷ = a⁽²⁾ = σ(w⁽²⁾a⁽¹⁾ + b⁽²⁾)

These notations help us systematically describe and implement neural networks, ensuring clarity and consistency in their construction and analysis.

commonly used neural network architectures make the following simplifying assumptions:

1. The neurons in an ANN are arranged in layers, and these layers are arranged sequentially.
2. The neurons within the same layer do not interact with each other.
3. The inputs are fed into the network through the input layer, and the outputs are sent out from the output layer.
4. Neurons in consecutive layers are densely connected, i.e., all neurons in layer l are connected to all neurons in layer l+1.
5. Every neuron in the neural network has a bias value associated with it, and each interconnection has a weight associated with it.
6. All neurons in a particular hidden layer use the same activation function. Different hidden layers can use different activation functions, but in a hidden layer, all neurons use the same activation function.

Assumptions for Simplifying Neural Networks

1. Linear Activation Functions:

   • Assuming linear activation functions (like identity functions) simplifies the mathematics, but this limits the network's ability to model complex, non-linear relationships.

2. Single Layer Perceptron:

   • Simplifying to a single-layer network (perceptron) helps in understanding basic concepts, although it's limited to linearly separable problems.

3. Small Network Size:

   • Working with smaller networks (fewer layers and neurons) simplifies the computation and conceptualization but might not capture the complexity of the data.

4. Uniform Weight Initialization:

   • Assuming all weights are initialized to small, random values or zeros makes the initialization process straightforward. However, this can lead to poor training performance.

5. Fixed Learning Rate:

   • Using a fixed learning rate simplifies training but may not be optimal for convergence.

6. Ignoring Regularization:

   • Ignoring techniques like dropout or L2 regularization can simplify the model, though it may lead to overfitting.

7. Simple Datasets:

   • Using simple, synthetic datasets (like XOR or AND gate problems) instead of real-world complex data makes it easier to visualize and understand the network's learning process.

8. Batch Gradient Descent:

   • Assuming the entire dataset is processed at once simplifies the gradient descent algorithm, though it's less efficient than mini-batch or stochastic gradient descent.

9. Static Model Architecture:

• Not considering dynamic changes in network architecture during training simplifies the model-building process.

Flow of Information Between Layers in a Neural Network

In a neural network, information flows from the input layer through the hidden layers to the output layer. Here's a step-by-step overview:

1. Input Layer:

   • The input layer receives the raw data or features. Each neuron in this layer corresponds to one feature of the input data.

   • Input Vector ($xx$): The data is represented as a vector $x=[x_1,x_2,\dots ,x_n]x\; =\; [x\_1,\; x\_2,....,\; x\_n]$.

2. Hidden Layers:

   • Each hidden layer consists of neurons that process the inputs. The processing involves computing the weighted sum of the inputs plus a bias and then applying an activation function.

   • Weighted Sum(z⁽ˡ⁾)

   • : For the l-th layer, the weighted sum is computed as:

                            z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾

Here, W⁽ˡ⁾ is the weight matrix, a⁽ˡ⁻¹⁾ is the activation output from the previous layer, and b⁽ˡ⁾ is the bias vector.

• Activation Output (a⁽ˡ⁾)

• : The activation function (e.g., ReLU, Sigmoid) is applied to z⁽ˡ⁾ to get the activation output:

$${\mathrm{<span> \; </span><span> \; </span><span> \; </span>a}}^{[l]}=\text{activation(z⁽ˡ⁾) a⁽ˡ⁾)} =activation(z⁽ˡ⁾)$$

            This output becomes the input for the next layer.

3. Output Layer:

   • The final layer produces the output predictions of the network.

   • Output Vector (ŷ): The output layer computes the final activation output, which represents the predictions of the network. This can be a single value (for regression) or a probability distribution (for classification).

Example of a Forward Pass

Consider a simple neural network with one hidden layer:

• Input Layer: $x=[x_1,x_2,\dots ,x_n]x\; =\; [x\_1,\; x\_2,....,\; x\_n]$

• Weights and Biases:

  • Hidden Layer: W⁽¹⁾, b⁽¹⁾

  • Output Layer: W⁽²⁾, b⁽²⁾

• Activation Functions:

  • Hidden Layer: ReLU

  • Output Layer: Sigmoid (for binary classification)

Step-by-Step Forward Pass:

1. Compute Weighted Sum for Hidden Layer:

                z⁽¹⁾ = W⁽¹⁾x + b⁽¹⁾

2. Apply Activation Function (ReLU):

                a⁽¹⁾ = ReLU(z⁽¹⁾)

3. Compute Weighted Sum for Output Layer:

                z⁽²⁾ = W⁽²⁾a⁽¹⁾ + b⁽²⁾

4. Apply Activation Function (Sigmoid):

                ŷ = Sigmoid(z⁽²⁾)

This forward pass results in the predicted output ŷ.

Summary

• Input Layer: Receives and normalizes the raw input data.

• Hidden Layers: Process the data through weighted sums and activation functions to extract features and learn patterns.

• Output Layer: Produces the final predictions based on the processed information.

let's walk through a concrete example of a forward pass in a simple neural network with one hidden layer.

Network Architecture:

• Input Layer: 2 input features (x₁, x₂)
• Hidden Layer: 3 neurons (using ReLU activation)
• Output Layer: 1 neuron (using Sigmoid activation for binary classification)

Weights and Biases (Randomly initialized for demonstration):

• W¹ (Weights between Input and Hidden Layer):

[[0.2, 0.5],
 [-0.3, 0.8],
 [0.1, -0.4]]

• b¹ (Biases for Hidden Layer):

[[0.1],
 [-0.2],
 [0.3]]

• W² (Weights between Hidden and Output Layer):

[[0.6, -0.7, 0.2]]

• b² (Bias for Output Layer):

[[0.5]]

Input Data:

• x:

[[2],
 [3]]

Forward Pass Calculation:

1. Hidden Layer Calculation:

   • z¹ = W¹ * x + b¹

   [[0.2, 0.5],   *   [[2],   +   [[0.1],
    [-0.3, 0.8],       [3]]        [-0.2],
    [0.1, -0.4]]                     [0.3]]
   
   =  [[0.4 + 1.5],   +   [[0.1],
       [-0.6 + 2.4],       [-0.2],
       [0.2 - 1.2]]        [0.3]]
   
   =  [[1.9],   +   [[0.1],
       [1.8],       [-0.2],
       [-1.0]]        [0.3]]
   
   =  [[2.0],
       [1.6],
       [-0.7]]
   

   • a¹ = ReLU(z¹) (ReLU activation)

   [[ReLU(2.0)],
    [ReLU(1.6)],
    [ReLU(-0.7)]]
   
   =  [[2.0],
       [1.6],
       [0.0]]
   

2. Output Layer Calculation:

   • z² = W² * a¹ + b²

   [[0.6, -0.7, 0.2]]   *   [[2.0],   +   [[0.5]]
                               [1.6],
                               [0.0]]
   
   =  [[1.2 - 1.12 + 0]]   +   [[0.5]]
   
   =  [[0.08]]   +   [[0.5]]
   
   =  [[0.58]]
   

   • ŷ = Sigmoid(z²) (Sigmoid activation)

   Sigmoid(0.58) = 1 / (1 + exp(-0.58)) ≈ 0.64
   

Result:

The output of the forward pass, ŷ, is approximately 0.64. This represents the network's prediction for the given input x.

Key Points:

• Matrix multiplication is used to efficiently calculate the weighted sums.
• Activation functions introduce non-linearity, which is crucial for the network to learn complex patterns.
• The output of each layer becomes the input to the next layer.

This example demonstrates a single forward pass. In a real training scenario, this forward pass would be followed by a backward pass (to calculate gradients) and an optimization step (to update the weights and biases). This process is repeated many times until the network learns to make accurate predictions.

Feedforward Algorithm Steps

1. Initialization:

   • Start with an input vector $xx$ and initialize weights $WW$ and biases $bb$ for each layer.

2. Input Layer:

   • The input data $xx$ is fed into the network through the input layer.

3. Hidden Layers:

   • Weighted Sum: For each hidden layer $ll$:

                        z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾ 

Here a⁽ˡ⁻¹⁾, is the activation from the previous layer (or input $xx$ for the first hidden layer).

• Activation Function: Apply an activation function (e.g., ReLU, Sigmoid) to the weighted sum to get the activation :

$${\mathrm{<span> \; </span><span> \; </span><span> \; </span>a}}^{[l]}=\text{activation }a⁽ˡ⁾ = activation(z⁽ˡ⁾)$$

4. Output Layer:

   • Weighted Sum: Compute the weighted sum for the output layer:

                z⁽ᴸ⁾ = W⁽ᴸ⁾a⁽ᴸ⁻¹⁾ + b⁽ᴸ⁾

• Activation Function: Apply the activation function (e.g., Sigmoid for binary classification) to get the final output ŷ:

                ŷ = activation(z⁽ᴸ⁾)

Example    

Consider a simple neural network with one hidden layer:

• Input Vector: x = [x₁, x₂]

• Weights and Biases:

  • Hidden Layer: W⁽¹⁾ = [ w₁₁⁽¹⁾ w₁₂⁽¹⁾ ] , b⁽¹⁾ = [ b₁⁽¹⁾ ] [ w₂₁⁽¹⁾ w₂₂⁽¹⁾ ] [ b₂⁽¹⁾ ]

    Output Layer: W⁽²⁾ = [ w₁₁⁽²⁾ w₁₂⁽²⁾ ] , b⁽²⁾ = b₁⁽²⁾

Step-by-Step Forward Pass:

1. Compute Weighted Sum for Hidden Layer:

        z⁽¹⁾ = W⁽¹⁾x + b⁽¹⁾

        z⁽¹⁾ = [ w₁₁⁽¹⁾ w₁₂⁽¹⁾ ] * [ x₁ ] + [ b₁⁽¹⁾ ] [ w₂₁⁽¹⁾ w₂₂⁽¹⁾ ] [ x₂ ] [ b₂⁽¹⁾ ]

2. Apply Activation Function (ReLU):

        a⁽¹⁾ = ReLU(z⁽¹⁾)

3. Compute Weighted Sum for Output Layer:

        z⁽²⁾ = W⁽²⁾a⁽¹⁾ + b⁽²⁾

        z⁽²⁾ = [ w₁₁⁽²⁾ w₁₂⁽²⁾ ] * [ a₁⁽¹⁾ ] + b₁⁽²⁾ [ a₂⁽¹⁾ ]