What is Fully Connected Layer in Deep Learning?

Fully Connected (FC) layers are also known as dense layers which are used in neural networks especially in of deep learning. They are a type of neural network layer where every neuron in the layer is connected to every neuron in the previous and subsequent layers. The "fully connected" descriptor comes from the fact that each of the neurons in these layers is connected to every activation in the previous layer creating a highly interconnected network.

• In CNNs fully connected layers often follow convolutional and pooling layers used to interpret the feature maps generated by these layers into the final output categories or predictions.
• In fully connected feedforward networks these layers are the main building blocks that directly process the input data into outputs.

Structure of Fully Connected Layers

The structure of FC layers is one of the most significant factors that define how it works in a neural network. This structure involves the fact that every neuron in one layer will interconnect with every neuron in the subsequent layer.

Key Components of Fully Connected Layers

A Fully Connected layer is characterized by its dense interconnectivity. Here’s a breakdown of its key components:

• Neurons: Basic units that receive inputs from all neurons of the previous layer and send outputs to all neurons of the subsequent layer.
• Weights: Each connection between neurons has an associated weight indicating the strength and influence of one neuron on another.
• Biases: A bias term for each neuron helps adjust the output along with the weighted sum of inputs.
• Activation Function: Functions like ReLU, Sigmoid or Tanh introduce non-linearity to the model helping it to learn complex patterns and behaviors.

Working and Structure of Fully Connected Layers in Neural Networks

The extensive connectivity allows for comprehensive information processing and feature integration making FC layers essential for tasks requiring complex pattern recognition.

Key Operations in Fully Connected Layers

1. Input Processing

Each neuron in an FC layer receives inputs from all neurons of the previous layer with each connection having a specific weight and each neuron incorporating a bias. The input to each neuron is a weighted sum of these inputs plus a bias:

z_j = \sum_i (w_{ij}.x_i) +b_j

Here w_{ij} is the weight from neuron i of the previous layer to neuron j, x_i​ is the input from neuron i and b_j​ is the bias for neuron j

2. Activation

The weighted sum is then processed through a non-linear activation function such as ReLU, Sigmoid or Tanh. This step introduces non-linearity enabling the network to learn complex functions:

a_j = f(z_j)

f denotes the activation function transforming the linear combination of inputs into a non-linear output.

Example Configuration

Consider a neural network transition from a layer with 4 neurons to an FC layer with 3 neurons:

• Previous Layer (4 neurons) → Fully Connected Layer (3 neurons)

Each neuron in the FC layer receives inputs from all four neurons of the previous layer resulting in a configuration that involves 12 weights and 3 biases. This design of FC layer helps in transforming and combining features from the input layer hence helping in network's ability to perform complex decision-making tasks.

Key Role of Fully Connected Layers in Neural Networks

The key roles of fully connected layers in neural network are discussed below:

1. Feature Integration and Abstraction

FC layers consolidate features extracted by earlier layers (e.g., convolutional or recurrent), transforming them into a form suitable for accurate prediction by capturing complex patterns and relationships.

2. Decision Making and Output Generation

Typically used as the final layer in classification or regression tasks, FC layers convert high-level features into output scores. For classification, these scores are passed through Softmax to yield class probabilities.

3. Introduction of Non-Linearity

Activation functions like ReLU, Sigmoid, or Tanh applied in FC layers introduce non-linearity, allowing the network to learn complex, non-linear patterns and generalize effectively.

4. Universal Approximation

According to the Universal Approximation Theorem, an FC layer with enough neurons can approximate any continuous function, showcasing its power in modeling diverse problems.

5. Flexibility across Domains

FC layers are input-agnostic and versatile, applicable to various domains like vision, speech, and NLP, supporting both shallow and deep architectures.

6. Regularization and Overfitting Control

Techniques like Dropout and L2 regularization are crucial in FC layers to prevent overfitting, promoting generalization by reducing reliance on specific neurons or large weights.

Advantages of Fully Connected Layers

• Integration of Features: They are capable of combining all features before making predictions, essential for complex pattern recognition.
• Flexibility: FC layers can be incorporated into various network architectures and handle any form of input data provided it is suitably reshaped.
• Simplicity: These layers are straightforward to implement and are supported by all major deep learning frameworks.

Limitations of Fully Connected Layers

Despite their benefits FC layers have several drawbacks:

• High Computational Cost: The dense connections can lead to a large number of parameters, increasing both computational complexity and memory usage.
• Prone to Overfitting: Due to the high number of parameters they can easily overfit on smaller datasets unless techniques like dropout or regularization are used.
• Inefficiency with Spatial Data: Unlike convolutional layers, FC layers do not exploit the spatial hierarchy of images or other structured data, which can lead to less effective learning.

Fully Connected layers are fundamental to the architecture of many neural networks, contributing to their ability to perform tasks ranging from simple classifications to complex pattern recognitions.