Dense Layers

This module provides specialized dense layer implementations for concept-based models.

Summary

Dense Layer Classes

Dense	A simple fully-connected layer implementing
MLP	Simple Multi-layer Perceptron encoder with optional linear readout.
ResidualMLP	Multi-layer Perceptron with residual connections.

Class Documentation

class Dense(input_size: int, output_size: int, activation: str = 'relu', dropout: float = 0.0, bias: bool = True)

Bases: Module

A simple fully-connected layer implementing

\[\mathbf{x}^{\prime} = \sigma\left(\boldsymbol{\Theta}\mathbf{x} + \mathbf{b}\right)\]

where \(\mathbf{x} \in \mathbb{R}^{d_{in}}, \mathbf{x}^{\prime} \in \mathbb{R}^{d_{out}}\) are the input and output features, respectively, \(\boldsymbol{\Theta} \in \mathbb{R}^{d_{out} \times d_{in}} \mathbf{b} \in \mathbb{R}^{d_{out}}\) are trainable parameters, and \(\sigma\) is an activation function.

Parameters:

• input_size (int) – Number of input features.

• output_size (int) – Number of output features.

• activation (str, optional) – Activation function to be used. (default: 'relu')

• dropout (float, optional) – The dropout rate. (default: 0)

• bias (bool, optional) – If True, then the bias vector is used. (default: True)

reset_parameters() → None

Reset layer parameters to initial random values.

forward(x)

Apply linear transformation, activation, and dropout.

Parameters:

x (torch.Tensor) – Input tensor of shape (batch_size, input_size).

Returns:

Output tensor of shape (batch_size, output_size).

Return type:

torch.Tensor

training: bool

class MLP(input_size, hidden_size=64, output_size=None, n_layers=1, activation='relu', dropout=0.0)

Bases: Module

Simple Multi-layer Perceptron encoder with optional linear readout.

Parameters:

• input_size (int) – Input size.

• hidden_size (int) – Units in the hidden layers.

• output_size (int, optional) – Size of the optional readout.

• n_layers (int, optional) – Number of hidden layers. (default: 1)

• activation (str, optional) – Activation function. (default: relu)

• dropout (float, optional) – Dropout probability.

reset_parameters() → None

Reset all layer parameters to initial random values.

forward(x)

Forward pass through MLP layers with optional readout.

Parameters:

x (torch.Tensor) – Input tensor of shape (batch_size, input_size).

Returns:

Output tensor of shape (batch_size, output_size)

if readout is defined, else (batch_size, hidden_size).

Return type:

torch.Tensor

training: bool

class ResidualMLP(input_size, hidden_size, output_size=None, n_layers=1, activation='relu', dropout=0.0, parametrized_skip=False)

Bases: Module

Multi-layer Perceptron with residual connections.

Parameters:

• input_size (int) – Input size.

• hidden_size (int) – Units in the hidden layers.

• output_size (int, optional) – Size of the optional readout.

• n_layers (int, optional) – Number of hidden layers. (default: 1)

• activation (str, optional) – Activation function. (default: relu)

• dropout (float, optional) – Dropout probability. (default: 0.)

• parametrized_skip (bool, optional) – Whether to use parametrized skip connections for the residuals.

forward(x)

Forward pass with residual connections.

Parameters:

x (torch.Tensor) – Input tensor of shape (batch_size, input_size).

Returns:

Output tensor of shape (batch_size, output_size)

if readout is defined, else (batch_size, hidden_size).

Return type:

torch.Tensor

Note

Each layer applies: x = layer(x) + skip(x), where skip is either Identity, a projection layer, or a parametrized transformation.

training: bool