Neurons, Activations & Forward Propagation

Why Deep Learning Architectures Matter

When physical systems exhibit strong non-linearities—such as turbulent aerodynamics or non-linear material plasticity—stacking linear layers with non-linear activation functions enables universal function approximation.

Note The Rectified Linear Unit (ReLU) Defined as $\text{ReLU}(z) = \max(0, z)$, ReLU introduces piecewise linearity while preserving clean gradients for positive activations.

Step-by-Step Worked Example

Execute the forward pass of a single hidden layer with ReLU activation in Python:

def relu(z):
    return max(0.0, z)

inputs = [2.5, -1.2, 0.8]
weights = [0.6, -0.5, 1.25]
bias = -0.1

linear_sum = sum(x * w for x, w in zip(inputs, weights)) + bias
activated = relu(linear_sum)
print(f"Pre-activation z = {linear_sum:.3f} -> ReLU output = {activated:.3f}")

Real-World Engineering Application

Surrogate neural networks trained on high-fidelity CFD/FEA simulations can predict stress concentrations in milliseconds during interactive design exploration.