Labs/Deep Learning/convolutional-networks

Deep LearningIntermediate

~25 min

CNNs & Convolution

Extract translation-invariant spatial features from multidimensional signals using sliding kernel convolutions, parameter sharing, and pooling.

#2D Discrete Cross-Correlation / Convolution#Sliding Kernels & Feature Maps#Stride & Zero-Padding (Valid vs Same)#Parameter Sharing & Translation Equivariance#Receptive Field Expansion#Spatial Downsampling & Max Pooling

Overview

The Problem Statement

Passing high-resolution images (e.g. 1000×1000 RGB pixels = 3,000,000 inputs) into a standard dense fully connected layer with 1,000 hidden units requires 3 billion weight parameters for a single layer—guaranteeing catastrophic overfitting and GPU memory exhaustion. Furthermore, dense layers are not spatially invariant: a dog recognized in the top-left corner must be re-learned from scratch if it appears in the bottom-right corner.

System Invariant:A convolutional layer enforces two structural inductive biases: Local Connectivity (neurons only connect to local spatial patches) and Parameter Sharing (the identical kernel weights W are slid across all spatial coordinates), guaranteeing Translation Equivariance: f(shift(x)) = shift(f(x)).

✓ When To Use

Computer vision: object detection, semantic segmentation, image classification.
Audio spectrogram processing: acoustic model feature extraction.
Spatial grid telemetry: climate modeling, medical imaging (MRI/CT), and physics simulations.

✕ When NOT To Use

Tabular business data where feature ordering is arbitrary (permuting columns breaks convolutional spatial locality).
Sequential language modeling where long-range dependencies across thousands of tokens exceed local receptive fields (use Transformers).
Graph-structured networks with non-Euclidean connectivity (use Graph Neural Networks).

Why It Exists

Catastrophic Outage Scenario

A defect inspection camera on an automated semiconductor assembly line uses unpadded valid convolutions across 12 deep layers. With a 5×5 kernel and no padding, each layer shrinks the spatial grid by 4 pixels ((32-5)/1 + 1 = 28). By layer 8, the feature map shrinks to 0×0 pixels, causing tensor dimension crash exceptions during deployment. Even worse, the outer 16 pixels of the silicon wafer were never inspected because valid convolution discards image borders.

Downstream System Degradation:

Spatial boundary degradation from unpadded valid convolutions.
Receptive field blindness: shallow CNNs failing to grasp global scene context.
Parameter explosion when flattening high-resolution feature maps into dense heads.
Information loss from aggressive non-invertible spatial pooling.

How It Works

In deep learning, 2D convolution is mathematically implemented as discrete cross-correlation. For input image I and kernel K of size (k_h, k_w): S(i, j) = (I * K)(i, j) = ∑_m ∑_n I(i + m, j + n) K(m, n).

At each coordinate (i, j), the kernel multiplies the local receptive field patch element-wise and sums the products into a single scalar in the output feature map.

Spatial Dimensions Formula: For input dimension W, kernel size K, padding P, and stride S, the output spatial dimension is: W_out = ⌊(W - K + 2P) / S⌋ + 1.

Receptive Field: As layers are stacked, a single pixel in a deep feature map corresponds to a progressively larger region of the original input image: RF_{l} = RF_{l-1} + (K_l - 1) · ∏_{i=1}^{l-1} S_i.

Single-Node vs Distributed Reality:

In production deep learning frameworks, 2D convolution is rarely computed with naive nested loops. Instead, it is lowered to General Matrix Multiply (GEMM) using the `im2col` algorithm: input image patches are rearranged into columns of a large matrix, allowing cuBLAS to execute the convolution as a single blazing-fast matrix product W · X_col.

Direct Spatial Convolution (im2col + GEMM)

Unrolls spatial receptive fields into columns and computes convolution via standard BLAS matrix multiplication.

+ Leverages highly optimized hardware tensor cores; industry standard in cuDNN.

- Requires temporary memory buffer to store unrolled im2col matrix.

Winograd Convolution

Applies minimal filtering algorithms to compute small convolutions (e.g. 3×3 kernels with stride 1) with up to 2.25x fewer multiplications.

+ Fastest algorithm for small kernels; significantly reduces arithmetic operations.

- Introduces numerical precision issues at larger tile sizes.

Depthwise Separable Convolution

Decomposes standard convolution into a depthwise spatial convolution (per channel) followed by a 1×1 pointwise convolution (across channels).

+ Reduces computational FLOPs and parameter count by 8x-9x; powers MobileNet architectures.

- Slightly lower peak FLOP utilization on GPU tensor cores compared to dense GEMM.

Engineering Prediction Challenge

Prove It: Predict the System Behavior

Reason before simulating

System State Given

An input image has dimensions 32×32 pixels. It passes through a convolutional layer with a 5×5 kernel, stride S = 1, and zero padding P = 0 (valid convolution).

Prediction Question:

What are the spatial dimensions (height × width) of the output feature map?

32×32

28×28

16×16

Interactive Visualizer

Interact directly with this distributed system primitive. Experiment with fault injection and observe state transitions.

2D Convolution Kernel Sliding & Feature Extraction

Sliding 3×3 receptive field computes local dot products, translating raw pixel matrices into invariant feature maps.

Kernel Filter:

1. Input Image (6×6 Grayscale)

240

Receptive Field: [0:2, 1:3]

2. Kernel Dot Product3×3 Convolution

p[0]: 10 × -1.00-10.0

p[1]: 10 × 0.000.0

p[2]: 240 × 1.00240.0

p[3]: 10 × -2.00-20.0

p[4]: 10 × 0.000.0

p[5]: 240 × 2.00480.0

p[6]: 10 × -1.00-10.0

p[7]: 10 × 0.000.0

p[8]: 240 × 1.00240.0

Sum + Bias:920.0

3. Feature Map (4×4)

920

4. Max Pooling 2×2 (Downsample to 2×2)

920

Build It Step-by-Step

The engineering blueprint. Expand each sequential milestone to examine requirements, architectural impacts, common pitfalls, and blueprint pseudocode.

Apply Zero-Padding to Input Tensor

For 'Same' padding that preserves spatial dimensions with stride 1, pad borders by P = (K - 1) // 2 on all sides.

def pad2d(X, pad):
  return np.pad(X, ((0, 0), (pad, pad), (pad, pad)), mode='constant')

Slide Kernel & Compute Local Dot Products

Iterate over output grid (out_h, out_w). Extract patch X[i*stride : i*stride + k_h, j*stride : j*stride + k_w], compute element-wise product with kernel K, sum products, and add bias.

for i in range(H_out):
  for j in range(W_out):
    patch = X[i*S:i*S+K, j*S:j*S+K]
    out[i, j] = np.sum(patch * kernel) + bias

Apply Spatial Pooling (Max Pooling)

Divide feature map into non-overlapping 2×2 blocks (stride 2). Retain the maximum scalar value in each block: out[i, j] = max(X[2i:2i+2, 2j:2j+2]).

def max_pool2d(X, pool_size=2):
  H, W = X.shape
  return X.reshape(H//2, 2, W//2, 2).max(axis=(1, 3))

Code Implementation

Production-ready, idiomatic reference implementations across Go, TypeScript, Python, and Java with sticky language switching, copy-to-clipboard, and source download.

Core Architectural Design Decisions

Invariants & Mechanics

Decision 01

Implemented integer dimension calculation (H - K + 2P)//S + 1.

Decision 02

Decoupled convolution from pooling for modular visual testing.

Algorithmic Complexity:Time Complexity: O(H_out · W_out · K_h · K_w). Memory: O(H_out · W_out) for output feature map.

cnn_convolution.py•65 LOC

import numpy as np
from typing import Tuple

class Conv2D:
    """
    2D Convolutional Layer with Stride, Padding, and Receptive Field math.
    """
    def __init__(self, kernel: np.ndarray, stride: int = 1, padding: int = 0):
        self.kernel = np.asarray(kernel, dtype=np.float32)
        self.k_h, self.k_w = self.kernel.shape
        self.stride = stride
        self.padding = padding

    def forward(self, X: np.ndarray) -> np.ndarray:
        """
        Executes 2D convolution over single-channel input matrix X [H, W].
        Returns: Feature map [H_out, W_out].
        """
        X = np.asarray(X, dtype=np.float32)
        H, W = X.shape

        # 1. Apply zero padding
        if self.padding > 0:
            X_padded = np.pad(X, self.padding, mode="constant", constant_values=0)
        else:
            X_padded = X

        H_pad, W_pad = X_padded.shape

        # 2. Compute output spatial dimensions
        H_out = (H_pad - self.k_h) // self.stride + 1
        W_out = (W_pad - self.k_w) // self.stride + 1
        output = np.zeros((H_out, W_out), dtype=np.float32)

        # 3. Slide kernel across spatial coordinates
        for i in range(H_out):
            r_start = i * self.stride
            r_end = r_start + self.k_h
            for j in range(W_out):
                c_start = j * self.stride
                c_end = c_start + self.k_w

                receptive_patch = X_padded[r_start:r_end, c_start:c_end]
                output[i, j] = np.sum(receptive_patch * self.kernel)

        return output

class MaxPool2D:
    """2x2 Max Pooling with Stride 2."""
    def __init__(self, pool_size: int = 2):
        self.pool_size = pool_size

    def forward(self, X: np.ndarray) -> np.ndarray:
        H, W = X.shape
        H_out = H // self.pool_size
        W_out = W // self.pool_size
        output = np.zeros((H_out, W_out), dtype=np.float32)

        for i in range(H_out):
            for j in range(W_out):
                patch = X[i * self.pool_size:(i + 1) * self.pool_size,
                          j * self.pool_size:(j + 1) * self.pool_size]
                output[i, j] = np.max(patch)

        return output

Implementation Strategy: Readable and mathematically explicit NumPy implementation of 2D cross-correlation and max pooling. Illustrates padding boundaries, sliding window indices, and dot-product accumulation.

Time Complexity: O(H_out · W_out · K_h · K_w). Memory: O(H_out · W_out) for output feature map.

Edge Cases & Failure Modes

Investigate real-world production incidents. Expand each failure mode to analyze symptoms, root causes, and defensive engineering mitigations.

Scenario: Border Erosion in Deep Unpadded Networks

Consequence: Without padding, every 3×3 convolution strips 2 pixels from each dimension. In a 10-layer network, 20 pixels are lost from height and width, clipping periphery objects.

Engineering Solution: Use 'Same' padding with P = (K - 1) // 2 to maintain constant spatial resolution across deep layers.

Scenario: Receptive Field Too Small for Global Context

Consequence: A shallow CNN classifies a medical image based on local texture rather than whole-organ geometry because its receptive field only covers 15×15 pixels of a 512×512 image.

Engineering Solution: Increase depth, introduce Dilated (Atrous) Convolutions, or append a Vision Transformer (ViT) self-attention layer for global context.

Scenario: Odd vs Even Kernel Sizes

Consequence: Even kernel sizes (e.g. 4×4) cannot be symmetrically padded around a center pixel, introducing directional phase drift across deep layers.

Engineering Solution: Always use odd-sized kernels (3×3, 5×5, 7×7) so padding is perfectly symmetric: P = (K - 1) / 2.

Production Considerations

Production checklist covering observability metrics, horizontal sharding, reliability fail-open patterns, and client identity security.

Scaling to 10x / 100x Traffic:

Use TensorRT / cuDNN FP16 fused Conv-BatchNorm-ReLU layers to double inference throughput on edge devices (NVIDIA Jetson).
Deploy MobileNetV3 / EfficientNet architectures with Depthwise Separable convolutions for mobile battery efficiency.

Telemetry & Observability:

PROMETHEUS METRICS:

➔ conv_layer_receptive_field_pixels{layer} (gauge)

➔ conv_feature_map_sparsity_ratio (gauge)

➔ conv_kernel_execution_ms (histogram)

DISTRIBUTED TRACES & LOGS:

➔ OpenTelemetry span for 'conv2d.gemm' measuring tensor execution duration.

Trade-offs

Select an algorithm below to dynamically compare memory footprints, burst tolerance, boundary precision, and distributed suitability.

Approach	Advantages	Disadvantages	Use When
Standard 2D Convolution	Strong spatial inductive bias; translation equivariance; excellent feature extraction.	Fixed local receptive field; requires deep stacking to observe global context.	Standard image classification, object detection, and localized pattern recognition.
Dilated (Atrous) Convolution	Expands receptive field exponentially without increasing parameter count or losing spatial resolution.	Prone to 'gridding artifacts' if dilation rates share common factors.	Semantic segmentation and real-time audio generation (WaveNet).
Vision Transformers (ViT / Self-Attention)	Global receptive field from Layer 1; scales exceptionally well with massive datasets.	Requires 10x more pre-training data to learn spatial biases; quadratic O(N²) attention cost.	Foundation models pre-trained on hundreds of millions of images (CLIP, DINOv2).