Neural Networks: From Fundamentals to Advanced Architectures

Core Concepts:

• Perceptron: Single neuron with inputs, weights, activation
• Forward Pass: Computation flow: X → W → σ → Output
• Backpropagation: Chain rule for gradient calculation
• Universal Approximation: 1 hidden layer can approximate any function

Python Implementation:

import numpy as np

class NeuralNetwork:
    def __init__(self):
        self.weights = np.random.randn(3, 1)  # 3 inputs to 1 output
        self.bias = np.random.randn()
        
    def sigmoid(self, x):
        return 1 / (1 + np.exp(-x))
        
    def forward(self, X):
        return self.sigmoid(np.dot(X, self.weights) + self.bias)
        

Key Insight:

A single neuron can make linear decisions; stacked neurons create complex nonlinear decision boundaries

Advanced Components:

Component	Purpose	Innovation
Hidden Layers	Feature hierarchy	Automatic feature engineering
Dropout	Regularization	Random deactivation prevents overfitting
BatchNorm	Training stability	Normalizes layer inputs

PyTorch Implementation:

import torch.nn as nn

class DeepNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(784, 256),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(256, 128),
            nn.BatchNorm1d(128),
            nn.Linear(128, 10)
        )
        
    def forward(self, x):
        return self.net(x)
        

Revolutionary Designs:

Transformer Block:

class TransformerBlock(nn.Module):
    def __init__(self, embed_size, heads):
        super().__init__()
        self.attention = nn.MultiheadAttention(embed_size, heads)
        self.norm1 = nn.LayerNorm(embed_size)
        self.norm2 = nn.LayerNorm(embed_size)
        self.ff = nn.Sequential(
            nn.Linear(embed_size, 4*embed_size),
            nn.ReLU(),
            nn.Linear(4*embed_size, embed_size))
        
    def forward(self, x):
        attn = self.attention(x, x, x)[0]
        x = self.norm1(attn + x)
        ff = self.ff(x)
        return self.norm2(ff + x)
        

Advanced Techniques:

• Mixed Precision: FP16/FP32 hybrid training
• Gradient Accumulation: Simulate larger batches
• Distributed Training: Data/model parallelism
• LoRA: Efficient fine-tuning

PyTorch Lightning Example:

import pytorch_lightning as pl

class LitModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = TransformerBlock(embed_size=512, heads=8)
        
    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.model(x)
        loss = F.cross_entropy(y_hat, y)
        return loss
        
trainer = pl.Trainer(accelerator="gpu", devices=4, strategy="ddp")
trainer.fit(LitModel(), train_loader)