Deep Learning from Scratch

2025-12-04

Code and docs

A feed-forward neural network in NumPy only, with manual backpropagation, optimizers, and WandB experiment tracking.

python · numpy · deep-learning · wandb · education

Repository

This repository is a feed-forward neural network in NumPy, with no autograd and no deep-learning framework. The models train on Fashion-MNIST and CIFAR-10. The project uses Weights & Biases (WandB) heavily for experiment tracking.

Engineering without abstractions

Dropping autograd means numerical stability is handled directly. The code does manual forward and backward passes, including:

  • Stable softmax and cross-entropy so gradients do not explode.
  • Weight-initialization scaling so variance stays usable across depth.
  • Manual optimizers (SGD with momentum, Adam) and how they change the loss landscape.
  • L2 regularization and gradient clipping.

The repo also has the operational pieces a real pipeline needs: minibatching, early stopping, checkpointing, and reproducible runs.

Experiment tracking over cherry-picking

The write-up does not lead with a single vanity metric. It commits to concrete baselines (~89% on Fashion-MNIST, ~45% on CIFAR-10 for an FFNN) and documents the hyperparameter search. WandB keeps the sweep history, failures, and tuning decisions visible. Measurement over cherry-picked peak accuracy.

Note: This was a DTU group project. The repository lives under the group’s GitHub organization.