A spring–block theory of feature learning in deep neural networks

This repository contains the source code and videos for the paper: A spring–block theory of feature learning in deep neural networks. More details can be found here.

Amusing vidoes of folding ruler expermenets and DNN training in different regimes

When slow and steady pulling:

ruler_lazy.mp4

and the corresponding training with small learning rate (or large batch size, low dropout, small noise, ...) of a non-linear DNN:

dnn_lazy.mp4

---

When quick and jerky pulling:

ruler_active.mp4

and the corresponding training (or small batch size, high dropout, large noise, ...) of a non-linear DNN:

dnn_active.mp4

Reproducing phase diagrams in real neural network

./bash/test_pd_ns_nl.sh   # data noise
./bash/test_pd_lr_nl.sh   # learning rate
./bash/test_pd_dp_nl.sh   # drop out
./bash/test_pd_bm_nl.sh   # batch size

CNN experiments

./bash/test_cnn.sh        # CNN experiments 

Spring block simulation

In the first notebook, we solve the first-order overdamping system by discrete iteration.

In the second notebook, we solve the second-order stochastic differential equation using sdeint.