A computational framework bridging Neural Resonance Theory and Deep Learning to develop AI systems that perceive temporal structure through oscillatory dynamics rather than statistical shortcuts.
Neurodynamics | EEG Integration | Visual Perception | ML Surrogate | Installation
The Neural Duet Engine transforms EEG signals into real-time music synthesis through coupled oscillator dynamics.
Contemporary AI systems achieve remarkable performance through statistical pattern matching, yet they fundamentally lack the dynamical principles that organize biological perception. This research addresses that gap by implementing Neural Resonance Theory (Large, 2010) and the ASHLE model (Roman et al., 2023) within a deep learning framework.
The core contribution is a Temporal Convolutional Network (TCN) surrogate trained with physics-informed loss to internalize the differential equations governing neural oscillators. This surrogate achieves 94.37% Phase-Locking Value (PLV) retention while enabling real-time audio synthesis driven by learned dynamics. The framework demonstrates applications across three modalities: auditory rhythm perception, EEG-driven brain-computer music interfaces, and cross-modal visual-to-audio entrainment.
TL;DR: Training a neural network to think like a brain oscillator, achieving 94.37% fidelity. The result: AI that genuinely perceives rhythm instead of pattern-matching.
Imagine tapping your foot to music—your brain naturally synchronizes with the beat through neural oscillations. Current AI systems lack this ability entirely; they recognize patterns but don't feel rhythm. This project teaches AI to perceive temporal structure the way humans do, using the same mathematical principles that govern your neurons. The result is a system that can:
- Listen to music and synchronize its internal "heartbeat" to the rhythm
- Watch videos and generate music that matches the visual motion
- Read brain waves (EEG) and compose music reflecting your mental state
Notebook: dedAI_Neurodynamics.ipynb
This module establishes the biophysical foundations of rhythm perception through three interconnected oscillator models: the Hopf oscillator (canonical bifurcation dynamics), GrFNN (Gradient Frequency Neural Network for beat perception), and ASHLE (Adaptive Synchronization with Hebbian Learning and Elasticity).
A critical validation of Neural Resonance Theory is its capacity to model individual differences in rhythm perception. By varying the Hebbian learning rate (λ₁) and elasticity (λ₂) parameters, the framework successfully distinguishes between musician and non-musician populations.
Figure 1: Statistically significant difference in phase-locking (p=0.0158, two-tailed t-test, n=50 simulated subjects per group) between modeled musician and non-musician populations. Musicians exhibit higher PLV (0.89 vs 0.72) and faster entrainment.
The GrFNN demonstrates robust beat detection through resonance rather than autocorrelation, successfully resolving the fundamental frequency even in the presence of strong harmonics.
Figure 2: Zero-error fundamental frequency detection (2.0 Hz) despite harmonic interference (n=30 oscillators, frequency range 0.5–8.0 Hz). The network's nonlinear dynamics naturally suppress subharmonic responses.
Western music theory's consonance hierarchy emerges naturally from oscillator coupling dynamics, providing a principled explanation for why certain intervals "sound good."
Figure 3: Dynamical stability (measured via PLV) correlates strongly with Western music theory interval rankings (r=0.834, p<0.001, n=12 intervals tested). Perfect consonances (unison, octave, fifth) exhibit highest stability.
Click to expand: Phase Portraits & Hopf Bifurcation
Phase space trajectories showing limit cycle behavior across different bifurcation parameters (α = -0.5, 0.0, 0.5, 1.0).
Resonance response of the Hopf oscillator to periodic forcing at various frequencies (0.5–4.0 Hz).
Notebook: dedAI_EEG_Integration.ipynb
This module implements a closed-loop Brain-Computer Music Interface (BCMI) using an Emotiv Epoc X digital twin—a simulated 14-channel EEG system that generates realistic brainwave patterns for different cognitive states, enabling development without physical hardware. Cognitive states extracted from EEG signals directly modulate oscillator parameters, creating music that reflects the user's mental state in real-time.
Figure 4: Real-time dashboard showing EEG biomarker extraction and mapping to Neural Duet synthesis parameters. Left: raw EEG timeseries (14 channels). Center: extracted cognitive metrics (Stress, Attention, Engagement, Relaxation). Right: resulting oscillator dynamics. End-to-end latency: <50ms.
The key insight is that relaxation inversely correlates with motor elasticity (λ₂)—relaxed states produce more adaptive, "loose" rhythms, while stressed states yield rigid, metronomic timing.
Figure 5: Mapping between EEG-derived cognitive states and ASHLE/Lorenz parameters (n=5 cognitive states tested, 30s per state). Note the inverse correlation between Relaxation and Elasticity (λ₂), producing more adaptive rhythms during relaxed states.
The following audio samples demonstrate how different cognitive states produce distinct musical characteristics:
| Cognitive State | Audio Sample | Characteristics |
|---|---|---|
| Stressed | neural_duet_stressed.wav | High ρ → chaotic melody, rigid timing |
| Focused | neural_duet_focused.wav | Balanced parameters → structured groove |
| Relaxed | neural_duet_relaxed_awake.wav | Low λ₂ → adaptive, flowing rhythm |
| Meditative | neural_duet_meditative.wav | Minimal chaos → ambient, spacious |
Notebook: dedAI_Visual_Perception.ipynb
This module extends Neural Resonance Theory to visual perception, demonstrating cross-modal entrainment where video motion drives musical rhythm. The framework extracts motion energy from video and couples it to the ASHLE drummer, while vertical position modulates the Lorenz melodist. Applications include: accessibility tools for the visually impaired, game audio, and experimental music composition.
Figure 6: Motion energy and vertical position extraction from three visual stimuli: bouncing ball (periodic), drifting grating (continuous), and biological motion walker (complex). Frame rate: 30 fps, duration: 10s per stimulus.
Figure 7: Complete visual-to-audio pipeline showing stimulus frames, extracted features, oscillator dynamics, and synthesized waveforms. Motion-rhythm correlation: r=0.91 for bouncing ball stimulus.
| Visual Stimulus | Audio Sample | Description |
|---|---|---|
| Bouncing Ball (Clean) | visual_duet_ball_clean.wav | Periodic motion → regular beat |
| Bouncing Ball (Noisy) | visual_duet_ball_noisy.wav | Stochastic → irregular groove |
| Drifting Grating | visual_duet_grating.wav | Continuous → drone-like |
| Biological Walker | visual_duet_walker.wav | Complex → humanized rhythm |
Notebook: dedAI_ML_Component.ipynb
The central machine learning contribution is a Temporal Convolutional Network (TCN) trained to approximate the differential equations of the ASHLE model. This surrogate enables gradient-based optimization and real-time inference while preserving the physical properties of neural entrainment.
The model is trained with a composite loss function that enforces physical constraints:
L_total = L_mse + λ_plv * L_plv + λ_freq * L_freq + λ_amp * L_ampWhere the PLV loss ensures the learned dynamics maintain phase-locking:
L_plv = 1 - |mean(exp(i * (φ_pred - φ_target)))|
Figure 8: Training convergence over 200 epochs showing negative generalization gap (validation loss below training loss)—a hallmark of physics-informed learning where the model learns generalizable dynamics rather than overfitting to noise. Training dataset: n=50 synthetic songs, 30s each, 80–160 BPM.
The critical validation is whether the surrogate preserves oscillator physics. The figure below compares ground truth ASHLE dynamics with TCN predictions across a novel test sequence.
Figure 9: AI Surrogate (red) accurately tracking Ground Truth ASHLE physics (blue) without temporal drift. Phase, frequency, and amplitude dynamics are faithfully reproduced over 30s test sequences (held-out data).
| Metric | Value | Description |
|---|---|---|
| PLV Retention | 94.37% | Phase-locking preservation (target: >90%) |
| Best Validation Loss | 0.740 | Composite physics-informed loss |
| Frequency Correlation | 0.606 | Pearson r for motor frequency |
| Phase Correlation | 0.417 | Pearson r for motor phase |
| Ground Truth PLV | 0.921 | Mean PLV from ASHLE simulation |
| AI Surrogate PLV | 0.763 | Mean PLV from TCN predictions |
| Parameter | Value |
|---|---|
| Architecture | 6-layer TCN + Causal Multi-Head Attention |
| Hidden Dimension | 128 |
| Kernel Size | 5 |
| Dilation Pattern | 2^i (exponential) |
| Dropout | 0.15 |
| Training Epochs | 200 |
| Batch Size | 64 |
| Learning Rate | 0.001 |
| Curriculum | 80-160 BPM tempo range |
| Resource | Specification |
|---|---|
| Training Time | ~45 minutes |
| GPU | NVIDIA RTX 3080 Ti (12GB VRAM, 16GB shared, 28GB total) |
| CPU | Intel Core i9-9900K (8 cores, 16 threads, 5.0 GHz turbo) |
| RAM | 32GB DDR4 |
Training Session Metadata
- Session ID:
ml_v3_20260102_162052 - ASHLE Parameters: f₀=2.0 Hz, λ₁=0.5, λ₂=2.0, γ=0.01
- Lorenz Parameters: σ=10.0, ρ=28.0, β=2.67, dt=0.02
- GrFNN: 30 oscillators, frequency range 0.5-8.0 Hz
- Sample Rates: 44.1 kHz (audio), 256 Hz (neural dynamics)
- Dataset: 50 synthetic songs × 30s = 25 minutes of training data
Ablation: Importance of PLV Loss
| Training Configuration | PLV Retention |
|---|---|
| Full loss (MSE + PLV + Freq + Amp) | 94.37% |
| Without PLV loss | 78.2% |
| Without Frequency loss | 89.1% |
| MSE only | 71.5% |
The PLV loss is critical for preserving synchronization behavior.
The synthesis pipeline couples three dynamical systems to generate coherent electronic music. The architecture is presented in three layers for clarity:
The following audio samples demonstrate the Neural Duet engine across different experimental conditions:
| Source | Audio Sample | Description |
|---|---|---|
| Neurodynamics Session | neural_duet.wav | Full ASHLE + Lorenz + GrFNN synthesis |
| ML Ground Truth | neural_music.wav | Ground truth ASHLE-driven drums |
| EEG-Driven | neural_duet_eeg.wav | Real-time EEG biomarker modulation |
DedAI-Neurodynamics/
│
├── dedAI_Neurodynamics.ipynb # Module 1: Foundations of rhythm perception
├── dedAI_EEG_Integration.ipynb # Module 2: Brain-Computer Music Interface
├── dedAI_Visual_Perception.ipynb # Module 3: Cross-modal entrainment
├── dedAI_ML_Component.ipynb # Module 4: TCN surrogate training
│
├── src/
│ └── pynrt/ # Neural Resonance Theory toolkit (v1.0.0)
│ ├── core/ # Oscillator implementations
│ │ ├── hopf.py # HopfOscillator - canonical bifurcation
│ │ ├── grfnn.py # GrFNN - beat perception network
│ │ └── ashle.py # ASHLE - dual oscillator model
│ ├── audio/ # Audio synthesis
│ │ ├── synthesis.py # NeuralSynthesizer, LorenzOscillator
│ │ ├── stimulus.py # Metronome, click generation
│ │ └── realtime.py # Real-time streaming
│ ├── analysis/ # Signal analysis
│ │ ├── entrainment.py # PLV, coherence metrics
│ │ └── stability.py # Bifurcation analysis
│ ├── visual/ # Visual perception
│ │ └── perception.py # Motion extraction from video
│ └── visualization/ # Plotting utilities
│
├── ml_utils/ # ML pipeline utilities
│ ├── run_inference_complete.py # End-to-end groove extraction
│ ├── sonification.py # Dynamics-to-audio mapping
│ └── mixing.py # Audio mastering
│
├── models/ # Trained model weights
│ └── ml_v3_*_best_model.pt # TCN surrogate checkpoint
│
├── outputs/
│ ├── figures/ # Publication-ready visualizations (18 PNGs)
│ └── sessions/ # Experimental session data
│ ├── 20251230_212609/ # Neurodynamics session (audio + data)
│ ├── eeg_bci_20251230_*/ # EEG-BCMI session (5 cognitive states)
│ ├── ml_v3_20260102_*/ # ML training session (metrics + model)
│ └── visual_duet_*/ # Visual perception session (4 stimuli)
│
├── tests/ # Unit tests (pytest)
│ ├── test_hopf.py # Hopf oscillator tests
│ ├── test_ashle.py # ASHLE model tests
│ ├── test_grfnn.py # GrFNN network tests
│ └── test_stability.py # Bifurcation analysis tests
│
├── docs/ # Documentation
│ ├── ARCHITECTURE.md # System design details
│ └── QUICKSTART.md # 5-minute getting started guide
│
├── EEGutils.py # EEG processing utilities
├── utils.py # Session management
├── pyproject.toml # Package configuration (PEP 517)
├── requirements.txt # Dependencies
├── Makefile # Common commands
├── CITATION.cff # Citation metadata
└── LICENSE # MIT License
# Clone the repository
git clone https://github.com/hawksp/DedAI-Neurodynamics
cd DedAI-Neurodynamics
# Create and activate virtual environment
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
# Install package in editable mode
pip install -e .
# Verify installation
python -c "from pynrt import ASHLE, GrFNN, HopfOscillator; print('Installation OK')"
# For development (testing, linting)
pip install -e ".[dev]"
# For EEG support (MNE-Python)
pip install -e ".[all]"- Python 3.10+
- PyTorch 2.0+
- NumPy, SciPy, Pandas
- librosa, soundfile
- matplotlib, Jupyter
Troubleshooting
librosa installation fails (FFmpeg not found)
# Ubuntu/Debian
sudo apt-get install ffmpeg
# macOS
brew install ffmpeg
# Windows (via conda)
conda install -c conda-forge ffmpegPyTorch CUDA issues
# Check CUDA availability
python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}')"
# Install CPU-only version if needed
pip install torch --index-url https://download.pytorch.org/whl/cpuImport errors after installation
# Ensure you're in the correct virtual environment
which python # Should show .venv/bin/python
# Reinstall in editable mode
pip install -e . --force-reinstallJupyter kernel not finding pynrt
# Install kernel with the virtual environment
python -m ipykernel install --user --name=dedai --display-name="DedAI"Extract neural dynamics and generate AI drums from any audio file:
python ml_utils/run_inference_complete.py input.wav \
-m models/ml_v3_20260102_162052_best_model.pt \
-o outputs/Generated outputs:
| File | Description |
|---|---|
*_groove_analysis.png |
5-panel visualization |
*_groove_metrics.csv |
Phase, frequency, amplitude time series |
*_drum_events.csv |
Detected kick, snare, hi-hat events |
*_ai_drums.wav |
Procedurally synthesized drums |
*_mix_preview.wav |
Original + AI drums mix |
from pynrt import ASHLE
# Create model with 2 Hz spontaneous motor tempo
model = ASHLE(f0=2.0, lambda1=0.5, lambda2=2.0)
# Simulate synchronization to a 2.2 Hz metronome
t, (z_s, z_m, f_s, f_m) = model.simulate_metronome(tempo=2.2, duration=20.0)
# Compute Phase-Locking Value
plv = model.compute_plv(z_s, z_m)
print(f"Phase-Locking Value: {plv:.3f}")make test # Run all unit tests
make lint # Check code style
make train # Retrain TCN model
make demo # Run inference on sample audio
make clean # Remove generated filesLarge, E. W. (2010). Neurodynamics of music. In M. Riess Jones, R. R. Fay, & A. N. Popper (Eds.), Music Perception (Springer Handbook of Auditory Research, Vol. 36, pp. 201-231). Springer. https://doi.org/10.1007/978-1-4419-6114-3_7
Roman, I. R., Washburn, A., Large, E. W., Chafe, C., & Fujioka, T. (2023). ASHLE: Adaptive synchronization for Hebbian learning and elasticity in sensorimotor interaction. bioRxiv. https://doi.org/10.1101/2023.07.06.548000
Kim, J. C., & Large, E. W. (2015). Signal processing in periodically forced gradient frequency neural networks. Frontiers in Computational Neuroscience, 9, Article 152. https://doi.org/10.3389/fncom.2015.00152
Large, E. W., Herrera, J. A., & Velasco, M. J. (2015). Neural networks for beat perception in musical rhythm. Frontiers in Systems Neuroscience, 9, Article 159. https://doi.org/10.3389/fnsys.2015.00159
Toussaint, G. (2005). The Euclidean algorithm generates traditional musical rhythms. Proceedings of BRIDGES: Mathematical Connections in Art, Music, and Science, 47-56.
Further Reading
- Hopf Bifurcation: Strogatz, S. H. (2015). Nonlinear Dynamics and Chaos. Westview Press.
- Music Cognition: Huron, D. (2006). Sweet Anticipation: Music and the Psychology of Expectation. MIT Press.
- EEG-Music: Daly, I., et al. (2016). Music-induced emotions can be predicted from a combination of brain activity and acoustic features. Brain and Cognition, 101, 1-11.
- Physics-Informed ML: Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019). Physics-informed neural networks. Journal of Computational Physics, 378, 686-707.
This project is licensed under the MIT License - see LICENSE for details.
Elliott Mitchell
This research explores the intersection of computational neuroscience, machine learning, and music cognition to develop AI systems that perceive temporal structure as humans do.
PhD Application: Neural Dynamics of Perceptually Aligned Artificial Intelligence
Queen Mary University of London
Supervisor: Dr. Iran Roman