A Friday 12:10–3:10 p.m. face-to-face undergraduate course for students with little or no deep learning background. The course starts each topic with a working demo and live Claude Code / PyTorch interaction, then reverse-engineers the math, systems, and modeling ideas behind the demo.
Every technical unit begins with a visible artifact: a classifier learning, an attention map, BPE token IDs, a tiny GPT generating text, MNIST reconstructions, a diffusion denoising path, or a toy adversarial attack.
After the demo works, the class unpacks the details: shapes, gradients, loss functions, optimization, data assumptions, inductive bias, compute/memory bottlenecks, and failure modes.
Slides are not the main artifact. Students build a paper reference manual in class. Missing class means learning from another student’s notes.
Claude Code is used as an instructor-operated live coding partner, not as a replacement for understanding. Typical prompts will ask it to scaffold a PyTorch file, add printouts for tensor shapes, generate small tests, convert equations into code, debug an error, or refactor a notebook into a readable script. Students are expected to critique the output: What is correct? What is hidden? What is inefficient? What would fail on a real dataset?
No student needs a paid AI account for graded work. All graded assessment is paper-based.
By the end of the course, students should be able to:
| Assessment | Weight | Format |
|---|---|---|
| Midterm exam | 50% | In class, paper, open paper materials, no electronics. |
| Final exam | 50% | In class during the final regular meeting, paper, open paper materials, no electronics. This requires department/dean confirmation before publication. |
| Optional bonus project | +0–10 pts | Extra credit only. GPT/Claude may be used as an advisory grading assistant under the rubric below; the instructor assigns the final score. |
| Attendance bonus | 0 or +5 pts | Extra credit only. Five random in-class attendance checks will be administered; students present for all five receive +5 points. No partial attendance bonus. |
| Homework | 0% | No required written homework is collected or graded. |
| In-class tasks | 0% | Used for attendance and practice, not as a grade component. |
Unless a question explicitly states a different rubric:
This rewards calibrated uncertainty. Students should not bluff. Leave a part blank if you cannot support the answer.
Students may complete one optional mini-project for up to 10 bonus points added after the midterm/final weighted average. The project should extend a live class demo, reproduce a small paper idea, build a clearer visualization, or critique a model failure mode. It is intended as enrichment, not a required workload.
| Criterion | Points | What earns credit |
|---|---|---|
| Novelty / ambition | 0–4 | A creative extension beyond copying a notebook: new data, new ablation, new visualization, new failure case, or a thoughtful paper-to-demo translation. |
| Technical solidity | 0–4 | The idea runs, the code or derivation is coherent, claims are supported by evidence, and limitations are stated honestly. |
| Clarity / reproducibility | 0–2 | The submission is easy to inspect: short README, clear figures, minimal commands, and a concise explanation of what was learned. |
AI-assisted bonus grading: GPT/Claude may be used to produce an initial rubric-based critique of de-identified project materials, especially for novelty and technical solidity. The model score is advisory; the instructor or teaching staff make the final decision, may override model output, and will not upload unnecessary student identifiers or private student records to public AI tools. Students may request human review of the bonus score.
Maximum extra credit: Students may earn up to +15 points total: up to +10 from the optional project and +5 from the attendance bonus.
Attendance is recorded through short in-class tasks. These tasks are meant to make students actively process the material: draw the computation graph, compute one gradient step, write BPE merges, label Q/K/V tensor shapes, explain one failure mode, or annotate a printed code snippet. Tasks are not homework and are not a regular grade component.
Attendance bonus: The instructor will take attendance during five random class meetings. Students who are present for all five random attendance checks receive +5 bonus points added after the midterm/final average and any project bonus. Missing one or more random checks earns no attendance bonus, but it is not a grade penalty. Rutgers-approved absences and accommodations will be handled under university and departmental procedures.
Rutgers calendar facts used: Fall 2026 classes begin Tuesday, September 1; regular classes end Thursday, December 10; Friday classes meet on Wednesday, November 25; Thanksgiving recess runs Thursday, November 26 through Sunday, November 29; reading days are Friday, December 11 and Monday, December 14; fall exams run Tuesday, December 15 through Tuesday, December 22. Regular class meetings for this section are Fridays, 12:10–3:10 p.m., in Lucy Stone Hall (LSH-A143), Livingston Campus.
Instructor-preferred final plan: This draft places the 50% final exam in class on Friday, December 4, 2026, 12:10–3:10 p.m. so the course has no separate registrar-period meeting. Rutgers policy normally places cumulative finals in the official final exam period unless the instructional dean approves an exception; obtain department/dean confirmation before using this line in the published syllabus.
Invited talks: The course will include two invited talks during regular class meetings. Tentatively, one talk will be paired with the video / multimodal generation unit and one with the world-model / agentic AI / systems unit; exact speakers and dates may change.
| Week | Date | Topic | Live demo / in-class task | Exam must-know |
|---|---|---|---|---|
| 1 | Fri Sep 4 | Core What is deep learning? Tensors, data, parameters, loss, gradients, and training loops. | Claude Code builds a tiny classifier; students draw the data → logits → loss → gradient loop. | Define model, parameter, loss, gradient, batch, epoch, overfitting. |
| 2 | Fri Sep 11 | Core Gradient descent, autograd, MLPs, backpropagation, handwritten PyTorch training loop. | Implement linear regression and MLP from scratch; inspect gradients before/after one update. | Compute one SGD update; trace forward/backward pass; identify learning rate effects. |
| 3 | Fri Sep 18 | Core From standard convolution to CNNs and ResNets; images as tensors; local connectivity, weight sharing, residual learning. | Visualize filters and feature maps; add residual connection to a toy network. | Convolution shape math; why CNNs generalize better than dense MLPs on images; why residuals help optimization. |
| 4 | Fri Sep 25 | Core Correlation learning, spurious features, robustness, adversarial examples, curse of dimensionality, Gaussian donuts. | Train on a spurious color cue, then test after cue flip; generate a small adversarial perturbation. | Explain shortcut learning, distribution shift, adversarial perturbation, high-dimensional intuition. |
| 5 | Fri Oct 2 | Core BPE tokenization, embeddings, attention, transformer block, RoPE positional encoding, ViT. | Print BPE integer tokens and embedding vectors; visualize one attention head. | Calculate attention tensor shapes; explain token IDs, embeddings, Q/K/V, softmax attention, positional information. |
| 6 | Fri Oct 9 | Core NanoGPT in class: train a tiny Shakespeare model; generation; transformer systems intro: KV cache, grouped KV, MoE, gradient checkpointing, FlashAttention. | Run NanoGPT-style mini training; inspect generated text; compare prefill vs decode. | Autoregressive loss; causal mask; KV cache purpose; why attention is memory/IO heavy. |
| 7 | Fri Oct 16 | Midterm In-class midterm exam, 50% of course grade. | Paper exam. Unlimited paper materials. No electronics or AI. | Weeks 1–6. |
| 8 | Fri Oct 23 | Core Self-supervised representation learning: SimCLR, triplet loss, contrastive learning, Siamese trivial solution, CLIP, SigLIP, distillation, GKD/OPD. | Build two augmented views; compute contrastive similarities; show collapse in a naive Siamese objective. | Positive/negative pairs; temperature; trivial solution; image-text alignment; teacher-student distillation. |
| 9 | Fri Oct 30 | Core Probabilistic generative models: ELBO derivation, VAE, VQ-VAE, latent bottlenecks, why VAEs/dimensionality reduction matter for image/video generation. | Train a small MNIST VAE; interpolate latent codes; compare reconstruction vs sampling. | ELBO terms; encoder/decoder; reparameterization trick; reconstruction vs KL; latent compression. |
| 10 | Fri Nov 6 | Core Diffusion and flows: denoising objective, DDPM intuition, score matching, normalizing flows, rectified flow, MeanFlow, DMD and one-step/few-step generation. | Animate noise → image denoising on 2D toy data; compare iterative sampling to one-step distillation. | Forward noising; reverse denoising; score; FID intuition; flow vs diffusion; distribution matching. |
| 11 | Fri Nov 13 | Frontier Invited talk 1 Video and unified multimodal generation: long-video challenge, teacher forcing, diffusion forcing, self forcing, causal forcing, RAE, Transfusion, UniFluid, Janus-style unified models. | Invited talk plus sketch of error accumulation in autoregressive video; discuss why low-dimensional video latents are necessary. | Teacher forcing vs exposure bias; why video is harder than images; continuous vs discrete tokens; unified modeling trade-offs. |
| 12 | Fri Nov 20 | Core Reinforcement learning and post-training: RL, policy, reward, RLHF, DPO, GRPO, preference learning, alignment, agent evaluation. | Paper game: students act as policy, reward model, preference judge, optimizer; derive why preference pairs can train behavior. | Policy/reward/value; preference data; DPO idea; GRPO group baseline idea; why post-training changes behavior. |
| 13 | Wed Nov 25 Rutgers Friday classes |
Frontier Invited talk 2 World models, agent models, Physical AI, VLA/WAM, System A/B/M, interpretability, and inference systems. | Invited talk plus structured debate on world/agent claims; then draw an LLM inference memory map and interpret one toy SAE feature. | World model vs policy/agent; VLA vs WAM; why interpretability and inference efficiency both depend on internal representations. |
| 14 | Fri Dec 4 | In-class Final Exam 50% of course grade. | Paper exam. Unlimited paper materials. No electronics or AI. | Cumulative, with emphasis on Weeks 8–13. Suggested split: 30% cumulative foundations, 70% post-midterm topics. |
These details are written for the instructor, not just students. Each week has a “demo → explanation → exam skill” structure.
Teach: supervised learning loop, tensors, parameters, logits, loss surfaces, train/test split, why neural networks are function approximators, and why gradients turn one data point into many parameter updates.
Demo: a two-class toy dataset and a tiny classifier. Use Claude Code to generate a minimal PyTorch script, then ask students to catch missing pieces: seed, train/test split, loss, optimizer, visualization.
Notebook task: draw arrows for x → model(x) → loss → backward() → optimizer.step().
Teach: scalar derivative, vector gradient, chain rule, linear layer shape, activation functions, cross entropy, learning rate, minibatches, and why manual backprop becomes tedious.
Demo: first implement a 1D regression update by hand, then switch to PyTorch autograd. Print gradients for each parameter. Show exploding/vanishing behavior with a bad learning rate or deep MLP.
Exam skill: one-step SGD update and shape tracing through Linear → ReLU → Linear.
Teach: convolution as a sliding dot product, padding, stride, channels, kernels, pooling, receptive fields, parameter sharing, and residual connections as an optimization trick.
Demo: Sobel/blur filters on images; then a small CNN. Let Claude Code add a residual block and shape assertions.
Exam skill: compute output shape, explain fewer parameters than an MLP, and explain why y = x + F(x) helps gradients.
Teach: empirical risk vs true risk, distribution shift, shortcut learning, confounders, invariance, adversarial perturbations, high-dimensional geometry, and Gaussian donut intuition.
Demo: train a classifier where color correlates with label; flip color at test time. Then apply a small gradient-sign perturbation.
Exam skill: distinguish correlation from causation in model behavior; explain why small per-pixel noise can matter in high dimension.
Teach: BPE merges, token IDs, embedding tables, Q/K/V projections, scaled dot-product attention, softmax, causal vs bidirectional masks, positional encoding, RoPE intuition, image patches in ViT.
Demo: print BPE integer tokens for a sentence; display embedding lookup; compute one attention head with small matrices; visualize attention weights.
Exam skill: label dimensions in QKᵀ / sqrt(d) and explain why attention mixes token information.
Teach: next-token prediction, causal mask, training vs generation, temperature, context length, prefill/decode, KV cache, grouped-query/multi-query attention, MoE routing, checkpointing, FlashAttention IO story.
Demo: train a tiny Shakespeare-like model; print generated samples as training progresses; inspect one layer’s attention; compare cached vs uncached generation conceptually.
Exam skill: explain why generation is sequential, why KV cache helps, and why attention is not only a FLOP problem but also a memory movement problem.
Format: in-class paper exam. Suggested structure: definitions and calibrated blanks; tensor-shape tracing; one gradient calculation; CNN/ResNet explanation; BPE/attention mechanics; small code-reading question.
Teach: augmentation as supervision, invariance, instance discrimination, triplet loss, SimCLR, collapse/trivial solution, CLIP, SigLIP, distillation, generalized/on-policy distillation intuition.
Demo: two views of same image; compute cosine similarity matrix; show how positives and negatives shape embeddings.
Exam skill: explain how a model learns from labels it created from data structure rather than human labels.
Teach: latent variable model, encoder posterior approximation, decoder likelihood, KL divergence, ELBO as reconstruction term minus regularizer, reparameterization, discrete latents in VQ-VAE, why image/video generators often operate in latent spaces.
Demo: MNIST VAE training and latent interpolation; compare reconstruction quality at different latent dimensions.
Exam skill: state the ELBO and explain each term in words.
Teach: forward noising, denoising network, score intuition, sampling steps, classifier-free guidance, FID as distribution-level metric, normalizing flows, rectified flow, average velocity in MeanFlow, and distribution matching distillation.
Demo: two-dimensional Gaussian donuts: corrupt with noise, learn denoising arrows, sample iteratively; then discuss how distillation compresses sampling steps.
Exam skill: compare autoregressive likelihood, VAE ELBO, diffusion denoising, and flow matching objectives.
Teach: why video is harder than image generation: temporal coherence, compute, data redundancy, long-horizon consistency, action/physics ambiguity. Connect teacher forcing, exposure bias, diffusion forcing, self forcing, causal forcing, RAE, Transfusion, UniFluid, and Janus-like unified understanding/generation.
Invited talk slot: 45–60 minutes for a researcher or industry guest on video generation, diffusion forcing, unified multimodal models, or evaluation of long-horizon generation. Reserve the final segment for student questions and exam-relevant synthesis.
Demo: a toy sequence predictor with compounding errors. Discuss how forcing choices change the train-test gap.
Exam skill: explain why a model that looks good for 2 seconds may fail for 2 minutes.
Teach: state, action, reward, policy, value, exploration, RLHF pipeline, reward models, preference pairs, DPO as direct preference optimization, GRPO as group-relative policy optimization without a separate critic in its common LLM use.
Demo: a small preference dataset: two answers, one preferred. Derive the intuition of making preferred outputs more likely while controlling drift.
Exam skill: distinguish supervised fine-tuning, RLHF, DPO, and GRPO conceptually.
Teach: world model vs policy vs agent model, agentic workflow vs agentive autonomy, VLA vs WAM, Physical AI, embodied prediction/action loops, System A/B/M framing, limitations of current agents, superposition, sparse autoencoders, transcoders, crosscoders, SelfIE, HBM vs SRAM, cache locality, LRU, KV cache strategies, ring attention, all-reduce/network delay, and gradient checkpointing review.
Invited talk slot: 45–60 minutes for a researcher or industry guest on agentic AI, world models, Physical AI, interpretability, or inference systems. Reserve the final segment for connecting the talk to the final-exam study guide.
Demo/debate: give students printed abstracts from “Critiques of World Models,” “Critique of Agent Model,” and “Why AI systems don’t learn…” Ask them to mark claims as definition, architecture, evidence, or speculation. Then use a toy activation matrix and sparse dictionary to explain features; sketch how FlashAttention avoids materializing attention; draw a distributed attention ring.
Exam skill: write a clean distinction between “predict what happens,” “choose what to do,” and “learn how to improve while acting,” then explain why interpretability and inference efficiency both depend on internal representations.
Format: in-class paper final exam during the final regular Friday meeting, Friday, December 4, 2026, 12:10–3:10 p.m. Suggested structure: post-midterm definitions; ELBO explanation; VAE/diffusion/flow comparison; forcing-method comparison; DPO/GRPO question; world-model/agent-model critique; interpretability/systems short answers; one cumulative tensor-shape or gradient question.
Exam skill: use the student’s paper notebook as a compact, searchable personal textbook. Correct answers earn full item credit, blank answers earn partial credit, wrong answers earn no credit unless the item states a different rubric.
Weeks 1–6. Students should bring paper notes that include: core definitions; SGD update formula; MLP and CNN diagrams; convolution shape examples; shortcut-learning example; BPE example; attention equation; causal mask diagram; NanoGPT training loop; KV cache explanation.
Cumulative, with emphasis on Weeks 8–13. Students should bring paper notes that include: contrastive loss diagrams; ELBO derivation; VAE diagram; diffusion/flow comparison; forcing methods; DPO/GRPO comparison; world model vs agent model distinction; interpretability tool definitions; HBM/SRAM and cache diagrams; key technical points from both invited talks.
AI-assisted coding is part of the course during supervised class demos. It is used to accelerate exploration and make debugging visible. Students must still understand every line of code and every equation that appears on the exam.
For the optional bonus project, students may use GPT/Claude or similar tools, but they must disclose meaningful AI assistance and remain responsible for correctness, reproducibility, and explanation. AI use on the bonus project is permitted because the purpose is exploration; AI use during exams is prohibited.
During exams, electronics and AI access are prohibited. This includes devices that are powered off but accessible at the desk. Devices must be stored away. Approved accommodations are handled through Rutgers procedures.
No required written homework is submitted or graded. Optional practice prompts, notebooks, papers, and the optional bonus project may be posted for enrichment and exam preparation. The course relies on live attendance, notebook-taking, and exam readiness. Attendance can earn the separate +5 bonus described above, but missed attendance checks do not subtract points from the base grade.
Rutgers takes academic dishonesty seriously. Suspected violations, including unauthorized electronic or AI assistance during exams, will be reported through Rutgers academic integrity procedures. A useful exam pledge is: “On my honor, I have neither received nor given any unauthorized assistance on this examination.”
Rutgers welcomes students with disabilities into its educational programs. Students who need accommodations should contact the appropriate Rutgers disability services office, complete the ODS process, and request a Letter of Accommodation as early as possible. Accommodations are not retroactive.
Students should notify the instructor as early as possible about conflicts due to religious observance or authenticated absence. Make-up arrangements for exams follow Rutgers policy and departmental procedures.
The course is intentionally undergraduate-first, but it borrows organization and topic coverage from graduate/frontier courses and public books. Readings are recommended, not graded.