NVIDIA · NCA-GENL

The efficiency thread

Representation and cost tradeoffs at the mechanical level: what tokenization throws away, why attention is quadratic, what an eval metric actually computes, how LoRA cuts fine-tuning memory, and how quantization and batching cut serving cost.

T4 · 48 lessons across 5 modules

  1. M0202-01Why text must be converted to numbers
  2. M0202-02Tokens, vocabulary, and subword tokenization
  3. M0202-03Counting tokens: why tokens are not words
  4. M0202-04BPE vs WordPiece vs SentencePiece: which model uses which
  5. M0202-05Stemming vs lemmatization, and stop-word removal
  6. M0202-06Bag-of-words, TF-IDF, and n-grams
  7. M0404-01Self-attention and why context length costs quadratically
  8. M0404-02Positional encoding: how transformers represent word order
  9. M0404-03Encoder-only vs decoder-only vs encoder-decoder models (BERT, GPT, T5)
  10. M0404-04Autoregressive generation: how an LLM produces text
  11. M0404-05Temperature, top-k, top-p, and greedy decoding
  12. M0404-06The context window: what it is and how to budget it
  13. M0909-01Scaling an evaluation set to a hundred items
  14. M0909-02Perplexity: what it measures and what it misses
  15. M0909-03Human evaluation: rubrics and inter-annotator agreement
  16. M0909-04BERTScore and embedding-based evaluation metrics
  17. M0909-05How to choose an evaluation metric: loss functions, R², precision vs recall
  18. M0909-06BLEU vs ROUGE vs exact match: which metric for which task
  19. M0909-07RAG evaluation metrics: faithfulness, relevance, and context recall
  20. M0909-08Cross-validation: k-fold, stratified, and when not to use it
  21. M0909-09Sample size and statistical significance in LLM evaluation
  22. M0909-10LLM-as-a-judge: how it works and where it fails
  23. M0909-11Reproducibility: why temperature 0 is not deterministic
  24. M0909-12Why LLMs hallucinate, and the types of hallucination
  25. M0909-13Error analysis: turning failures into a fix list
  26. M1111-01Pretraining vs continued pretraining vs instruction tuning
  27. M1111-02Supervised fine-tuning (SFT): what it can and cannot change
  28. M1111-03Catastrophic forgetting when fine-tuning
  29. M1111-04GPU memory requirements for training an LLM
  30. M1111-05LoRA and parameter-efficient fine-tuning (PEFT)
  31. M1111-06RLHF: reinforcement learning from human feedback explained
  32. M1111-07Reward models, reward hacking, and preference data
  33. M1111-08Prompt vs RAG vs fine-tuning: the full decision rule
  34. M1111-09Choosing a model adaptation strategy under real constraints
  35. M1212-01Numeric precision: FP32, TF32, FP16, BF16, and INT8
  36. M1212-02Quantization: PTQ vs QAT and recovering accuracy
  37. M1212-03Reading loss curves to diagnose a training run
  38. M1212-04Distributed training: data parallelism, AllReduce, and NCCL
  39. M1212-05The KV cache and why LLM generation is memory-bound
  40. M1212-06Batching for inference: static, dynamic, and continuous
  41. M1212-07PagedAttention and vLLM: virtual memory for the KV cache
  42. M1212-08ONNX, TensorRT, and TensorRT-LLM: compiling a model for inference
  43. M1212-09LLM cost per million tokens, per request, and per month
  44. M1212-10Latency and throughput: TTFT, tokens per second, and p95
  45. M1212-11Multi-turn chat history and query rewriting
  46. M1212-12Keeping a vector index fresh: re-embedding and migration
  47. M1212-13Deploying with NVIDIA Triton Inference Server and NIM
  48. M1212-14Monitoring an LLM in production and detecting drift

Part of the throughlines running across the NCA-GENL prep course.