NVIDIA · NCA-GENL

The measurement thread

How you'd know the model is wrong. Opens in M01 building a crude 20-item eval set, re-runs that same instrument through RAG, metrics, experimentation and fine-tuning, and closes in M12 when production monitoring uses it to catch drift — the build's validator becomes the decay detector.

T1 · 95 lessons across 12 modules

  1. M0101-01Next-token prediction: what a language model is trained to do
  2. M0101-02LLM parameters: what they are and where knowledge is stored
  3. M0101-03Tensor shapes in transformers: batch, sequence, and hidden size
  4. M0101-04Vectors, dot products, and cosine similarity
  5. M0101-05Loss functions and cross-entropy explained
  6. M0101-06Gradient descent and backpropagation
  7. M0101-07Train, validation, and test splits
  8. M0101-08How to build an evaluation set for an LLM project
  9. M0202-01Why text must be converted to numbers
  10. M0202-02Tokens, vocabulary, and subword tokenization
  11. M0202-03Counting tokens: why tokens are not words
  12. M0202-04BPE vs WordPiece vs SentencePiece: which model uses which
  13. M0202-05Stemming vs lemmatization, and stop-word removal
  14. M0202-06Bag-of-words, TF-IDF, and n-grams
  15. M0303-01What text embeddings are: learned dense vectors
  16. M0303-02Token embeddings vs sentence and document embeddings
  17. M0303-03How to choose an embedding model
  18. M0303-04How to test retrieval quality by hand
  19. M0303-05Vector arithmetic and word analogies (word2vec)
  20. M0505-01Zero-shot vs few-shot prompting and in-context learning
  21. M0505-02How to structure a prompt: instruction, context, and format
  22. M0505-03Chain-of-thought prompting: when it helps and when it misleads
  23. M0505-04Prompt templates, versioning, and testing
  24. M0505-05Getting structured JSON output from an LLM
  25. M0505-06Prompt, RAG, or fine-tune? A first decision rule
  26. M0606-01Document parsing for RAG: PDFs, tables, and silent failures
  27. M0606-02Chunking strategies for RAG: fixed, recursive, and semantic
  28. M0606-03Metadata in RAG: what to embed versus what to return
  29. M0606-04Deduplication and corpus cleaning for RAG
  30. M0707-01Sparse retrieval and BM25 keyword search
  31. M0707-02Dense retrieval with embeddings
  32. M0707-03Limits of embedding search: negation, recency, and authority
  33. M0707-04Vector databases and ANN indexes (HNSW, IVF)
  34. M0707-05Access control and permissions in RAG retrieval
  35. M0707-06Hybrid search: combining keyword and vector retrieval
  36. M0707-07Reranking with a cross-encoder
  37. M0707-08Assembling context: chunk order and the lost-in-the-middle problem
  38. M0707-09The complete RAG pipeline, stage by stage
  39. M0707-10Debugging RAG: retrieval failure versus generation failure
  40. M0707-11Grounding, citations, and letting a model say I don't know
  41. M0707-12When RAG is the wrong tool
  42. M0808-01How to curate a dataset for an LLM task
  43. M0808-02Data quality problems: label noise, leakage, imbalance, and drift
  44. M0808-03Exploratory data analysis (EDA) on a text corpus
  45. M0808-04Choosing the right chart: histogram, box plot, scatter, bar, heatmap, line
  46. M0808-05NVIDIA RAPIDS: cuDF, cuML, and cuGraph for GPU data science
  47. M0909-01Scaling an evaluation set to a hundred items
  48. M0909-02Perplexity: what it measures and what it misses
  49. M0909-03Human evaluation: rubrics and inter-annotator agreement
  50. M0909-04BERTScore and embedding-based evaluation metrics
  51. M0909-05How to choose an evaluation metric: loss functions, R², precision vs recall
  52. M0909-06BLEU vs ROUGE vs exact match: which metric for which task
  53. M0909-07RAG evaluation metrics: faithfulness, relevance, and context recall
  54. M0909-08Cross-validation: k-fold, stratified, and when not to use it
  55. M0909-09Sample size and statistical significance in LLM evaluation
  56. M0909-10LLM-as-a-judge: how it works and where it fails
  57. M0909-11Reproducibility: why temperature 0 is not deterministic
  58. M0909-12Why LLMs hallucinate, and the types of hallucination
  59. M0909-13Error analysis: turning failures into a fix list
  60. M1010-01Public benchmarks (GLUE, MMLU) and data contamination
  61. M1010-02Zero-shot and few-shot capability testing
  62. M1010-03A/B testing an LLM feature in production
  63. M1010-04Regression testing an LLM system in CI/CD
  64. M1111-01Pretraining vs continued pretraining vs instruction tuning
  65. M1111-02Supervised fine-tuning (SFT): what it can and cannot change
  66. M1111-03Catastrophic forgetting when fine-tuning
  67. M1111-04GPU memory requirements for training an LLM
  68. M1111-05LoRA and parameter-efficient fine-tuning (PEFT)
  69. M1111-06RLHF: reinforcement learning from human feedback explained
  70. M1111-07Reward models, reward hacking, and preference data
  71. M1111-08Prompt vs RAG vs fine-tuning: the full decision rule
  72. M1111-09Choosing a model adaptation strategy under real constraints
  73. M1212-01Numeric precision: FP32, TF32, FP16, BF16, and INT8
  74. M1212-02Quantization: PTQ vs QAT and recovering accuracy
  75. M1212-03Reading loss curves to diagnose a training run
  76. M1212-04Distributed training: data parallelism, AllReduce, and NCCL
  77. M1212-05The KV cache and why LLM generation is memory-bound
  78. M1212-06Batching for inference: static, dynamic, and continuous
  79. M1212-07PagedAttention and vLLM: virtual memory for the KV cache
  80. M1212-08ONNX, TensorRT, and TensorRT-LLM: compiling a model for inference
  81. M1212-09LLM cost per million tokens, per request, and per month
  82. M1212-10Latency and throughput: TTFT, tokens per second, and p95
  83. M1212-11Multi-turn chat history and query rewriting
  84. M1212-12Keeping a vector index fresh: re-embedding and migration
  85. M1212-13Deploying with NVIDIA Triton Inference Server and NIM
  86. M1212-14Monitoring an LLM in production and detecting drift
  87. M1313-01NVIDIA's four pillars of trustworthy AI and how to implement each
  88. M1313-02NVIDIA NeMo Guardrails and content moderation for LLM applications
  89. M1313-03Prompt injection and indirect injection through RAG
  90. M1313-04Bias in AI: sources, measurement, and mitigation
  91. M1313-05Data privacy, consent, and why model weights cannot forget
  92. M1313-06Transparency, explainability, and model cards for LLM systems
  93. M1313-07AI energy use, carbon, and efficient inference
  94. M1313-08A trustworthy AI checklist for your own LLM service
  95. M1313-09How to read an AI research paper and track new LLM trends

Part of the throughlines running across the NCA-GENL prep course.