Data Modalities and Making Them Neural-Network Ready: Text, Image, Audio, Time-Series, and Geospatial

A modality is neural-network-ready once it has been converted from its native file format or raw signal into a numeric tensor of a fixed, predictable shape that a network's first layer can accept. That definition has three load-bearing words. "Numeric" rules out raw text, raw audio waveforms treated as arbitrary-length byte streams, or an image file's compressed on-disk encoding — none of those are directly usable, whatever their apparent structure. "Tensor" means the representation is an array of numbers with a defined number of dimensions — a vector, a matrix, a stack of matrices — not a variable, freeform data structure. "Fixed, predictable shape" is the part people most often skip past: a network's first layer is built for a specific input size, so "convert to numbers" is not enough on its own; the conversion has to also produce a consistent shape across every example, or padding, truncation, or resampling has to be added on top to force consistency.

This is a stronger requirement than "the data is clean." A dataset can be free of missing values, correctly deduplicated, and perfectly labeled — the Module 2 data-cleaning checklist — and still be nowhere near neural-network-ready, because cleaning operates on values a spreadsheet or dataframe can already represent, while "ready" is about the further step of turning a modality-specific raw signal into the tensor shape a specific model architecture expects. A cleaned CSV of product reviews is not neural-network-ready until the review text has been tokenized; a cleaned folder of product photos is not neural-network-ready until the images have been resized, normalized, and possibly cut into patches.

[GROUND TRUTH] (Sources/nca-genm/domain-1-core-ml-ai.md) states the general principle this lesson specializes: "In multimodal settings each modality needs its own preprocessing — tokenization for text, pixel normalization/patching for images, spectrograms or waveforms for audio — before features can be combined." The phrase "before features can be combined" is the important clause. Readiness is a precondition for fusion, not a replacement for it — this lesson stops at the point where each modality has its own tensor, and the next lesson picks up the separate question of when those tensors get combined into one model's reasoning.

L1 — Intuition: five different raw signals, one common destination

Text arrives as a sequence of characters with no inherent numeric meaning. Images arrive as a grid of pixel intensities that already look numeric but are not yet in a form a network's layers expect. Audio arrives as a continuous pressure wave sampled at some rate, which is numeric from the start but at a resolution and length no network can practically consume raw. Time-series arrives as a sequence of measurements taken at intervals, structurally similar to audio but usually at a far lower sampling rate and often with multiple correlated channels. Geospatial data arrives as coordinates, polygons, or already-gridded raster layers, and depending on which of those it is, it needs a different conversion again. Five different starting points, and in every case the destination is the same kind of object: a fixed-shape numeric tensor.

L2 — Mechanism: the conversion pipeline per modality

Text becomes tokens, then embeddings. [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) names the representation directly: "Tokens → embeddings (BPE/WordPiece subwords)." A tokenizer first splits a string into subword units using a scheme like byte-pair encoding or WordPiece — chosen over whole-word tokenization specifically because it handles a word the tokenizer has never seen by falling back to smaller, previously-seen fragments, rather than mapping every unknown word to a single generic "unknown" token. Each subword token maps to an integer id from a fixed vocabulary, and an embedding layer then maps each integer id to a dense vector. The output is a sequence of vectors, one per token — the tensor shape a transformer's first layer expects.

Images become normalized pixels or patch embeddings. [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) names this as "Normalized pixels / patches → feature maps or patch embeddings." Normalization rescales each pixel's raw 0–255 intensity value onto a smaller, more numerically stable range — commonly 0–1 or a mean-zero, unit-variance range matched to the statistics the encoder was originally trained on. A convolutional pipeline then typically consumes the normalized pixel grid directly, sliding filters over it to build feature maps. A vision-transformer pipeline instead first cuts the image into fixed-size square patches — 16×16 pixels is a common choice — and flattens and linearly projects each patch into a single embedding vector, so an image becomes a sequence of patch embeddings rather than a single dense grid, which is what lets a transformer's attention mechanism operate on it the same way it operates on a sequence of text tokens.

Audio becomes a waveform or spectrogram, turned into frame features. [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) names both options: "Waveform or spectrogram → frame features." A raw waveform is already numeric — a sequence of amplitude samples at, say, 16,000 samples per second — but that sequence is both extremely long and dominated by high-frequency detail a model does not need directly. The far more common route converts the waveform into a spectrogram: the signal is cut into short overlapping windows (commonly 20–25 milliseconds each), a Fourier transform is applied to each window to reveal which frequencies are present and how strongly, and the result is stacked into a 2-D grid of frequency-versus-time — visually and structurally similar to an image, which is part of why convolutional architectures originally built for vision transfer reasonably well to spectrograms. Each column of that grid is one "frame," and the frame-by-frame sequence of frequency-content vectors is the tensor a downstream model actually consumes.

Time-series becomes windowed, normalized sequences. [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) states this plainly: "Windowed/normalized sequences." A raw time-series — sensor readings, stock prices, a patient's vital signs over time — is cut into fixed-length windows (say, the last 60 readings), each window normalized independently or against a running statistic, so the network always receives a same-shaped chunk of history rather than an unboundedly long stream. Multiple correlated channels — temperature and humidity and pressure from the same sensor array, say — stack into extra dimensions of the same windowed tensor rather than requiring separate pipelines.

Geospatial data becomes coordinates or grids, often rasterized. [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md) names this as "Coordinates/grids, often rasterized." Point data — a single latitude/longitude pair, or a set of them — can go into a model directly as a small numeric vector, sometimes after a coordinate-system transform to make distances behave more sensibly. Polygon or region data (a city boundary, a field outline) more often gets rasterized: overlaid onto a regular grid, where each grid cell is assigned a value based on what falls inside it, turning an irregular shape into the same kind of fixed-size 2-D grid a convolutional vision model already knows how to consume. This is the direct reason geospatial and image pipelines share so much tooling in practice — once rasterized, a satellite image and a rasterized population-density map are the same kind of tensor to a network, even though they started as completely different data types.

L3 — The exam-relevant edge case: representation is not a free choice for every downstream use

A single modality does not always get exactly one canonical representation, and the exam's own framing leaves room for a scenario question that hinges on this. Text bound for a classic RNN or an older CNN-based text classifier is sometimes represented as a bag-of-words or TF-IDF vector rather than as a token-embedding sequence — a representation with no notion of order at all, useful for some tasks and useless for others. Audio bound for a task that genuinely needs fine time resolution (detecting a very short click or transient) sometimes stays closer to the raw waveform, because a spectrogram's windowing necessarily smooths over anything shorter than one window. The exam's own scope note treats this domain as foundational — you are expected to recognize the standard representation per modality, not to relitigate every edge case — but "the standard representation" is itself a claim you should hold loosely enough to notice when a described scenario's stated goal (fine time resolution, no positional information needed, extreme sparsity) points toward the less common alternative instead.

⭐ THE EARNED INSIGHT: > "Neural-network-ready" is never a property of the raw data alone — it is a property of a raw signal relative to the architecture that will consume it. The same audio clip is ready as a waveform for one model and ready only as a spectrogram for another; the same image is ready as a normalized pixel grid for a convolutional model and ready only as a sequence of patch embeddings for a transformer. Asking "is this modality ready" without asking "ready for what" is an incomplete question, and it is exactly the gap a well-built scenario question will probe.

Two patterns are worth reading off this table directly rather than memorizing row by row. First, three of the five destinations — image patches, spectrogram frames, and a rasterized geospatial grid — are all, structurally, a 2-D (or patch-sequence) grid of numbers, which is why the same convolutional or transformer building blocks recur across modalities the source material otherwise treats as unrelated. Second, text is the one modality here whose ready-form is a sequence of discrete symbols mapped through a learned embedding table, rather than a signal that is continuous from the start — every other modality's "ready" tensor is a numeric transform of an already-numeric or already-continuous quantity, while text's tokens are categorical before the embedding layer ever touches them.

Treat the following as a constructed scenario built to make the arithmetic legible, not a measurement from a real production pipeline. A support platform logs one record per customer interaction: a free-text description of the issue, a photo of a damaged product the customer uploaded, a 12-second voice-memo the customer recorded describing the fault, a 60-reading window of the product's own sensor telemetry leading up to the failure, and the store location where the product was purchased.

Record: ticket_48213

1. Text ("the screen flickers whenever the charger is plugged in")
   -> BPE tokenizer: 13 subword tokens
   -> embedding lookup: 13 x 512 tensor (one 512-dim vector per token)

2. Image (2448 x 3264 pixel photo, RGB)
   -> resize to 224 x 224, normalize to mean-zero/unit-variance per channel
   -> patch size 16 x 16 -> (224 / 16) x (224 / 16) = 14 x 14 = 196 patches
   -> patch embedding: 196 x 768 tensor (one 768-dim vector per patch)

3. Audio (12-second voice memo, 16,000 samples/second, mono)
   -> total raw samples: 12 x 16,000 = 192,000
   -> spectrogram: 25ms window, 10ms hop
      window length in samples = 0.025 x 16,000 = 400
      hop length in samples    = 0.010 x 16,000 = 160
      number of frames = floor((192,000 - 400) / 160) + 1 = 1,199 frames
   -> spectrogram tensor: 1,199 x 128 (128 frequency bins per frame)

4. Time-series (sensor telemetry: temperature + vibration, 2 channels)
   -> window: last 60 readings, both channels
   -> normalize each channel independently (z-score against its own training stats)
   -> tensor shape: 60 x 2

5. Geospatial (store location, single lat/lon pair)
   -> project to a local planar coordinate system
   -> tensor shape: 1 x 2 (a bare 2-dimensional vector)

Five completely different raw inputs, and five tensors of five completely different shapes — a 13×512, a 196×768, a 1,199×128, a 60×2, and a 1×2. None of that mismatch is a problem yet, because this lesson stops exactly here: at "ready," not at "combined." The next lesson's entire subject is what happens when these five differently-shaped tensors need to inform one prediction — whether they get merged before any of them are processed further (early fusion), after some independent processing (intermediate fusion), or only at the very end once each has produced its own decision (late fusion). Nothing above chose a fusion point; it only produced the five inputs a fusion strategy would need.

A voice-assistant team needs to decide between two representations for a wake-word detector — the small model that constantly listens for a specific short phrase and triggers the larger, expensive pipeline only when it hears it. Alternative A represents each audio chunk as a spectrogram; Alternative B represents each chunk as the raw waveform fed directly into a lightweight 1-D convolutional network.

The deciding constraint is stated directly in the requirements: the wake word is very short (under 400 milliseconds) and the detector must run continuously on a low-power always-on chip with a strict latency budget. Compute the spectrogram's own frame count for a chunk this short at the same window/hop settings used above: frames = floor((6,400 - 400) / 160) + 1 = 38 frames, where 6,400 samples is 400ms at 16,000 samples/second. Thirty-eight frames is already usable, but producing them costs a Fourier transform per window, computed continuously, on a chip chosen specifically for its low power budget — a cost Alternative A pays on every single window, forever, whether or not the wake word is ever spoken.

Alternative B skips that transform entirely and feeds the 1-D convolutional network the 6,400 raw samples directly, at the cost of needing a network expressive enough to learn frequency-like structure on its own rather than receiving it pre-extracted. For an always-on, power-constrained, single-short-phrase task, that tradeoff usually favors Alternative B: the compute saved by never running a Fourier transform matters more here than the modeling convenience a spectrogram would offer, precisely because the task is narrow (one phrase, not general speech understanding) and the power budget is the binding constraint. A general-purpose speech-to-text system serving the same platform, needing to transcribe arbitrary long-form speech rather than detect one short phrase, would very plausibly make the opposite choice, because its binding constraint is transcription accuracy across open vocabulary, not continuous low-power operation — which is exactly the kind of scenario-dependent reasoning a question about audio representation is built to test, rather than a fixed universal rule that one representation always wins.

Getting a modality into tensor form resolves exactly one problem: a network's first layer now has something it can mathematically operate on. It does not resolve, and is not meant to resolve, three further questions the rest of this module answers in turn. It does not decide when two or more modalities' tensors get combined — that is fusion, the next lesson's subject, and the five differently-shaped tensors in section 4 above are a direct illustration of why that question is nontrivial: you cannot simply concatenate a 13×512 text tensor and a 196×768 image tensor without first deciding where in the pipeline, and by what mechanism, they are allowed to interact. It does not decide whether two modalities' representations land in a comparable space at all — a text embedding and an image patch embedding from two independently-trained encoders have no reason to be comparable by similarity, even though both are, formally, dense numeric vectors; that is the shared-embedding-space problem M4-03 solves for text and images specifically, via CLIP's joint contrastive training. And it does not decide what to do when one of these five tensors is simply absent for a given example — a support ticket with no photo attached, a sensor with no telemetry logged — which is the subject the module turns to right after fusion.

Multimodal Data sits at 15% of the NCA-GENM blueprint [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md), and this lesson's material is the domain's own opening subsection — every later Domain 4 topic assumes you can already name each modality's standard representation without re-deriving it. The domain's scope note frames the whole module at foundational depth: know the modalities and how to fuse them, not how to hand-build a production preprocessing pipeline from first principles [GROUND TRUTH] (Sources/nca-genm/domain-4-multimodal-data.md). That framing predicts the shape of the question this material tends to generate: less "compute the number of spectrogram frames," more "given this described modality, which representation is correct, and which of the four options is borrowed from a different modality's pipeline."

The most common trap the exam sets is exactly that cross-modality substitution: a question describing audio data offers "normalized pixel patches" as a distractor option, or a question describing geospatial polygon data offers "BPE subword tokens" as a distractor — both real, correct representations, just for the wrong modality entirely. A second recognizable shape names a modality and asks which architecture typically consumes its ready-form representation, testing the same table from section 3 in the other direction: given "spectrogram frames," which model type is the natural fit.

What the distractors typically look like

The reliable distractor pattern borrows a real, correct representation from a modality other than the one described in the question stem — offering "coordinates/grids" for an audio question, or "tokens/embeddings" for a geospatial one. Each option is true of some modality, which is exactly what makes it plausible rather than obviously wrong; the fix is simply to hold the five-row table in section 3 firmly enough that a mismatched pairing stands out immediately rather than requiring re-derivation under time pressure.

Data cleaning, the Module 2 topic, fixes problems within a value's own representation — filling a missing entry, capping an outlier, encoding a category — while operating on data that is already, in some loose sense, structured as rows and columns a spreadsheet tool can display. Making a modality neural-network-ready is a separate, later step that converts an entire raw signal — a string, an image file, an audio waveform — into the specific fixed-shape numeric tensor a model's input layer is built to accept. A dataset can pass every cleaning check and still be nowhere close to ready, because cleaning never addresses tokenization, patching, spectrogram extraction, windowing, or rasterization; those are modality-specific transforms cleaning does not touch. In practice the two steps run in sequence, cleaning first, because a bad value is easier to catch and fix while it is still human-readable text or an easily-inspected numeric column, before it disappears into an embedding or a patch grid where it is no longer legible at a glance.

Because both, in their neural-network-ready form, are grids — a normalized pixel grid for an image, a rasterized value grid for geospatial polygon or point-density data — and a convolutional network or a vision transformer does not know or care what a grid cell's value originally represented, whether it is a color intensity or a population count. The moment a raw signal has been converted into "a regular 2-D array of numbers," it becomes eligible for the exact same architectural toolkit that vision models were originally built for, which is why remote-sensing and geospatial-ML tooling borrows so heavily from computer-vision libraries. The similarity is a consequence of the shape two different modalities happen to share after preprocessing, not a claim that images and maps are the same kind of thing before preprocessing.

No, and this is a common misreading of "shared destination." Every modality's ready-form is a numeric tensor, but the five tensors in section 4's worked example have five different shapes — 13×512, 196×768, 1,199×128, 60×2, and 1×2 — and that mismatch is completely normal at this stage. Forcing every modality into one identical shape before fusion is neither required nor, in most architectures, even desirable; a fusion strategy's whole job, taken up in the next lesson, is to define a mechanism (concatenation after independent processing, cross-attention, decision-level combination) that can operate across tensors of different shapes, rather than requiring the tensors themselves to already match.

• Five modalities, five different raw signals, one shared destination: a numeric tensor of a fixed, predictable shape.
• Text becomes subword tokens then dense embeddings; images become normalized pixels or patch embeddings; audio becomes a waveform or, more often, a spectrogram; time-series becomes windowed and normalized sequences; geospatial data becomes coordinates or a rasterized grid.
• "Neural-network-ready" is relative to the consuming architecture, not an absolute property of the raw data — the same signal can have more than one valid ready-form depending on what will process it next.
• Cleaning and readiness are two different steps; a spotless dataframe is not automatically neural-network-ready, because cleaning never performs the modality-specific tensor conversion.
• Producing five ready tensors resolves none of fusion, shared-space alignment, or missing-modality handling — those are the module's next three questions, in order.
• The most common exam trap borrows a real representation from the wrong modality; hold the five-row mapping firmly enough to catch the mismatch under time pressure.

Every tensor in this lesson's worked example is ready and none of them has been combined with any other yet.

Next: M4-02 picks up exactly there — model fusion in depth, the question of when, at raw inputs, at hidden layers, or only at final decisions, two or more of these now-ready modalities actually start talking to each other, and what each choice costs in accuracy, latency, and tolerance for a modality that simply is not there.