Building and Connecting Custom Tools, APIs, and Functions

A tool is a capability an agent can invoke that reaches outside the model's own generation process — typically a function call, an API request, or a database query — and returns an observation the agent can then reason over. The distinction from a prompt-chain stage (M2-01 covered chains as sequences of model calls) is that a tool call is not asking a model to produce text; it is asking some external system to do something or report something, and the agent's job is to decide when to call which tool with which arguments, then incorporate whatever comes back.

This "reaching outside" property is exactly why tools need a stricter discipline than prompts do. A prompt that is worded slightly ambiguously produces text that is slightly off — annoying, but self-contained; nothing outside the model call was touched. A tool call that is invoked with the wrong arguments, or invoked twice when it should have run once, can charge a customer twice, send a duplicate notification, or overwrite a record with stale data — effects that persist in the outside world well after the agent's turn has ended. The contract a tool exposes is what keeps that gap between "the agent asked for the wrong thing" and "something in the world is now actually wrong" as narrow as possible.

A well-designed tool has a clear input/output contract: a defined shape for what arguments it accepts (their names, types, and valid ranges) and a defined shape for what it returns (success payload, and how failure is signaled). [GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md): "Well-designed tools have clear input/output contracts and are, where possible, idempotent." This clarity matters for the agent calling the tool — a language model deciding whether and how to call a function is more reliable when the function's expected arguments are explicit and narrow, the same way a human developer writes fewer bugs against a clearly typed API than an undocumented one — and it matters for anything wrapping the tool afterward, because a resilience layer (a retry policy, a timeout, a circuit breaker) needs to know unambiguously what counts as a failure response versus a valid result before it can decide anything about when to reattempt.

L1 — Intuition

Think of a tool's contract the way you would think of a vending machine's interface: you put in a specific coin and press a specific button, and you get back a specific item or your coin, never something in between and never something undefined. A tool with a vague contract is a vending machine with no labels on its buttons — sometimes pressing one gets you a snack, sometimes an error, and neither the person pressing the button nor anything watching the machine can predict which, or tell after the fact which outcome actually happened.

L2 — Mechanism

Mechanically, a contract is typically expressed as a schema: named parameters with types (a string, an integer, an enum of allowed values) for the input side, and a structured response shape — a success object with defined fields, or an error object with a defined error code or category — for the output side. Frameworks that support tool/function calling generally require this schema up front, because the model itself uses the schema to decide how to fill in arguments when it chooses to call the tool, and the calling code uses the schema to validate what came back before doing anything further with it. A tool without this schema forces every caller, human or model, to guess at the shape of what a call will accept and return, which is exactly the ambiguity a contract exists to remove.

L3 — The exam-relevant edge case: a clear contract does not by itself make a tool idempotent

It is tempting to treat "well-defined inputs and outputs" as the whole of what a good tool contract needs, but a tool can have an impeccably clear, fully typed contract and still not be safe to call twice — a charge_payment(account_id, amount) function can have perfectly explicit typed parameters and a perfectly structured success/failure response, and still double-charge a customer if called twice because the first response never arrived. Contract clarity and idempotency are two separate properties a tool needs, addressing two separate problems: clarity is about whether a caller (model or human) can use the tool correctly the first time; idempotency is about what happens if the tool ends up being called more than once for what was meant to be a single logical operation. The next section is entirely about that second property, because it is the one the rest of this module depends on.

An operation is idempotent if running it more than once produces the same end state as running it exactly once — repeating it does not compound the effect. This property is what makes an operation safe to retry: if a network blip means the caller never received the first attempt's response, calling the same operation again with an idempotent tool produces no harm, because a second application changes nothing beyond what the first already did.

Naturally idempotent operations

A read — fetch this record, look up this weather forecast, retrieve this document — is idempotent by construction: reading the same thing twice returns the same data and changes nothing about the world either time. Setting a value to an absolute state is also idempotent even though it is a write: set_account_tier(account_id, "gold") run five times leaves the account in exactly the same state as running it once, because each call fully specifies the end state rather than an increment relative to whatever the current state happens to be.

Naturally non-idempotent operations

An operation whose effect compounds with each call is not idempotent by default. Incrementing a counter (add_to_balance(account_id, +50) run twice adds 100, not 50), sending a notification (run twice, the recipient gets two emails), creating a record with no duplicate check (run twice, there are now two records where there should be one), and — the example this domain's own source material uses directly — charging a payment, are all operations where a second, unintended application changes the outcome beyond what the first call alone would have produced.

Making a non-idempotent operation safe through a safeguard

A naturally non-idempotent operation is not permanently off-limits to retry logic; it needs a safeguard that makes it effectively idempotent before retry logic is allowed near it. The standard mechanism is an idempotency key: the caller generates a unique token for the logical operation once — before the first attempt — and attaches that same token to every attempt, including retries. The receiving service checks the token against operations it has already applied; if it recognizes the token, it returns the original result without reapplying the effect, and if it does not recognize the token, it applies the operation and records the token against the result for next time. This turns "charge this payment" into something that behaves like an idempotent operation from the caller's perspective, even though the underlying effect (money moving) is inherently a one-time thing that a naive retry would otherwise double up.

⭐ THE EARNED INSIGHT Idempotency is not a property you notice once a tool misbehaves — it is a property you have to decide on before the tool is ever called twice, because by the time a retry has actually duplicated a side effect, the question "was this safe to call again" has already been answered the hard way. A tool's contract can be flawless and its idempotency status can still be the one thing nobody checked, precisely because a clear, well-typed contract looks like thoroughness and creates a false sense that the tool has been fully specified. The actual discipline is asking one more question past the contract, every time a tool changes state: if this exact call happens twice, what changes the second time that didn't the first? A read answers "nothing." An absolute-value write answers "nothing." Anything else answers "something," and that something is precisely what an idempotency-key safeguard exists to prevent.

Consider an agent that manages warehouse inventory and needs a tool to adjust stock levels when an order ships.

Naive version (relative adjustment, non-idempotent):
  adjust_inventory(sku: string, delta: int) -> {new_quantity: int}
  Call: adjust_inventory("SKU-4821", -3)
  Effect: subtracts 3 from SKU-4821's current quantity.
  Problem: if the caller times out waiting for a response and retries the exact same
  call, the second attempt subtracts another 3 -- five units shipped are recorded as
  eight units removed from stock.

Idempotency-key version (safe to retry):
  adjust_inventory(sku: string, delta: int, operation_id: string) -> {new_quantity: int}
  Call: adjust_inventory("SKU-4821", -3, operation_id="ship-order-88213")
  Effect (receiving service): checks whether "ship-order-88213" has already been
  applied. First call: applies the -3 delta, records the operation_id, returns
  new_quantity. Retried call with the same operation_id: recognizes it has already
  been applied, does not subtract again, and returns the same new_quantity as before.

Constructed scenario — the SKU and operation-id values are illustrative, not drawn from a real inventory system. The contract's shape did not need to change in a way that made the tool harder for the agent to call correctly — the agent still passes a SKU and a delta, plus one additional field it can generate once per logical shipment. What changed is that the tool is now safe to wrap in a retry policy: a lost response no longer risks double-decrementing stock, because the receiving side, not the caller's memory of whether the first attempt "probably" succeeded, is what enforces the one-time effect.

The inventory example showed a numeric adjustment turning unsafe under retry. A second common category worth tracing separately is a notification-send tool, because it is easy to misjudge as harmless even when it is not — sending a duplicate message feels less dangerous than double-charging money, but it is the same underlying failure mode.

Naive version (no dedupe, non-idempotent):
  send_shipping_notification(order_id: string, email: string) -> {sent: bool}
  Call: send_shipping_notification("ORDER-7734", "customer@example.com")
  Effect: sends one email to the customer confirming the order shipped.
  Problem: if the caller's connection drops after the email provider accepts the
  send but before the "sent: true" response reaches the caller, the caller sees
  a timeout and, treating this as a transient fault, retries -- the customer now
  receives two identical shipping-confirmation emails for the same order.

Idempotency-key version (safe to retry):
  send_shipping_notification(order_id: string, email: string,
    operation_id: string) -> {sent: bool}
  Call: send_shipping_notification("ORDER-7734", "customer@example.com",
    operation_id="notify-shipped-order-7734")
  Effect (receiving service): checks whether "notify-shipped-order-7734" has
  already triggered a send. First call: sends the email, records the
  operation_id. Retried call with the same operation_id: recognizes the send
  already happened, does not send again, and returns {sent: true} reflecting
  the original send.

Constructed scenario — the order id and email address are illustrative, not drawn from a real order. The reasoning that makes the payment and inventory examples unsafe under retry applies identically here, even though a duplicate email is a much lower-stakes mistake than a duplicate charge: the operation's effect (a message actually leaving the system and reaching a real inbox) compounds with each successful call, and nothing about the failure that triggered the retry — a dropped connection after the email provider already accepted the send — tells the caller whether the first attempt actually landed. The lesson to take from comparing this example against the payment case is that severity of the mistake and whether the mistake happens at all are two separate questions: a notification tool without an idempotency-key safeguard is exactly as retry-unsafe as a payment tool without one, even though a duplicate email is a far smaller problem to clean up after than a duplicate charge.

NVIDIA's own tooling reflects this contract discipline at the framework level. [GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md): "In the NeMo Agent Toolkit, agents, tools, and workflows are modeled as reusable, composable function calls ('build once, reuse'), so a tool written for one workflow can be dropped into another." This "build once, reuse" model is only possible because each tool exposes a defined contract in the first place — a function call with a specified signature is exactly the unit that can be composed into a different workflow without modification, whereas a capability with no defined boundary (an ad hoc block of prompt text describing "some external lookup, however it happens to work this time") cannot be lifted out of one workflow and dropped into another with any confidence about what it will do there.

This composability has a direct consequence for the idempotency discipline covered above: a tool built once and reused across multiple workflows needs its idempotency properties to hold in every context it gets reused in, not just the one it was originally written for. A payment-charging tool that is safe to retry inside one workflow because that workflow happens to call it only once per user action is not automatically safe inside a different workflow that might legitimately call it multiple times in quick succession for different reasons — the idempotency-key safeguard, once built into the tool itself rather than assumed by a specific caller, is what makes the tool's safety travel with it across every workflow that reuses it.

Agent Development carries 15% of the NCP-AAI blueprint, tied with Agent Architecture for the heaviest weight in the exam, and objective 2.3 — building and connecting custom tools, APIs, and functions — is where this domain introduces the tool boundary that objective 2.4's error-handling material (Retry, and the Circuit Breaker pairing already covered in M2-05) exists to protect. [GROUND TRUTH] (Sources/ncp-aai/domain-2-agent-development.md) states the connection directly rather than leaving it implicit: "Well-designed tools have clear input/output contracts and are, where possible, idempotent... which is what makes safe retries possible." That sentence is close to exam phrasing on the causal relationship being tested: idempotency does not merely correlate with safe retries, it is the stated precondition for them.

Expect a scenario question that describes a tool operation (a payment, a notification, a counter increment, a record lookup) and asks whether it is safe to wrap in a retry policy as-is, with the correct answer turning on whether the operation is naturally idempotent, needs a safeguard, or is a pure read. Expect a second question shape naming the NeMo Agent Toolkit's composability model directly, since "reusable, composable function calls" and "build once, reuse" are close enough to the source material's own wording to be a plausible direct-recall item rather than only a scenario one.

A tool's input/output contract defines what arguments it accepts and what shape its response takes — it is about whether a caller (model or human) can use the tool correctly the first time. Idempotency is about what happens if the tool ends up being called more than once for what was meant to be a single logical operation — it is about whether a second call is safe, not whether the first call is well-specified. A tool can have an excellent, fully typed contract and still not be idempotent, which is why the two properties have to be checked separately rather than treating a clear contract as proof that retrying the tool is safe.

Attach an idempotency key: a unique token generated once per logical operation, sent with every attempt including retries, that the receiving service checks against operations it has already applied. If the token has been seen before, the service returns the original result instead of reapplying the effect; if it has not, the service applies the operation and records the token. This adds one field to the tool's contract without changing the operation's actual behavior on a single, successful call — the only behavior that changes is what happens on a repeated call with the same token, which is exactly the case a retry produces.

Modeling tools as function calls with defined signatures is what makes "build once, reuse" possible: a function with a specified contract can be lifted out of one workflow and dropped into a different one with a clear expectation of what it accepts and returns in the new context, whereas a capability with no defined boundary carries no such guarantee. This composability also means a tool's safety properties, including idempotency, need to be built into the tool itself rather than assumed by whichever workflow happened to write it first, since the same tool is expected to be reused under calling patterns its original author never anticipated.

Work through each item before checking the answer key.

1. A tool has a fully typed, well-documented input/output schema. Is it automatically safe to wrap in a retry policy?
   • A. Yes — a clear contract guarantees safe retries.
   • B. Not necessarily — contract clarity and idempotency are separate properties, and a clearly typed tool can still be unsafe to call twice.
   • C. Only if the tool is written in a statically typed language.
   • D. Yes, as long as the tool returns a success/failure flag.
2. Which of the following operations is naturally idempotent without any safeguard?
   • A. Incrementing a counter by a fixed amount.
   • B. Setting an account's tier to a specific named value.
   • C. Sending a notification email.
   • D. Appending a new row to a log with no dedupe check.
3. What does an idempotency key actually do?
   • A. Encrypts the tool call's arguments.
   • B. Lets a receiving service recognize a repeated attempt of the same logical operation and return the original result instead of reapplying the effect.
   • C. Speeds up the tool call's execution.
   • D. Converts a non-transient fault into a transient one.
4. A payment-charging tool is safe to retry inside Workflow A because Workflow A only ever calls it once per user action. Is it automatically safe inside Workflow B, which might call it multiple times in quick succession for different reasons?
   • A. Yes, safety is a property of the workflow, not the tool.
   • B. Not automatically — idempotency needs to be built into the tool itself so it holds across every workflow that reuses it.
   • C. Yes, as long as both workflows use the same programming language.
   • D. No tool can ever be reused safely across workflows.
5. What is the "build once, reuse" model the NeMo Agent Toolkit uses for agents, tools, and workflows?
   • A. Immutable binaries deployed once per cluster.
   • B. Reusable, composable function calls with defined signatures that can be reused across workflows without modification.
   • C. SQL stored procedures shared across databases.
   • D. A single monolithic prompt reused for every task.
6. Why is a notification-send tool without an idempotency-key safeguard considered exactly as retry-unsafe as a payment tool without one, even though the consequences differ in severity?
   • A. It is not — notifications are always safe to retry.
   • B. Both operations compound their effect with each successful call, and nothing about a dropped-connection failure tells the caller whether the first attempt already landed.
   • C. Notifications are idempotent by definition.
   • D. Severity determines safety, not the operation's underlying effect.
7. Which property does idempotency specifically determine?
   • A. Whether a tool's contract is well-documented.
   • B. Whether it is safe to call an operation more than once for what was meant to be a single logical operation.
   • C. How fast a tool call executes.
   • D. Whether a model can parse the tool's output.
8. What should happen before retry logic is allowed to wrap any state-changing tool?
   • A. The tool should be rewritten in a faster language.
   • B. The tool's idempotency should be classified — naturally idempotent, non-idempotent, or safeguarded — and confirmed safe before retry logic touches it.
   • C. The tool's contract should be left undocumented for flexibility.
   • D. The tool should be merged with a different tool.

Answers

1. B. A tool's contract clarity is about whether a caller can use it correctly the first time; idempotency is a separate question about what happens on a second call, and one property does not guarantee the other.
2. B. Setting a field to an absolute value fully specifies the end state regardless of how many times it runs, unlike the other three options, which all compound with repetition.
3. B. The idempotency key is what lets a receiving service tell "this is the same logical operation retried" apart from "this is a new operation," and act accordingly.
4. B. A tool's idempotency safeguard has to be enforced on the receiving side, built into the tool itself, so it travels with the tool regardless of which workflow's calling pattern happens to invoke it.
5. B. This is the exact composability model the source material names for the NeMo Agent Toolkit.
6. B. The mechanism that makes a retry unsafe — a compounding effect combined with an ambiguous failure signal — is identical in both cases; only the real-world cost of the resulting duplicate differs.
7. B. Idempotency is specifically about the safety of a repeated call, not about documentation, execution speed, or output parseability.
8. B. Classifying idempotency explicitly, before any retry policy is applied, is the gate that prevents a well-intentioned resilience layer from silently duplicating a side effect.

• A tool is what lets an agent act on or query the world rather than only generate text, and that "reaching outside" property is exactly why tools need a stricter discipline than prompts.
• A well-designed tool has a clear input/output contract — defined inputs, defined outputs, defined failure signaling — which is a separate property from idempotency, not a substitute for checking it.
• Idempotency means a second call produces the same end state as the first; reads and absolute-value writes are naturally idempotent, while relative writes, notifications, and record creation are not, unless safeguarded with an idempotency key.
• Idempotency is the specific precondition that determines whether it is safe to retry a tool call at all — this is the direct setup for M2-04's Retry pattern, which assumes this question has already been answered for whatever it wraps.
• The NeMo Agent Toolkit's "build once, reuse" model treats tools as reusable, composable function calls, which means a tool's idempotency safeguards need to travel with the tool itself across every workflow that reuses it, not live only in one caller's assumptions.

Every idempotency classification this lesson worked through — naturally safe, naturally unsafe, safeguarded — exists to answer one question a retry policy asks before it ever reattempts a call: is this safe to run again?

Next: M2-04 takes that question and builds the full Retry pattern on top of it — the three retry strategies (cancel, retry immediately, retry after backoff), and why never retrying a non-idempotent operation without a safeguard is the rule this lesson's idempotency-key mechanism exists to satisfy.