ConstructiCat Logo
CodeBust.
Browse section ▾

Hallucinated API.

AI-generated code that calls functions, methods, parameters, config keys, or packages that look plausible but do not exist in the actual library version you depend on.

##Signs and Symptoms

A reviewer spots a Hallucinated API when the code reads fluently and "looks like" idiomatic use of a library, yet the specific call, option, or import cannot be found in that library's real surface. Tell-tale signs:

  • A method or option whose name is too convenient — it does exactly what the prompt asked, with a name that blends two real APIs (e.g. findLastWhere, includesAll, parseDateSafe).
  • Invented parameters on a real function (the dangerous variant — it often compiles and only fails at runtime, or silently ignores the option).
  • An import of a package that is not in package.json / requirements.txt, or a named import that the module never exports.
  • API shapes that mix versions: a v2 signature called on a v5 client, or a method removed/renamed several releases ago.
  • Confident inline comments asserting the call is correct ("// returns the refunded charge").
// AI-generated — fluent, plausible, and wrong
import { formatRelative } from 'date-fns';

// `roundingMethod` is invented — date-fns exposes no such option, it's silently ignored
const label = formatRelative(date, new Date(), { roundingMethod: 'floor' });

// `findLastWhere` does not exist on Array — runtime TypeError
const lastActive = items.findLastWhere(i => i.active);

// Blended SDK call: real Stripe shape is stripe.refunds.create({ charge })
await stripe.charges.refund(chargeId, { amount: 500 });

A fast heuristic: if you cannot point to the doc entry or type definition for a third-party call in under a minute, treat it as hallucinated until proven otherwise.

##Reasons for the Problem

Why models produce it

  • Next-token plausibility, not lookup. An LLM predicts the most statistically likely continuation, which is effectively the average of every similar API it has seen. That average is often a method that ought to exist — the model emits "the convenient name" rather than the real one. Studies of API recommendation find 58.1%–84.1% of recommended APIs do not exist in the named package, and the dominant error is non-existent method names (arXiv 2404.00971, ACM TOSEM 2025).
  • Version blending. Training corpora mix many versions of a library, so the model fuses v2 and v5 signatures into one that matches neither. These "Knowledge-Conflicting Hallucinations" (e.g. non-existent parameters) are specifically the kind that slip past linters and fail at runtime.
  • Training-cutoff staleness. The model confidently uses APIs renamed, deprecated, or removed since its cutoff, and reaches for whatever appeared most in the corpus — which OX Security notes means older, sometimes vulnerable, package versions get recommended (OX report, Oct 2025).
  • No repository/dependency context. Without your package.json or the actual module source, the model invents helpers that "feel" like they belong to your stack.
  • Sycophancy / eagerness to answer. The assistant almost never says "I'm not sure that method exists" — it produces confident, runnable-looking code, which lowers reviewer suspicion.
  • Determinism makes it exploitable. Package hallucination is not random noise: across 16 models and 2.23M generations, 19.7% of recommended packages were fictitious (205,474 unique names), and 58% of hallucinations recurred within 10 re-prompts (USENIX Security 2025; SecurityWeek summary).

Why it hurts

  • Correctness. Invented parameters and silently-ignored options produce wrong behavior on code paths that tests rarely cover; the failure surfaces in production, not at compile time.
  • Security / supply chain. A hallucinated package name is a registration target: attackers publish malware under the predicted name, so the next developer who accepts the suggestion installs it — the slopsquatting attack (term coined by PSF's Seth Larson). The staleness variant silently reintroduces deprecated or CVE-bearing APIs.
  • Review load. Plausible code shifts the burden onto reviewers to verify every unfamiliar call against docs; fluent prose makes that verification less likely to happen.
  • Tech-debt accrual. Developers often paste the "almost working" hallucinated stub and patch around it rather than fix the root call — feeding the broader AI-era trend of rising copy/paste and falling refactoring (GitClear 2025: clones up ~8×, refactor-driven moved lines down from 25% to <10%). This pattern is now catalogued among AI-specific code smells (arXiv 2509.20491).

##Treatment

Process & prompting tactics

  • Ground the model in real surfaces. Paste the actual type stubs, the relevant doc page, or the installed version's source into context, or use a docs-retrieval tool (Context7-style / RAG). Tell it the exact dependency versions from your lockfile.
  • Require citations. Ask the model to name the official doc entry or type signature for every third-party call it uses. Practitioner rule of thumb: every method call against a third-party library should be traced to its documentation entry before the PR is approved.
  • Make it run, in the loop. Require tsc / mypy / pylint / the build and the test suite to pass, and have the agent actually npm install / pip install so a hallucinated package fails fast instead of reaching review.
  • Ask it to reuse, not invent. Feed it a grep of the existing module and instruct it to call existing helpers (reuse the helpers in utils/http.ts) rather than conjure new ones — this also counters the related reinvented-helper duplication smell.
  • Gate dependencies. Use a lockfile + an install allowlist and a supply-chain scanner (Socket/Snyk) before any new package is added, so slopsquatted names cannot slip in.

The code refactor

Replace the invented call with the verified real one. If you genuinely want the convenience the model imagined, implement it once against the real API behind an Extract Function wrapper instead of scattering the fake call.

// Before — hallucinated parameter + blended SDK shape
async function refundLast(chargeId: string) {
  // `stripe.charges.refund` and this option shape do not exist
  return stripe.charges.refund(chargeId, { amount: 500, reason: 'requested' });
}

// After — verified against the installed Stripe SDK's types/docs,
// and the "convenience" wrapped once so the real API isn't repeated
async function refundCharge(chargeId: string, amountCents: number) {
  return stripe.refunds.create({
    charge: chargeId,
    amount: amountCents,
    reason: 'requested_by_customer',
  });
}

For the staleness variant, treat a flagged deprecated call as a real upgrade task: move to the current API and pin the version, rather than silencing the warning.

Limits. Type-checkers and resolvers catch the resolvable subset (typed objects, unresolved imports). The dynamic/untyped remainder — invented options on any, string config keys, REST payload fields — has no reliable automated detector and must be verified by a human against the real docs.

##Detected by