Skip to main content
Prompt-to-Playable Character
All kit documentsView Markdown

Long-form article

Prompt to playable: how the pipeline was built

Technical readers1,811 words

An endpoint that turns text into a 3D mesh is genuinely useful. It is also not a product. The gap between "I called an API and got a task ID" and "a player can pick up a controller and move this thing" is where the real integration work lives: a human checkpoint before you spend geometry credits on the wrong direction, a durable orchestration that survives a redeploy mid-run, state you can trust without polling leaking a credential, and a scene that loads the result without falling over on a stale signed URL.

This is a walkthrough of how Prompt-to-Playable Character does that with the Meshy API — one prompt in, a textured, rigged, animated GLB in a browser scene out. Everything below is taken directly from the repository: the request bodies, the workflow steps, the error taxonomy, and the loader code are all real.

Why an endpoint wrapper is not enough

A thin wrapper around POST /openapi/v1/image-to-3d gets you a task ID. It does not get you a guarantee that the mesh can be rigged at all (Meshy's rigging step rejects unclear or non-bipedal anatomy), a way to resume a run after the process restarts without re-paying for stages that already succeeded, a safe place to hold the API key, or — the part most demos skip — any say in what the model actually looks like before you commit real geometry credits to it.

The actual developer problem is orchestration plus a trust boundary, not generation. Once you accept that, the architecture follows: one module that knows Meshy's wire format, one durable workflow that owns the order of the pipeline and can be resumed, and a scene that renders whatever the pipeline produces.

Architecture overview

Browser (R3F scene, poll hook)
        │
        ▼
/api/orchestrations/* route handlers
        │
        ▼
workflows/character-generation.ts  — durable steps, approval webhook, snapshots
        │
        ▼
server/meshy/client.ts             — HTTP, retries, auth, request builders
        │
        ▼
Meshy API (text-to-image, image-to-3d, rigging, animations)

Three modules do almost all of the work:

  • src/server/meshy/client.ts — the only file that knows Meshy's wire format: endpoints, request bodies, retry policy, and the API key. Everything above it works in normalized domain types (src/types/pipeline.ts).
  • src/workflows/character-generation.ts — a Vercel Workflow SDK function ("use workflow") that owns the entire run: two parallel concept generations, an approval webhook that pauses the run until a human picks one, then base geometry, rigging, and animation as sequential durable steps ("use step"). Each step's result is persisted, so the workflow can resume after a redeploy without re-creating tasks or re-spending credits.
  • src/server/workflow/snapshots.ts and the src/features/* components — read the latest published snapshot and render it; toPublicSnapshot() strips the owner key and approval token before anything reaches the browser.

The same provider-neutral orchestration types also power the pre-generated showcase path, which never calls Meshy and needs no workflow run. Live generation runs through the durable workflow.

The concept-approval gate — the interesting design decision

Every stage in this pipeline before the approval gate is generation; every stage after it is spend. Rigging and animation cost credits on geometry that already exists, so the design puts a real human-in-the-loop checkpoint between the cheap step (two images) and the expensive one (a textured mesh), rather than letting the model guess which direction to commit to.

The workflow kicks off two text-to-image tasks in parallel — direction A ("faithful," restrained) and direction B ("bolder," more contrast) — both built from the same prompt:

// src/server/meshy/client.ts
export function buildConceptRequest(
  input: ConceptTaskInput,
): Record<string, unknown> {
  const direction =
    input.direction === "A"
      ? "Faithful direction: practical materials, clean silhouette, restrained details."
      : "Alternate direction: bolder costume shapes and material contrast while preserving the core character identity.";

  return {
    ai_model: "nano-banana-2",
    prompt: `${input.prompt}\n\n${direction} Full-body single humanoid character concept, centered and isolated on a plain neutral background, entire body visible, clean A-pose, arms and legs clearly separated, no text, no frame, no scenery. Designed as production reference for a riggable 3D game asset.`,
    aspect_ratio: "1:1",
    pose_mode: "a-pose",
  };
}

Once both images succeed, the workflow opens a webhook and blocks:

// src/workflows/character-generation.ts
using approval = createWebhook({ respondWith: "manual" });
snapshot = touched({
  ...snapshot,
  status: "AWAITING_CONCEPT",
  approvalToken: approval.token,
});
await emitSnapshot(progress, snapshot);
const { conceptId } = await acknowledgeApproval(await approval);

POST /api/orchestrations/[id]/concept is the only thing that can resume that webhook — it validates the caller owns the run, checks the chosen concept actually succeeded, and calls resumeWebhook(token, ...). Nothing about which image is "better" is decided by the model; a person looks at both and picks. The approved image stays on screen the whole time base geometry is being built, so the user never loses the thing they actually approved.

Only the approved concept's task ID crosses into geometry generation, as input_task_id on image-to-3d. The rejected direction is never built into a mesh.

The durable workflow, and what it buys you

generateCharacterWorkflow is a single function marked "use workflow". Every side-effecting call inside it — creating a task, polling one, emitting a progress snapshot — is its own "use step" function, and the Workflow SDK persists each step's result as it completes:

// src/workflows/character-generation.ts
async function createConceptStep(
  prompt: string,
  direction: "A" | "B",
  apiKey: string,
) {
  "use step";
  return createConceptTask({ prompt, direction }, apiKey);
}
createConceptStep.maxRetries = 0;

maxRetries = 0 here is deliberate, not an oversight: Meshy documents no idempotency key for task creation, so retrying a POST after an ambiguous network failure risks creating and charging for the same task twice. Polling steps (pollConceptStep, pollStageStep) get maxRetries = 5 instead, because a GET is safe to retry freely.

What this buys you in practice: if the process restarts mid-run — a redeploy, a crash — the workflow run itself is durable state. A client reattaches by polling GET /api/orchestrations/[id], which resolves the workflow run (wrun_...) through workflow/api's getRun(), reads the latest published snapshot off the run's readable stream, and returns exactly what it would have returned before the restart. No stage re-runs, no task gets created twice, and the approval webhook — if the run is sitting at AWAITING_CONCEPT — is still valid and waiting.

Polling without exposing the key

A developer-owned key can come from the server environment. On the public deployment, a visitor enters a key into the credential form; the browser submits it to a same-origin route, the server validates it against Meshy's balance endpoint, and JavaScript never receives it back or stores it in local storage. Active workflows keep their credential encrypted so a cold function or refresh does not strand paid work. Every route handler that touches Meshy imports server/meshy/client.ts, which is gated by the server-only package.

The client polls its own server, never Meshy directly:

// src/app/api/orchestrations/[id]/route.ts
if (isWorkflowRunId(id)) {
  const snapshot = await getOwnedWorkflowSnapshot(id, ownerKey);
  if (!snapshot) {
    return jsonError(notFoundError(`No session "${id}" for this client.`), 404);
  }
  return jsonOk(toPublicSnapshot(snapshot), {
    headers: { "Cache-Control": "no-store" },
  });
}

getOwnedWorkflowSnapshot checks the run's stored ownerKey against the caller's derived key before returning anything, and toPublicSnapshot strips ownerKey and approvalToken on the way out — the only thing that ever crosses into the browser is the normalized OrchestrationSnapshot.

Task state, without drift

FieldMeaningWhere it shows up
status (orchestration)GENERATING_CONCEPTS, AWAITING_CONCEPT, GENERATING, RIGGING, ANIMATING, PLAYABLE, PARTIAL_FAILURE, FATAL_FAILUREDrives which screen renders
activeStageconcept, preview, rig, animate, load, readyStage rail highlight
conceptCandidates[].statusPer-image QUEUED / RUNNING / SUCCEEDED / FAILEDConcept screen cards
tasks[].statusPer-Meshy-task QUEUED / RUNNING / SUCCEEDED / FAILED / CANCELEDDeveloper panel
outputs.expiresAtISO timestamp the current signed URLs are assumed stale byDownload gating

A failure before base geometry succeeds is FATAL_FAILURE — there's nothing durable to hand back. A failure at rigging or animation is PARTIAL_FAILURE: the textured mesh from the prior stage is still a real, downloadable asset, so the run doesn't get thrown away.

Loading the animated GLB in R3F

Once the animate stage succeeds, the browser has a signed GLB URL. CharacterModel loads it with drei's useGLTF, clones it with SkeletonUtils.clone (a plain .clone() doesn't preserve skinned-mesh bone bindings), and disposes every GPU resource on unmount:

// src/features/scene/character-model.tsx
const gltf = useGLTF(url, false, false);
const normalized = useMemo(() => {
  const clone = SkeletonUtils.clone(gltf.scene) as THREE.Object3D;
  return normalizeCharacter(clone);
}, [gltf.scene]);

Draco and Meshopt decoding are deliberately off (useGLTF(url, false, false)) — those loaders fetch decoder WASM from a Google CDN that the app's CSP doesn't allow, and Meshy's baseline GLB export doesn't need them.

The zero-key showcase ships a project-owner-supplied Meshy GLB with a real skeleton, authored Walk and Run clips, and a generated planted Idle. It runs through the same animation mixer, scene, controls, local history, and download path as a live result, while remaining clearly labelled as pre-generated. Its provenance and separate media terms are recorded in docs/ASSETS.md rather than being implied by the repository's MIT source-code license.

Error recovery, signed URLs, and rate limits

Every failure funnels through one function, normalizeError() in src/server/meshy/errors.ts. HTTP status maps to intent: 401/403 disables live generation and points at the showcase; 402 does the same without implying anything is broken; 422 is retryable with a cleaner-anatomy prompt; 429 backs off automatically; 5xx is retryable because completed stages are untouched:

// src/server/meshy/errors.ts
if (status === 402) {
  return {
    code: "MESHY_INSUFFICIENT_CREDITS",
    userMessage: "Live generation credits are unavailable.",
    recovery:
      "The showcase character is still fully interactive. Run the project with your own API key to generate live.",
    retryable: false,
  };
}

The HTTP client (server/meshy/client.ts) retries 429/500/502/503/504 up to three times with exponential backoff, honoring Retry-After — but only for GET, for the same idempotency reason the workflow's createStageStep sets maxRetries = 0.

Signed asset URLs expire; Meshy returns expires_at on completed tasks, and the app trusts that when present, falling back to a conservative ASSET_URL_TTL_MS — 71 hours, one hour inside Meshy's documented three-day non-enterprise retention ceiling — when it isn't. GET /api/assets/download never accepts a raw URL from the client: it takes a session ID and an asset name, looks the current signed URL up server-side from that session, and refuses to stream from anywhere outside an explicit host allow list (assets.meshy.ai, asset.meshy.ai, the Meshy S3/GCS buckets) — the SSRF guard isAllowedAssetUrl() enforces.

Fork and extend

The seams are exactly where the code is already split: add a Meshy request builder to client.ts, add a step and a stage transition to character-generation.ts, and add its PipelineTask shape to the state types. Nothing downstream needs to know the pipeline grew a step — it all consumes the same normalized OrchestrationSnapshot, and the developer panel and history sidebar render whatever it contains without modification.