RunPod's InfiniteTalk endpoint takes its output orientation from the reference image, not from the size you send. Pass a portrait reference with size: "landscape" and you get a portrait 512x768 clip back, with no error and no warning. The fix is to keep every reference in a pool the same orientation and run ffprobe on the raw clip before compositing.

We run InfiniteTalk as a talking-head picture-in-picture (PiP) layer on top of our automated product-walkthrough videos: screen recording, AI narration, captions, a music bed, and a cloned presenter in the corner. Across several multi-minute client productions it showed us six behaviors the docs don't mention. None of them raised an error. Each one is below with the exact check we now run.

Why does InfiniteTalk return a portrait clip when you asked for size: "landscape"?

When the reference image's aspect ratio and the size parameter disagree, InfiniteTalk follows the image. One portrait reference (848x1264) in a round-robin pool of 11 landscape ones (1376x768) produced a 512x768 clip despite size: "landscape". Our scale=420:-1 overlay then rendered a 420x630 PiP box instead of 420x234.

Nothing in the pipeline complained. The clip duration matched the narration and ffmpeg exited 0. The logs looked identical to every successful render before it. A human caught it by looking at a frame of the finished video, which is the worst possible detection layer.

At the time of writing, RunPod's InfiniteTalk public endpoint docs describe size as a resolution tier (480p or 720p), not an orientation. That fits what we saw: the endpoint accepted "landscape" without complaint, but orientation was never controlled by that parameter. It always came from the image. Treat the reference image as the only orientation control you have.

The pool-level fix: reject orientation outliers before they can be picked

Our presenter has a pool of outfit variants. A per-character journal.json ({order: [...], next: int, history: [...]}) picks them round-robin, so every video looks slightly different. The outlier was a tighter, closer crop that had slipped into the pool. It was easy to miss in a thumbnail grid. Round-robin meant the bug only fired on the one render in twelve that drew it.

We removed the outlier from order rather than force-cropping it, because its framing genuinely differed from the set. A crop would have fixed the shape and broken the visual consistency. We also run this check before any new reference joins a pool:

from collections import Counter
from pathlib import Path
from PIL import Image

def orientation(path):
    w, h = Image.open(path).size
    return "landscape" if w > h else "portrait" if h > w else "square"

def check_reference_pool(ref_dir):
    shapes = {p.name: orientation(p) for p in sorted(Path(ref_dir).glob("*.png"))}
    majority, _ = Counter(shapes.values()).most_common(1)[0]
    outliers = {name: o for name, o in shapes.items() if o != majority}
    if outliers:
        raise SystemExit(f"reference pool has orientation outliers: {outliers}")
    return majority

The render-level fix: ffprobe the raw clip before compositing

The pool check prevents the problem. This guard catches it if something slips through anyway. It runs automatically when our poller collects a finished job, before the clip gets near the overlay step:

dims=$(ffprobe -v error -select_streams v:0 \
  -show_entries stream=width,height -of csv=p=0 talk_raw.mp4)

if [ "$dims" != "832,464" ]; then
  echo "InfiniteTalk clip is ${dims}, expected 832,464 (landscape). Refusing to composite." >&2
  exit 1
fi

A related point: don't pre-crop a landscape reference to square to be safe. An early render did exactly that, assuming only the 464x832 portrait and 624x624 square buckets were supported. An uncropped 1376x768 landscape reference renders fine at 832x464. Cropping just threw away part of the locked framing for nothing.

How long does an InfiniteTalk render take, and what does it cost in 2026?

Render time scales roughly linearly with audio length, at about 7 to 8 minutes of wall-clock per minute of audio. Our 15-second test clip took about 2 minutes. A 234-second full-narration clip took about 29 minutes. RunPod prices per video by resolution tier, $0.25 at 480p and $0.50 at 720p, so cost is trivial and wall-clock is what you plan around.

The "2–10 minutes" figure we had pencilled in from short tests was useless for a four-minute walkthrough. We now render the whole narration as a single job instead of per scene. We submit it the moment the final voiceover exists, then record the screen while the avatar renders remotely. That overlaps a roughly 35-to-40-minute avatar render with a 25-to-30-minute recording phase, which cuts total production time roughly in half.

The precondition is strict: submit only after the narration is final. Any script edit after submission wastes the render, because the in-flight clip's lips are synced to audio that no longer exists.

Poll in bounded loops, not with a background waiter

A 29-minute job outlives most tool-call and request timeouts. The RunPod job runs server-side and /status/<id> is stateless, so we poll in bounded loops and simply call again if the job isn't done. A local "wait for it" subprocess that has to survive a process restart is the part that breaks.

import os, time, requests

API = "https://api.runpod.ai/v2/infinitetalk"
HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}

def submit(image_url, audio_url, prompt):
    r = requests.post(f"{API}/run", headers=HEADERS, json={"input": {
        "prompt": prompt,
        "image": image_url,   # public URL; orientation comes from THIS
        "audio": audio_url,   # 16 kHz mono WAV, public URL
        "size": "landscape",  # see above: not what decides orientation
        "enable_safety_checker": True,
    }}, timeout=30)
    r.raise_for_status()
    return r.json()["id"]     # persist this; the job outlives your process

def poll(job_id, iterations=25, interval=20):
    for _ in range(iterations):           # ~8 min per call, then re-invoke
        s = requests.get(f"{API}/status/{job_id}", headers=HEADERS, timeout=30).json()
        if s["status"] in ("COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"):
            return s
        time.sleep(interval)
    return None                           # not done yet: call poll() again later

The endpoint fetches inputs by URL, not upload, so both the reference image and the audio have to be hosted publicly. We put them in an S3-compatible bucket. Audio has to be converted first: ffmpeg -i voiceover.mp3 -ar 16000 -ac 1 talk_input.wav.

Why does amix-ing the avatar audio with the base narration give you two voices?

Because the talking-head clip's audio track is already the narration, the same source voiceover file you fed InfiniteTalk. Mixing it with the screen recording's narration plays the same script twice, slightly out of phase. Our first composite used amix. The reviewer's verdict was immediate: "it sounds really freaky and weird." Replace the audio. Never mix it.

# WRONG: two copies of one narration, a few ms apart
-filter_complex "[0:a][1:a]amix=inputs=2[a]" -map "[a]"

There is one legitimate case for touching both tracks. If a short test clip only covers part of the runtime, mute the base narration inside the PiP window with a time-gated volume instead of summing the two:

# base narration silent between t=12s and t=27s, where the PiP clip speaks
[0:a]volume='if(between(t,12,27),0,1)':eval=frame[base]

Why did our music bed silently disappear after adding the PiP?

Because we fixed the double voice by mapping the PiP clip's audio. Once captions and music are on the base recording, its track is narration plus ducked music, and it's the only place the music exists. InfiniteTalk's output is voice-only. Mapping 1:a discards the bed with no error. Map the scored base's audio, 0:a, instead.

These two audio failures pull in opposite directions, and that's why the second one shipped once. Fixing "two voices" by picking the avatar's track looks correct. Both tracks carry identical narration, so it sounds fine on a skim. It's quietly wrong on every production that has a music pass.

The rule is simple once you see it. When both tracks carry the same narration, take the one that also carries everything else, which is always the scored base and never the raw InfiniteTalk output. This is the composite we run now:

ffmpeg -i scored_base.mp4 -i talk_raw.mp4 \
  -filter_complex "[1:v]scale=420:-2[pip];[0:v][pip]overlay=W-w-24:H-h-24:shortest=1[v]" \
  -map "[v]" -map 0:a \
  -c:v libx264 -crf 18 -c:a copy \
  final.mp4

Note scale=420:-2 rather than -1: it keeps the clip's own aspect ratio and rounds the height to an even number, which libx264 needs. Don't force-fit a landscape clip into a square box.

The pre-ship check: volumedetect

"Plays fine but the music is gone" can't be heard on a quick skim, so we measure it. FFmpeg's volumedetect filter reports mean and peak levels without modifying the stream:

ffmpeg -hide_banner -i final.mp4 -af volumedetect -f null - 2>&1 \
  | grep -E "mean_volume|max_volume"

A ducked music bed under speech should land around -15 to -25 dB mean. The stricter version runs the same command on scored_base.mp4 and on a stretch where the narrator is silent, such as the intro. Because the final track is a stream copy of the base, the numbers should match. If the music-only window reads near silence, the bed is gone.

Why does the avatar keep snapping back to the same pose?

InfiniteTalk anchors every clip to the reference image's pose. Whatever the reference is doing becomes a "home position" the animation keeps returning to between gestures. A reference with a raised hand gives you a presenter whose hand keeps floating back up mid-sentence. Use a neutral reference: arms down, hands out of frame, head level, mouth at rest.

We learned this on our first character and haven't broken the rule since. When we generate outfit variants for a pool, we keep lighting, pose and framing identical and swap only the garment with an image-edit model. That's also why the orientation outlier above stood out once we found it: it was the one variant whose framing had changed.

Why wasn't plain loudnorm enough to fix a quiet voice-clone source?

Because a quiet recording has a level problem and a dynamics problem, and loudnorm only fixes the first. Our source measured -40 LUFS, from a good mic at low gain. After loudnorm alone the clone still sounded thin. What worked was a chain: acompressor to lift the quiet passages, alimiter as a safety ceiling, then loudnorm as the final polish.

ffmpeg -i raw_voice.wav \
  -af "acompressor=threshold=-30dB:ratio=4:makeup=12dB,alimiter=limit=0.9,loudnorm=I=-16:TP=-1.5:LRA=11" \
  -ac 1 -ar 48000 \
  voice_clone_source.wav

Two details matter here. First, loudnorm resamples to 192 kHz internally, so pin -ar back to the source rate. Second, don't downsample to 16 kHz at this stage. That conversion is only for InfiniteTalk's audio input, which is a separate downstream file.

We also chose Professional Voice Clone over Instant once we had more than about 90 seconds of clean audio. ElevenLabs' IVC vs PVC guidance positions Instant for short samples. In practice, the 11 MB upload cap on the Instant endpoint holds about 90 seconds of 16-bit mono WAV, so a longer recording never gets used. PVC trains on the full sample and won a six-way A/B comparison outright. The best IVC result was "close but a little odd."

One character should get one voice. If the presenter in the corner is also the narrator, point the whole video's TTS at that character's locked voice ID. An early production paired one face with a different narration voice, and the mismatch was the first thing flagged in review.

What other avatar-pipeline bug causes lip-sync drift that grows over time?

Building the avatar's audio from per-scene metadata instead of from the real rendered files. Each scene's sync.json reported timeline_duration as narration plus a 2-second tail, but the tail was never baked into the muxed scene.mp4. Padding six scenes added about 12 seconds of phantom silence. Lips were in sync at 0:33 and drifted further every scene after that.

The fix is to treat the rendered files as ground truth. Extract each scene's audio with ffmpeg -i scene.mp4 -vn ..., concatenate those, then ffprobe both the concatenated track and the base video. The difference should be well under a second, not ten. For one-shot productions, where TTS generates the full narration in one pass, the voiceover file is already the final audio track byte for byte, so this class of bug disappears.

The common thread is documentation versus reality. Every gotcha here came from trusting a description of the output (a parameter name, a metadata field, a clean exit code) instead of measuring the output itself. It's the same lesson we wrote up in why you shouldn't trust API docs but should trust shipping code. The screen-recording half of this pipeline had its own silent failure, covered in why time.sleep stalls your Playwright CDP screencast.

What is the full checklist for a RunPod InfiniteTalk PiP layer?

Six checks, each of which turned a silent failure into a loud one:

  • Keep every reference in the pool the same orientation and run a pool check before adding a new one.
  • ffprobe the raw clip's dimensions before compositing.
  • Budget 7–8 minutes of render per minute of audio, and submit as soon as the narration is final.
  • Replace the audio rather than mixing it, and map the scored base's track (-map 0:a).
  • Run volumedetect before shipping.
  • Use a neutral-pose reference and a compressed, limited, then normalized clone source.

InfiniteTalk itself is excellent and costs well under a dollar per video. The fragile parts are the seams around it: what decides orientation, which track carries the music, and which file is the real timeline. None of them fail loudly, so you have to make them.