Appearance

AI Video Sound Design & Foley Generation ​

Transform silent video clips (AI-generated video from Veo 3, Sora 2, Kling, Wan, or 3D animations) into cinematic, immersive audio-visual experiences with synchronized Foley sound effects, ambient environmental audio, and dynamically ducked background scores.

This recipe coordinates frame-level visual action detection, GPU-accelerated Foley synthesis via MMAudio on FOTOhub GPU infrastructure, cloud-scale soundtrack composition via MiniMax Music, and hardware-accelerated FFmpeg sidechain ducking and stream remuxing.

Why automated sound design?

Silent AI videos lack emotional impact. Hiring a Foley artist costs upwards of $150/hour, while manually syncing sound effects in Adobe Premiere or DaVinci Resolve takes hours per minute of footage. The FOTOhub API fully automates this pipeline for ~$0.045 USD per video.

Production Architecture ​

The system utilizes a 4-stage pipeline: Visual Perception, Foley Synthesis, Music Generation, and DSP Assembly.

Workflow Sequence ​

mermaid
sequenceDiagram
    autonumber
    actor Client as Automation Pipeline / Video Studio
    participant API as FotoHUB API (/v1/ai)
    participant Vision as Vision Perception Engine
    participant MMAudio as MMAudio Foley Engine
    participant Music as Music Engine (MiniMax / ElevenLabs)
    participant Mixer as FFmpeg NVENC Remuxer

    Client->>API: 1. Ingest silent video (MP4)
    API->>Vision: Extract keyframes & detect visual kinetic events
    Vision-->>API: Sound Cue Sheet (Timestamps, Actions, SFX prompts)
    loop For Each Sound Cue (Footsteps, Impact, Ambience)
        API->>MMAudio: 2. POST /v1/ai/generate/sfx (Prompt, Duration, CFG)
        MMAudio-->>API: 44.1 kHz Studio WAV stem
    end
    API->>Music: 3. POST /v1/ai/generate/music (Mood, Genre, BPM, Duration)
    Music-->>API: Finished Instrumental Music Track (MP3)
    API->>Mixer: 4. Assemble stem timeline & apply sidechain ducking (-16dB)
    Mixer->>Mixer: Master EBU R128 loudness (-14 LUFS) & copy video stream
    Mixer-->>API: Completed master video with synchronized audio
    API-->>Client: Final Video URL + Audio Stems Breakdown

System Architecture Flowchart ​

mermaid
flowchart TD
    A[Client Request] --> B(API Gateway / Rate Limiter)
    B --> C{Pipeline Orchestrator}
    C -->|1. Analyze| D[Vision Perception Engine]
    D --> E[Cue Sheet JSON]
    E -->|2. Parallel Generation| F(MMAudio Engine)
    E -->|3. Parallel Generation| G(MiniMax Music Engine)
    E -->|3. Parallel Generation| H(Narrator Voice Engine)
    F --> I[SFX Stems]
    G --> J[Music Bed]
    H --> K[Voice Over Stems]
    I --> L(DSP & Assembly Engine)
    J --> L
    K --> L
    L -->|Sidechain Ducking| M[FFmpeg NVENC Remuxer]
    M --> N[Master Rendered MP4]
    N --> O(AWS S3 / R2 Export)
    O --> P[Webhook Notification to Client]

1. Visual Cue Detection Deep-Dive ​

To automatically synthesize audio, the system must first "see" what is happening in the video. The POST /v1/ai/analyze/image endpoint analyzes video keyframes to detect kinetic events and map them to timestamps.

Kinetic Event Types Detected ​

  • Impacts: Collisions, punches, doors slamming, objects dropping.
  • Footsteps: Walking or running on various surfaces (concrete, water, snow, gravel).
  • Water/Liquid: Splashes, pouring, rain, underwater ambience.
  • Explosions: Fire, pyrotechnics, blasts.
  • Vehicles: Engine revs, tire screeches, aircraft flyovers.
  • Atmospheric: Wind, rustling leaves, background city noise.

Confidence Threshold

The Vision engine requires a confidence score of 0.85+ to trigger a discrete Foley event. If a scene is too dark or motion is blurry, the engine falls back to generating a continuous Ambient track instead of precise Foley hits.

2. Sound Cue Taxonomy ​

The extracted cue sheet categorizes sounds into distinct architectural layers:

  1. Ambient: Continuous background noise that sets the spatial context. (e.g., "Heavy rain in a cyberpunk city"). Usually loops or spans the entire duration.
  2. Foley: Synchronized, everyday sounds created by character actions. (e.g., "Leather boots walking on wet pavement").
  3. Impact: High-energy transient sounds denoting significant collisions. (e.g., "Deep metallic sci-fi explosion").
  4. Voice: Generated dialogue or narration using /v1/ai/generate/voice.
  5. Music: The emotional background score, generated via /v1/ai/generate/music.

3. MMAudio Technical Specification ​

FOTOhub utilizes Sony's MMAudio (mmaudio-large_44k_v2) architecture for Foley and SFX synthesis.

  • Hardware: Deployed on dedicated FOTOhub GPU workers.
  • Architecture: Flow-matching diffusion model.
  • Output Quality: Studio-grade 44.1 kHz / 48 kHz, 16-bit PCM WAV.
  • CFG Scale (Classifier-Free Guidance): Controls prompt adherence. Default is 4.5. Lower values (2.0-3.0) produce more abstract, creative sounds; higher values (6.0-8.0) strictly follow the prompt but may introduce artifacting.

4. API Endpoints & Parameter Tables ​

Video Analysis (POST /v1/ai/analyze/image) ​

Extracts the sound cue sheet from a silent video.

ParameterTypeRequiredDefaultDescription
image_urlstringYes—URL of the silent video (MP4/WebM) to analyze.
featuresarrayYes—Array of features to extract. Must include "actions", "scene_detection", "motion_tracking".
confidence_thresholdfloatNo0.85Minimum confidence to register a kinetic event.

Complete Sound Cue Sheet JSON Format ​

The API returns a meticulously structured cue sheet:

json
{
  "duration_s": 15.0,
  "cues": [
    {
      "cue_id": "cue_8f9a2b",
      "timestamp_s": 0.0,
      "duration_s": 15.0,
      "category": "ambient",
      "prompt": "Continuous heavy night rain pouring on asphalt with distant city traffic rumble",
      "volume_db": -12.0,
      "pan": 0.0
    },
    {
      "cue_id": "cue_1c3d4e",
      "timestamp_s": 2.4,
      "duration_s": 3.0,
      "category": "foley",
      "prompt": "Heavy combat boots walking steadily through water puddles with distinct wet splashes",
      "volume_db": -6.0,
      "pan": 0.2
    },
    {
      "cue_id": "cue_9f8e7d",
      "timestamp_s": 11.5,
      "duration_s": 3.5,
      "category": "impact",
      "prompt": "Low sub-bass cinematic impact braam boom with electrical sparks crackle",
      "volume_db": 0.0,
      "pan": 0.0
    }
  ]
}

MMAudio Foley Generation (POST /v1/ai/generate/sfx) ​

ParameterTypeRequiredDefaultDescription
promptstringYes—Description of the sound effect.
duration_sfloatNo5.0Duration in seconds (range: 0.5–30.0s).
stepsintegerNo25Diffusion steps. Higher = better quality. Range: 10-50.
cfg_strengthfloatNo4.5Classifier-free guidance scale.
seedintegerNorandomInteger seed for reproducibility.
sample_rateintegerNo44100Output sample rate (44100 or 48000 Hz).
formatstringNo"wav"Output format ("wav" or "mp3").
negative_promptstringNo"music, speech, singing"Audio attributes to avoid.

Music Generation (POST /v1/ai/generate/music) ​

Generates background scores via MiniMax.

ParameterTypeRequiredDefaultDescription
promptstringYes—Musical description (instruments, tempo, style).
duration_sintegerNo30Duration in seconds (range: 10–300s). Billed per started minute.
genrestringNo—Genre hint ("cinematic", "electronic", "ambient").
moodstringNo—Mood modifier ("dark", "energetic", "mysterious").
bpmintegerNoautoBeats per minute tempo target (range: 60–200).
keystringNoautoMusical key (e.g., "C minor", "D major").
instrumentalbooleanNotrueTrue guarantees no vocal hallucinations.
loopbooleanNofalseRenders seamless looping boundaries.

There is no server-side "compose" endpoint ​

Corrected 2026-09

Earlier revisions of this page documented a POST /v1/ai/compose/video-audio "master orchestrator" that took a silent video plus a cue sheet and returned a fully mixed, ducked, loudness-mastered video in one async job — with webhook notification and a BYOB stem export. That endpoint does not exist. There is no server-side mixing, sidechain ducking, or loudness mastering capability on the public API.

What is real: POST /v1/ai/generate/sfx and POST /v1/ai/generate/music (documented above) each return a finished audio file — synchronously, no job_id, no polling. Assembling those stems onto the silent video — timeline placement, sidechain ducking, and EBU R128 mastering — is your own FFmpeg step, same as the "DSP & Assembly Engine" box in the architecture diagram above describes; it is not something you can hand to the API in a single call.

5. Genre-Specific Presets ​

To streamline music generation, you can use these proven parameter combinations:

  1. Cinematic Action:
    • Prompt: "Massive orchestral hybrid score, aggressive string ostinatos, heavy brass, huge taiko drums."
    • Mood: energetic, Genre: cinematic, BPM: 130, Key: D minor
  2. Horror Ambient:
    • Prompt: "Unsettling atonal drone, scraping metallic textures, low rumbly sub-bass, dissonant string clusters."
    • Mood: dark, Genre: ambient, BPM: 60, Key: C minor
  3. Nature Documentary:
    • Prompt: "Sweeping orchestral, gentle harp, soaring woodwinds, majestic and awe-inspiring."
    • Mood: uplifting, Genre: orchestral, BPM: 85, Key: G major
  4. Corporate Explainer:
    • Prompt: "Light acoustic guitar, gentle marimba, upbeat hand claps, optimistic corporate tech."
    • Mood: happy, Genre: acoustic, BPM: 110, Key: C major
  5. Lo-Fi Chill:
    • Prompt: "Dusty vinyl boom bap beat, warm rhodes piano chords, relaxed mellow groove."
    • Mood: chill, Genre: lofi, BPM: 75, Key: F major

6. The 8-Stage DSP Audio Chain ​

Once all audio stems (Foley, Ambience, Music, Voice) are generated, FOTOhub employs a broadcast-grade Digital Signal Processing (DSP) chain using FFmpeg filters.

  1. High-Pass Filter (80 Hz): highpass=f=80 removes sub-sonic rumble from Foley tracks to prevent muddying the mix.
  2. FFT Denoiser: afftdn=nr=12:nf=-25 cleans up minor artifacts from the diffusion synthesis.
  3. Noise Gate: agate=threshold=0.01:ratio=10 silences the noise floor between discrete impacts.
  4. De-Esser: firequalizer dynamically reduces harsh sibilance in the 4-9 kHz range.
  5. Compressor: acompressor=threshold=-20dB:ratio=4:makeup=2 glues the stems together for consistent dynamics.
  6. Presence EQ: A gentle 2-5 kHz boost (equalizer=f=3500:width_type=o:width=2:g=3) enhances vocal intelligibility and transient punch.
  7. Brickwall Limiter: Ensures peaks never exceed the absolute digital ceiling, capping at -0.95 dBFS.
  8. 2-Pass EBU R128: loudnorm=I=-14:TP=-1.5:LRA=11 standardizes the final master to broadcast loudness specs (-14 LUFS is the standard for YouTube/Spotify).

Sidechain Ducking Mechanics ​

The most critical aspect of the mix is Sidechain Ducking. When an explosion or voice-over occurs, the music must temporarily reduce in volume. We use the sidechaincompress filter:

  • Music is routed to Input 1.
  • Foley/Voice is routed to Input 2 (the sidechain key).
  • When Foley exceeds the threshold, the music ducks by -16dB.
  • Attack: 20ms (fast dip). Release: 250ms (smooth recovery).

7. Timeline Assembly ​

Merging multiple audio tracks precisely to their action timestamps is critical for realistic Foley.

FFmpeg uses the adelay filter to offset stems to their correct timestamp. For example, a sound cue scheduled at 2.4 seconds receives a delay of 2400 milliseconds (adelay=2400|2400). Both the left and right channels are delayed. After delaying each stem, they are mixed together with the amix filter before the sidechain compression phase.

8. Production Code Examples ​

Here is how to automate the real, synchronous part of the pipeline using the FOTOhub API.

Each real call below is synchronous — there is no job_id to poll and no webhook for these two endpoints. Mixing the returned stems onto the video is a local FFmpeg step, not an API call.

python
import os
import requests

API_KEY = os.environ.get("FOTOHUB_API_KEY", "fh_live_your_api_key")
BASE_URL = "https://apis.fotohub.app/v1"
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

def run_automated_sound_design(video_url: str):
    print("1. Extracting Cue Sheet...")
    analyze_res = requests.post(
        f"{BASE_URL}/ai/analyze/image",
        headers=HEADERS,
        json={"image_url": video_url, "features": ["actions"]}
    ).json()

    cue_sheet = analyze_res.get("cues", [])

    print("2. Synthesizing an SFX stem per cue (synchronous)...")
    sfx_urls = []
    for cue in cue_sheet:
        sfx_res = requests.post(
            f"{BASE_URL}/ai/generate/sfx",
            headers=HEADERS,
            json={"prompt": cue["sfx_prompt"], "duration": cue.get("duration", 3)}
        ).json()
        sfx_urls.append({"start_s": cue["timestamp"], "url": sfx_res["audio_url"]})

    print("3. Generating the music bed (synchronous)...")
    music_res = requests.post(
        f"{BASE_URL}/ai/generate/music",
        headers=HEADERS,
        json={"prompt": "Cinematic dark synthwave, 110 bpm", "duration": 15, "instrumental": True}
    ).json()
    music_url = music_res["audio_url"]

    print("4. Mixing stems onto the video (local FFmpeg — not an API call)...")
    # Build an `adelay`+`amix`+sidechain `ffmpeg` filtergraph from `sfx_urls`
    # and `music_url` here; see "Timeline Assembly" above for the filter shape.
    return {"sfx": sfx_urls, "music": music_url}
typescript
import fetch from "node-fetch";

const API_KEY = process.env.FOTOHUB_API_KEY || "fh_live_your_api_key";
const BASE_URL = "https://apis.fotohub.app/v1";
const HEADERS = {
  "Authorization": `Bearer ${API_KEY}`,
  "Content-Type": "application/json"
};

async function runSoundDesign(videoUrl: string) {
  console.log("1. Extracting Cue Sheet...");
  const analyzeRes = await fetch(`${BASE_URL}/ai/analyze/image`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({ image_url: videoUrl, features: ["actions"] })
  });
  const analyzeData = await analyzeRes.json();

  console.log("2. Synthesizing SFX stems (synchronous, one call per cue)...");
  const sfxUrls: string[] = [];
  for (const cue of analyzeData.cues ?? []) {
    const sfxRes = await fetch(`${BASE_URL}/ai/generate/sfx`, {
      method: "POST",
      headers: HEADERS,
      body: JSON.stringify({ prompt: cue.sfx_prompt, duration: cue.duration ?? 3 })
    });
    const sfxData = await sfxRes.json();
    sfxUrls.push(sfxData.audio_url);
  }

  console.log("3. Generating the music bed (synchronous)...");
  const musicRes = await fetch(`${BASE_URL}/ai/generate/music`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({ prompt: "Cinematic dark synthwave, 110 bpm", duration: 15, instrumental: true })
  });
  const musicData = await musicRes.json();

  console.log("4. Mix sfxUrls + musicData.audio_url onto the video locally with FFmpeg.");
  return { sfxUrls, music: musicData.audio_url };
}

runSoundDesign("https://storage.fotohub.app/raw/silent_cyberpunk.mp4");
bash
# 1. Analyze Video for Cues
curl -X POST https://apis.fotohub.app/v1/ai/analyze/image \
  -H "Authorization: Bearer fh_live_your_api_key" -H "Content-Type: application/json" -d '{
    "image_url": "https://storage.fotohub.app/raw/silent.mp4",
    "features": ["actions"]
  }'

# 2. Generate one SFX stem (synchronous — the audio URL comes back in this response)
curl -X POST https://apis.fotohub.app/v1/ai/generate/sfx \
  -H "Authorization: Bearer fh_live_your_api_key" -H "Content-Type: application/json" -d '{
    "prompt": "Heavy footstep on wet concrete",
    "duration": 3
  }'

# 3. Generate the music bed (also synchronous)
curl -X POST https://apis.fotohub.app/v1/ai/generate/music \
  -H "Authorization: Bearer fh_live_your_api_key" -H "Content-Type: application/json" -d '{
    "prompt": "Cinematic orchestral",
    "duration": 15,
    "instrumental": true
  }'

# 4. Mix the returned stems onto the video with your own FFmpeg pipeline (no API call)

9. Webhook Handling & Security ​

The SFX and music generation calls used in this recipe are synchronous and do not fire webhooks — there is no webhook_url parameter on either, and no job to notify you about. This section is a general reference for webhooks elsewhere on the platform (e.g. generation.completed, shorts.job.completed — register via POST /v1/console/webhooks, see the Webhooks guide); it does not apply to the pipeline built above.

FOTOhub signs webhook payloads using HMAC-SHA256. Verify the signature using the per-webhook secret returned when you create the webhook (not your API key).

Python FastAPI Webhook Handler ​

python
from fastapi import FastAPI, Request, HTTPException
import hmac
import hashlib
import os

app = FastAPI()
API_KEY = os.environ.get("FOTOHUB_API_KEY", "fh_live_your_api_key").encode()

@app.post("/webhooks/fotohub")
async def handle_webhook(request: Request):
    signature = request.headers.get("X-FotoHUB-Signature")
    payload = await request.body()
    
    # Verify HMAC-SHA256 signature
    expected_sig = hmac.new(API_KEY, payload, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected_sig, signature):
        raise HTTPException(status_code=401, detail="Invalid signature")
    
    data = await request.json()
    if data["status"] == "completed":
        print(f"Video ready: {data['result']['video_url']}")
    
    return {"received": True}

TypeScript Express Webhook Handler ​

typescript
import express from 'express';
import crypto from 'crypto';

const app = express();
const API_KEY = process.env.FOTOHUB_API_KEY || 'fh_live_your_api_key';

app.post('/webhooks/fotohub', express.raw({ type: 'application/json' }), (req, res) => {
  const signature = req.headers['x-fotohub-signature'] as string;
  const expectedSig = crypto
    .createHmac('sha256', API_KEY)
    .update(req.body)
    .digest('hex');

  if (signature !== expectedSig) {
    return res.status(401).send('Invalid signature');
  }

  const data = JSON.parse(req.body.toString());
  if (data.status === 'completed') {
    console.log(`Video ready: ${data.result.video_url}`);
  }

  res.send({ received: true });
});

app.listen(3000);

10. There is no async job, DLQ, or batch endpoint for this pipeline ​

POST /v1/ai/generate/sfx and POST /v1/ai/generate/music are both synchronous: no 202, no job_id, no polling, no Dead Letter Queue, no auto-retry, and no POST /v1/ai/compose/batch to submit many at once. If a call fails you get a 4xx/5xx directly and the wallet is refunded — retry it yourself. For genuinely large studio runs, rate-limit your own client to the per-endpoint cap in Rate Limits rather than relying on a server-side batch queue that does not exist.

11. Delivering stems to your own storage ​

There is no export_s3 parameter on these endpoints, and no capability that writes an unmixed stem breakdown to your bucket automatically. What is real: download each audio_url FOTOhub returns and upload it to your own storage, or provision a FOTOhub-managed S3 bucket via GET,POST /v1/storage/s3/buckets and push the files there yourself with the standard object endpoints.

12. Manual FFmpeg Filter Chain Examples ​

If you prefer to download the raw stems and perform the mixing locally on your own infrastructure, here is the raw FFmpeg command used by the pipeline:

bash
ffmpeg -y -i silent_video.mp4   -i bg_music.mp3   -i foley_1.wav   -i foley_2.wav   -filter_complex "     [2:a]adelay=2400|2400[sfx1];     [3:a]adelay=8100|8100[sfx2];     [sfx1][sfx2]amix=inputs=2:normalize=0[foley_mix];     [1:a][foley_mix]sidechaincompress=threshold=0.08:ratio=4:attack=20:release=250[ducked_music];     [ducked_music][foley_mix]amix=inputs=2:duration=first:weights=0.8 1.2,loudnorm=I=-14:TP=-1.5:LRA=11[aout]   "   -map 0:v -map "[aout]"   -c:v copy -c:a aac -b:a 320k   -shortest final_output.mp4

Delay Explanation

The adelay=2400|2400 filter shifts the audio stem exactly 2.4 seconds (2400 milliseconds) into the timeline for both left and right channels to align perfectly with the visual action.

13. Common Failure Modes & Troubleshooting ​

Error CodeMeaningResolution
VISION_SCENE_DARKThe video is too dark for the perception engine to detect actions.Pass a manual cue_sheet or fall back to ambient-only generation.
MMAUDIO_OOMThe GPU worker ran out of memory. Usually caused by requesting a single SFX duration >30s.Split the Foley request into multiple smaller segments (e.g., 2x 15s).
MUSIC_GENRE_INVALIDElevenLabs rejected the genre tag.Use standard genres (cinematic, electronic, rock).
MIXER_CLIPPINGAudio exceeded 0 dBFS during stem summation.Lower volume_db parameters in the cue sheet, or rely on the loudnorm filter.
JOB_TIMEOUTThe FFmpeg mix took longer than 60s.Usually implies the source video is >10 minutes long. Chunk processing is recommended.

14. Unit Economics & ROI Table ​

All operations strictly deduct from your USD balance (wallet.available_usd). No credits, no synthetic tokens.

ComponentProvider / EnginePrice in USDBilling Unit
Video Cue PerceptionFOTOhub Vision$0.005Per 10 seconds of video
Foley / SFX SynthesisMMAudio$0.008Per 5 seconds of audio
MiniMax Music BedMiniMax Cloud$0.015Per started minute (1–60s)
ElevenLabs MusicElevenLabs$0.050Per started minute
DSP / Remux PassFOTOhub Worker$0.010Per 30 seconds of video

ROI Scenario: 1-Minute Marketing Video ​

  • Manual Foley Artist + Studio Time: ~$150.00
  • FOTOhub AI Full Pipeline:
    • Perception (60s): $0.030
    • 10x SFX (50s total): $0.080
    • 60s Music (MiniMax): $0.015
    • DSP Mixing (60s): $0.020
    • Total Cost: $0.145 USD

Cost Optimization

You can cache and reuse background music tracks across multiple videos by passing an existing music_track_url instead of generating a new one every time, saving $0.015 per video.

© 2026 FOTOhub. All rights reserved.