AI Video Sound Design & Foley Generation

Transform silent video clips (AI-generated video from Veo 3, Sora 2, Kling, Wan, or 3D animations) into cinematic, immersive audio-visual experiences with synchronized Foley sound effects, ambient environmental audio, and dynamically ducked background scores.

This recipe coordinates frame-level visual action detection, GPU-accelerated Foley synthesis via MMAudio (server/mmaudio-server/ on GPU2), cloud-scale soundtrack composition via MiniMax Music, and hardware-accelerated FFmpeg sidechain ducking and stream remuxing.

Why automated sound design?

Silent AI videos lack emotional impact. Hiring a Foley artist costs upwards of $150/hour, while manually syncing sound effects in Adobe Premiere or DaVinci Resolve takes hours per minute of footage. The FOTOhub API fully automates this pipeline for ~$0.045 USD per video.

---

Production Architecture

The system utilizes a 4-stage pipeline: Visual Perception, Foley Synthesis, Music Generation, and DSP Assembly.

Workflow Sequence

sequenceDiagram
    autonumber
    actor Client as Automation Pipeline / Video Studio
    participant API as FotoHUB API (/v1/ai)
    participant Vision as Vision Perception Engine
    participant MMAudio as MMAudio Foley Engine (GPU2 :8799)
    participant Music as Music Engine (MiniMax / ElevenLabs)
    participant Mixer as FFmpeg NVENC Remuxer

    Client->>API: 1. Ingest silent video (MP4)
    API->>Vision: Extract keyframes & detect visual kinetic events
    Vision-->>API: Sound Cue Sheet (Timestamps, Actions, SFX prompts)
    loop For Each Sound Cue (Footsteps, Impact, Ambience)
        API->>MMAudio: 2. POST /v1/ai/generate/sfx (Prompt, Duration, CFG)
        MMAudio-->>API: 44.1 kHz Studio WAV stem
    end
    API->>Music: 3. POST /v1/ai/generate/music (Mood, Genre, BPM, Duration)
    Music-->>API: Finished Instrumental Music Track (MP3)
    API->>Mixer: 4. Assemble stem timeline & apply sidechain ducking (-16dB)
    Mixer->>Mixer: Master EBU R128 loudness (-14 LUFS) & copy video stream
    Mixer-->>API: Completed master video with synchronized audio
    API-->>Client: Final Video URL + Audio Stems Breakdown

System Architecture Flowchart

flowchart TD
    A[Client Request] --> B(API Gateway / Rate Limiter)
    B --> C{Pipeline Orchestrator}
    C -->|1. Analyze| D[Vision Perception Engine]
    D --> E[Cue Sheet JSON]
    E -->|2. Parallel Generation| F(MMAudio Engine - GPU2)
    E -->|3. Parallel Generation| G(MiniMax Music Engine)
    E -->|3. Parallel Generation| H(Narrator Voice Engine)
    F --> I[SFX Stems]
    G --> J[Music Bed]
    H --> K[Voice Over Stems]
    I --> L(DSP & Assembly Engine)
    J --> L
    K --> L
    L -->|Sidechain Ducking| M[FFmpeg NVENC Remuxer]
    M --> N[Master Rendered MP4]
    N --> O(AWS S3 / R2 Export)
    O --> P[Webhook Notification to Client]

---

1. Visual Cue Detection Deep-Dive

To automatically synthesize audio, the system must first "see" what is happening in the video. The POST /v1/ai/analyze/image endpoint analyzes video keyframes to detect kinetic events and map them to timestamps.

Kinetic Event Types Detected

• Impacts: Collisions, punches, doors slamming, objects dropping.
• Footsteps: Walking or running on various surfaces (concrete, water, snow, gravel).
• Water/Liquid: Splashes, pouring, rain, underwater ambience.
• Explosions: Fire, pyrotechnics, blasts.
• Vehicles: Engine revs, tire screeches, aircraft flyovers.
• Atmospheric: Wind, rustling leaves, background city noise.

Confidence Threshold

The Vision engine requires a confidence score of 0.85+ to trigger a discrete Foley event. If a scene is too dark or motion is blurry, the engine falls back to generating a continuous Ambient track instead of precise Foley hits.

---

2. Sound Cue Taxonomy

The extracted cue sheet categorizes sounds into distinct architectural layers:

1. Ambient: Continuous background noise that sets the spatial context. (e.g., "Heavy rain in a cyberpunk city"). Usually loops or spans the entire duration.
2. Foley: Synchronized, everyday sounds created by character actions. (e.g., "Leather boots walking on wet pavement").
3. Impact: High-energy transient sounds denoting significant collisions. (e.g., "Deep metallic sci-fi explosion").
4. Voice: Generated dialogue or narration using /v1/ai/generate/voice.
5. Music: The emotional background score, generated via /v1/ai/generate/music.

---

3. MMAudio Technical Specification

FOTOhub utilizes Sony's MMAudio (mmaudio-large_44k_v2) architecture for Foley and SFX synthesis.

• Hardware: Deployed on dedicated NVIDIA A10G instances (GPU2).
• Architecture: Flow-matching diffusion model.
• Output Quality: Studio-grade 44.1 kHz / 48 kHz, 16-bit PCM WAV.
• CFG Scale (Classifier-Free Guidance): Controls prompt adherence. Default is 4.5. Lower values (2.0-3.0) produce more abstract, creative sounds; higher values (6.0-8.0) strictly follow the prompt but may introduce artifacting.

GPU Affinity

MMAudio strictly runs on GPU2. Ensure your workloads do not attempt to invoke MuseTalk (GPU3) or 3D generation (GPU4/5) simultaneously if you are self-hosting on a single cluster. FOTOhub Cloud handles this load balancing automatically.

---

4. API Endpoints & Parameter Tables

Video Analysis (POST /v1/ai/analyze/image)

Extracts the sound cue sheet from a silent video.

Parameter	Type	Required	Default	Description
image_url	string	Yes	—	URL of the silent video (MP4/WebM) to analyze.
features	array	Yes	—	Array of features to extract. Must include "actions", "scene_detection", "motion_tracking".
confidence_threshold	float	No	0.85	Minimum confidence to register a kinetic event.

Complete Sound Cue Sheet JSON Format

The API returns a meticulously structured cue sheet:

{
  "duration_s": 15.0,
  "cues": [
    {
      "cue_id": "cue_8f9a2b",
      "timestamp_s": 0.0,
      "duration_s": 15.0,
      "category": "ambient",
      "prompt": "Continuous heavy night rain pouring on asphalt with distant city traffic rumble",
      "volume_db": -12.0,
      "pan": 0.0
    },
    {
      "cue_id": "cue_1c3d4e",
      "timestamp_s": 2.4,
      "duration_s": 3.0,
      "category": "foley",
      "prompt": "Heavy combat boots walking steadily through water puddles with distinct wet splashes",
      "volume_db": -6.0,
      "pan": 0.2
    },
    {
      "cue_id": "cue_9f8e7d",
      "timestamp_s": 11.5,
      "duration_s": 3.5,
      "category": "impact",
      "prompt": "Low sub-bass cinematic impact braam boom with electrical sparks crackle",
      "volume_db": 0.0,
      "pan": 0.0
    }
  ]
}

MMAudio Foley Generation (POST /v1/ai/generate/sfx)

Parameter	Type	Required	Default	Description
prompt	string	Yes	—	Description of the sound effect.
duration_s	float	No	5.0	Duration in seconds (range: 0.5–30.0s).
steps	integer	No	25	Diffusion steps. Higher = better quality. Range: 10-50.
cfg_strength	float	No	4.5	Classifier-free guidance scale.
seed	integer	No	random	Integer seed for reproducibility.
sample_rate	integer	No	44100	Output sample rate (44100 or 48000 Hz).
format	string	No	"wav"	Output format ("wav" or "mp3").
negative_prompt	string	No	"music, speech, singing"	Audio attributes to avoid.

Music Generation (POST /v1/ai/generate/music)

Generates background scores via MiniMax.

Parameter	Type	Required	Default	Description
prompt	string	Yes	—	Musical description (instruments, tempo, style).
duration_s	integer	No	30	Duration in seconds (range: 10–300s). Billed per started minute.
genre	string	No	—	Genre hint ("cinematic", "electronic", "ambient").
mood	string	No	—	Mood modifier ("dark", "energetic", "mysterious").
bpm	integer	No	auto	Beats per minute tempo target (range: 60–200).
key	string	No	auto	Musical key (e.g., "C minor", "D major").
instrumental	boolean	No	true	True guarantees no vocal hallucinations.
loop	boolean	No	false	Renders seamless looping boundaries.

Video/Audio Composition (POST /v1/ai/compose/video-audio)

The master orchestrator endpoint that handles the full pipeline in a single async job.

Parameter	Type	Required	Default	Description
video_url	string	Yes	—	The source silent video.
cue_sheet	object	Yes	—	The JSON cue sheet from the perception pass.
music_track_url	string	No	—	Pre-generated music track, or omitted to auto-generate.
ducking_db	float	No	-16.0	Amount of sidechain ducking applied to music.
target_lufs	float	No	-14.0	Master EBU R128 loudness target.
export_s3	object	No	—	BYOB S3/R2 export configuration.
webhook_url	string	No	—	URL to notify upon completion (HMAC-SHA256 secured).

---

5. Genre-Specific Presets

To streamline music generation, you can use these proven parameter combinations:

1. Cinematic Action:
   • Prompt: "Massive orchestral hybrid score, aggressive string ostinatos, heavy brass, huge taiko drums."
   • Mood: energetic, Genre: cinematic, BPM: 130, Key: D minor
2. Horror Ambient:
   • Prompt: "Unsettling atonal drone, scraping metallic textures, low rumbly sub-bass, dissonant string clusters."
   • Mood: dark, Genre: ambient, BPM: 60, Key: C minor
3. Nature Documentary:
   • Prompt: "Sweeping orchestral, gentle harp, soaring woodwinds, majestic and awe-inspiring."
   • Mood: uplifting, Genre: orchestral, BPM: 85, Key: G major
4. Corporate Explainer:
   • Prompt: "Light acoustic guitar, gentle marimba, upbeat hand claps, optimistic corporate tech."
   • Mood: happy, Genre: acoustic, BPM: 110, Key: C major
5. Lo-Fi Chill:
   • Prompt: "Dusty vinyl boom bap beat, warm rhodes piano chords, relaxed mellow groove."
   • Mood: chill, Genre: lofi, BPM: 75, Key: F major

---

6. The 8-Stage DSP Audio Chain

Once all audio stems (Foley, Ambience, Music, Voice) are generated, FOTOhub employs a broadcast-grade Digital Signal Processing (DSP) chain using FFmpeg filters.

1. High-Pass Filter (80 Hz): highpass=f=80 removes sub-sonic rumble from Foley tracks to prevent muddying the mix.
2. FFT Denoiser: afftdn=nr=12:nf=-25 cleans up minor artifacts from the diffusion synthesis.
3. Noise Gate: agate=threshold=0.01:ratio=10 silences the noise floor between discrete impacts.
4. De-Esser: firequalizer dynamically reduces harsh sibilance in the 4-9 kHz range.
5. Compressor: acompressor=threshold=-20dB:ratio=4:makeup=2 glues the stems together for consistent dynamics.
6. Presence EQ: A gentle 2-5 kHz boost (equalizer=f=3500:width_type=o:width=2:g=3) enhances vocal intelligibility and transient punch.
7. Brickwall Limiter: Ensures peaks never exceed the absolute digital ceiling, capping at -0.95 dBFS.
8. 2-Pass EBU R128: loudnorm=I=-14:TP=-1.5:LRA=11 standardizes the final master to broadcast loudness specs (-14 LUFS is the standard for YouTube/Spotify).

Sidechain Ducking Mechanics

The most critical aspect of the mix is Sidechain Ducking. When an explosion or voice-over occurs, the music must temporarily reduce in volume. We use the sidechaincompress filter:

• Music is routed to Input 1.
• Foley/Voice is routed to Input 2 (the sidechain key).
• When Foley exceeds the threshold, the music ducks by -16dB.
• Attack: 20ms (fast dip). Release: 250ms (smooth recovery).

---

7. Timeline Assembly

Merging multiple audio tracks precisely to their action timestamps is critical for realistic Foley.

FFmpeg uses the adelay filter to offset stems to their correct timestamp. For example, a sound cue scheduled at 2.4 seconds receives a delay of 2400 milliseconds (adelay=2400|2400). Both the left and right channels are delayed. After delaying each stem, they are mixed together with the amix filter before the sidechain compression phase.

---

8. 4-Way Production Code Examples

Here is how to automate the full pipeline using the FOTOhub API.

Python

import os
import requests
import hmac
import hashlib
import time

API_KEY = os.environ.get("FOTOHUB_API_KEY", "fh_live_your_api_key")
BASE_URL = "https://apis.fotohub.app/v1"
HEADERS = {"Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"}

def run_automated_sound_design(video_url: str):
    print("1. Extracting Cue Sheet...")
    analyze_res = requests.post(
        f"{BASE_URL}/ai/analyze/image",
        headers=HEADERS,
        json={"image_url": video_url, "features": ["actions"]}
    ).json()
    
    cue_sheet = analyze_res.get("cues", [])
    
    print("2. Submitting Async Composition Job...")
    compose_res = requests.post(
        f"{BASE_URL}/ai/compose/video-audio",
        headers=HEADERS,
        json={
            "video_url": video_url,
            "cue_sheet": {"duration_s": 15.0, "cues": cue_sheet},
            "music_prompt": "Cinematic dark synthwave, 110 bpm",
            "webhook_url": "https://your-server.com/webhooks/fotohub"
        }
    ).json()
    
    job_id = compose_res.get("job_id")
    print(f"Job {job_id} submitted. Polling for completion...")
    
    # 3. Async Job Polling Pattern
    while True:
        status_res = requests.get(f"{BASE_URL}/jobs/{job_id}", headers=HEADERS).json()
        if status_res["status"] == "completed":
            print(f"Success! Final Video: {status_res['result']['video_url']}")
            break
        elif status_res["status"] in ["failed", "dlq"]:
            print(f"Job failed: {status_res['error']}")
            break
        time.sleep(5)

TypeScript

import fetch from "node-fetch";

const API_KEY = process.env.FOTOHUB_API_KEY || "fh_live_your_api_key";
const BASE_URL = "https://apis.fotohub.app/v1";
const HEADERS = {
  "Authorization": `Bearer ${API_KEY}`,
  "Content-Type": "application/json"
};

async function runSoundDesign(videoUrl: string) {
  console.log("1. Extracting Cue Sheet...");
  const analyzeRes = await fetch(`${BASE_URL}/ai/analyze/image`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({ image_url: videoUrl, features: ["actions"] })
  });
  const analyzeData = await analyzeRes.json();
  
  console.log("2. Submitting Async Composition Job...");
  const composeRes = await fetch(`${BASE_URL}/ai/compose/video-audio`, {
    method: "POST",
    headers: HEADERS,
    body: JSON.stringify({
      video_url: videoUrl,
      cue_sheet: { duration_s: 15.0, cues: analyzeData.cues },
      music_prompt: "Cinematic dark synthwave, 110 bpm",
      export_s3: {
        bucket: "my-studio-bucket",
        endpoint: "s3.amazonaws.com",
        access_key: "AKIA...",
        secret_key: "..."
      }
    })
  });
  
  const composeData = await composeRes.json();
  const jobId = composeData.job_id;
  
  console.log(`Job ${jobId} submitted. Awaiting Webhook or Polling...`);
  // Note: For production, rely on webhooks rather than active polling.
}

runSoundDesign("https://storage.fotohub.app/raw/silent_cyberpunk.mp4");

Go

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
	"time"
)

const apiKey = "fh_live_your_api_key"
const baseURL = "https://apis.fotohub.app/v1"

func main() {
	client := &http.Client{Timeout: 30 * time.Second}

	// Submit Job
	reqBody, _ := json.Marshal(map[string]interface{}{
		"video_url": "https://storage.fotohub.app/raw/silent.mp4",
		"music_prompt": "Cinematic ambient score",
	})
	
	req, _ := http.NewRequest("POST", baseURL+"/ai/compose/video-audio", bytes.NewBuffer(reqBody))
	req.Header.Set("Authorization", "Bearer "+apiKey)
	req.Header.Set("Content-Type", "application/json")
	
	resp, _ := client.Do(req)
	var result map[string]interface{}
	json.NewDecoder(resp.Body).Decode(&result)
	jobID := result["job_id"].(string)
	
	fmt.Printf("Job ID: %s. Polling...
", jobID)
	
	for {
		statusReq, _ := http.NewRequest("GET", baseURL+"/jobs/"+jobID, nil)
		statusReq.Header.Set("Authorization", "Bearer "+apiKey)
		statusResp, _ := client.Do(statusReq)
		
		var status map[string]interface{}
		json.NewDecoder(statusResp.Body).Decode(&status)
		
		if status["status"] == "completed" {
			fmt.Printf("Done! URL: %v
", status["result"].(map[string]interface{})["video_url"])
			break
		}
		time.Sleep(5 * time.Second)
	}
}

cURL

# 1. Analyze Video for Cues
curl -X POST https://apis.fotohub.app/v1/ai/analyze/image   -H "Authorization: Bearer fh_live_your_api_key"   -H "Content-Type: application/json"   -d '{
    "image_url": "https://storage.fotohub.app/raw/silent.mp4",
    "features": ["actions"]
  }'

# 2. Submit Compose Job
curl -X POST https://apis.fotohub.app/v1/ai/compose/video-audio   -H "Authorization: Bearer fh_live_your_api_key"   -H "Content-Type: application/json"   -d '{
    "video_url": "https://storage.fotohub.app/raw/silent.mp4",
    "music_prompt": "Cinematic orchestral",
    "webhook_url": "https://api.yourdomain.com/webhook"
  }'

# 3. Poll Job Status (202 Accepted -> 200 OK)
curl -X GET https://apis.fotohub.app/v1/jobs/job_12345abc   -H "Authorization: Bearer fh_live_your_api_key"

---

9. Webhook Handling & Security

For long-running videos, do not use active polling. Provide a webhook_url to receive a POST request when the job completes.

FOTOhub signs webhook payloads using HMAC-SHA256. Verify the signature using your API key.

Python FastAPI Webhook Handler

from fastapi import FastAPI, Request, HTTPException
import hmac
import hashlib
import os

app = FastAPI()
API_KEY = os.environ.get("FOTOHUB_API_KEY", "fh_live_your_api_key").encode()

@app.post("/webhooks/fotohub")
async def handle_webhook(request: Request):
    signature = request.headers.get("X-FotoHUB-Signature")
    payload = await request.body()
    
    # Verify HMAC-SHA256 signature
    expected_sig = hmac.new(API_KEY, payload, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected_sig, signature):
        raise HTTPException(status_code=401, detail="Invalid signature")
    
    data = await request.json()
    if data["status"] == "completed":
        print(f"Video ready: {data['result']['video_url']}")
    
    return {"received": True}

TypeScript Express Webhook Handler

import express from 'express';
import crypto from 'crypto';

const app = express();
const API_KEY = process.env.FOTOHUB_API_KEY || 'fh_live_your_api_key';

app.post('/webhooks/fotohub', express.raw({ type: 'application/json' }), (req, res) => {
  const signature = req.headers['x-fotohub-signature'] as string;
  const expectedSig = crypto
    .createHmac('sha256', API_KEY)
    .update(req.body)
    .digest('hex');

  if (signature !== expectedSig) {
    return res.status(401).send('Invalid signature');
  }

  const data = JSON.parse(req.body.toString());
  if (data.status === 'completed') {
    console.log(`Video ready: ${data.result.video_url}`);
  }

  res.send({ received: true });
});

app.listen(3000);

---

10. Async Job Patterns, DLQ, & Retries

Complex multi-track compositions can take 10-30 seconds depending on video length. The API uses a standard async pattern:

1. 202 Accepted: The compose endpoint immediately returns a job_id.
2. Polling: Make GET /v1/jobs/{job_id} requests.
3. Dead Letter Queue (DLQ): If a job fails (e.g., MMAudio GPU OOM, or invalid FFmpeg filter), it is routed to the DLQ. You can query GET /v1/jobs/failed to review.
4. Auto-Retry: FOTOhub automatically retries transient network errors (like MiniMax API timeouts) up to 3 times before failing the job.

---

11. BYOB (Bring Your Own Bucket) S3/R2 Export

By default, FOTOhub stores renders for 24 hours. For production pipelines, configure direct export to your AWS S3 or Cloudflare R2 bucket. FOTOhub will write the master video AND the individual unmixed audio stems to your bucket.

"export_s3": {
  "provider": "aws",
  "bucket": "studio-assets-prod",
  "endpoint": "s3.us-east-1.amazonaws.com",
  "prefix": "projects/cyberpunk/",
  "access_key": "AKIA...",
  "secret_key": "..."
}

The resulting bucket will contain:

• master_mix.mp4
• stem_music.wav
• stem_foley_01.wav
• stem_ambient.wav

---

12. Batch Processing for Studios

Need to process 50 silent clips overnight? Do not dispatch 50 concurrent API calls as you may hit rate limits (default: 10 concurrent jobs).

Implement a local queue (RabbitMQ / Redis) or use the FOTOhub Batch API:

POST https://apis.fotohub.app/v1/ai/compose/batch
Authorization: Bearer fh_live_your_api_key

{
  "batch_name": "Nightly Render Pass",
  "jobs": [
    {"video_url": "vid1.mp4"},
    {"video_url": "vid2.mp4"}
  ],
  "webhook_url": "https://api.domain.com/batch-complete"
}

---

13. Manual FFmpeg Filter Chain Examples

If you prefer to download the raw stems and perform the mixing locally on your own infrastructure, here is the raw FFmpeg command used by the pipeline:

ffmpeg -y -i silent_video.mp4   -i bg_music.mp3   -i foley_1.wav   -i foley_2.wav   -filter_complex "     [2:a]adelay=2400|2400[sfx1];     [3:a]adelay=8100|8100[sfx2];     [sfx1][sfx2]amix=inputs=2:normalize=0[foley_mix];     [1:a][foley_mix]sidechaincompress=threshold=0.08:ratio=4:attack=20:release=250[ducked_music];     [ducked_music][foley_mix]amix=inputs=2:duration=first:weights=0.8 1.2,loudnorm=I=-14:TP=-1.5:LRA=11[aout]   "   -map 0:v -map "[aout]"   -c:v copy -c:a aac -b:a 320k   -shortest final_output.mp4

Delay Explanation

The adelay=2400|2400 filter shifts the audio stem exactly 2.4 seconds (2400 milliseconds) into the timeline for both left and right channels to align perfectly with the visual action.

---

14. Common Failure Modes & Troubleshooting

Error Code	Meaning	Resolution
VISION_SCENE_DARK	The video is too dark for the perception engine to detect actions.	Pass a manual cue_sheet or fall back to ambient-only generation.
MMAUDIO_OOM	GPU2 ran out of memory. Usually caused by requesting a single SFX duration >30s.	Split the Foley request into multiple smaller segments (e.g., 2x 15s).
MUSIC_GENRE_INVALID	ElevenLabs rejected the genre tag.	Use standard genres (cinematic, electronic, rock).
MIXER_CLIPPING	Audio exceeded 0 dBFS during stem summation.	Lower volume_db parameters in the cue sheet, or rely on the loudnorm filter.
JOB_TIMEOUT	The FFmpeg mix took longer than 60s.	Usually implies the source video is >10 minutes long. Chunk processing is recommended.

---

15. Unit Economics & ROI Table

All operations strictly deduct from your USD balance (wallet.available_usd). No credits, no synthetic tokens.

Component	Provider / Engine	Price in USD	Billing Unit
Video Cue Perception	FOTOhub Vision	$0.005	Per 10 seconds of video
Foley / SFX Synthesis	MMAudio (GPU2)	$0.008	Per 5 seconds of audio
MiniMax Music Bed	MiniMax Cloud	$0.015	Per started minute (1–60s)
ElevenLabs Music	ElevenLabs	$0.050	Per started minute
DSP / Remux Pass	FOTOhub Worker	$0.010	Per 30 seconds of video

ROI Scenario: 1-Minute Marketing Video

• Manual Foley Artist + Studio Time: ~$150.00
• FOTOhub AI Full Pipeline:
  • Perception (60s): $0.030
  • 10x SFX (50s total): $0.080
  • 60s Music (MiniMax): $0.015
  • DSP Mixing (60s): $0.020
  • Total Cost: $0.145 USD

Cost Optimization

You can cache and reuse background music tracks across multiple videos by passing an existing music_track_url instead of generating a new one every time, saving $0.015 per video.