multimedia-production-mastery

Multimedia Production Mastery

Automation recipes, mathematical formulations, and Python/FFmpeg templates for edits that are *driven by a signal or a curve*: audio beats, easing functions, and time-remap profiles. The unit of work here is a computed timeline, not a manual cut.

When to Use

Use this skill when one or more of the following are true:

• Beat synchronization — cutting or transitioning clips to align with the tempo, beats, or onsets of an audio track (librosa analysis feeding MoviePy/FFmpeg cuts).
• Speed ramping — smooth, non-linear acceleration/deceleration profiles (fast → slow-mo → fast) via time remapping.
• Zoom & pan curves — Ken Burns motion using sinusoidal or exponential ease-in/ease-out instead of robotic linear interpolation.
• Camera shake — procedural handheld or bass-impact shake generated from noise or onset strength.
• Styled dynamic subtitles — .ass files with word-by-word highlight colors, custom fonts, borders, and margins.
• Pipeline orchestration — wiring librosa + MoviePy 2.x + FFmpeg around pre-rendered clips (including AI-generated ones).

When NOT to use — routing table

| Request | Route to |

|---|---|

| Cut, trim, join, or re-encode segments with no timing math | ffmpeg-video-editing |

| Constant-rate scale, crop, watermark, blur, or uniform speed change | ffmpeg-video-filters |

| Assemble finished clips, audio, and titles into a final deliverable | video-assembly-pipeline |

| Extract, normalise, or mix audio tracks | ffmpeg-audio-processing |

| Programmatic timeline edits inside CapCut Desktop drafts | capcut-video-automation |

| Container/format conversion with no timeline changes | ffmpeg-format-conversion |

Prerequisites

Python environment

All bundled scripts require the following packages in the active Python environment:

| Package | Version constraint | Purpose |

|---|---|---|

| librosa | ≥ 0.11 (tempo returns NumPy array) | Beat/onset detection |

| moviepy | 2.x only — 1.x API is incompatible | Video composition, time remapping |

| numpy | any recent | Numerical integration, interpolation |

| scipy | any recent | cumulative_trapezoid for speed profiles |

Verify before running any script:

python -c "import librosa, moviepy, numpy, scipy; print(librosa.__version__, moviepy.__version__)"

Expected output should show moviepy version starting with 2..

FFmpeg

FFmpeg must be on PATH. Verify:

ffmpeg -version

Bundled scripts location

All scripts live in scripts/ relative to this skill directory:

| Script | Purpose | Key flags |

|---|---|---|

| beat_sync.py | Detect beats/onsets in a track and cut a video (or stitch a clip folder) on them | --audio (req), --video \| --dir, --output (req), --onset, --export-beats beats.json |

| speed_ramp.py | Three-phase non-linear speed ramp | --video (req), --output (req), --start-speed 2.0, --mid-speed 0.2, --end-speed 2.0, --ramp-start 0.2, --ramp-end 0.8 |

| subtitles_styled.py | Timed-words JSON → styled .ass with per-word highlights | --input (req), --output (req), --font, --size, --color #RRGGBB, --highlight-color, --outline-color, --outline, --shadow, --alignment, --margin-v |

Bundled references

Load references/effects-landscape.md when the user asks about available AI-enhanced or classical video effects, their parameters, tooling, or underlying math — it is a catalog you can search for effect names and implementation details.

Procedure

Step 1 — Identify the driving signal

Determine what drives the edit:

| Signal | Tool | Script |

|---|---|---|

| Musical beats or tempo | librosa.beat.beat_track | beat_sync.py |

| Onsets (percussive/transient) | librosa.onset.onset_detect | beat_sync.py --onset |

| Custom speed curve | scipy.integrate.cumulative_trapezoid | speed_ramp.py |

| Easing curve for zoom/pan | FFmpeg zoompan with cosine easing | Recipe 1 below |

| Word timings for subtitles | External JSON (whisper or manual) | subtitles_styled.py |

Step 2 — Run or compose the matching recipe

Recipe A: Beat-synced cut from a folder of clips

python scripts/beat_sync.py --audio music.mp3 --dir .\raw_clips --output synced.mp4

Export detected beats for inspection or reuse:

python scripts/beat_sync.py --audio music.mp3 --dir .\raw_clips --output synced.mp4 --export-beats beats.json

For sparse or syncopated tracks where beat tracking misses, use onset detection:

python scripts/beat_sync.py --audio ambient.mp3 --dir .\raw_clips --output synced.mp4 --onset

Recipe B: Three-phase speed ramp (fast → slow-mo → fast)

python scripts/speed_ramp.py --video in.mp4 --output ramped.mp4 --mid-speed 0.15

Full parameter control:

python scripts/speed_ramp.py `
  --video in.mp4 `
  --output ramped.mp4 `
  --start-speed 2.0 `
  --mid-speed 0.2 `
  --end-speed 2.0 `
  --ramp-start 0.2 `
  --ramp-end 0.8

Recipe C: Word-highlight styled subtitles

Input JSON format (word-level timings, highlight flags the accent color):

[
  { "start": 0.5, "end": 1.2, "text": "Welcome", "highlight": false },
  { "start": 1.8, "end": 3.0, "text": "Future!", "highlight": true }
]

Generate the .ass file:

python scripts/subtitles_styled.py `
  --input words.json `
  --output subs.ass `
  --highlight-color "#00FFFF"

Step 3 — Core formulas (reference when composing custom code)

Non-linear motion curves

HARD RULE: Never linearly interpolate zoom or pan — it reads as robotic. Use sinusoidal ease-in-out:

z(t) = z_start + (z_end - z_start) * (1 - cos(pi * t / d)) / 2

where t is current time, d the ease duration. To keep pixel (x_f, y_f) centered on a W x H frame while zooming by z(t):

x(t) = x_f - W / (2 * z(t))
y(t) = y_f - H / (2 * z(t))

Time remapping (speed ramps)

Define S(t) = source time at output time t; the derivative S'(t) is the instantaneous speed multiplier. Build S(t) by integrating a speed profile.

• MoviePy 2.x: clip.time_transform(time_func) — the 1.x fl_time is gone. HARD RULE: time_transform() leaves the clip's duration undefined. Call .with_duration(target) immediately after, or output is truncated.
• FFmpeg: dynamic speed requires splitting into segments and applying setpts (video) + atempo (audio) per segment. HARD RULE: atempo only accepts [0.5, 2.0]; chain filters for anything outside (e.g. atempo=2.0,atempo=2.0 for 4x, atempo=2.0,atempo=1.5 for 3x).

ASS color formats

Two different hex conventions inside one file:

| Context | Format | Example |

|---|---|---|

| Style definitions ([V4+ Styles]) | &HAABBGGRR — 8 digits, no trailing & | &H0000FFFF = opaque yellow |

| Inline override tags | \c&HBBGGRR& — 6 digits, trailing &, no alpha | {\c&H00FFFF&} = yellow from here on |

HARD RULE: Alpha is inverted relative to RGBA: 00 = fully opaque, FF = fully transparent. Invisible text usually means FF (transparent) where 00 (opaque) was intended.

Step 4 — FFmpeg recipes

Recipe 1: Jitter-free FFmpeg zoompan (Ken Burns)

zoompan evaluates positions in integer pixels, causing visible jitter at low zoom speeds. Pre-upscale to 4K first so the truncation error becomes sub-pixel:

ffmpeg -i in.jpg -vf "scale=3840:-2,zoompan=z='1+0.5*(1-cos(PI*on/149))/2':x='iw/2-(iw/zoom/2)':y='ih/2-(ih/zoom/2)':d=150:fps=30:s=1920x1080" -c:v libx264 -pix_fmt yuv420p out.mp4

Recipe 2: The even-dimensions rule

HARD RULE: Most encoders (yuv420p especially) fail on odd width/height. After any crop or scale, force even dimensions:

ffmpeg -i input.mp4 -vf "scale='trunc(iw/2)*2:trunc(ih/2)*2'" output.mp4

Recipe 3: MoviePy 2.x non-linear speed ramp (custom code)

Integrate a speed profile to get the time-remap function, then apply it:

import numpy as np
from scipy.integrate import cumulative_trapezoid
from moviepy import VideoFileClip

src = VideoFileClip("in.mp4")
fps = src.fps
out_dur = 6.0
t_out = np.linspace(0, out_dur, int(out_dur * fps) + 1)

# Speed profile: fast -> slow -> fast
speed = 1.0 + 0.5 * (np.cos(2 * np.pi * t_out / out_dur) * 0.5 + 0.5)

# Integrate speed to get source timestamps for each output frame
s_src = np.clip(cumulative_trapezoid(speed, t_out, initial=0.0), 0, src.duration - 1e-3)
time_func = lambda t: float(np.interp(t, t_out, s_src))

ramped = src.time_transform(time_func).with_duration(out_dur)
ramped.write_videofile("out.mp4", fps=fps, audio=False)

src.close()
ramped.close()

Recipe 4: Proxy workflow for heavy footage

Validate cuts and beat alignment on a cheap proxy, render once on the original:

1. Generate proxy:

```powershell

ffmpeg -i input.mp4 -vf scale=-2:720 -c:v libx264 -crf 20 -c:a copy proxy.mp4

```

2. Iterate timing/beat scripts against proxy.mp4.

3. Swap the source path back to the 4K original for the final render only.

Step 5 — Onset fallback for difficult tracks

Standard beat tracking misses beats in sparse or heavily syncopated material. Fall back to onsets in custom code:

onset_frames = librosa.onset.onset_detect(y=y, sr=sr, backtrack=True)
times = librosa.frames_to_time(onset_frames, sr=sr)

(beat_sync.py --onset does this automatically.)

Pitfalls

1. MoviePy duration reset — time_transform() leaves the clip's duration undefined. Call .with_duration(target) immediately after, or output is truncated.

2. ASS inverted alpha — invisible text usually means FF (transparent) where 00 (opaque) was intended. Alpha is inverted relative to RGBA.

3. atempo range — multipliers outside [0.5, 2.0] silently fail unless chained (atempo=2.0,atempo=1.5 for 3x, atempo=2.0,atempo=2.0 for 4x).

4. librosa tempo type — beat_track() returns a NumPy array for tempo in librosa 0.11+; cast with float() before formatting.

5. Beat tracking on ambient/vocal tracks — standard beat tracking misses beats in sparse or heavily syncopated material. Use --onset flag or librosa.onset.onset_detect with backtrack=True.

6. Windows file locks — unclosed VideoFileClip/AudioFileClip handles keep files locked and crash subsequent runs. HARD RULE: Always .close() every clip before the script exits, including in error paths.

7. Odd output dimensions — yuv420p and most encoders fail on odd width/height. Always force even dimensions after any crop or scale operation.

8. MoviePy 1.x vs 2.x API — only use 2.x methods: time_transform / with_duration / subclipped / with_audio. The 1.x fl_time / set_duration / subclip methods do not exist in 2.x and will raise AttributeError.

9. zoompan integer jitter — zoompan evaluates positions in integer pixels. Without pre-upscaling to 4K, low zoom speeds produce visible jitter. Always pre-scale to a high resolution before zoompan.

Verification

Run through this checklist before considering the task complete:

• [ ] Environment check: librosa, moviepy (2.x), numpy, and scipy import successfully:

```powershell

python -c "import librosa, moviepy, numpy, scipy; print(librosa.__version__, moviepy.__version__)"

```

Output must show moviepy version starting with 2..

• [ ] MoviePy 2.x API only: Only these methods are used: time_transform / with_duration / subclipped / with_audio. No 1.x fl_time / set_duration / subclip appears anywhere in the code.

• [ ] Even dimensions: Every scale/crop expression forces even output dimensions. Verify with:

```powershell

ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=p=0 output.mp4

```

Both width and height must be even numbers.

• [ ] ASS color formats: Inline ASS tags use 6-digit hex with trailing & (\c&HBBGGRR&); style lines use 8-digit &HAABBGGRR; alpha semantics are inverted (00 = opaque, FF = transparent).

• [ ] Clip handles closed: All VideoFileClip/AudioFileClip handles are .close()d before the script exits. On Windows, verify no file locks remain by attempting to delete or rename the input file after the script finishes.

• [ ] atempo within range: Any atempo filter values are within [0.5, 2.0] or properly chained. Verify by inspecting the FFmpeg filtergraph string.

• [ ] Output plays: Quick sanity check:

```powershell

ffprobe -v error -show_entries format=duration -of csv=p=0 output.mp4

```

Duration should match the expected target duration (not the source duration for ramped clips).

Related skills

• ffmpeg-video-editing — plain cut/trim/concat with no timing math
• ffmpeg-video-filters — constant-rate scale, crop, watermark, blur, uniform speed
• video-assembly-pipeline — stitching finished clips into a deliverable
• ffmpeg-audio-processing — audio extraction, normalisation, mixing
• capcut-video-automation — CapCut Desktop draft manipulation
• ffmpeg-format-conversion — container/format conversion with no timeline changes