cloudy with chance of smiles
← All posts

Building DORA Songsmith: A Multi-Agent Music Video Generator

How we designed and built DORA Songsmith, a decoupled multi-agent architecture utilizing Gemini, Lyria, and Veo to generate custom music videos.

AI Dev ToolsGoogle CloudGeminiMulti-Agent

Awkward silence at the beginning of group online meetings is not great. The DORA Community’s solution for this has been to listen to a tech parody video that related to the discussion topic for the first few minutes of the call letting people join and get settled. It is actually amazing the collection of songs the tech community has generated over the last 15 years. Unfortunately, after a few years of discussions and the emerging trend of topics related to a new technology (AI), it was getting harder and harder to find videos that had not been used in a prior meeting.

Building DORA Songsmith took over a year—but that wasn’t a year of complex engineering; it was a year of waiting for AI developer tools and models to mature. Today, the barriers are gone. With modern agentic frameworks, you can build a solution like this in days.

Let’s dig into the DORA Songsmith MVP —an automated pipeline that transforms free-form event descriptions into fully produced, 3-minute high-fidelity music videos, complete with synchronized lyrics and cinematic backgrounds, tailored for the DORA community.

The Vision: Decoupled Multi-Agent Architecture

When designing a system that handles natural language processing, audio synthesis, video generation, and video editing, it's tempting to write one giant, monolithic LLM prompt. However, monoliths suffer from "prompt drift," safety filter bottlenecks, and poor execution control. By leveraging the Antigravity planning capabilities, we systematically evaluated these architectural trade-offs to arrive at our final design.

Instead, we designed DORA Songsmith around the Google Agent Development Kit (ADK) and the Antigravity Orchestrator, creating a team of highly specialized, loosely coupled agents:
DORA Songsmith Multi-Agent Architecture Each agent has a specific persona, a constrained output schema, and its own set of tools.

Agent A: The Lyricist (Researcher & Writer)

The Lyricist Agent is the creative brain of the operation. It is specialized in DORA research, DORA metrics (like Lead Time for Changes, Deployment Frequency, MTTR, and Change Failure Rate), generative culture, and software engineering themes.

1. The Tools & Model

2. The Original System Prompt

To ensure the agent adopted the right tone and adhered to security guidelines, we used this system prompt:

You are The Lyricist, an L1-L3 Specialized Agent in DORA Research, High Performing Software Development Teams, DORA AI capabilities, and Generative Culture (specifically the 2025 focus), and Software Delivery metrics.

Your task is to take a free-form event description and turn it into a 3-minute high-fidelity music video song map.

Include a Mood Board with BPM and generic genre names. 
CRITICAL: DO NOT include names of specific artists, bands, or copyrighted songs in the mood board or lyrics, as this will trigger safety filters.

Include a segmented visual timeline mapping the song's sections (Intro, Verse, Chorus, Outro) to visual prompts and durations.

Include full raw lyrics for a 3 minute song. Ensure the output reflects DORA community values and technical accuracy and does not resemble existing copyrighted songs.

3. Two-Step Execution Pipeline

Because generating high-quality lyrics and mapping accurate millisecond-level timestamps simultaneously is too computationally heavy for a single inference step, we implemented a two-step pipeline:

[!NOTE] The Timestamp Prompt:

You are an Audio Engineer. Map the following lyrics to a 3-minute timeframe.
The song is exactly 180,000 milliseconds long.
Provide a start_ms and end_ms for each line of lyrics.

Lyrics:
{raw_lyrics_text}

Agent B: The Audio Engineer (Lyria 3 Pro)

The Audio Engineer Agent takes the output of the Lyricist and turns it into sound.

Agent C: The Video Producer (Original Veo 3.1 Pipeline)

In our original steel-thread implementation, the Video Producer Agent was built on Google's Veo 3.1 video generation model.

operation = self.genai_client.models.generate_videos(**kwargs)
while not operation.done:
    print("Waiting for video generation to complete...")
    time.sleep(10)
    operation = self.genai_client.operations.get(operation=operation)

Once the operation finished, the generated clips were downloaded, written to /tmp, and uploaded to the GCS media bucket.

The Secure Compositor (FFmpeg Cloud Run Job)

Once the Orchestrator has the .wav audio from the Audio Engineer and the .mp4 video segments from the Video Producer, it initiates a Cloud Run Job running FFmpeg.

  1. Duration Scaling: It reads the actual duration of the audio using ffprobe and scales the video segments proportionally.
  2. Looping & Standardizing: It loops each background video segment using -stream_loop -1 and normalizes the resolution (1080p, 30fps) so the transitions are seamless.
  3. Crossfading: It applies the xfade filter graph to fade between sections (Intro → Verse → Chorus) with a 1-second crossfade duration.
  4. Burning Subtitles: It parses the final lyric_map.json, converts it to a standard .srt subtitle file, and uses the subtitles filter with custom styling to burn dynamic lyrics directly onto the video.

In my next post, we will explore how to deploy and leverage Google Cloud to bring DORA Songsmith to life!