Footage and sound

Beats in sound

The tempo, beats, downbeats and accents of an uploaded sound, found when it arrives, and how to cut, title and time sound to them.

Anything uploaded with sound in it has its beats found when it arrives, the same way its waveform is measured: its tempo and where the tempo changes, every beat and how strongly the sound marks it, the downbeats that begin each measure and how many beats a measure holds, the accents, and where the energy changes for good. The answer is stored beside the upload. Reading it costs a read rather than a new analysis, and asking twice gives the same answer to the byte.

This page covers how to read a sound's beats, how their times relate to the timeline, and a few ways to use them. Placing sound is covered in Sound, music and sound effects.

Reading a sound's beats

First inspect uploads with ymotion_media {"action":"list"}. If your tool list declares the beats action, ask by the upload's id or by the src a document's asset carries:

ymotion_media {"action":"beats","id":"<upload id>"}

The reply has two faces that say the same thing:

The text starts with the tempo, how many beats a measure holds, how periodic the sound is and how clearly its downbeats are marked, then the tempo map, the sections when there are any, and the beats themselves: one measure a line, a | before each downbeat, every time followed by its strength from 0 to 1. Last come the accents.
The structured data carries bpm, meter, confidence, downbeatConfidence and steady; tempo, a list of { from, to, bpm } spans in time order; beats, each { t, strength, count }; downbeats, the times whose count is 1; accents, each { t, strength }; and sections, each { from, to, energy }, when the energy changes for good.

A beat's count is where it falls in its measure: 1 on a downbeat, up to meter. A meter of 2 is a compound time such as 6/8, counted in its dotted beats. A count of 0 means no meter was read, so no beat is given a place in a measure nothing measured.

The tempo map has one span when the tempo holds. A song that speeds up or slows down has more, and bpm is the tempo most of it is at. A span runs from about a beat before the first beat it covers to about a beat after its last, so a stretch with no beat, such as speech before or after a song, belongs to no span.

confidence is how periodic the sound is, 0 to 1: how clearly it has a steady beat, or how clearly it has none. It doesn't grade the grid. A grid at twice or half the tempo a person would tap scores just as high, because the sound is just as periodic either way. downbeatConfidence is how clearly the sound marks which beat is the first of the measure: the low end and the harmony on it, against every other place in the measure. It's low when a kick hits 1 and 3 alike and nothing else tells them apart; then the downbeats may be half a measure off, and it's worth checking them by ear before cutting on them.

Long sounds, a window at a time

One reply holds 1000 beats. A longer answer is cut there: the text says so and ends with the call for the rest, and the structured data carries cut: true and the same call as more. First inspect uploads with ymotion_media {"action":"list"}; then, if your tool list declares the beats action, ask for a window in seconds of the file:

ymotion_media {"action":"beats","id":"<upload id>","from":60,"to":120}

A window holds every beat that starts in it, from from up to but not including to. The first 120 minutes of a file are analysed and no more; a longer file says so in every reply, because a beat past that point isn't missing, only not looked for.

A layer's beats, on its timeline

Every time in the reply above is seconds from the start of the file. A layer plays a slice of that file on a clock of its own: it starts somewhere on its composition, it may begin partway into the file, and it may play faster, slower, backwards or through a speed ramp. A client with an older tool list can first inspect uploads with ymotion_media {"action":"list"}; then, if its list declares the beats action and layerId, ask with the layer instead, and the reply does that arithmetic:

ymotion_media {"action":"beats","layerId":"<layer id>"}

This works for a sound layer and a footage layer whose video has sound, on the main timeline or inside a composition. It returns only the beats the layer shows, each at the time it lands on the layer's composition, in the order they are seen, which for a reversed layer is the file's order backwards. In the structured data each beat also carries file, its time in the file, and layer says what the layer shows: its start, its trim, its speed and whether it plays backwards. The text names the clock the times are on.

On the main timeline, the times are the ones ymotion_render and splitLayer take.
Inside a composition, the times are on that composition's own clock, which is the one every edit made inside it uses (ymotion_edit with its compId). Wherever the composition is placed, what is timed inside it moves with it.
A speed change moves the beats with it: at 2x a 120 bpm song's beats come 240 times a minute, and the reply says so. Through a ramp their spacing follows the speed.
A freeze frame holds one moment of the file, so no beat falls inside it. The structured data says frozen: true, and bpm is null.
Before the composition starts. A layer that starts before 0 shows nothing before 0, so beats that would land there aren't listed. The reply says how many were left out, and the structured data carries the count as beforeStart.
A composition layer plays everything inside it mixed, with no one file, so it has no beats of its own. Ask for a layer inside the composition by its id.

A window here is still in seconds of the file: "from":30,"to":60 asks for the beats of that stretch of the music, wherever the layer puts them.

The layer's own clock, which a split carries on so a wiggle doesn't jump, plays no part. Which moment of the file is showing has only ever been the layer's trim and its speed.

Using them

Beats are numbers to build with. Nothing on the timeline follows them by itself: a cut, a title or a sound effect placed on a beat is an ordinary layer at an ordinary time.

Cutting to the beat

Ask for the music layer's beats, then end each shot on the downbeat you want. A cut on a beat is a change of shot: a split alone leaves the same picture playing on, so it shows nothing. trimLayer takes a time on the composition, which is exactly what the layer form gives. An older list can first inspect uploads with ymotion_media {"action":"list"}; then, if it declares the beats action and layerId, use the layer call:

ymotion_media {"action":"beats","layerId":"<music layer id>"}
ymotion_edit {"ops":[{"verb":"trimLayer","args":{"layerId":"<first shot id>","edge":"end","t":6.244,"opts":{"ripple":true}}}]}

With shots end to end on one track, work from the first shot to the last. ripple brings the later shots on that track along, so the next one starts on that downbeat and is ended on the one after; the music, on a track of its own, stays where it is. Within one take, take a part out from the downbeat on with removeTime, leaving the music's track out of its trackIds. In a performance, cut only between real takes or angles.

Ask with the music layer, not the footage: the cut should follow what is heard, and the music is often trimmed or moved independently of the picture. Cutting on every beat is usually too fast to read; every downbeat, or every second one, is a common rhythm for a montage.

Landing a title on a downbeat

A title that arrives on a downbeat reads as part of the music. Start its layer so its entrance finishes on the downbeat. If an entrance takes 0.4 seconds and the downbeat is at 12.244, the layer starts at 11.844. A title that simply appears is started on the downbeat itself. See Text animation for entrances.

Captions and sound effects to the music

A sound effect lands its own moment on at, so a hit placed on a downbeat is one call. For example, a whoosh on the downbeat at 16.244:

ymotion_edit {"ops":[{"verb":"placeSfx","args":{"sfx":"<sfx id>","at":16.244}}]}

The accents are strong hits off the grid that break the measure's own pattern, such as a stab that the other measures don't have, or a door slamming. A hit at the same place in most measures is the groove and isn't listed, even when it's loud: a horn line that stabs the same off-beat in every measure is read as part of the beat. So an accent is a candidate for a flash or a sound effect rather than a certainty; listen before relying on one. A reply lists at most the strongest 300, and says how many there were when there are more. Captions over music can change on the beat rather than on a pause; over speech, time them to the words instead (What an upload says).

Markers on the beats

A marker makes a beat visible on the timeline for the person. A composition marker stays at a time in the edit:

ymotion_edit {"ops":[{"verb":"addMarker","args":{"t":12.244,"label":"downbeat"}}]}

A layer marker travels with its layer, and its time is measured from the layer's start, so take the layer's start off a timeline time first:

ymotion_edit {"ops":[{"verb":"addLayerMarker","args":{"layerId":"<music layer id>","t":9.244,"label":"downbeat"}}]}

When there is no steady beat

Speech, room tone, a crowd, a piano played freely: these have no steady beat, and the reply says so rather than inventing one. steady is false, bpm and meter are null, and there are no beats, downbeats or tempo spans. confidence is how clearly the sound has no steady beat. The strong hits are still listed as accents, so a sound effect or a cut can land on a slam or a laugh.

The same holds inside one file. A podcast that opens with a sting and then talks has a grid over the sting and none over the talk: the beats stop where the music stops, and the tempo map covers only the music.

A grid that is close but not quite right is worse than none, because something gets cut to it. When a sound has a beat a person can hear and the reply says it has none, it's usually a quiet or very free one. Ask the person, and time by ear from the waveform instead: ymotion_render {"what":"waveform"} draws the film's sound against a seconds grid.

How it was found

The beats come from arithmetic on the sound's samples, with no model: the same file gives the same answer on every run. The sound is decoded at 48 kHz on the file's own clock, so a video whose sound starts a little after its picture keeps every beat where it is in the file rather than early by the gap, and it's read as one channel.

It runs once when the file arrives, behind the reply: by the uploads page, by a ChatGPT attachment, by an upload link and by a base64 put alike, on a server with storage and on a person's own machine. A song takes a second or two. Files uploaded before this existed have none yet, and the first request finds them.

What the status means

list marks each upload with what it knows: the tempo and how many beats a measure holds when they're found, no steady beat when there is none, finding beats while it runs, and beats not found when it failed. ymotion_media {"action":"describe","id":"<upload id>"} shows the same mark in full and names the call that reads the beats. The mark is a summary; the beats always come from the beats action.

Still being found. The reply says so, with roughly how long and the call to make. There is nothing to wait for: carry on with the rest of the film and ask once more when that's done. Asking doesn't hurry it, and asking again joins the same work rather than starting it over.
Failed. The reply gives the reason that was recorded. Asked again 10 minutes or more after the failure, it's tried once more.
No sound. A picture or a silent video has no beats, and the reply says that.