m64 format documentation

[simonlindholm]

Through reverse-engineering of the SM64 audio library, I figured some more details of the m64 format.

The hack64 forum software doesn't seem to support tables, but here's an external markdown doc: https://hackmd.io/opEB-OmxRa26P8h8pA-x7w as well as a decoder: https://gist.github.com/simonlindholm/30...22c291f878

The list of commands should be complete, though some of the descriptions are a bit vague. I could look closer into the code if there's some particular aspect that people are interested in.

[lezg_g]

Through reverse-engineering of the SM64 audio library, I figured some more details of the m64 format.

The hack64 forum software doesn't seem to support tables, but here's an external markdown doc: https://hackmd.io/opEB-OmxRa26P8h8pA-x7w as well as a decoder: https://gist.github.com/simonlindholm/30...22c291f878

The list of commands should be complete, though some of the descriptions are a bit vague. I could look closer into the code if there's some particular aspect that people are interested in.

Man, thank you! this helps a lot! I added some commands in seq64 and works really great. 

[Sauraen]

Through reverse-engineering of the SM64 audio library, I figured some more details of the m64 format.

The hack64 forum software doesn't seem to support tables, but here's an external markdown doc: https://hackmd.io/opEB-OmxRa26P8h8pA-x7w as well as a decoder: https://gist.github.com/simonlindholm/30...22c291f878

The list of commands should be complete, though some of the descriptions are a bit vague. I could look closer into the code if there's some particular aspect that people are interested in.

Awesome work simonlindholm! How did you figure the rest of the commands out? Just by reading the disassembly? There are a lot of data structures used by the sequence parsing, and even with stepping through the code in an emulator it is very difficult to figure out what all the fields would be.

I'm the author of SEQ64, which is software for importing/exporting this sequence file format from/to MIDI, both in SM64 and the other first-party Nintendo games which use versions of the same file format (most notably OoT and MM). SEQ64 is also the backend for N64 Sound Tools for the games using these formats. I figured out many of these commands years ago, starting from previous work by Deathbasket, when I was developing the program, but I was never able to figure out the rest of them. I hope you were able to build on my work--I see you're using my notation of "Q" for the accumulator, so I'm hoping you were able to start from the sequence definition in SEQ64 and figure out the rest from there.

The reason SEQ64 is 20,000 lines of code rather than a few hundred is because it allows the definition of the sequence format to be specified by the user, rather than being hard-coded into the program. This was primarily intended to avoid minor differences between games causing the program to only be usable for certain games or versions, because nobody wants to change the code and recompile it. But that does mean it became large and unwieldy in some ways, and it was never able to use the scripting capabilities of the format besides optimizing data with calls and loops. It would be nice to have a full script editor that's complementary to SEQ64, where you could import a MIDI with SEQ64 and then edit it as a script with the other tool. Of course, this tool should also work on OoT and all other games using similar formats, which is a lot to ask!

[nutta]

Great job on this though there's some little things you should change, the 0xFD command doesn't actually delay based on frames, it delays based on ticks though they might work the same way.

Code:

32nd note - 12 ticks 0x0c

16th note - 24 ticks 0x18

8th note - 48 ticks 0x30

4th note - 72 ticks 0x48

2nd note - 96 ticks 0x60

Whole note = 192 ticks 0xc0

4 bars =  0x300

[simonlindholm]

Through reverse-engineering of the SM64 audio library, I figured some more details of the m64 format.

The hack64 forum software doesn't seem to support tables, but here's an external markdown doc: https://hackmd.io/opEB-OmxRa26P8h8pA-x7w as well as a decoder: https://gist.github.com/simonlindholm/30...22c291f878

The list of commands should be complete, though some of the descriptions are a bit vague. I could look closer into the code if there's some particular aspect that people are interested in.

Awesome work simonlindholm! How did you figure the rest of the commands out? Just by reading the disassembly? There are a lot of data structures used by the sequence parsing, and even with stepping through the code in an emulator it is very difficult to figure out what all the fields would be.

By reading the disassembly, yeah. (Helped by decompilation into C using https://github.com/matt-kempster/mips_to_c.) The SM64 audio lib indeed has a lot of interconnected structures, so it was non-trivial...

I'm the author of SEQ64, which is software for importing/exporting this sequence file format from/to MIDI, both in SM64 and the other first-party Nintendo games which use versions of the same file format (most notably OoT and MM). SEQ64 is also the backend for N64 Sound Tools for the games using these formats. I figured out many of these commands years ago, starting from previous work by Deathbasket, when I was developing the program, but I was never able to figure out the rest of them. I hope you were able to build on my work--I see you're using my notation of "Q" for the accumulator, so I'm hoping you were able to start from the sequence definition in SEQ64 and figure out the rest from there.

I definitely used SEQ64 to help understand some of the structures involved and the meaning of various commands in non-code terms, and to verify the sanity of other parts. The "Q" notation was to align with SEQ64, though I actually found out about SEQ64 only after I got the Q-related commands down.

The reason SEQ64 is 20,000 lines of code rather than a few hundred is because it allows the definition of the sequence format to be specified by the user, rather than being hard-coded into the program. This was primarily intended to avoid minor differences between games causing the program to only be usable for certain games or versions, because nobody wants to change the code and recompile it. But that does mean it became large and unwieldy in some ways, and it was never able to use the scripting capabilities of the format besides optimizing data with calls and loops. It would be nice to have a full script editor that's complementary to SEQ64, where you could import a MIDI with SEQ64 and then edit it as a script with the other tool. Of course, this tool should also work on OoT and all other games using similar formats, which is a lot to ask!

Yeah, the flexible design of SEQ64 certainly makes a lot of sense, and it's always convenient to have things as data instead code in general. I agree that a good standalone script editor would be neat.

Re scripting capabilities, one of the fun parts of having this documentation was to see the tricks they used for sound effects in SM64, in sequence 0. For instance, here's how a red coin sound gets played:

- the game calls SetSound(0x78289081 + (coin number << 16)), which writes 0x28 + coin number to IO slot 4 of channel 7 of sequence 0, and 1 ("play") to IO slot 0.
- the write to slot 0 causes the script for channel 7 to break out of the following loop:

Code:

...
.chan_FD:
chan_delay1
chan_ioreadval 0
chan_bltz .chan_FD

and do

Code:

chan_setdyntable .table_2B1D
chan_ioreadval 4
chan_dyncall

- at 0x2B1D in the sequence is a table of data, which at positions 0x28..0x2F all contain 0x3122. At location 0x3122 is the following channel script:

Code:

.chan_3122:
chan_setinstr 128
chan_setnotepriority 14
chan_setpanchanweight 0
chan_setenvelope .envelope_3378
chan_ioreadval 4
chan_subtract 0x28
chan_readseq .addr_313E
chan_writeseq 0, .layer_fn_3188, 1
chan_setlayer 0, .layer_3146
chan_setlayer 1, .layer_3168
chan_setlayer 2, .layer_3148
chan_end

.addr_313E:
.byte 0x0
.byte 0x2
.byte 0x4
.byte 0x5
.byte 0x7
.byte 0x9
.byte 0xb
.byte 0xc

.layer_3146:
layer_delay 0x6

.layer_3148:
layer_call .layer_fn_3188
layer_note0 46, 0xc, 75, 20
layer_note0 45, 0xc, 75, 20
layer_note0 46, 0xc, 75, 20
layer_note0 58, 0x10, 80, 80
layer_note0 58, 0x10, 45, 80
layer_note0 58, 0x10, 20, 80
layer_note0 58, 0x10, 15, 80
layer_end

.layer_3168:
layer_call .layer_fn_3188
layer_note0 41, 0xc, 75, 20
layer_note0 40, 0xc, 75, 20
layer_note0 41, 0xc, 75, 20
layer_note0 53, 0x10, 80, 80
layer_note0 53, 0x10, 45, 80
layer_note0 53, 0x10, 20, 80
layer_note0 53, 0x10, 15, 80
layer_end

.layer_fn_3188:
layer_transpose 0
layer_end

which reads IO slot 4, subtracts 0x28, maps it to the major scale using a lookup table, overwrites the "transpose" command of a function in the sequence script, and calls that function from the channel layers before finally playing the coin sound, now with a pitch corresponding to the index of the red coin. I found that pretty neat. :)

Great job on this though there's some little things you should change, the 0xFD command doesn't actually delay based on frames, it delays based on ticks though they might work the same way.

Thanks. Ticks vs. frames was actually a typo, fixed.

[Sauraen]

overwrites the "transpose" command of a function in the sequence script

So not only does the sequence format contain a Turing-complete scripting language, it uses self-modifying code? :O

[simonlindholm]

Yep. :) Makes a little bit of sense, so they didn't have to implement dynamic versions of all commands, though it's also somewhat terrifying. You could also use the sequence read/write commands to implement memory, making sequence scripts even more clearly Turing complete.

[Sauraen]

Have you checked what happens if the sequence write command attempts to write outside the sequence itself? If that isn't checked and prohibited, and somehow arbitrary sequence execution was obtained, that could be used to obtain arbitrary code execution...

[simonlindholm]

No checks against it, so it could totally aid in arbitrary code execution if you were able to manipulate sequence data somehow. Though it's limited to a 2^16-byte range, and in SM64's case there's nothing interesting after the sequence data, so you would have to get creative with other unchecked commands as well.

[Sauraen]

There's nothing interesting in the entire almost-64 kB after the sequence data? Really? Have you checked whether the address value is sign-extended, so an address of e.g. FFFF accesses memory before the sequence? If the memory allocation works similarly to malloc, writing to somewhere around FFF8 or FFFC could corrupt the heap after the sequence is freed.

[simonlindholm]

Not sign extended, but you might be right in that I was too hasty in saying that everything afterwards is uninteresting -- I was thinking of ROM and not RAM. Indeed from the RAM location that sequences are in it might be possible to reach stack memory! Anyhow, I think celebrating over this would be too early; I think achieving ACE and writing to sequence data are probably about equally hard.

[Sauraen]

Of course. I was more thinking, if we were reversing the sequences, and we found a substantial bug in one of the code-heavy sequences, we should keep our eyes open for possible exploitation of it.

Perhaps this isn't the right place to bring this up, but I have been thinking about the relationship of music to ACE in another context for a while. Any successful ACE requires not only some sort of bug where execution goes somewhere it shouldn't (e.g. stack smash, jump table out of range, etc.) but also a relatively large area of memory which can be manipulated through in-game actions (e.g. Koopa shell X-coordinate table in SMW or inventory in Pokemon). Ideally, in-game actions would be able to set every byte of this memory uniquely to write whatever instructions you wanted, but it may be able to work even if there are constraints on what you can write there.

The second, "secret" scarecrow song in Ocarina of Time--the one you teach to Pierre, not the well-known one you teach to Bonooru which allows you to call Pierre--is not eight notes, but a seemingly unlimited arbitrary song that stores your pitch bends, vibrato, etc. This is the song that's played after the end credits in various different instruments (there actually may be a sequence for this--I'll have to take another look at something). Anyway, this song is stored in a unique format, different from the other scarecrow song. It starts at 0x0F60 in SRA and ends at 0x12b8 or 0x12c0, which is enough to store 108 8-byte-long events. The events are aligned to 0x2/0xA, and the format is:

struct event{
u8 note_number; // FF for rest, 00 is middle C, counts up for half steps; Z/R modifier affects this (possible range: FF, 01, 02, 03, 04, 05, 06, 08, 09, 0A, 0B, 0C, 0D, 0E, 0F)
u8 flags; //Unknown; appears to be 00 if the event is slurred, c0 if it's rearticulated, 01 for the first event, also have seen 80
s8 pitch_bend; //Signed, with emulator set to 66% of control stick, gives 3C or c4, but probably could be set to larger range
u8 vibrato; //Unsigned, range 00 to 0F, same if you go L or R on the stick
u8 unknown_57 = 0x57; //Unknown, always 0x57
u16 timestamp; //Amount of time before this event, possibly continues into the next byte
u8 zero = 0; //Unknown, either always zero or a continuation of the timestamp
}

The decompile will definitely help with this!

Interpreting this as 216 4-byte words, you get alternating words of

[00-FF] (timestamp byte 2)
[00] or [00-FF] (zero, or timestamp byte 3)
[FF-0F] (note, excluding 00 and 07]
[??] (flags)

and

[C4-3F] (pitch bend)
[00-0F] (vibrato)
[57] (unknown)
[00-FF] (timestamp byte 1)

That is quite a decent amount of control, and 216 instructions is a LOT to have at your disposal to write a bootloader. And, the data here (within the bounds on each value) can be easily recorded via TAS, simply by going to the scarecrow and then making the correct changes to the controller state at the correct times. It even has the bonus that it's stored in the save file in the same format--this isn't some transient RAM state which will get changed when something else spawns in!