~ajxs

Hacking the Yamaha DX9 To Turn It Into a DX7

2023.04.10
A brief account of hacking the Yamaha DX9's firmware ROM to make its functionality more closely match the DX7.
Tagged as:
TL;DR: I created a new firmware ROM for the Yamaha DX9, which restores many of the synth's missing features, making the synth able to load and play DX7 patches. The new firmware ROM's source code is freely available here.

Updates

In light of reaching the milestone of releasing Version 1.0 of the DX9/7 firmware, I've made a few updates to this article. The most significant update is the addition is about some interesting leftover data found in the Yamaha DX9 ROM. Also, a big thanks to the amazing Olli Niemitalo for correcting an error in how I was calculating the DX7/9's voice update frequency!

Foreword

Like many vintage synth enthusiasts, I keep a keen eye on the local classified sites for the odd bargain that might pop up. Late last year, amidst the usual sea of second-hand Roland grooveboxes, and Korg Volca synths was something I'd never seen before: A Yamaha DX9.

After my 2021 project creating a detailed technical analysis of the DX7, and my subsequent disassembly of its firmware, I figured there wasn't too much more I needed to know about the inside of the DX7... Seeing the DX9 —advertised at half the price of a DX7, no less— an interesting thought crossed my mind: Could the DX9 firmware be hacked to restore its missing features?

Background

In 1983, Yamaha released the groundbreaking DX7 synthesiser. It featured a revolutionary new sound generation technology which would change the course of popular music, and help define the signature sound of the 80s: Frequency Modulation synthesis. Simultaneously, Yamaha launched the DX9: A more affordable alternative featuring the same FM synth engine, but substantially more limited features than its more popular sibling.

Despite being built around the same FM sound chips1, and sharing the DX7's famously rugged metal chassis, the DX9 possessed considerably less functionality than its iconic sibling. It featured only four operators —two less than the DX7's six— greatly limiting the range of sounds it could create. Its keyboard was not velocity-sensitive, or aftertouch capable, and the DX7's pitch envelope generator had been removed. Its patch storage had also been greatly reduced, with internal patch memory cut down from 32 patches to only 20, and the cartridge system swapped out for a cassette tape interface.

Reverse-Engineering the DX9

A common misconception exists that under the hood the DX9's hardware is identical to the DX7. While the DX9 does use the same FM voice chips, and CPU as the DX7, that's about where the similarities end. This misunderstanding seems to have been exacerbated by an incomplete copy of the DX7/9 service manual without the DX9 board schematics having circulated widely online, leading people to assume that without having its own schematics the two synths must be identical inside. It was this very misconception that led me to purchase one and see for myself...

A single look inside the DX9 told me that simply dropping the DX7 firmware in wasn't going to cut it. My first step to hacking the DX9 would then have to be disassembling its firmware.

At a glance, the structure of the DX9's ROM seemed very different from the DX7 firmware that I was familiar with. The designs are so distinct that there's no way they were written by the same team. The two synths share very little code in common: Only the code for working with the serial interface, and a few common utility routines. I tried to look for a common ancestor that both synths might have inherited this code from, but the DX7/9 appear to be Yamaha's first devices to use the 6303 CPU architecture2.

The DX9 has only 4 kilobytes of RAM compared to the DX7's 6. To save space, the DX9 uses as a different, abridged patch format to that of the DX7. Reduced in size from 128, to only 64 bytes. To accommodate this, the DX9 uses a simplified keyboard scaling implementation, and removes patch names.

Interestingly, the DX9 sends and receives patches over SysEx in a DX7-compatible format, translating patches between the two formats as they're read and written. Patches written to tape via the cassette interface however are serialised as-is in the DX9's native format.

Fortunately, despite all of the differences between the two ROMs, there were enough little commonalities that I was able to hit the ground running...

MIDI

A great place to start reverse-engineering any synthesiser's firmware is to track down the code for handling incoming MIDI messages. From here you can identify a broad cross-section of internal functionality —such as note on/off handlers, parameter changes, and patch functions— by tracing how the software handles the associated MIDI status messages.

Interestingly, the DX series actually allow one synth to remotely trigger keypresses on another via SysEx messages3. Analysing how these messages were processed was how I reverse-engineered the DX7's UI logic.

MIDI data is always sent and received over a serial interface. Finding code that uses a chip's serial interface is usually fairly straightforward, even on relatively more complicated modern hardware. In the case of the Hitachi 63B03 CPU used by the DX7/9, the serial interface is configured via a standard control register, and a function pointer to an interrupt routine to handle I/O is placed at a standard location in the ROM. The interrupt handler in the DX9 uses two ring buffers to store incoming and outgoing MIDI data, before processing incoming data in a sub-routine of the firmware's main loop.

The main MIDI processing routine in any synth is always going to be implemented as a state-machine: After receiving a MIDI status byte, the synth then processes each incoming data byte until all the required data for this particular message type is received. Receiving a new status byte when a data byte is expected resets this state.

Since the DX7 and DX9 were both released prior to MIDI's standardisation in 1985, early versions of the DX7 firmware prior to V1.5 had an incomplete, non-standard MIDI implementation. DX7 Service Notes Bulletin No. E-325 from November 1984 describes this update in full detail. The TX7 owner's manual implies the existence of a firmware update for the DX9 that addressed these issues, however I've never seen any actual proof of its existence.

Keyboard

While the DX9 does use the same Hitachi HD63B03RP microprocessor as its main CPU, it doesn't share the DX7's 6805S 'sub-CPU' for interfacing with its analog peripherals. Instead it uses a multiplexing circuit wired into the CPU's address bus, through which the synth's analog peripherals are polled for updates in the firmware's main loop.

The code interfacing with what the service manual calls the 'KBD/SW Scan Driver' was by far the most difficult code in the ROM to parse. This code is responsible for reading the front-panel switch line signals, and converting them into the internal representation used by the user-interface routines. To make the work of understanding this great tangled mess of assembly easier on myself, I wrote a DX9 driver for MAME so that I could step through the firmware's code instruction by instruction in MAME's debugger. I got this idea from David Viens from Plogue's fantastic video Emulating the DX7 the HARD way.

Unlike the DX7, rather than keyboard events triggering note events directly they set a global variable which is tested by Note On/Off event handling routines called later in the main loop. A similar approach is used for loading patch data, with a global flag being used to trigger reloading patch data to the voice chips after parameters are edited via the front-panel.

String Fragments

Yamaha employed an interesting programming technique in the DX9 firmware to preserve ROM space4: The LCD string copy routine treats any byte it encounters with a value of '128' or above to be an index into a table of commonly repeated 'string fragments'. When encountered, the copy routine will recursively call itself with the table offset as a source string pointer. This will copy the 'string fragment' to the destination, then return to print the remainder of the original string. This technique was not used in the DX7 ROM, however it appears in the firmware of many subsequent Yamaha synths, such as the DX100, and TX81Z.

For more specific technical documentation on this technique, refer to the implementation in the Yamaha DX9/7 firmware here, as well as the accompanying string table documentation here.

Creating the New Firmware

The full task of reverse-engineering the original firmware and writing the new one was too large to exhaustively detail in writing here. I'll use this section to cover a few small areas that I think are meaningful and interesting.
For anyone interested in more specific information, the DX9/7 Firmware's repository contains more technical details about the firmware.

Unfortunately, hacking the DX9 to restore the use of all six operators wasn't going to be as straightforward as just changing a '4' to a '6' in a few crucial places. Ambitious as ever, I figured I could dispense entirely with the DX9 firmware's eccentricities and just write a new ROM from scratch with the full range of features I wanted.

Creating a 'vertical slice' of the ROMs functionality turned out to be more involved than I'd anticipated: Unsurprisingly, getting from zero to handling to MIDI events and assigning notes to voices requires a considerable amount of code, leaving plenty of surface area for things to go wrong... And go wrong they did.

My initial attempt at writing the firmware from scratch was not exactly a roaring success. However, for what it was worth, I did manage to squeeze some never before heard sounds from the poor EGS and OPS chips as my buggy code hammered the pitch-modulation registers with invalid data. You can hear the machine crying out in pain for yourself here5.

Around this point I decided to rethink my strategy.

My first step hacking the existing DX9 firmware to restore the missing functionality was to change the patch initialisation, and loading routines. Importing the code from the DX7 ROM used to serialise, and deserialise stored patches was pretty straightforward. This completed one goal of the project: To restore the DX7's rate scaling implementation.

I took the opportunity to rewrite the synth's keyboard event handlers, simplifying how key events were scanned and processed. The original implementation had different handlers for Note On, and Note Off events. It was straightforward to combine these into a single, more efficient routine.

Portamento and Pitch EG

The synth's portamento processing routine is run as part of the system's periodic interrupt handler. This routine is responsible for 'transitioning' the frequency of each voice6 from its current frequency towards its final target frequency. The updated voice frequency is then written to the EGS chip. The routine alternates between updating voices 1-8, and 9-16 with each interrupt, updating each voice's pitch at a rate of roughly 187Hz7. Similarly, the DX7's pitch EG is updated once every two periodic interrupts.

The YM21290 EGS chip inside the synth is responsible for handling all note events and setting their pitch. It uses a logarithmic representation of note frequencies, with 4096 'units' per octave. In the DX7 and DX9's firmware, Yamaha used a novel method for transforming the 'key codes' used by incoming MIDI messages and keyboard events into this format. On the DX7, the key transpose offset is added to the key code before this transformation is performed. The final logarithmic frequency value written to the EGS is calculated by combining the logarithmic value of the played note together with the current level of the pitch EG (A +-4 octave range in the same representation). The constant 0x1BA8 is then added, which probably represents the logarithmic value of the note A0.

The DX9 uses a different process to calculate the final frequency value in the portamento routine: It calculates the logarithmic frequency of the key transpose offset ahead of time, and adds this value to the frequency of each new note. Since the DX9 doesn't feature a pitch EG, it compensates by adjusting the constants added to the final frequency, and in the calculation of the 'Key Transpose Base Frequency' described above.

After some trial and error, I decided it was much easier to port the DX7's portamento processing code to the new firmware rather than amending the existing implementation to support the pitch EG. As a result, the new firmware ended up featuring the DX7's 'glissando' functionality, which was a nice bonus.

MIDI and Note Handling

I decided to rewrite the synth's MIDI processing routine and individual MIDI event handlers. This gave me the opportunity to store the incoming MIDI velocity, which is used to calculate the final operator volumes when adding an individual note.

As mentioned earlier, the DX9 firmware predates the finalisation of the first MIDI standard, so certain MIDI features needed to be re-implemented from scratch, such as the 'Active Sensing' functionality. I completely rewrote the SysEx parameter change code to be compliant with the implementation in the DX7's v1.8 ROM. This was essential to ensure the DX7's many patch editors would support the firmware.

I also ported the routine used for adding individual new notes from the DX7. When a new note is triggered, this function is responsible for initialising the voice's pitch EG; calculating the volume of each individual operator according to the note's velocity; and calculating the note's initial frequency according to the synth's polyphony, and portamento settings. Porting this code over also restored the 'Follow' portamento mode setting missing from the DX9. While there isn't any way to restore the keyboard's missing velocity sensitivity, this restored the synth's sensitivity to the velocity of incoming MIDI notes.

Current Status

The new ROM binary, and full source code can be found here: https://github.com/ajxs/yamaha_dx97

The main new features added are as follows:

Afterword

Since releasing the project online, I've been extremely fortunate to receive a positive response from not only the synth hacking community, but DX9 owners keen to try it for themselves. I've also been lucky enough to receive positive coverage from several online publications.

One response in particular on Synthopia gave me a good laugh:

Awesome. This is a dream come true for somewhere between 2 to 3 people all over the world.
It's true. I fully admit that this is an extremely niche project. Despite this, the project was a labour of love. I really enjoyed the experience of hacking vintage synth hardware, and being able to squeeze a little bit of extra value out of the 'Worst Ever' FM synth.

In spite of its narrow appeal, I feel very fortunate to have had the opportunity to create something other people can enjoy, even if only for a very small audience.

As for the DX9 itself, it's hard to imagine exactly what kind of success Yamaha expected. As yamahadx9.com so eloquently notes: "...the DX9 couldn't be described in terms of what it offered; it was defined by what it lacked compared to the DX7" 8. With an intimidating price tag of USD$1395 at its launch, the DX9 was more affordable than its bigger brother, but not by much. For an investment of just $600 more, a prospective buyer could have it all9. For some added perspective, at the time of its release in 1984 the now iconic Roland Juno 106 was retailing for only USD$1,095. Particularly amusing in light of the intimidating prices commanded by the 106 today, and the staggering divergence of their prices in the polar opposite directions.

My personal theory about the DX9's design is that developing the FM sound chips incurred such a high up-front cost that Yamaha needed to get them moving off shelves by any means necessary10. Maybe a budget FM synth was always part of Yamaha's plan, but high manufacturing costs limited what kind of discount they could offer? Whatever the case, the DX9 couldn't match the breakaway success of the DX7, and was out of production by mid-1985. By this point Yamaha evidently weren't too concerned with their first-generation FM chips, having since moved on to producing a new line of 4-operator synths built on an entirely new technological foundation.

Yamaha had bet the house on their new chip manufacturing technology. A bold gamble that —In spite of the DX9's lack of success— had paid off spectacularly, catapulting Yamaha to the head of the pack for years to come.

Appendix: Leftover Development Artifacts in the DX9 ROM

During my work reverse-engineering the DX9 ROM binary, I noticed something unusual: The DX9 mask ROM contains several blocks of unused data: A large block of orphaned 6303 code located at offset 0xC84D, and two fragments of strange ASCII data, located at 0xEE7E, and 0xFF85.

The block of orphaned, unreferenced code at 0xC84D definitely doesn't belong to the DX9. All of the DX9's functionality is accounted for in the reachable ROM code, and this unknown code references addresses in the 0x4000 - 0x6000 range, which is outside of the DX9's physical address space. As far as I can tell, none of the other Yamaha devices with a 6800-series processor have an address space with low enough addresses for this code either.

The two fragments of ASCII data appear in conspicuous places, one at the end of the ROM's main string table, and the other between the end of the code and the start of the vector table, located at the end of the binary. The data seemed to consist of a series of strings, with each entry starting with a '`' character (ASCII 0x60, backtick), followed by a six character string of valid ASCII, two non-ASCII bytes, and a terminating '0'11.

At first I didn't think much of the data. My initial suspicion was that the unprintable characters were part of an alternate character encoding, possibly to render Kanji (The DX9 predating Unicode by several years). I dismissed it as being a remnant of the development system and moved on. I had plenty of work left to do on the DX9/7 ROM, after all. Much later, I was doing a cleanup of my disassembly, and I noticed something funny about those unprintable characters... Suddenly it dawned on me that the two trailing non-ASCII bytes weren't part of the string at all... My disassembler had been rendering each block as a single garbled, null-terminated string, but the two trailing bytes were actually an offset into the ROM's address space, and the preceding 6 character strings actually described what they pointed to...
I was looking at a symbol table!

The symbol table fragment located at 0xEE7E, as seen in Ghidra.
Note that the labels such as "`CASS2 " are the ASCII labels contained in the binary, while the accompanying symbol ui_yes_no_cassette_functions::.exit is my own label.

How did this get here? These fragments of the symbol table give us a little bit of insight into the synth's development. Potentially even a clue as to what tools the developers used. It's no coincidence that all the symbol names are 6 characters in length. The 6301/6801 Assembler Text Editor User's Manual states that labels are limited to 6 characters.

The Hitachi cross-assemblers used various different formats for serialising binary data to punched tape, such as Intel HEX, Motorola SREC, and the Intel Absolute Object Module Format, among others. This data doesn't match any of these, or any other format I'm aware of.

The Motorola M68MASR Macro Assemblers Reference Manual mentions ten byte symbol table entries:

'Each unique label, undefined symbol, and external reference symbol in a program is allocated a ten-byte block in the symbol table. In addition, a ten-byte block is allocated for every four references to a symbol, if the cross reference option (paragraph 4,20) is in effect.'
Motorola M68MASR Macro Assemblers Reference Manual, 2-2
The entries in this mysterious DX9 symbol table are also ten bytes. This could just be a coincidence. After all, it's unlikely that Yamaha used this exact cross-assembler. However whatever they did use could very well be built on the same codebase? I'd love to check for myself, but I wouldn't even know where to start looking for this software.

I've checked the ROM of pretty much every other Yamaha device that uses the 6303 looking for a similar leftover symbol table, but as far as I'm aware this is the only device where this has happened.

My guess is that the orphaned 6303 code and this symbol table were in the working memory of whatever development system Yamaha were using, and somehow ended up mixed in with the actual code before being sent off to create the mask ROM. It's anyone's guess how something like this would happen. It's clear that the DX9 binary was written over this data, so maybe it was just what happened to be in the development system's memory before? Maybe they took a snapshot of the whole memory space of their development system after running their final tests?

If you have any ideas, or have any insight to share about what tools Yamaha might have used, please reach out! I'd love to hear any theories you might have.

  1. The DX9 is built around the same two proprietary LSI chips as the DX7: The YM21290 EGS, which handles note events and envelope generation; and the YM21280 OPS, which is responsible for tone generation.
    For more information regarding the DX7's hardware, please refer to this article. ↲
  2. At first I thought the DX series may have been Yamaha's first commercially released instruments to use any microprocessor, however I was incorrect. Yamaha's GS1 —the first commercially released FM synthesiser— used the NEC µPD8035 CPU, a second-source version of the Intel MCS-48 microcontroller.
    A copy of the GS1's service manual is available here. ↲
  3. It's interesting to see the novel ideas that synth designers had for SysEx prior to MIDI's standardisation, and the industry eventually converging on an expected set of capabilities. The DX7, and DX9 offer a clue to the kinds of ideas that manufacturers may have had for where MIDI would lead. ↲
  4. For all I know, this could very well have been a common technique when the DX9 firmware was developed. If this is the case, please forgive me for not knowing the right terminology. Tracking down whether this, and other routines, are well known can be difficult.
    It's quite likely that over time Yamaha developed an internal toolset of bespoke routines that they reused throughout their firmware. If anyone has any insight into this routine, or anything else I may have misnamed, please contact me. I would love to be corrected. ↲
  5. This sounds a little like some kind of crude formant synthesis? I confess, I'm not entirely sure how I created this effect. My best guess was that I was filling the pitch/amp modulation registers on the EGS with bad data with each interrupt. ↲
  6. The DX7's official literature uses the term 'voice' to refer to what is more commonly known as a 'patch' in modern synthesiser parlance, and 'note' to refer to what is more commonly known as a 'voice'. I decided for the sake of minimising confusion to stick to the more modern terminology.
    Similarly, Yamaha's technical literature also features some interesting terminology for programming concepts, using words such as 'job', and 'routine' interchangeably. I use the terms 'routine', and 'subroutine' throughout to stay consistent with Hitachi's official HD6303 documentation. ↲
  7. According to the schematics, the CPU in the DX7 is clocked by a 9.4265 MHz crystal, which is halved by IC48. The HD63B03RP has built in divide-by-4 circuitry, so the synth's actual clock rate is 1.178 MHz. The DX7's OCF interrupt resets the CPU's 'Output Compare' register to '3140'. From this we can calculate the rate of the periodic interrupt: 9.4265 MHz / 2 / 4 / 3140 = 375.258758 Hz.
    The DX9's CPU is clocked by a 3.77 MHz crystal, and the OCF interrupt resets the CPU's 'Output Compare' register to '2500'. 3.77 MHz / 4 / 2500 = 377 Hz. ↲
  8. Yamaha's contemporary marketing literature for the DX9 seems comically tragic in retrospect:
    'If you're used to conventional synthesizers, one look at the DX9 will tell you something unique has happened. There are no knobs, just two linear controls, and a small liquid-crystal digital display panel, and a number of flat-panel touch switches.'
    Today this control scheme is foremost among the synth community's many criticisms. It's likely that Yamaha experimented with a more conventional interface, and realised that real-time control didn't necessarily result in musical effects, and wasn't worth the extra investment. ↲
  9. Steve Howell, in a contemporary article published in the July 1984 issue of Electronics & Music Maker, noted that: 'the difference in price between the two could buy you a second MIDI synthesiser'. Going on to say that he 'bought a JX3P with the money I saved [opting for a DX9 over the DX7]'. Again, like the aforementioned Juno 106, this is highly amusing considering the prices these iconic Roland polysynths fetch today.
    Mind you, the Roland Jupiter-6, also released in 1983, had the hefty price tag of USD$2995. Considerably more than the DX7. ↲
  10. Another theory of mine is that the TX7 —released a year after the DX9 ceased production— was a clever business move by Yamaha to continue profiting from their stockpile of first-generation FM chips. Who knows? I'd love to know more folklore about Yamaha's product development from this era. ↲
  11. For all I know, the '0' byte might actually be the start of an entry. ↲