Ding ding ding! We have a winner folks!
The combination of using the right RS232 interface, at the right speed, with the right wiring.
Here are the instructions I could have used to save myself from a headache and getting snippy on the internet
When performing a firmware update "over the air" using either the Java Firmware Upgrader or the GEM Network Utility connecting to port 8000, you *must* first go to setup and change the baud rate of the gem down to 19200. Whatever device is connecting to that port (dashbox, xbee, network interface, etc) may also need to be reconfigured to that speed.
However, when performing a recovery of the firmware using a direct RS232 connection, you must connect at 115200... not 19200 as used for OTA / standard firmware updates. The board won't respond at 19200.
Also, USB-TTL adapters will *not* work, and will likely blow out your TTL adapter's chip as it is expecting 0-3.3v or 0-5v and RS232 can range from 0 to negative 12 or 15 volts
Steps to make it go (some settings may be superfluous and may get overridden by the network utility in any case):
You need to use a windows laptop, as the GEM Network Utility is the only current tool capable of sending a recovery firmware, and it is windows-only.
Windows in a VM on top of linux or mac *might* work if you have a really good usb to rs232 interface and the VM is configured to have raw access to that USB port, but this is not recommended.
Best case: RS232 port from a real (old) laptop motherboard, set to 115,200 / 8 data bits, 1 stop bits, no parity, no flow control, FIFO buffers enabled and set to "high"
RS232 pin 2 (RX) to GEM COM1 (TX)
RS232 pin 3 (TX) to GEM COM1 (RX)
RS232 pin 5 (GND) to GEM RS232 GND
In the GEM Network Utility, choose serial mode, hit Find Serial Ports, select the correct laptop port (com1 because it has just 1). Serial Port Baud 115200
Make sure connection status is "closed" and the background is red
In the COM firmware section, Select Firmware File and choose the C firmware, then enable the recovery mode checkbox
With the GEM powered off, unplug any network cables, dash box/add-on serial interfaces, etc.
Remove the CR2032 battery (a pair of needle-nose pliers or a sharp edged flat blade screwdriver may help here to get it out. Careful not to damage the board!)
Next, with the battery removed and the RS232-com connection in place, power up the GEM with the bottom "PB Switch" button held down. Look for green LED on COMM LED. Release the button, the SYS LED will go from off to solid red.
Hit start in the Network Utility to deploy the firmware. The progress bar should nearly immediately begin to move. You will know the GEM is receiving data because the SYS LED's green LED will flicker while the red LED stays on. Network utility should say completed successfully, and the GEM's SYS LED will start a blink sequence instead of being solid red. Mine went green-red, red, red, red and repeated.
At this point the Get GEM info button should be working.
If the progress bar doesn't move, the laptop is not getting a correct signal from the GEM. Double check the pins, wires, and port settings.
If the progress bar moves, but the green SYS LED does not flicker during the transfer, the GEM isn't actually receiving any data. Again, check the pins, wires, and port settings.
If the memory may have been corrupt due to a bad power supply or power brownouts/blackouts/etc (did you notice any strange characters in any of the settings screens before you started?) it will be best to reset factory defaults. If you can't connect because of an incorrect password, check the Debug tab of network utility, there is a password reset option.
