GEM abruptly reset its serial ports from 19200 bps to 115200

Ask questions about the GEM here.
Post Reply
jnh
Posts: 23
Joined: Mon Jul 22, 2013 6:22 am

GEM abruptly reset its serial ports from 19200 bps to 115200

Post by jnh » Mon Dec 14, 2020 1:51 pm

At 11:50pm last night, a few minutes before its midnight log rollover, my direct-serial-connected GEM (no network modules) abruptly stopped responding to polling commands (^^^APISPT once/second from my custom logger; using packet format 8, BIN32NET). I was asleep then, and didn't notice the problem until about 11am this morning, when I checked whether the heater had run overnight and noticed a lack of data. No bad-checksum or other errors were logged at the time it went down.

After various troubleshooting attempts, power-cycling the GEM, switching serial ports, etc. I moved its connection to an x86 PC to run the setup utility and discovered both GEM serial ports had somehow reset themselves from 19200 bps to the factory-default 115200 (I believe my cable normally connects to GEM's port 2, with nothing on 1, but don't want to remove the cover to verify since it's a little hard to get to, and I usually end up losing one of those tiny screws. In any case, for some reason changing the baud rate of port 1 changed both 1 and 2 to 19200). I use the lower rate because my long cable length (approx 25m / 80ft over Cat5) prevents 115.2k from being fully reliable.

No other config options appear to have been lost or changed, only the baud rates, and the GEM appears to have continued recording watt-hour data accurately for the duration of its communications outage.

Has anyone seen this happen before? Are there any known bugs in older firmware that could be relevant? My GEM's been running well since late December 2019, and so I haven't tried to update it since that time. It is running

Com firmware ver: 5.13
Engine1 firmware ver: 1.49

I did have to power-cycle it once over the summer, following a VERY close lightning strike which caused it to seize up, but thankfully there was no permanent damage. I have resistors & clamp diodes on the serial line for protection, and typical MOV-based suppression on the AC subpanel it feeds from, which is behind an Outback Power solar PV inverter system that doubles as whole-house UPS. There were no apparent power disturbances last night, though - the Outback logs those separately, and its communications to another serial port on the same monitoring computer were not interrupted.

Is it possible the GEM's 5V adapter is starting to fail? I've read here about that being a common source of trouble, but my GEM was ordered in late 2019, and so should have the newer, more reliable design. Still, I've seen various random flakiness from things like Raspberry Pi's over the years which turned out to be from a noisy or poorly-regulated 5V supply.

One oddity that makes me wonder: over the past month or two, the 12V DC fuse panel supplying most computer & network equipment in my home office area (other end of that 25m cable run) as well as various radio equipment has started to make a faint, high-pitched beeping/squeaking sound in perfect time to the GEM's serial port transmissions - disconnecting the GEM or stopping its logger process kills this noise. It don't quite understand a simple, mostly-passive DC power distribution panel making audible noise at all, but it does have filter capacitors across all terminals, as well as an over/under-voltage alerting circuit, so maybe a capacitor or inductor in there has started "singing". This is a Rigrunner 8012 from West Mountain Radio. The grounded DC- of this panel, and computers feeding from it is common with serial ground. Serial data to/from the Outback system over other wires in the same cable, into another port of the same computer (also at 19200 bps) does not trigger or correlate with any sounds, but that's not quite comparable because there's an active control-panel device in between the computer & remote Outback that provides line buffering and optoisolation. I should go over this DC panel & wiring with a highly directional mic to help narrow down the exact source of its sound.

Anyway, if someone has insight/guesses into either issue I'd appreciate hearing them :)
ben
Site Admin
Posts: 4262
Joined: Fri Jun 04, 2010 9:39 am

Re: GEM abruptly reset its serial ports from 19200 bps to 115200

Post by ben » Mon Dec 14, 2020 1:58 pm

Pauls going to look into the issue. When the GEM boots it pulls some values from memory, might be getting switched in that period.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
jnh
Posts: 23
Joined: Mon Jul 22, 2013 6:22 am

Re: GEM abruptly reset its serial ports from 19200 bps to 115200

Post by jnh » Mon Dec 14, 2020 3:33 pm

Thanks! For now I went ahead and bumped it back up to 115200, which is working surprisingly well despite the long line, and had the benefit of eliminating that annoying chirping from my DC panel (pushed it up into ultrasonic frequencies, perhaps) as I'd hoped. I can't remember now whether there were actual problems before at this rate, or I'd merely assumed there would be, and stayed with 19200 to match the ECM-1240. I'll watch it closely for a few days, but wouldn't care much it comes down to a few dropped samples/day, since accuracy of the internal joule-counters shouldn't be affected.
ben
Site Admin
Posts: 4262
Joined: Fri Jun 04, 2010 9:39 am

Re: GEM abruptly reset its serial ports from 19200 bps to 115200

Post by ben » Tue Dec 15, 2020 12:10 pm

jnh wrote:
Mon Dec 14, 2020 3:33 pm
Thanks! For now I went ahead and bumped it back up to 115200, which is working surprisingly well despite the long line, and had the benefit of eliminating that annoying chirping from my DC panel (pushed it up into ultrasonic frequencies, perhaps) as I'd hoped. I can't remember now whether there were actual problems before at this rate, or I'd merely assumed there would be, and stayed with 19200 to match the ECM-1240. I'll watch it closely for a few days, but wouldn't care much it comes down to a few dropped samples/day, since accuracy of the internal joule-counters shouldn't be affected.
Shouldn't be any dropped samples, especially when using binary packets given the packet size.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Post Reply