Firmware upgrade - Oh Noes!!

[mwall]

teken,

thank you for the suggestions.

of the 10 GEMs that i have installed so far, only two have been problematic. unfortunately they are both at the same site, and i typically only get there to physically diagnose/fix issues maybe once in a year.

one of them decided to change its serial number. one day the serial number simply changed from a normal 01000xxx number to 02904626. nothing i have done has changed it back: updating firmware, resetting the hardware, etc. so i now have a special case in btmon to map that serial back to its proper number. that GEM also has a flaky power connector - if you push the barrel connector down, the power cycles. so over time, gravity pulls the connector down, and the GEM randomly power cycles. so we switched to mini-usb connector to power that GEM.

the other one is the one i have been describing in this thread. i had comm problems with 5 of the GEM when the Adaptor caps started going bad. but this one seems to have not ever really recovered.

i usually install direct serial connections, with btmon pulling the data from the GEM and sending it to influx or emoncms or mysql. all of the environments are windows-free, so having to spin up a windows vm to run some windows-only software is a pain. one of these days i'll rewrite btcfg so that we have a simple, run-anywhere tool to get the entire GEM configuration into a .ini file, and to configure the entire GEM from a single .ini file, that works whether you run it on a mac, linux, or windows machine.

i've got 3 sites using WIZnet just fine, but at one site they simply won't work reliably. we've tried rewiring, recrimping, changing network topology to use dedicated switches, VLANs to isolate the GEMs, ... they just stop talking to the GEM after a day or a month. you can still ping the WIZnet, and you can still talk to the WIZnet, but the WIZnet won't talk to the GEM.

i have considered releasing a weeWX driver for the GEM (and ecm1240). weeWX has more overhead than the single btmon script, but it also introduces better logging, more error handling/recovery, local reporting, local storage, more upload opportunities.

anyway, my frustration in this case is that i have not been able to reliably make this particular GEM communicate, and the process for resetting seems to be much more complicated than it should be.

i've got btmon collecting data from the GEM via WIZnet right now. but it is *not* a clean communication (every third packet or so has unexpected characters).

i'm trying to set up both serial (COM2) and WIZnet (COM1) on each GEM so that when the WIZnets fail again in a month or two, i can get in with a serial connection to reset/diagnose/whatever.

so far i have been able to get data from COM1 or COM2, but not both.

i have one more day at this site, so i'll probably have a go with the latest "GEM Network Utility" instead of the old version 6 i was using.

m

[Teken]

RE: The barrel connector that was an issue on some boards where there was cold solder joint at that point. A ten second reflow of that area corrected that problem area. Please do follow up with the group as to how you get along with this trouble unit.

I know from past experience how frustrating it is to see error messages or not being able to access the GEM's main web page.

This is why the process I outlined is done all the time along with banking the allotted time to complete this task on site.

Happy New Years and keep us in the loop as I am sure any solution you find will help someone else down the road.

[ben]

The supplies with the bad caps have caused a lot of weird/bad issues with the GEM. We've had a few units completely rewrite their EEPROM values to random numbers. The bad SN is probably a consequence of this.

I would do a factory reset as you noted before, then set the hardware option as Teken said, and start from there.

The weird SN likely needs to be reset. I'll see if I can get a program off Paul to do just that.

[ben]

Emailed over the commands for the serial number.

[mwall]

everything is working now.

tcp/ip comms are still a bit wobbly on the two GEM that suffered most from the Adaptor bad caps. i'm still seeing random bogus bytes every few packets. it seems to be coming from the WIZnet, since the direct serial connection on each GEM is flawless. it might be related to the keepalive character - the keepalive strings were also corrupted and had to be reset.

anyway, this is what i did to get everything working, on each of the two GEM (using "GEM Network Utility 7.4"):

1) disconnect
unplug GEM power (but not PT)
remove GEM battery
disconnect ethernet cable from WIZnet

2) establish physical connection to GEM COM1
usb-serial converter to three-wire 'probe' to GEM COM1

3) start "GEM Network Utility" in a windows vm
choose the COM port corresponding to the usb-serial device

4) power up GEM into recovery mode
press and hold button on bottom of GEM, plug in GEM power, release button
left LED steady red, right LED steady green

5) (re)flash the 4.33 COM firmware
progress indicator worked properly, GEM rebooted itself

6) insert battery

7) do 'Get GEM info' in "GEM Network Utility"
the small panel indicated all zeros for every setting, but there were no failures

8) do 'Open in web browser' in "GEM Network Utility"
- reset all counters
- verify and reprogram the channel, CT, temperature
- verify realtime frequency, packet type
- set both COM1 and COM2 to 115200

9) using "GEM Network Utility", save a snapshot of the configuration

10) reconnect
network cable to the WIZnet ethernet port
connect 3 wires to COM2 on the GEM

now i can pull data over the network, and i can pull data over the COM2 port. on the (linux) computer that gets data from the GEM, i can see the packets in real time from both ethernet and serial (using btmon). that should make it easier to debug WIZnet issues. if the WIZnet loses contact with the GEM, i 'll still be able to pull data over the COM2 port (unless the WIZnet tries to send data *to* the GEM, in which case the serial data could be corrupted - we'll cross that bridge when we come to it).

i was able to use the commands on the 'debug' tab of "GEM Network Utility" to reset the serial number of the GEM with bogus serial number (thanks ben!)

i did not have to re-flash the ENG firmware on either device.

i did not have time to re-solder the wonky barrel connector, so that one is still powered using the mini-usb port. (it also has a PT plugged in - the brultech docs make it sound like the mini-usb port will both power the GEM and do the PT magic, but confounding those two seems odd to me, so i left the PT plugged in)

each GEM now has both ethernet cable and serial cable from it to the server racks (about a 10 meter run).

basically the key to eliminating frustration in this exercise was to use a direct serial connection instead of trying to go through the ethernet (WIZnet, in this case) interface.

m

we were considering installation of yet another GEM this month at a somewhat remote location (completely off the grid), but after reading about teken's cold weather fun and experiencing some rather choppy penobscot bay conditions, we'll probably hold off on that until summertime

[ben]

i did not have time to re-solder the wonky barrel connector, so that one is still powered using the mini-usb port. (it also has a PT plugged in - the brultech docs make it sound like the mini-usb port will both power the GEM and do the PT magic, but confounding those two seems odd to me, so i left the PT plugged in)

We haven't had a chance to make a combination PT/Power Supply yet. The Mini-USB jack has 5 pins, 1 and 5 are used for power (DC), and 2 of the other pins are used for the PT (AC).