GEM suddenly not responding in dashbox, last packet in April?

[JPhi1618]

I have a single GEM (sn 01011293) and a dashbox connected via serial cable. The other day, I opened up the dashbox page, and all my readings seemed to be frozen. The "Status" tab at the top shows a red stop sign next to it, but when I click on the tab heading, nothing happens. The tab does't load. When I click on the settings tab, it shows the GEM with an orange stop sign, and it says the last packet was delivered on 2023-04-16, but I know I've used it since April so that can't be right.

On the GEM itself, the left LED blinks red once every few seconds and the right LED is solid green. As far as I know, my GEM doesn't have WiFi or ethernet. I've always used the serial cable.

I will say that a few months ago, I did upgrade the SD card for the dashbox to a larger card, but everything has worked fine since then. I have restarted the GEM and the dashbox, and I've checked the serial connection on the board and the headphone jack. The system has been rock solid since I bought it, which is great, but now I find myself not remembering anything about troubleshooting because I've never had to touch it. The status page not loading is the biggest red flag to me, and makes me think the dashbox might be acting up. I'm an experienced software engineer, so don't hold back if there are any technical things I can try.

Additional: On the dashbox, the Raw Log page shows some events on the system log with the same April date (the date at the top of the dashbox page is correct, not April). Also the <serial number>.log file next to that is just filled with garbage characters.

[ben]

I have a single GEM (sn 01011293) and a dashbox connected via serial cable. The other day, I opened up the dashbox page, and all my readings seemed to be frozen. The "Status" tab at the top shows a red stop sign next to it, but when I click on the tab heading, nothing happens. The tab does't load. When I click on the settings tab, it shows the GEM with an orange stop sign, and it says the last packet was delivered on 2023-04-16, but I know I've used it since April so that can't be right.

On the GEM itself, the left LED blinks red once every few seconds and the right LED is solid green. As far as I know, my GEM doesn't have WiFi or ethernet. I've always used the serial cable.

I will say that a few months ago, I did upgrade the SD card for the dashbox to a larger card, but everything has worked fine since then. I have restarted the GEM and the dashbox, and I've checked the serial connection on the board and the headphone jack. The system has been rock solid since I bought it, which is great, but now I find myself not remembering anything about troubleshooting because I've never had to touch it. The status page not loading is the biggest red flag to me, and makes me think the dashbox might be acting up. I'm an experienced software engineer, so don't hold back if there are any technical things I can try.

Additional: On the dashbox, the Raw Log page shows some events on the system log with the same April date (the date at the top of the dashbox page is correct, not April). Also the <serial number>.log file next to that is just filled with garbage characters.

What F/W is the DashBox running? Start by updating that to the latest (https://www.brultech.com/software).

After that check under Settings->System->Change Baudrates to make sure nothing got misconfigured there. If it says "Unknown Device" it's likely either the DashBox or GEM side got toggled.

[JPhi1618]

Thanks Ben. My firmware was so old that I had to upgrade it three times, but I'm now on the latest version. The status page does display now, and it just says "no recent packets". Baud rates are 115200. I tried to change it to 19200 and that didn't help. I changed it back to the 115200. I verified the voltage of the power supply to the GEM and it is ~5.2vdc. The headphone connector did seem to have some black corrosion or something on it, so I cleaned that off. I've tried COM1 and 2 with no change (on the dashbox and the GEM). The GEM LED behavior is the same as before. There does seem to be a communication issue between the GEM and dashbox, but I don't know how to debug further. Right now, the Raw Logs page shows the page header with a control to clear or update the logs, but there is no log information below that - just blank page.

[ben]

Thanks Ben. My firmware was so old that I had to upgrade it three times, but I'm now on the latest version. The status page does display now, and it just says "no recent packets". Baud rates are 115200. I tried to change it to 19200 and that didn't help. I changed it back to the 115200. I verified the voltage of the power supply to the GEM and it is ~5.2vdc. The headphone connector did seem to have some black corrosion or something on it, so I cleaned that off. I've tried COM1 and 2 with no change (on the dashbox and the GEM). The GEM LED behavior is the same as before. There does seem to be a communication issue between the GEM and dashbox, but I don't know how to debug further. Right now, the Raw Logs page shows the page header with a control to clear or update the logs, but there is no log information below that - just blank page.

Log into the DashBox and go to http://<dashboxip>/index.php/settings/clearMemory to get rid of any old temp files from the old F/W.

What did the box at the top of Change Baudrates say? One way you can confirm whether or not the serial connection works is to try the http://<dashbox-ip>:8001/ or http://<dashbox-ip>:8002/ pages (COM1/2). If the baud rates are correct it'll pull the GEM Setup Webpage.

If you can't get the webpage to pull on either baud we need to investigate further.

[JPhi1618]

That clearMemory URL gives me a 404 error. I'm currently on Dashbox firmware 4.2.4c17c with "Updater" 4.2.4b.

Additional strangeness this morning. It looks like from 2am to 3am last night, the GEM sent packets to the dashbox and all channels show data from that time. At the top of the change baudrates screen, it says "GreenEye Monitor detected", but it did used to say "No device detected". Trying to hit the serial connection using ip:port does not work, and there seems to be no data currently coming from the GEM. Changing the baud rate doesn't help.

[ben]

That clearMemory URL gives me a 404 error. I'm currently on Dashbox firmware 4.2.4c17c with "Updater" 4.2.4b.

Additional strangeness this morning. It looks like from 2am to 3am last night, the GEM sent packets to the dashbox and all channels show data from that time. At the top of the change baudrates screen, it says "GreenEye Monitor detected", but it did used to say "No device detected". Trying to hit the serial connection using ip:port does not work, and there seems to be no data currently coming from the GEM. Changing the baud rate doesn't help.

Sorry, it's http://<dashboxip>/index.php/pages/settings/clearMemory

Try unplugging/replugging the GEM and hitting it again.

Are you on COM1 or COM2 GEM-side? If COM1, do you have a communication module at all (Ethernet, wifi, etc).

[JPhi1618]

Ok, cleared memory (got a 200, blank response from that URL) and power cycled the GEM. That didn't seem to help. After my last message, I had left the Baud at 19200 and when I came back, the page said an unknown device was detected. I think it must have received another burst of data from the GEM at 115200 and couldn't interpret it. I've changed the baud back to 115200 for the time being.

So, the COM port story could provide helpful information, or it could be extra info that just confuses the issue at hand. The COM I'm using is #2 on the GEM and the Dashbox. A few months ago, I was adding a circuit and moving some CTs (with the GEM off) and when I brought it all back up, the Dashbox was having trouble starting. I think the light would stay red and the ethernet port would not light up. I disconnected the headphone serial cable and the dashbox would then start up. So, suspecting I loosened a wire or something while working on the GEM, I just move the serial cable to COM 2, and plugged it into COM 2 on the dashbox, and it all started up and has been happy since then.

During this outage, I've tried COM 1 again on the GEM and dashbox, and I didn't get the same startup issue. The Dashbox seems to come up just fine regardless of the COM port I'm using.



I honestly don't remember if I have a communication module, but I don't think I do. I've always had it serial-connected to the dashbox. Image of connections included.

[ben]

Ok, cleared memory (got a 200, blank response from that URL) and power cycled the GEM. That didn't seem to help. After my last message, I had left the Baud at 19200 and when I came back, the page said an unknown device was detected. I think it must have received another burst of data from the GEM at 115200 and couldn't interpret it. I've changed the baud back to 115200 for the time being.

So, the COM port story could provide helpful information, or it could be extra info that just confuses the issue at hand. The COM I'm using is #2 on the GEM and the Dashbox. A few months ago, I was adding a circuit and moving some CTs (with the GEM off) and when I brought it all back up, the Dashbox was having trouble starting. I think the light would stay red and the ethernet port would not light up. I disconnected the headphone serial cable and the dashbox would then start up. So, suspecting I loosened a wire or something while working on the GEM, I just move the serial cable to COM 2, and plugged it into COM 2 on the dashbox, and it all started up and has been happy since then.

During this outage, I've tried COM 1 again on the GEM and dashbox, and I didn't get the same startup issue. The Dashbox seems to come up just fine regardless of the COM port I'm using.

I honestly don't remember if I have a communication module, but I don't think I do. I've always had it serial-connected to the dashbox. Image of connections included.

No comm module on that unit.

Since you have no comm. module your best bet is to stick with COM1 there. It's the main COM port.

Send over the Status page (either post here or email me it) of the GEM and I'll see if you got any issues/suggestions there.

[JPhi1618]

Thanks, but just so we’re clear, what status page do you want?

[ben]

Thanks, but just so we’re clear, what status page do you want?

In the GEM Setup Webpage (the :8001 or :8002 site) there's a tab called Status that'll give an Overview of your settings.