GEM suddenly not responding in dashbox, last packet in April?
-
- Posts: 8
- Joined: Tue Sep 19, 2023 2:23 pm
GEM suddenly not responding in dashbox, last packet in April?
I have a single GEM (sn 01011293) and a dashbox connected via serial cable. The other day, I opened up the dashbox page, and all my readings seemed to be frozen. The "Status" tab at the top shows a red stop sign next to it, but when I click on the tab heading, nothing happens. The tab does't load. When I click on the settings tab, it shows the GEM with an orange stop sign, and it says the last packet was delivered on 2023-04-16, but I know I've used it since April so that can't be right.
On the GEM itself, the left LED blinks red once every few seconds and the right LED is solid green. As far as I know, my GEM doesn't have WiFi or ethernet. I've always used the serial cable.
I will say that a few months ago, I did upgrade the SD card for the dashbox to a larger card, but everything has worked fine since then. I have restarted the GEM and the dashbox, and I've checked the serial connection on the board and the headphone jack. The system has been rock solid since I bought it, which is great, but now I find myself not remembering anything about troubleshooting because I've never had to touch it. The status page not loading is the biggest red flag to me, and makes me think the dashbox might be acting up. I'm an experienced software engineer, so don't hold back if there are any technical things I can try.
Additional: On the dashbox, the Raw Log page shows some events on the system log with the same April date (the date at the top of the dashbox page is correct, not April). Also the <serial number>.log file next to that is just filled with garbage characters.
On the GEM itself, the left LED blinks red once every few seconds and the right LED is solid green. As far as I know, my GEM doesn't have WiFi or ethernet. I've always used the serial cable.
I will say that a few months ago, I did upgrade the SD card for the dashbox to a larger card, but everything has worked fine since then. I have restarted the GEM and the dashbox, and I've checked the serial connection on the board and the headphone jack. The system has been rock solid since I bought it, which is great, but now I find myself not remembering anything about troubleshooting because I've never had to touch it. The status page not loading is the biggest red flag to me, and makes me think the dashbox might be acting up. I'm an experienced software engineer, so don't hold back if there are any technical things I can try.
Additional: On the dashbox, the Raw Log page shows some events on the system log with the same April date (the date at the top of the dashbox page is correct, not April). Also the <serial number>.log file next to that is just filled with garbage characters.
-
- Site Admin
- Posts: 4262
- Joined: Fri Jun 04, 2010 9:39 am
Re: GEM suddenly not responding in dashbox, last packet in April?
What F/W is the DashBox running? Start by updating that to the latest (https://www.brultech.com/software).JPhi1618 wrote: ↑Tue Sep 19, 2023 3:06 pmI have a single GEM (sn 01011293) and a dashbox connected via serial cable. The other day, I opened up the dashbox page, and all my readings seemed to be frozen. The "Status" tab at the top shows a red stop sign next to it, but when I click on the tab heading, nothing happens. The tab does't load. When I click on the settings tab, it shows the GEM with an orange stop sign, and it says the last packet was delivered on 2023-04-16, but I know I've used it since April so that can't be right.
On the GEM itself, the left LED blinks red once every few seconds and the right LED is solid green. As far as I know, my GEM doesn't have WiFi or ethernet. I've always used the serial cable.
I will say that a few months ago, I did upgrade the SD card for the dashbox to a larger card, but everything has worked fine since then. I have restarted the GEM and the dashbox, and I've checked the serial connection on the board and the headphone jack. The system has been rock solid since I bought it, which is great, but now I find myself not remembering anything about troubleshooting because I've never had to touch it. The status page not loading is the biggest red flag to me, and makes me think the dashbox might be acting up. I'm an experienced software engineer, so don't hold back if there are any technical things I can try.
Additional: On the dashbox, the Raw Log page shows some events on the system log with the same April date (the date at the top of the dashbox page is correct, not April). Also the <serial number>.log file next to that is just filled with garbage characters.
After that check under Settings->System->Change Baudrates to make sure nothing got misconfigured there. If it says "Unknown Device" it's likely either the DashBox or GEM side got toggled.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Brultech Research Inc.
E: ben(at)brultech.com
-
- Posts: 8
- Joined: Tue Sep 19, 2023 2:23 pm
Re: GEM suddenly not responding in dashbox, last packet in April?
Thanks Ben. My firmware was so old that I had to upgrade it three times, but I'm now on the latest version. The status page does display now, and it just says "no recent packets". Baud rates are 115200. I tried to change it to 19200 and that didn't help. I changed it back to the 115200. I verified the voltage of the power supply to the GEM and it is ~5.2vdc. The headphone connector did seem to have some black corrosion or something on it, so I cleaned that off. I've tried COM1 and 2 with no change (on the dashbox and the GEM). The GEM LED behavior is the same as before. There does seem to be a communication issue between the GEM and dashbox, but I don't know how to debug further. Right now, the Raw Logs page shows the page header with a control to clear or update the logs, but there is no log information below that - just blank page.
-
- Site Admin
- Posts: 4262
- Joined: Fri Jun 04, 2010 9:39 am
Re: GEM suddenly not responding in dashbox, last packet in April?
Log into the DashBox and go to http://<dashboxip>/index.php/settings/clearMemory to get rid of any old temp files from the old F/W.JPhi1618 wrote: ↑Wed Sep 20, 2023 3:54 pmThanks Ben. My firmware was so old that I had to upgrade it three times, but I'm now on the latest version. The status page does display now, and it just says "no recent packets". Baud rates are 115200. I tried to change it to 19200 and that didn't help. I changed it back to the 115200. I verified the voltage of the power supply to the GEM and it is ~5.2vdc. The headphone connector did seem to have some black corrosion or something on it, so I cleaned that off. I've tried COM1 and 2 with no change (on the dashbox and the GEM). The GEM LED behavior is the same as before. There does seem to be a communication issue between the GEM and dashbox, but I don't know how to debug further. Right now, the Raw Logs page shows the page header with a control to clear or update the logs, but there is no log information below that - just blank page.
What did the box at the top of Change Baudrates say? One way you can confirm whether or not the serial connection works is to try the http://<dashbox-ip>:8001/ or http://<dashbox-ip>:8002/ pages (COM1/2). If the baud rates are correct it'll pull the GEM Setup Webpage.
If you can't get the webpage to pull on either baud we need to investigate further.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Brultech Research Inc.
E: ben(at)brultech.com
-
- Posts: 8
- Joined: Tue Sep 19, 2023 2:23 pm
Re: GEM suddenly not responding in dashbox, last packet in April?
That clearMemory URL gives me a 404 error. I'm currently on Dashbox firmware 4.2.4c17c with "Updater" 4.2.4b.
Additional strangeness this morning. It looks like from 2am to 3am last night, the GEM sent packets to the dashbox and all channels show data from that time. At the top of the change baudrates screen, it says "GreenEye Monitor detected", but it did used to say "No device detected". Trying to hit the serial connection using ip:port does not work, and there seems to be no data currently coming from the GEM. Changing the baud rate doesn't help.
Additional strangeness this morning. It looks like from 2am to 3am last night, the GEM sent packets to the dashbox and all channels show data from that time. At the top of the change baudrates screen, it says "GreenEye Monitor detected", but it did used to say "No device detected". Trying to hit the serial connection using ip:port does not work, and there seems to be no data currently coming from the GEM. Changing the baud rate doesn't help.
-
- Site Admin
- Posts: 4262
- Joined: Fri Jun 04, 2010 9:39 am
Re: GEM suddenly not responding in dashbox, last packet in April?
Sorry, it's http://<dashboxip>/index.php/pages/settings/clearMemoryJPhi1618 wrote: ↑Thu Sep 21, 2023 10:13 amThat clearMemory URL gives me a 404 error. I'm currently on Dashbox firmware 4.2.4c17c with "Updater" 4.2.4b.
Additional strangeness this morning. It looks like from 2am to 3am last night, the GEM sent packets to the dashbox and all channels show data from that time. At the top of the change baudrates screen, it says "GreenEye Monitor detected", but it did used to say "No device detected". Trying to hit the serial connection using ip:port does not work, and there seems to be no data currently coming from the GEM. Changing the baud rate doesn't help.
Try unplugging/replugging the GEM and hitting it again.
Are you on COM1 or COM2 GEM-side? If COM1, do you have a communication module at all (Ethernet, wifi, etc).
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Brultech Research Inc.
E: ben(at)brultech.com
-
- Posts: 8
- Joined: Tue Sep 19, 2023 2:23 pm
Re: GEM suddenly not responding in dashbox, last packet in April?
Ok, cleared memory (got a 200, blank response from that URL) and power cycled the GEM. That didn't seem to help. After my last message, I had left the Baud at 19200 and when I came back, the page said an unknown device was detected. I think it must have received another burst of data from the GEM at 115200 and couldn't interpret it. I've changed the baud back to 115200 for the time being.
So, the COM port story could provide helpful information, or it could be extra info that just confuses the issue at hand. The COM I'm using is #2 on the GEM and the Dashbox. A few months ago, I was adding a circuit and moving some CTs (with the GEM off) and when I brought it all back up, the Dashbox was having trouble starting. I think the light would stay red and the ethernet port would not light up. I disconnected the headphone serial cable and the dashbox would then start up. So, suspecting I loosened a wire or something while working on the GEM, I just move the serial cable to COM 2, and plugged it into COM 2 on the dashbox, and it all started up and has been happy since then.
During this outage, I've tried COM 1 again on the GEM and dashbox, and I didn't get the same startup issue. The Dashbox seems to come up just fine regardless of the COM port I'm using.
I honestly don't remember if I have a communication module, but I don't think I do. I've always had it serial-connected to the dashbox. Image of connections included.
So, the COM port story could provide helpful information, or it could be extra info that just confuses the issue at hand. The COM I'm using is #2 on the GEM and the Dashbox. A few months ago, I was adding a circuit and moving some CTs (with the GEM off) and when I brought it all back up, the Dashbox was having trouble starting. I think the light would stay red and the ethernet port would not light up. I disconnected the headphone serial cable and the dashbox would then start up. So, suspecting I loosened a wire or something while working on the GEM, I just move the serial cable to COM 2, and plugged it into COM 2 on the dashbox, and it all started up and has been happy since then.
During this outage, I've tried COM 1 again on the GEM and dashbox, and I didn't get the same startup issue. The Dashbox seems to come up just fine regardless of the COM port I'm using.
I honestly don't remember if I have a communication module, but I don't think I do. I've always had it serial-connected to the dashbox. Image of connections included.
-
- Site Admin
- Posts: 4262
- Joined: Fri Jun 04, 2010 9:39 am
Re: GEM suddenly not responding in dashbox, last packet in April?
No comm module on that unit.JPhi1618 wrote: ↑Thu Sep 21, 2023 5:39 pmOk, cleared memory (got a 200, blank response from that URL) and power cycled the GEM. That didn't seem to help. After my last message, I had left the Baud at 19200 and when I came back, the page said an unknown device was detected. I think it must have received another burst of data from the GEM at 115200 and couldn't interpret it. I've changed the baud back to 115200 for the time being.
So, the COM port story could provide helpful information, or it could be extra info that just confuses the issue at hand. The COM I'm using is #2 on the GEM and the Dashbox. A few months ago, I was adding a circuit and moving some CTs (with the GEM off) and when I brought it all back up, the Dashbox was having trouble starting. I think the light would stay red and the ethernet port would not light up. I disconnected the headphone serial cable and the dashbox would then start up. So, suspecting I loosened a wire or something while working on the GEM, I just move the serial cable to COM 2, and plugged it into COM 2 on the dashbox, and it all started up and has been happy since then.
During this outage, I've tried COM 1 again on the GEM and dashbox, and I didn't get the same startup issue. The Dashbox seems to come up just fine regardless of the COM port I'm using.
I honestly don't remember if I have a communication module, but I don't think I do. I've always had it serial-connected to the dashbox. Image of connections included.
Since you have no comm. module your best bet is to stick with COM1 there. It's the main COM port.
Send over the Status page (either post here or email me it) of the GEM and I'll see if you got any issues/suggestions there.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Brultech Research Inc.
E: ben(at)brultech.com
-
- Posts: 8
- Joined: Tue Sep 19, 2023 2:23 pm
Re: GEM suddenly not responding in dashbox, last packet in April?
Thanks, but just so we’re clear, what status page do you want?
-
- Site Admin
- Posts: 4262
- Joined: Fri Jun 04, 2010 9:39 am
Re: GEM suddenly not responding in dashbox, last packet in April?
In the GEM Setup Webpage (the :8001 or :8002 site) there's a tab called Status that'll give an Overview of your settings.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Brultech Research Inc.
E: ben(at)brultech.com