btmon stopped processing data

[Rocquestar]

So the other day, I noticed that my graphs were empty, and when I checked, the GEM was still running, but btmon wasn't updating the rra's.

So, I love that the GEM and the btmon that I have running just seems to work most of the time, but that means that the time between debugging issues is very large, and when something does go wrong, I feel like I'm starting again. Anyway, I was running btmon311, so the first thing I did was grab the latest version (320) and give it a try, but the result was the same.

I'm struggling to gather useful (to me) information, so I'm reaching out here. The logs gave a little info about when it stopped- everything seemed to be fine until 23:59:55 on the 18th, but when newsyslog rolled the logs, btmon gave no more log entries (as in, it crashed, or didn't restart) until almost 48 hours later, when I discovered it and manually tried to restart.

Here's (the top of) the log file from earlier this week, when it was working properly. This isn't debug output, because it was running fine:

Code: Select all

2018/09/18 13:18:04 btmon: 3.1.1
2018/09/18 13:18:05 python: 2.7.3 (default, Mar 26 2013, 17:57:50) 
[GCC 4.2.1 20070831 patched [FreeBSD]]
2018/09/18 13:18:05 platform: freebsd8
2018/09/18 13:18:05 polarity is reversed
2018/09/18 13:18:05 device type: gem
2018/09/18 13:18:05 device list: ['']
2018/09/18 13:18:05 packet format: gem48ptbin
2018/09/18 13:18:05 schema: counters
2018/09/18 13:18:05 SOCKET: bind host: 
2018/09/18 13:18:05 SOCKET: bind port: 5000
2018/09/18 13:18:05 RRD: update period: 60
2018/09/18 13:18:05 RRD: dir: /usr/local/www/apache22/data/cacti/rra
2018/09/18 13:18:05 RRD: step: 10
2018/09/18 13:18:05 RRD: heartbeat: 20
2018/09/18 13:18:23 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch26_pws.rrd with 1 values
2018/09/18 13:18:23 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch17_aws.rrd with 1 values
...

...and those "updating [...] with 1 values" lines repeat for the '_pws.rrd' and '_aws.rrd' for each of my channels, then after a minute, they all repeat, updating "...with 6 values" which is what I would expect, since it's gathering six 10-second intervals before updating the rras.

But now, it stops after the first packet. Here's the (non-debug) log file from two nights ago, when it failed:

Code: Select all

2018/09/20 23:55:42 btmon: 3.1.1
2018/09/20 23:55:42 python: 2.7.3 (default, Mar 26 2013, 17:57:50) 
[GCC 4.2.1 20070831 patched [FreeBSD]]
2018/09/20 23:55:42 platform: freebsd8
2018/09/20 23:55:42 polarity is reversed
2018/09/20 23:55:42 device type: gem
2018/09/20 23:55:42 device list: ['']
2018/09/20 23:55:42 packet format: gem48ptbin
2018/09/20 23:55:42 schema: counters
2018/09/20 23:55:42 SOCKET: bind host: 
2018/09/20 23:55:42 SOCKET: bind port: 5000
2018/09/20 23:55:42 RRD: update period: 60
2018/09/20 23:55:42 RRD: dir: /usr/local/www/apache22/data/cacti/rra
2018/09/20 23:55:42 RRD: step: 10
2018/09/20 23:55:42 RRD: heartbeat: 20
2018/09/20 23:55:55 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch26_pws.rrd with 1 values
2018/09/20 23:55:55 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch17_aws.rrd with 1 values
2018/09/20 23:55:55 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch6_aws.rrd with 1 values
2018/09/20 23:55:55 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch25_pws.rrd with 1 values
2018/09/20 23:55:55 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch31_pws.rrd with 1 values
2018/09/20 23:55:55 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch29_aws.rrd with 1 values
2018/09/20 23:55:55 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch3_aws.rrd with 1 values

...and that's it; just seven entries (I have 32 channels, plus a few temp probes, so maybe 70 lines in a normal update (with one for each of the pws and aws lines). The next log entry is when I tried to restart it again 10 minutes later.

When i run it in debug mode, there's a little more info, but I can't make anything of it:

Code: Select all

2018/09/21 23:02:18 btmon: 3.2.0
2018/09/21 23:02:18 python: 2.7.3 (default, Mar 26 2013, 17:57:50) 
[GCC 4.2.1 20070831 patched [FreeBSD]]
2018/09/21 23:02:18 platform: freebsd8
2018/09/21 23:02:18 polarity is reversed
2018/09/21 23:02:18 device type: gem
2018/09/21 23:02:18 device list: ['']
2018/09/21 23:02:18 packet format: gem48ptbin
2018/09/21 23:02:18 schema: counters
2018/09/21 23:02:18 buffer size: 120
2018/09/21 23:02:18 SOCKET: bind host: 
2018/09/21 23:02:18 SOCKET: bind port: 5000
2018/09/21 23:02:18 RRD: update period: 60
2018/09/21 23:02:18 RRD: dir: /usr/local/www/apache22/data/cacti/rra
2018/09/21 23:02:18 RRD: step: 10
2018/09/21 23:02:18 RRD: heartbeat: 20
2018/09/21 23:02:18 packet format is GEM48PTBinaryPacket
2018/09/21 23:02:18 using collector SocketServerCollector
2018/09/21 23:02:18 using 1 processors:
2018/09/21 23:02:18   RRDProcessor
2018/09/21 23:02:18 setup SocketServerCollector
2018/09/21 23:02:18 SOCKET: binding to :5000
2018/09/21 23:02:18 setup RRDProcessor
2018/09/21 23:02:18 SOCKET: waiting for connection
2018/09/21 23:02:21 waiting for data from device
2018/09/21 23:02:21 reading 1 of 1 packets
2018/09/21 23:02:21 SOCKET: read 1 of 1 bytes from socket: fe
2018/09/21 23:02:21 SOCKET: read 1 of 1 bytes from socket: ff
2018/09/21 23:02:21 SOCKET: read 1 of 1 bytes from socket: 05
2018/09/21 23:02:21 SOCKET: read 8 of 619 bytes from socket: 04 b7 8e d5 cc ba 51 8c
2018/09/21 23:02:21 SOCKET: read 611 of 611 bytes from socket: 82 a6 c8 61 3c 51 bd 6b 39 d8 2c e1 7f 6b 19 b1 e8 69 64 05 62 d7 2e 70 3d bf 1a e4 02 2b 79 f2 96 00 46 e8 ce d1 01 16 f3 87 a2 01 d1 7d 89 a4 00 60 1e 8d f9 02 45 ec 48 9c 00 4b 1f 20 d2 00 08 d4 30 c5 02 96 72 36 10 00 09 4d f5 ab 00 e8 13 07 5c 05 8f 8c 6f 34 00 a9 44 3c c2 01 64 65 ee 14 01 15 c4 51 00 00 f0 b5 4f 00 00 52 26 59 00 01 f8 f0 50 00 00 22 1a a5 02 01 8b ea 48 00 00 39 9e 4a 00 00 45 e8 40 00 00 3d 1c 46 00 00 7e a5 6b 24 00 40 ed 68 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e5 01 00 00 00 4a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 4d 57 20 00 00 b2 81 26 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a4 05 db 00 01 00 01 00 01 00 00 00 00 00 0b 00 71 02 48 00 01 00 00 00 00 00 0b 00 cb 00 00 00 02 00 72 00 04 00 1b 00 2c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 00 10 0f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 01 cf 00 0a ea 02 4f 00 00 00 00 00 05 00 04 00 0a 00 05 00 71 01 2a 00 00 00 00 00 04 00 1a 00 5e 00 00 00 04 00 37 00 07 00 14 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00 1a f3 80 d8 02 80 22 01 80 b9 00 80 d7 00 80 e0 01 80 10 05 00 00 db a7 81 00 2c 80 04 01 80 bf 03 38 01 00 00 00 00 00 01 00 00 02 00 00 2c 00 00 02 00 02 00 00 00 00 00 00 00 02 1e 00 12 09 15 17 05 08
2018/09/21 23:02:21 SOCKET: read 1 of 1 bytes from socket: ff
2018/09/21 23:02:21 SOCKET: read 1 of 1 bytes from socket: fe
2018/09/21 23:02:21 SOCKET: read 1 of 1 bytes from socket: 59
2018/09/21 23:02:21 adding buffer for 01000XXX
2018/09/21 23:02:21 buffering packet ts:1537585341 sn:01000XXX
2018/09/21 23:02:21 SOCKET: closing connection
2018/09/21 23:02:21 buffer info:
2018/09/21 23:02:21   01000XXX:   1 of 120 (1537585341)
2018/09/21 23:02:21 processing with RRDProcessor
2018/09/21 23:02:21 not enough data for 01000XXX
2018/09/21 23:02:21 SOCKET: waiting for connection
2018/09/21 23:02:31 waiting for data from device

That third last line is the one that suggests to me that something isn't right: "Not enough data for " and my serial number (which I'm not sure if it's really private, but I've obfuscated it hopefully everywhere, although I suppose it may be in that hex sequence.

Can anyone enlighten me as to what may be going on, and how to rectify it?

[Rocquestar]

Whoops. It may or may not be relevant, but I left out the part that followed the log file there.

After that one attempt, it performs another with slightly different output, then logs the same 7 lines of rra updates:

Code: Select all

2018/09/22 23:12:18 waiting for data from device
2018/09/22 23:12:18 reading 1 of 1 packets
2018/09/22 23:12:19 SOCKET: read 1 of 1 bytes from socket: fe
2018/09/22 23:12:19 SOCKET: read 1 of 1 bytes from socket: ff
2018/09/22 23:12:19 SOCKET: read 1 of 1 bytes from socket: 05
2018/09/22 23:12:19 SOCKET: read 619 of 619 bytes from socket: 04 c9 b8 f7 6c c3 51 33 01 c2 c9 61 ea 5c 26 6d 39 38 46 e2 7f 6b 56 fe 2c 6a 64 6e b4 2d 2f 70 55 02 45 e4 02 28 e8 00 97 00 0a 10 e5 d1 01 7e 7e e7 a2 01 a2 3f a3 a4 00 8f bb ec f9 02 6b ec 48 9c 00 14 c0 3b d2 00 07 e5 a2 c5 02 b2 29 38 10 00 2b 4f 0a ac 00 60 80 ca 5c 05 51 2a 76 34 00 be a7 36 c3 01 64 d0 2a 15 01 15 c4 51 00 00 f0 b5 4f 00 00 52 26 59 00 01 f8 f0 50 00 00 22 1a a5 02 01 8b ea 48 00 00 39 9e 4a 00 00 45 e8 40 00 00 3d 1c 46 00 00 26 8d 96 24 00 87 23 76 06 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e5 01 00 00 00 62 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 3f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 4d 57 20 00 00 b2 81 26 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ee 03 d6 00 01 00 01 00 00 00 00 00 00 00 0b 00 02 00 47 00 00 00 7c 00 00 00 01 00 be 00 00 00 00 00 75 00 04 00 21 00 2c 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 00 10 0f 0e 0d 0c 0b 0a 09 08 07 06 05 04 03 02 01 01 cf 00 0a e2 01 4c 00 00 00 00 00 04 00 04 00 0b 00 05 00 04 00 29 00 00 00 38 00 04 00 00 00 56 00 00 00 05 00 38 00 08 00 16 00 1f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 07 00 00 1a f3 80 d8 02 80 22 01 80 b9 00 80 d7 00 80 e0 01 80 10 05 00 00 70 27 81 02 2b 80 dd 00 80 7b 56 39 01 00 00 00 00 00 01 00 00 02 00 00 29 00 00 02 00 02 00 00 00 00 00 00 00 02 17 00 12 09 16 17 0b 19
2018/09/22 23:12:19 SOCKET: read 1 of 1 bytes from socket: ff
2018/09/22 23:12:19 SOCKET: read 1 of 1 bytes from socket: fe
2018/09/22 23:12:19 SOCKET: read 1 of 1 bytes from socket: fc
2018/09/22 23:12:19 buffering packet ts:1537672339 sn:01000XXX
2018/09/22 23:12:19 SOCKET: closing connection
2018/09/22 23:12:19 buffer info:
2018/09/22 23:12:19   01000XXX:   2 of 120 (1537672339)
2018/09/22 23:12:19 processing with RRDProcessor
2018/09/22 23:12:19 2 buffered packets sn:01000XXX
2018/09/22 23:12:19 1 calculated packets sn:01000XXX
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch26_pws.rrd with 1 values
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch17_aws.rrd with 1 values
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch6_aws.rrd with 1 values
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch25_pws.rrd with 1 values
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch31_pws.rrd with 1 values
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch29_aws.rrd with 1 values
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch3_aws.rrd with 1 values

Then, nothing more. When I check, btmon isn't running anymore.

I'm at a loss...

[ben]

What's the GEM COM/ENG F/W versions?

Is the GEM AC Adapter one of the bad ones as per the notice here: http://www.brultech.com/contact?

Have you checked your CT/PT Settings (GEM Setup Webpage, Channel Option for PT, Chan CT for CTs)?

If the PT settings have been altered (the bad AC adapters can cause memory corruption), your voltage may be off.

2018/09/21 23:02:21 01000XXX: 1 of 120 (1537585341)
2018/09/21 23:02:21 processing with RRDProcessor
2018/09/21 23:02:21 not enough data for 01000XXX

My guess here is there's only 1 packet processed so far. You need a 2nd packet to perform calculations.

[Rocquestar]

Hi Ben-

Ack- I wonder if this (and another similar problem that I've been seeing for a long while) is related to taht notice, which I hadn't seen before now. I did buy my GEM well before 2015 I think, so it could be. I'll check the label, and try a USB power supply to see if that rectifies it.

Anyway, the versions are 4.33 (COM) and 1.49 (ENG)

I haven't checked the CT or PT settings, as I had no reason to think that they might have changed, but if there's been a possible memory corruption, then it's worth looking at.

2018/09/21 23:02:21 01000XXX: 1 of 120 (1537585341)
2018/09/21 23:02:21 processing with RRDProcessor
2018/09/21 23:02:21 not enough data for 01000XXX

My guess here is there's only 1 packet processed so far. You need a 2nd packet to perform calculations.

It gets a second packet before failing, shown here, just before the end of the log:

Code: Select all

2018/09/22 23:12:19   01000XXX:   2 of 120 (1537672339)
2018/09/22 23:12:19 processing with RRDProcessor
2018/09/22 23:12:19 2 buffered packets sn:01000XXX
2018/09/22 23:12:19 1 calculated packets sn:01000XXX
2018/09/22 23:12:19 RRD: updating /usr/local/www/apache22/data/cacti/rra/01000XXX_ch26_pws.rrd with 1 values
<... repeats ~7 times>

Anyway, let me check the AC adaptor (ter?) and see if that's it.

I appreciate it. Will update when I know...

[Rocquestar]

Well, that's one thing.

I had the bad adaptor [sic]. I'm now running on USB power, but btmon is still crashing on receiving some data, or trying to send it to the rra.

I did look at the CT and PT settings, and as best as I could figure, put them to the way I thought they should be (there were two corrupt entries, with '?' where the CT-size value should have been.

Still, something is crashing btmon, but I'm too tired tonight to keep looking. Will hopefully hit it again tomorrow.

[ben]

Try wrapping line 3036 of btmon.py with

Code: Select all

try:
    rc = rrdtool.update(fn, values)
    if rc:
        wrnmsg("RRD: failed to update '%s': %d" % (fn, rc))
except:
    traceback.print_exc()
    raise

[Rocquestar]

Try wrapping line 3036 of btmon.py with

Code: Select all

try:
    rc = rrdtool.update(fn, values)
    if rc:
        wrnmsg("RRD: failed to update '%s': %d" % (fn, rc))
except:
    traceback.print_exc()
    raise

That doesn't seem to have made any difference to the output, I guess it's not getting to that part of the code prior to the crash.

FWIW (Ben, the location of your debugging code suggests that you already know this) but the problem is encountered in the RRD Processor, as btmon runs as expected and continuously in --print mode.

[ben]

Can you post the contents of:

/usr/local/www/apache22/data/cacti/rra

[Rocquestar]

Can you post the contents of:

/usr/local/www/apache22/data/cacti/rra

Yup, provided you mean a listing of it, and not to attach the files. (or did you?)
What are you looking for?

I didn't hide the SN here, I suspect that's not a great violation of my privacy.

Code: Select all

mercury:/usr/local/www/apache22/data/cacti/rra> ls -l
total 158996
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch10_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch10_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch11_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch11_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch12_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch12_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch13_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch13_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch14_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch14_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch15_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch15_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch16_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch16_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 25 21:56 01000463_ch17_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch17_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch18_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch18_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch19_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch19_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch1_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch1_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch20_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch20_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch21_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch21_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch22_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch22_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch23_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch23_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch24_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch24_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch25_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 25 21:56 01000463_ch25_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch26_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 25 21:56 01000463_ch26_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch27_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch27_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch28_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch28_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 25 21:56 01000463_ch29_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch29_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch2_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch2_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch30_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch30_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch31_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 25 21:56 01000463_ch31_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch32_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch32_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 25 21:56 01000463_ch3_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch3_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch4_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch4_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch5_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch5_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 25 21:56 01000463_ch6_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch6_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch7_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch7_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch8_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch8_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch9_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch9_pws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_p1.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_p2.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_p3.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_p4.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t1.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t2.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t3.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t4.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t5.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t6.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t7.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_t8.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_volts.rrd
-rw-r--r--  1 rocquestar  cacti    68320 Nov  9  2017 junk.rrd
mercury:/usr/local/www/apache22/data/cacti/rra> 

[ben]

Can you post the contents of:

/usr/local/www/apache22/data/cacti/rra

Yup, provided you mean a listing of it, and not to attach the files. (or did you?)
What are you looking for?

I didn't hide the SN here, I suspect that's not a great violation of my privacy.

Code: Select all

mercury:/usr/local/www/apache22/data/cacti/rra> ls -l
total 158996
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch10_aws.rrd
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_ch10_pws.rrd
....
-rw-r--r--  1 rocquestar  wheel  2087776 Sep 19 19:25 01000463_volts.rrd
-rw-r--r--  1 rocquestar  cacti    68320 Nov  9  2017 junk.rrd
mercury:/usr/local/www/apache22/data/cacti/rra> 

I'm curious what would happen if you moved the rrd files and let btmon.py recreate them. Can you try that?