BTMon.py - problem creating rrd database?

[Rocquestar]

I'm experiencing some issues with the creation of the rrd from within btmon. I'm not much of a python coder, so debugging isn't really an issue for me, but I can't beleive that I'm the only one using it with rrd, so I suspect this is a config issue, not a problem with the code - although, I understand that the rrd functionality is somewhat new (last fall?)

I read in another post that it's as easy to try to create a new rrd database and fail gracefully, than to check for it's existence and conditionally create it, so I took from that that there isn't a 'one-time initialization step' of an rrd database that I need to do before starting. Also, the error I'm getting isn't 'no database', rather it's failing during a create stage:

RRD: creating rrd file /var/btmon/rrd/01000463_volts.rrd
Exception in RRDProcessor: argument 5 must be string

Here's my btmon.cfg:

Code: Select all

>cat btmon.cfg
[source]
device_type = gem
ip_read = true
ip_host = 10.0.0.2
ip_port = 5000
ip_mode = client
[rrd]
rrd_out = true
rrd_dir = /var/btmon/rrd
>

and here's the debug output leading up to the error, when running against that config:

Code: Select all

>./btmon.py -c btmon.cfg --debug
2013/03/26 19:20:01 device type: gem
2013/03/26 19:20:01 device list: ['']
2013/03/26 19:20:01 packet format: gem48ptbin
2013/03/26 19:20:01 schema: counters
2013/03/26 19:20:01 buffer size: 120
2013/03/26 19:20:01 SOCKET: timeout: 15
2013/03/26 19:20:01 SOCKET: server host: 10.0.0.2
2013/03/26 19:20:01 SOCKET: server port: 5000
2013/03/26 19:20:01 RRD: update period: 60
2013/03/26 19:20:01 RRD: dir: /var/btmon/rrd
2013/03/26 19:20:01 RRD: step: 10
2013/03/26 19:20:01 RRD: heartbeat: 20
2013/03/26 19:20:01 packet format is GEM48PTBinaryPacket
2013/03/26 19:20:01 using collector BlockingSocketClientCollector
2013/03/26 19:20:01 using 1 processors:
2013/03/26 19:20:01   RRDProcessor
2013/03/26 19:20:01 setup BlockingSocketClientCollector
2013/03/26 19:20:01 opening socket connection to 10.0.0.2:5000
2013/03/26 19:20:01 setup RRDProcessor
2013/03/26 19:20:01 SOCKET: waiting for data
2013/03/26 19:20:01 reading 1 of 1 packets
2013/03/26 19:20:08 adding buffer for 01000463
2013/03/26 19:20:08 buffering packet ts:1364340008 sn:01000463
2013/03/26 19:20:08 buffer info:
2013/03/26 19:20:08   01000463:   1 of 120 (1364340008)
2013/03/26 19:20:08 processing with RRDProcessor
2013/03/26 19:20:08 not enough data for 01000463
2013/03/26 19:20:08 SOCKET: waiting for data
2013/03/26 19:20:08 reading 1 of 1 packets
2013/03/26 19:20:17 buffering packet ts:1364340017 sn:01000463
2013/03/26 19:20:17 buffer info:
2013/03/26 19:20:17   01000463:   2 of 120 (1364340017)
2013/03/26 19:20:17 processing with RRDProcessor
2013/03/26 19:20:17 2 buffered packets sn:01000463
2013/03/26 19:20:17 1 calculated packets sn:01000463
2013/03/26 19:20:17 RRD: creating rrd file /var/btmon/rrd/01000463_volts.rrd
2013/03/26 19:20:17 Exception in RRDProcessor: argument 5 must be string
Traceback (most recent call last):
  File "./btmon.py", line 1987, in process
    p.process_compiled(self.packet_collector.packet_buffer)
  File "./btmon.py", line 2557, in process_compiled
    self.process_calculated(packets)
  File "./btmon.py", line 2853, in process_calculated
    self._process_all(packets)
  File "./btmon.py", line 2862, in _process_all
    self._update_files(packets)
  File "./btmon.py", line 2835, in _update_files
    self._create_rrd(p, x, t)
  File "./btmon.py", line 2810, in _create_rrd
    "RRA:MIN:0.5:%d:%d" % (RRD_STEPS[3],RRD_RESOLUTIONS[3]))
TypeError: argument 5 must be string
2013/03/26 19:20:17 SOCKET: waiting for data
2013/03/26 19:20:17 reading 1 of 1 packets
^C2013/03/26 19:20:22 cleanup RRDProcessor
2013/03/26 19:20:22 cleanup BlockingSocketClientCollector
2013/03/26 19:20:22 closing socket connection
>

I checked that btmon is communicating with the GEM, and it is:

Code: Select all

>./btmon.py --device-type=gem --ip --ip-host=10.0.0.2 --ip-port=5000 --ip-mode=client --print 
2013/03/26 19:26:03 device type: gem
2013/03/26 19:26:03 device list: ['']
2013/03/26 19:26:03 packet format: gem48ptbin
2013/03/26 19:26:03 schema: counters
2013/03/26 19:26:03 SOCKET: timeout: 15
2013/03/26 19:26:03 SOCKET: server host: 10.0.0.2
2013/03/26 19:26:03 SOCKET: server port: 5000

2013/03/26 19:26:17: Serial: 01000463
2013/03/26 19:26:17: Voltage:  121.10V
2013/03/26 19:26:17: Ch01:   -220.718599KWh (-4437W)
2013/03/26 19:26:17: Ch02:    -86.934899KWh ( -478W)
2013/03/26 19:26:17: Ch03:    -15.590492KWh (   -3W)
2013/03/26 19:26:17: Ch04:    -25.306183KWh (-2211W)
2013/03/26 19:26:17: Ch05:     -0.176301KWh (   -7W)
2013/03/26 19:26:17: Ch06:     -0.019482KWh (    0W)
2013/03/26 19:26:17: Ch07:     -0.988829KWh ( -267W)
<snipped for brevity>

So it's getting data that looks reasonable. (Looking closer, it looks like I have the polarity reversed on all my CT's, but that's another issue.)

Any thoughts? Is this a configuration problem?

I'm running python 2.7.3 on FreeBSD, if it matters.

[mwall]

which version of btmon are you running? could you try 3.0.6b3?

http://lancet.mit.edu/mwall/projects/po ... .0.6-b3.py

btmon will create the rrd file if it does not already exist.

fwiw, i have been running btmon with gem and ecm devices for quite some time now without issues.

as for polarity, simply add '--reverse-polarity' to the command line invocation, or 'reverse_polarity=true' to the config file.

m

[Rocquestar]

Hey Matt- thanks for your reply. You wrote this script, yes?

reversed polarity - thanks.

I'm running v3.0.5, but I actually tried 3.0.6-b3, with the same results. (I ran a diff of the two, and grepped for 'rrd', which returned nothing, so I suspected that little or nothing in the rrd realm changed)

so...

I'm thinking that I'll manually create the rrd database (with rrdtool on the command line), but I think that it will ultimately be trying to create a different file for each channel (does it?) Once I could do, but not if it simply advances me to a problem creating the next db.

[Rocquestar]

Matt-

What version of rrdtool are you using? I'm on port 1.2.30_2

[mwall]

i forgot about the rrdtool version. i am using 1.4.3. i seem to recall having issues with 1.2 and perhaps 1.3 when i was experimenting with rrdtool and btmon on windows. btmon does not use any special rrdtool features, but if i recall correctly the bindings changed significantly between 1.2 and 1.4.

btmon creates a file for each channel, pulse counter, volts, current.

you might be able to hack _create_rrd so that it uses 1.2 binding syntax. if so, please let me know so i can incorporate it into btmon.

m

[mwall]

btw, one of the things i intend to change is the data store name in the rrd files (in the RRA definitions). currently it is the channel/volt/current label, but a generic 'data' name will be much easier to use. the channel/volt/current identifier is already in the rrd filename, so putting it in the ds identifier is redundant. and when you try to write a generic plotter using tools such as javascriptrrd it just gets in the way.

[Rocquestar]

Right after I sent that message, I realized how old my rrdtool was. It's updating now. I fully expect this will resolve the issue; I'll post when I know.

EDIT: That was it. Collecting data now.

[Peter_V]

btw, one of the things i intend to change is the data store name in the rrd files (in the RRA definitions). currently it is the channel/volt/current label, but a generic 'data' name will be much easier to use. the channel/volt/current identifier is already in the rrd filename, so putting it in the ds identifier is redundant. and when you try to write a generic plotter using tools such as javascriptrrd it just gets in the way.

I've been wondering about that. Why did you put each channel into a different RRD database? From what I understand each database can contain multiple DS, so couldn't you just use a single database for all the channels?

As it is when I'm doing my combined graphs I have to simultaneously open 8 different databases. I figure you must have had a reason, but it seems awkward

[mwall]

I've been wondering about that. Why did you put each channel into a different RRD database? From what I understand each database can contain multiple DS, so couldn't you just use a single database for all the channels?

peter,

putting 32+ channels into a single file makes for a pretty monstrous RRA definition.

the number of channels is fixed, but not all may be used. so my intent was to have one RRD file for each channel that actually contains data. disk space is cheap, but there are still embedded systems where storage is something of a premium (for operational reasons if not for cost).

single-file vs multiple-files is a persistent issue with RRD. we went through this with nagiosgraph, and there is no one-size-fits-all strategy when you are targeting both large, distributed, redundant systems as well as small, low-power, sd-based systems.

i would consider adding a single-file-for-RRD option to btmon, but it will take some thought and design work to get the RRA definition right.

m

[Peter_V]

I keep forgetting about the GEM, I was just thinking in relation to the ECM-1240.

Speaking of systems with limited resources...have you considered making the program modular? seems like an excellent candidate for a modular approach, then you could just import the modules you need.

I've been trying to go through the program and weed out all the stuff I don't use: the uploads to online systems, mysql, sqlite, ethernet packets (I use serial), GEM specific sections, etc. but I'm not familiar with python so it's going slowly.