GEM & Raspberry Pi: getting started?

[cramer]

Hi All,

I'm working on trying to use a raspberry pi to get data from a GEM. I know Brultech's DashBox is now available but I started this project months ago and want to see if I can make it work. I'm trying to monitor energy use in a home, as well as integrate OneWire temp & RH sensors and a water meter.

So I've got the GEM hooked up with an FTDI serial-usb adapter, and it seems to work fine when connected to a windows machine and using Brultech's network tool.

On the RPi side I'm totally lost. I can run btmon, but I've no idea what to do with it. For now I'd be happy if it wrote the data to a csv file that I physically retrieved once a month. MySQL, RRD, Cosm, and all those other services are either out of my a) price range or b) programming ability. I don't even know if I'm using the right port (/dev/ttyUSB0?). But lsusb does recognize the FTDI adapter, so at least that's working. But this is where my working knowledge of this stuff, linux included, ends (I'm a mechanical engineer, with only a smattering of programming experience).

I've read through Matt Wall's guide on setting up btmon but it's almost all over my head. And I find the Brultech documentation a bit sparse for someone like me starting from very little programming knowledge. Obviously this device is very versatile, but all I want is to the get the data from the GEM and put into a format I can read in Excel (ie csv) where I know what I'm doing. Can anyone out there point me towards a more step-by-step guide, or just example files, on how to get the GEM and RPi to talk to each other, for someone starting from scratch?

I'm fully prepared to give up and buy the DashBox but I'd really like to see if can get this to work.

Thanks,
Cramer

[depasseg]

Can you get btmon.py to display any data when testing on RPi?

My understanding of it (based on reading; I haven't ever used it) is that btmon will listen for packets from the GEM, and then do 1 or more things with it. You can:

Collect data from Brultech ECM-1240, ECM-1220, and GEM power monitors. Print
the data, save the data to database, or upload the data to a server.

Includes support for uploading to the following services:
* MyEnerSave/Bidgely * SmartEnergyGroups * pachube/cosm * WattzOn
* PlotWatt * PeoplePower * thingspeak * Eragy
* emoncms

from http://lancet.mit.edu/mwall/projects/po ... n/btmon.py

Saving to a CSV isn't one of those options. The closest would be "Print the data". Print isn't necessarily to paper. You could print it to the screen or to a file. But my guess is that the file would look like this over and over down the page (meaning not in a table):

Example Usage:

btmon.py --serial --serialport=/dev/ttyUSB0 -p

Example Output:

2010/06/07 21:48:37: ECM ID: 355555
2010/06/07 21:48:37: Volts: 120.90V
2010/06/07 21:48:37: Ch1 Amps: 0.25A
2010/06/07 21:48:37: Ch2 Amps: 3.24A
2010/06/07 21:48:37: Ch1 Watts: -124.586KWh ( 1536W)
2010/06/07 21:48:37: Ch1 Positive Watts: 210.859KWh ( 1536W)
2010/06/07 21:48:37: Ch1 Negative Watts: 335.445KWh ( 0W)
2010/06/07 21:48:37: Ch2 Watts: -503.171KWh ( 0W)
2010/06/07 21:48:37: Ch2 Positive Watts: 0.028KWh ( 0W)
2010/06/07 21:48:37: Ch2 Negative Watts: 503.199KWh ( 0W)
2010/06/07 21:48:37: Aux1 Watts: 114.854KWh ( 311W)
2010/06/07 21:48:37: Aux2 Watts: 80.328KWh ( 523W)
2010/06/07 21:48:37: Aux3 Watts: 13.014KWh ( 35W)
2010/06/07 21:48:37: Aux4 Watts: 4.850KWh ( 0W)
2010/06/07 21:48:37: Aux5 Watts: 25.523KWh ( 137W)

So then you could play around with SED/AWL/Grep and a bunch of other tools to modify that text into a table.

Or you could play with RRD. RRD is a file format used by MRTG and now Cacti to graph. Both are free and I'm sure are available for RPi.

Or you could dump it to a database, and something like myswl workbench to access it from your windows machine. You might be able to export the data that way.

Or you could forward the data to somewhere like SmartEnergyGroups.com and have your data graphed there for free.

[Peter_V]

I'd recommend RRDtool.

While many people use Cacti to run RRDtool, you don't need it. Which means you also don't need MySQL. It always seemed a little silly to me to use MySQL just to store config data for RRDtool.

RRDtool is a nice little program and the database files are reasonably small (depending on how much data you store). It's fairly simple to install if you aren't using Cacti and MySql.

Not sure which Linux distro you are using, but if it's something like Raspbian then you can install rrdtool with:

apt-get update
apt-get install rrdtool

To get btmon to log to rrd add the following to the config.cfg file for btmon:

[rrd]
rrd_out = true
rrd_dir = /var/rrd

This assumes you want the data stored in a directory called /var/rrd
RRDTool normally exports data as XML files, I'm pretty sure Excel can import XML data.

I wrote a bunch of perl scripts to use rrdtool to create graphs like this:


I also use rrdtool for logging weather data:


I can share the scripts with you if you're interseted. However, they will probably need to be modified since I use an ECM-1240 rather than a GEM.

[ben]

Another option is using EmonCMS (http://emoncms.org/) and btmon.py together.

The instructions here: http://emoncms.org/site/docs/installlinux will get EmonCMS installed.

Then you want to do something like:

[source]
device_type = gem
serial_read = true
serial_port = /dev/ttyUSB0
[oem]
oem_out = false
oem_url=http://RPiIP/input/post
oem_token = XXXXXX

In your config.cfg file.

Ben

[cramer]

Thanks for the tips, all. emoncms looks pretty nice. Got it working, apparently, but still no evidence of life from the GEM

Ben, where do I find the oem_token for the config file?

So I'm still not sure I have basic communication functioning. The GEM is not hooked up to any CTs yet, but I should get some basic connectivity, right? All channels=0, etc?

Attached is what I get when I run btmon.

Any ideas what to do next?

[mwall]

1) log in to your emoncms server

2) click the 'feeds' link

3) click the 'input api help' link

your token will be there. the help page also links URLs that you can try to verify that your emoncms installation is working properly.

[mwall]

So I'm still not sure I have basic communication functioning. The GEM is not hooked up to any CTs yet, but I should get some basic connectivity, right? All channels=0, etc?

use the --verbose and --debug options for more feedback about what btmon is doing, at least until you ensure that everything is working properly.

[cramer]

Matt,

This is what I'm getting:

"read failed: expected START_HEADER0 0xfe, got 0x6c"

With the hex number at the end changing a bit - 0xf9, 0xff, 0xfd///

Any ideas?

Thanks,
Cramer

[ben]

Matt,

This is what I'm getting:

"read failed: expected START_HEADER0 0xfe, got 0x6c"

With the hex number at the end changing a bit - 0xf9, 0xff, 0xfd///

Any ideas?

Thanks,
Cramer

What packet format are you set to? btmon.py uses Bin-48-NET-Time if I remember correctly.

Ben

[cramer]

Ben,

GEM48PTBinaryPacket is what it comes up with. Does that need to be set to something else with a command line option?