Setting up GEM + btmon.py on raspbery pi

Working on a home project? Post it here.
Post Reply
scyto
Posts: 258
Joined: Sun Apr 06, 2014 7:24 pm

Setting up GEM + btmon.py on raspbery pi

Post by scyto » Thu Mar 19, 2015 10:07 pm

Teken challenged me to document some of this, so here goes. I will add more details as folks need to each step as and when folks need it, I will assume you know how to use SSH, I used this SSH client http://www.bitvise.com/ssh-client-download as I liked the SFTP, terminal and RDP integration and its free.
  1. copy fresh raspbian image (http://www.raspberrypi.org/downloads/) on to SD card, insert in pi and boot (use this tool to copy to SD card http://sourceforge.net/projects/win32diskimager/)
  2. connect with SSH (default username is pi and the password is raspberry
  3. run 'sudo raspi-config' and choose to expand file system
  4. run 'sudo reboot'
  5. log back in with SSH
  6. run 'sudo apt-get update' and then 'sudo apt-get upgrade'
  7. run 'sudo reboot'
  8. log back in with SSH
  9. python is installed by default - so need to worry about that
  10. run mkdir btmon
  11. copy over the btmon.py file into the btmon directory (download latest from http://lancet.mit.edu/mwall/projects/power/)

    personally I use the bitvise SSH client on windows that comes with an easy to use SFTP GUI function that I use to make directories, copy files and rename them
  12. rename file to btmon.version.py - mine is currently btmon.3.1.1.py
Ok that's the basics over, now to configure the config.cfg file for the services you want to access. This requires that you register on certain websites to get api keys etc. The details can be found in btmon.py so it is worth reading that to get the details. I will try and cover most of it here, I may have to keep coming back to edit post as I can't do it all in one sitting.
  1. CD into the directory where you copied the renamed btmon script and run 'nano config.cfg' this starts the text editor. Use the cursor keys and ctrl+ letter combinations to navigate
  2. the first section should look something like this:

    [source]
    device_type = gem
    ip_read = true
    ip_host = 192.168.1.100
    ip_port = 8000
    ip_mode = client
    reverse_polarity = 1
  3. this connects to the ethernet adapter of a GEM that shared com1. the IP address is the the IP address the GEM got issued by your DHCP server, router, or that you manually assigned. The port of 8000 is the default, if you changed it you will need to change this, the reverse polarity was need in my system or some of the sites registered negative value YMMV.
  4. next you need to configure the services you want to connect to, you will need a variety of information to configure these, where to go and how to register is all covered in the btmon.py script up front, so read it. If you have question I will try and answer.

    Anything in <> brackets is a parameter you get from registering on the service in question and is unique to you and your installation.

    For the GEM you will see xx replacing my SN digits - use you right sn

    [openenergymonitor]
    oem_url = http://emoncms.org/input/post.json
    oem_out = true
    oem_token = <token>
    oem_node = 1

    [bidgely]
    bidgely_out = true
    by_url = https://api.bidgely.com/v1/users/<token ... s/1/upload
    by_map = 010000xx_ch1,mains,0,

    [plotwatt]
    plotwatt_out = true
    pw_house_id = <housed>
    pw_api_key = <apikey>
    pw_map=010000xx_ch1,<meterid1>,01000029_ch3,<meterid2>

    [thingspeak]
    thingspeak_out = true
    ts_tokens = 010000xx,<token>
    ts_fields = 010000xx_ch1,1

    [wattvision]
    wattvision_out = true
    wv_api_id = <apid>
    wv_api_key = <apikey>
    wv_sensor_id = <sensorid>
    wv_channel = 010000xx_ch1
  5. open energy monitor - this is the one that for me showed the negative values that caused me to reverse polarity so be warned, by default the configuration load all 32 channels
  6. bidgely - this is my favourite that I am using with btmon.py I am only loading my main channel. the main and 0 are to tell it this. I am waiting to see how well their API does disaggregation before I think about uploading the rest of my channels.
  7. plotwatt is hard to setup due to the need to use curl to create the meters, however curl is installed on pi by default so if you follow the instructions in the script notes you should have no issues. while I am uploading 2 channels as two meters (my main and a load, I am not sure there is any need to do more than just the main ch1 (or ch1 and ch2 if you are monitoring the different sides of your supply separately).
  8. thingspeak and wattvision I am still figuring out
Once you have configured you config.cfg (and I recommend starting with just one service and get that working before you configure the others - so only put the sections in you need or use the _out=false to turn the sections off) you to run and test.
  1. from within your btmon directory run 'python ./btmon.3.1.1.py -c config.cfg

    this will let you see the console log output and you will get errors if you did something wrong

    if you want this to run at startup then:
  2. sudo nano /etc/rc.local
  3. in the file add this command anywhere above the exit 0 command

    #run btmon.py to send data to services
    python /home/pi/btmon/btmon.3.1.1.py -c /home/pi/btmon/config.cfg > /dev/null 2>&1 &
hopefully this is enough to bootstrap someone else, it took me a while to figure out what the folks in the btmon.py threads were talking about and I couldn't find a guide that started from getting the Linux OS download. I have to thank io_guy on the UD forums who hand held me through my first pi install and has a great guide on this and installing his dsclink app. You may find that helpful if you have not used SSH http://automationshack.com/Files/Raspbian_Setup.pdf

My system has been uploading data to all of these services ~24hours at this point.
Carl
Posts: 9
Joined: Fri Oct 21, 2011 12:45 pm

Re: Setting up GEM + btmon.py on raspbery pi

Post by Carl » Fri Feb 09, 2018 8:55 am

I appreciate the detailed post from 2+ years ago. Could you also please go into detail about how the GEM is configured so BTMON.py works with it?

Questions I would have is:
What COM port are you using for the btmon communication?
What baud rate for the com port?
What packet type?
Real Time status (On or Off)
Server or Client (I think Server based upon btmon config file)
Any specific settings on the "Data Post" tab.
What settings do you have on the Network page under "Remote Host"?
Anything else I need to configure on GEM?

Thanks,
Carl
ben
Site Admin
Posts: 4254
Joined: Fri Jun 04, 2010 9:39 am

Re: Setting up GEM + btmon.py on raspbery pi

Post by ben » Fri Feb 09, 2018 11:11 am

Carl wrote: Questions I would have is:
What COM port are you using for the btmon communication?
What baud rate for the com port?
It looks scyto is using Ethernet or WiFi/Ethernet to connect, they aren't using Serial so COM and Baud aren't needed.
Carl wrote:What packet type?
Bin-48-NET-Time
Carl wrote:Real Time status (On or Off)
On.
Carl wrote:Server or Client (I think Server based upon btmon config file)
I've used Client in my tests.

Client -> BTMon.py will connect to the GEM at the specified IP Address/Port.
Server -> The GEM will have to connect to BTmon.py, you'll have to configure the Network page on the GEM for this.
Carl wrote:Any specific settings on the "Data Post" tab.
Nothing needed.
Carl wrote:What settings do you have on the Network page under "Remote Host"?
Nothing should be needed with the above config.

If you're running this on a Windows system, I've packed up my test setup here:

http://brultech.com/community/viewtopic ... 786#p11786

The included Grafana and InfluxDB may be a bit out of date at this point. It uses a version of BTMon.py I've forked to include watt and kwh calculations for the GEM.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Carl
Posts: 9
Joined: Fri Oct 21, 2011 12:45 pm

Re: Setting up GEM + btmon.py on raspbery pi

Post by Carl » Fri Feb 09, 2018 2:48 pm

Thanks Ben,

How do I know if the GEM is in Client or Server mode and what port The GEM is listening for connections on?

Thanks,
Carl
ben
Site Admin
Posts: 4254
Joined: Fri Jun 04, 2010 9:39 am

Re: Setting up GEM + btmon.py on raspbery pi

Post by ben » Mon Feb 12, 2018 12:03 pm

Carl wrote:Thanks Ben,

How do I know if the GEM is in Client or Server mode and what port The GEM is listening for connections on?

Thanks,
Carl
With the WiFi/Ethernet module, the default configuration is it's always running in Server mode and you use the Network tab in the GEM Setup Webpage to setup a Client connection.

In terms of btmon:
If the GEM connects to btmon, the GEM is considered to be in Client mode.
If BTMon connects to the GEM, the GEM is considered to be in Server mode.

The reverse setting in BTMon is:
BTMon.py will connect to the GEM at the specified IP Address/Port when in Client mode.
The GEM will have to connect to BTmon.py when in Server mode.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
Post Reply