btmon for python 3?

[ian351c]

4197 - 4207:

def _add_msg(self, packet, channel, payload):
if payload == None:
return
key = mklabel(packet['serial'], channel)
if key in self.map:
key = self.map[key]
self._msgs.append({'topic': '%s/%s' % (self.base_topic, key),
'payload': round(payload, 3),
'qos': self.qos,
'retain': self.retain})

[ben]

I've got the btmon script for python3 up and running, with one issue: all of my Watts measurements are 0. The Watt Hours measurements seem to come in fine. What information would be useful for troubleshooting?

EDIT:

So, I've done some more sleuthing and it really looks like btmon is not sending the Watts measurements to InfluxDB.
I see the measurements coming from btmon fine:
2021/05/13 13:40:51: Voltage: 119.20V
2021/05/13 13:40:51: Ch01: 81972.869791KWh ( 1070W)
Note that Channel 1 is my entire panel, so there is always some load there.

But Watts are not getting sent to InfluxDB (this is from a packet capture):
energy,gemlocation=main_panel,host=gem.home XXX338_volts=119.4 1620926627000000000
energy,gemlocation=main_panel,host=gem.home XXX338_ch1_aws=295101134484.0 1620926627000000000
energy,gemlocation=main_panel,host=gem.home XXX338_ch1_pws=0.0 1620926627000000000

END EDIT:

My Config:
Old Monitoring Setup (works fine)
btmon 3.1.1
2021/05/13 12:04:49 python: 2.7.18 (default, Feb 25 2021, 01:19:10)
[GCC FreeBSD Clang 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-
2021/05/13 12:04:49 platform: freebsd12
2021/05/13 12:04:49 polarity is reversed
2021/05/13 12:04:49 device type: gem
2021/05/13 12:04:49 device list: ['']
2021/05/13 12:04:49 packet format: gem48ptbin
2021/05/13 12:04:49 schema: counters
2021/05/13 12:04:49 SOCKET: timeout: 15
2021/05/13 12:04:49 SOCKET: server host: 172.16.100.137
2021/05/13 12:04:49 SOCKET: server port: 8000
2021/05/13 12:04:49 OEM: upload period: 1
2021/05/13 12:04:49 OEM: timeout: 15
2021/05/13 12:04:49 OEM: url: http://localhost/emoncms/input/post.json
2021/05/13 12:04:49 OEM: token: xxxx
2021/05/13 12:04:49 OEM: node: 20

New Monitoring Setup (works, but all Watts measurements are 0):
2021/05/13 12:03:35 btmon: 4.0.0
2021/05/13 12:03:35 python: 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0]
2021/05/13 12:03:35 platform: linux
2021/05/13 12:03:35 polarity is reversed
2021/05/13 12:03:35 device type: gem
2021/05/13 12:03:35 device list: ['']
2021/05/13 12:03:35 packet format: gem48ptbin
2021/05/13 12:03:35 schema: counters
2021/05/13 12:03:35 SOCKET: timeout: 60
2021/05/13 12:03:35 SOCKET: server host: 172.16.100.137
2021/05/13 12:03:35 SOCKET: server port: 8000
2021/05/13 12:03:35 InfluxDB: upload period: 10
2021/05/13 12:03:35 InfluxDB: host: localhost
2021/05/13 12:03:35 InfluxDB: port: 8086
2021/05/13 12:03:35 InfluxDB: username: btmon
2021/05/13 12:03:35 InfluxDB: map:
2021/05/13 12:03:35 InfluxDB: schema: counters

Config for new setup:
[source]
device_type = gem
ip_read = true
ip_host = 172.16.100.137
ip_port = 8000
ip_mode = client
reverse_polarity = 1
[influxdb]
influxdb_out = true
influxdb_host = localhost
influxdb_port = 8086
influxdb_upload_period = 10
influxdb_username = xxxxx
influxdb_password = xxxxx
influxdb_database = btmon_greeneye
influxdb_measurement = energy
influxdb_mode = row
influxdb_tags = host,gem.home,gemlocation,main_panel
influxdb_db_schema = counters

Change your Influxdb schema from counters to ecmreadext. Counters just logs the wattseconds counters.

[ian351c]

Change your Influxdb schema from counters to ecmreadext. Counters just logs the wattseconds counters.

Thanks Ben!

If I do that, I get the following error:

Exception in InfluxDBProcessor: 'ch1_a'
Traceback (most recent call last):
File "btmon3.py", line 2249, in process
p.process_compiled(self.packet_collector.packet_buffer)
File "btmon3.py", line 2838, in process_compiled
self.process_calculated(packets)
File "btmon3.py", line 4299, in process_calculated
values['fields'][value_name] = p[c] * 1.0
KeyError: 'ch1_a'

BTW: In reading how the watts are calculated in btmon, it's not strictly necessary to have the watts data in the database since InfluxDB has a "derivative" function that will derive watts from watt-seconds. So I've got a work around for now. However, it would be good to either the watts values working or to just not create the watt fields in InfluxDB in the first place.

[ben]

Change your Influxdb schema from counters to ecmreadext. Counters just logs the wattseconds counters.

Thanks Ben!

If I do that, I get the following error:

Exception in InfluxDBProcessor: 'ch1_a'
Traceback (most recent call last):
File "btmon3.py", line 2249, in process
p.process_compiled(self.packet_collector.packet_buffer)
File "btmon3.py", line 2838, in process_compiled
self.process_calculated(packets)
File "btmon3.py", line 4299, in process_calculated
values['fields'][value_name] = p[c] * 1.0
KeyError: 'ch1_a'

BTW: In reading how the watts are calculated in btmon, it's not strictly necessary to have the watts data in the database since InfluxDB has a "derivative" function that will derive watts from watt-seconds. So I've got a work around for now. However, it would be good to either the watts values working or to just not create the watt fields in InfluxDB in the first place.

Are you running this copy?

https://github.com/BenK22/mtools/blob/i ... /btmon3.py

[ian351c]

Yep. That's the one.

[ben]

Yep. That's the one.

I think you need "include_current = true" in your source setup.

[wci68]

Apologies for the necropsy on this thread, but I thought this better than creating a new thread for the same topic.

I updated my linux-based OS running btmon which no longer includes python2.x. I had been putting off the OS upgrade largely for this reason but it has come time. I dropped in the version of btmon3.py from Ben's influxdb branch (https://github.com/BenK22/mtools/blob/i ... /btmon3.py) and it mostly works, but I do have one issue that is common for python2 -> python3 migrations:

Code: Select all

Dec 26 17:51:06 rpi3p btmon3.py[839288]: 2023/12/26 17:50:54 OpenEnergyMonitorProcessor Error: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

This should be an easy fix similar to other legacy-str related issues noted earlier in this thread. Before I jump in to look for this, has this one been seen yet? I do use OEM to feed a local instance of emoncms and I have not yet moved to influxdb yet so I'd like to get this one fixed. I'm a little surprised I may be the only one using OpenEnergyMonitor configuration though.

[ben]

Apologies for the necropsy on this thread, but I thought this better than creating a new thread for the same topic.

I updated my linux-based OS running btmon which no longer includes python2.x. I had been putting off the OS upgrade largely for this reason but it has come time. I dropped in the version of btmon3.py from Ben's influxdb branch (https://github.com/BenK22/mtools/blob/i ... /btmon3.py) and it mostly works, but I do have one issue that is common for python2 -> python3 migrations:

Code: Select all

Dec 26 17:51:06 rpi3p btmon3.py[839288]: 2023/12/26 17:50:54 OpenEnergyMonitorProcessor Error: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

This should be an easy fix similar to other legacy-str related issues noted earlier in this thread. Before I jump in to look for this, has this one been seen yet? I do use OEM to feed a local instance of emoncms and I have not yet moved to influxdb yet so I'd like to get this one fixed. I'm a little surprised I may be the only one using OpenEnergyMonitor configuration though.

Can you try switching Line 3959 from

Code: Select all

result = self._urlopen(url, '')

to

Code: Select all

result = self._urlopen(url, None)

[wci68]

Thanks Ben!

Unfortunately using None didn't work:

Code: Select all

Dec 28 19:27:42 rpi3p btmon3.py[1260959]: 2023/12/28 19:26:18 OpenEnergyMonitorProcessor Error: object of type 'NoneType' has no len()

However encoding a null string to a utf-8 bytestr worked instead:

Code: Select all

 result = self._urlopen(url, ''.encode('utf-8'))

My local emoncms instance appears to be getting fed correctly now, but if I find any other problems I'll post back here.

[ben]

Thanks Ben!

Unfortunately using None didn't work:

Code: Select all

Dec 28 19:27:42 rpi3p btmon3.py[1260959]: 2023/12/28 19:26:18 OpenEnergyMonitorProcessor Error: object of type 'NoneType' has no len()

However encoding a null string to a utf-8 bytestr worked instead:

Code: Select all

 result = self._urlopen(url, ''.encode('utf-8'))

My local emoncms instance appears to be getting fed correctly now, but if I find any other problems I'll post back here.

Got it, updated the script.