A good way to store data for WiFi GEM

Ask questions about the GEM here.
Post Reply
ecosang
Posts: 8
Joined: Thu Jun 15, 2017 8:51 am

A good way to store data for WiFi GEM

Post by ecosang » Thu Jun 22, 2017 2:39 am

Hello, Ben.
I really appreciate your help and support for this product.
Currently, I installed one GEM (wifi), and collect data using btmon.py.
I am not quite confident, but I thought the GEM is server, so I used following script for btmon.py
"btmon.py --device-type=gem --ip --ip-mode=client --ip-host=192.168.0.16 --ip-port=8000 --print".
My question is how this works (I know there is source code btmon.py, but I have a hard time to interpret this).
In my understanding, the GEM sends (emit) data by a certain interval, and my computer(btmon.py) is waiting for the data and store it when the data is sent (i am using sqlite). Is this correct? If so, what is the difference between GEM as a server and a client?

I think I will install 17~20 GEM in a building. In the building, there are 50 units(households). We will only monitor 17-20 households out of 50. The building has WiFi network, but it doesn't have wired internet for each unit (household). So, I think GEM with WiFi is the solution for this site.
Another question is what is a good way to get the data in this case?
I think using 17-20 btmon.py from one computer is not a good solution.
I think the convenient way is to collect data using HTTP GET method from the server computer. Many products provides REST API (with OAUTH 2.0), and I can use HTTP GET in this situation, but I am not sure how I can do this with GEM.

Sorry for long question. I really appreciate your help.
Thanks
Sang woo Ham
ben
Site Admin
Posts: 4254
Joined: Fri Jun 04, 2010 9:39 am

Re: A good way to store data for WiFi GEM

Post by ben » Thu Jun 22, 2017 9:41 am

ecosang wrote:Currently, I installed one GEM (wifi), and collect data using btmon.py.
I am not quite confident, but I thought the GEM is server, so I used following script for btmon.py
"btmon.py --device-type=gem --ip --ip-mode=client --ip-host=192.168.0.16 --ip-port=8000 --print".
My question is how this works (I know there is source code btmon.py, but I have a hard time to interpret this).
In my understanding, the GEM sends (emit) data by a certain interval, and my computer(btmon.py) is waiting for the data and store it when the data is sent (i am using sqlite). Is this correct? If so, what is the difference between GEM as a server and a client?
Correct.

If the GEM is set as a Server, the software would have to connect to it. Therefore, each GEM would require a static IP.

The GEM in Client mode connects to the software itself.

SQLite is likely a poor choice for what you want to do. You'll want to use something like MySQL when dealing with that many connections. We've used SQLite in the past and had some issues with concurrency, corruption, and size constraints.
ecosang wrote: I think I will install 17~20 GEM in a building. In the building, there are 50 units(households). We will only monitor 17-20 households out of 50. The building has WiFi network, but it doesn't have wired internet for each unit (household). So, I think GEM with WiFi is the solution for this site.
Another question is what is a good way to get the data in this case?
I think using 17-20 btmon.py from one computer is not a good solution.
I think the convenient way is to collect data using HTTP GET method from the server computer. Many products provides REST API (with OAUTH 2.0), and I can use HTTP GET in this situation, but I am not sure how I can do this with GEM.
It might not be too bad to run that many instances of btmon. You'd want some sort of watchdog in case of any issues with the scripts.

The GEM doesn't have OAUTH or anything like that, it's very limited in space due to trying to make it as cost effective as possible.

If you just want a simple data logger, you could use our Standalone Dashboard software, the MySQL version. http://brultech.com/software/files/down ... 1_PHP7.zip.

It supports data logging on the minute and hour.

It requires XAMPP to be installed, or similar web server (https://www.apachefriends.org/index.html).

Once XAMPP is installed, delete everything from the htdocs folder in the installation path and extract the zip contents there.

Bring up phpmyadmin (http://localhost/phpmyadmin), go to Import in the top menu, and select the .sql file located in your XAMPP install htdocs folder. Importing that file will create the database.

Once that's done go to http://localhost/.

- Use Settings to input your Serial Numbers, then connect to the GEM and switch it to HTTP Get mode under Packet Format.
- Switch Data Post to Standalone Dashboard/DashBox and fill in the IP boxes.
- You may need to switch your Network section to use whatever port you've configured XAMPP to, usually Port 80.

It should shortly start data logging.
Ben
Brultech Research Inc.
E: ben(at)brultech.com
ecosang
Posts: 8
Joined: Thu Jun 15, 2017 8:51 am

Re: A good way to store data for WiFi GEM

Post by ecosang » Sun Jul 02, 2017 3:38 pm

Thank you for the answer.
I have an additional question about the standalone dashboard.

For your comment,
- Use Settings to input your Serial Numbers, then connect to the GEM and switch it to HTTP Get mode under Packet Format.
- Switch Data Post to Standalone Dashboard/DashBox and fill in the IP boxes.
- You may need to switch your Network section to use whatever port you've configured XAMPP to, usually Port 80.

I think the Apache server's port is 80. I switched to HTTP GET mode. I wrote down ip address of my desktop (Apache server) on URL Address under Data Post (i.e., 192.168.0.7). But, the standalone dashboard doesn't show anything (I added GEM with serial number).

I think I did something wrong in Data post.
The packet is (Ir emoved SN)

GET /?SN=*******&SC=330264&V=0&c1=130925981,0&c2=0,0&c3=0,0&c4=0,0&c5=0,0&c6=0,0&c7=
0,0&c8=0,0&c9=0,0&c10=0,0&c11=1,0&c12=0,0&c13=0,0&c14=0,0&c15=0,0&c16=0,0&c17=0,0
&c18=1,0&c19=3,0&c20=7,0&c21=0,0&c22=0,0&c23=3,0&c24=0,0&c25=0,0&c26=2,0&c27=0,0
&c28=0,0&c29=0,0&c30=0,0&c31=0,0&c32=7,0&PL=0,0,0,0&T=nc,,,,,,,&key=&Resp= HTTP/1.1
Host: 192.168.0.7

Is there any manual or discussion thread regarding current version of standalone dashboard?

Thanks
Attachments
51.png
51.png (10.3 KiB) Viewed 1601 times
ecosang
Posts: 8
Joined: Thu Jun 15, 2017 8:51 am

Re: A good way to store data for WiFi GEM

Post by ecosang » Sun Jul 02, 2017 11:46 pm

sorry.
I resolved this issue
I put 8 sec to packet send interval, and put host ip address and port number under Network tab.
Thanks Ben.
This dashboard is very good.
Post Reply