emonBase Internet Traffic Download is High

Emoncms
1

As we are using a metered internet connection I have been monitoring the upload and download traffic from/to my emonBase and emonCMS and was surprised to see that the download is much higher than the upload. The emonBase is connected to my emonCMS account and is set for every 10 seconds.

As you can see in the screenshot below, over 10 minutes, the number of packets being downloaded is just over 13,600 vs 700 uploaded. What is being downloaded to the emonBase and why so many packets? If my math is correct, the average packet size on the download is 190 bytes, and 130 bytes on the upload.

image

Thanks,

Gordon

2

I think it’s likely to be the overhead of Https, there’s quite a bit of back and forth performance - HTTPS overhead compared to HTTP - Server Fault

Il check on a system here for comparison

3

Trystan - thanks for the response. I will be curious to see what you learn. Can emonCMS use HTTP vs HTTPS? I’m not a hosting expert so this may be a dumb question.

I do know that some of the other devices we use cannot connect to HTTPS and we have had to connect via HTTP.

If emonCMS.org cannot support HTTP perhaps we could/should host a version ourselves if it can use HTTP. Thoughts?

Thanks,
Gordon

4

Yes it’s easy to switch to http, it’s just a matter o changing the url in the EmonHubEmoncmsHTTPInterfacer:


[[emoncmsorg]]
        Type = EmonHubEmoncmsHTTPInterfacer
        [[[init_settings]]]
        [[[runtimesettings]]]
            pubchannels = ToRFM12,
            subchannels = ToEmonCMS,
            url = https://emoncms.org
            apikey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            senddata = 1    # Enable sending data to Emoncms.org
            sendnames = 1    # Send full input names (compression will be automatically enabled)
            interval = 30    # Bulk send interval to Emoncms.org in seconds

You lose out on encrypted data transfer of course but it is an option.

This might not be the whole story in your example, will get some logging running here and get back to you. I have investigated this before but that was a while back.

5

OK. I am going to make this change and see the impact.

Also, I have been doing a bit of research and see that MQTT may be an alternative. I see this as an option in emonHub. Is this a viable connection method to emonCMS or is this only for non-emonCMS.org solutions? (I know that I should look this up first, but I’m being a bit lazy and just asking - I will ask for forgiveness later).

Thank you,

6

MQTT is not currently supported by emoncms.org. It is possible to self-host an emoncms instance remotely to use MQTT, but it’s probably simpler to solve this by changing the emonHub settings.

Im running a band width test at the moment with a periodic 30s upload of the following inputs to emoncms.org with the standard emonHub settings:

image
image1838×821 54.6 KB

Removed the initial results - had an active SSH connection

7

Results 1 HTTPS (30s interval, with names & compression):

  • TX: 72.4 bytes/s
  • RX: 340 bytes/s

Result 2: HTTP only (30s interval, with names & compression):

  • TX: 31 bytes/s
  • RX: 65 bytes/s

Result 3: HTTP only (300s interval, with names & compression):

  • TX: 7.3 bytes/s
  • RX: 48 bytes/s

Slightly puzzled by the last results as I dont see 5 minute jumps as Id expect, will have to dig into this some more.

Result 4: HTTP only (900s interval,with names & compression):

  • TX: 3.1 bytes/s
  • RX: 46 bytes/s

Result 5: HTTP only (300s interval, with names & compression):

  • TX: 6.8 bytes/s
  • RX: 77 bytes/s
8

hmmm. I notice that your upload interval is every 30 seconds vs mine is at 10 seconds. Also, I continue to get phantom nodes appearing despite changing the emonhub autoconf = 0.

Further, I have Emoncms Sync: Feeds turned on to sync every 10 seconds. I think this may be the culprit. I do not see how can I change the interval, turn it off, or disable it to see how much this contributes. (once a day sync is probably sufficient - not sure what I was thinking at 10 seconds).

image

I should only have 3 nodes (17. 18, and 19)

image
image971×786 18 KB

Thanks,

9

Are you sure your using emoncms sync? Emoncms sync uploads the feed data directly and doesnt upload via inputs and input processing…

If you do have Emoncms sync, how is it configured to upload every 10s?

10

I can’t recall how I set this up initially. But here is a screenshot from two of my emonBases. There is also a long series of remote locations that are all the feeds I have on emonCMS.org

image
image1597×666 40.1 KB

image
image1554×680 39.8 KB

11

Could you check what you have configured in:

sudo crontab -e

and:

crontab -e
12

When I try to ssh pi@IP address it takes me right into emonCMS.
Not sure how to see the system files (I am a Linux-challenged Windows user!)

1 Like
13

No problem. the program you want is called Putty, it can be downloaded from here:
Download PuTTY: latest release (0.80)

It’s then possible to use this to SSH into the Pi. It would be useful to have some screenshots to share with you showing how to configure. Il try and get some tomorrow.

14

It doesn’t look like I have anything in crontab

image
image766×557 14 KB

15

Thanks Gordon, could you try this again, the spelling wasn’t quite right:

sudo crontab -e
1 Like
16

Example of the emonhub log sending data to emoncms.org with 300s interval:

  • TX: 6.8 bytes/s
  • RX: 77 bytes/s

2024-01-23 09:29:10,251 DEBUG    emoncmsorg Buffer size: 1
2024-01-23 09:29:10,253 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706002150&cb=1 (117 bytes of data, 1 frames, compressed)
2024-01-23 09:29:10,253 INFO     emoncmsorg compression ratio: 67%
2024-01-23 09:29:10,300 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (45 ms)
2024-01-23 09:34:10,321 DEBUG    emoncmsorg Buffer size: 30
2024-01-23 09:34:10,324 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706002450&cb=1 (377 bytes of data, 30 frames, compressed)
2024-01-23 09:34:10,325 INFO     emoncmsorg compression ratio: 7%
2024-01-23 09:34:10,373 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (47 ms)
2024-01-23 09:39:10,377 DEBUG    emoncmsorg Buffer size: 31
2024-01-23 09:39:10,382 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706002750&cb=1 (389 bytes of data, 31 frames, compressed)
2024-01-23 09:39:10,383 INFO     emoncmsorg compression ratio: 7%
2024-01-23 09:39:10,424 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (39 ms)
2024-01-23 09:44:10,499 DEBUG    emoncmsorg Buffer size: 31
2024-01-23 09:44:10,502 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706003050&cb=1 (379 bytes of data, 31 frames, compressed)
2024-01-23 09:44:10,502 INFO     emoncmsorg compression ratio: 7%
2024-01-23 09:44:10,548 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (45 ms)
2024-01-23 09:49:10,638 DEBUG    emoncmsorg Buffer size: 30
2024-01-23 09:49:10,641 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706003350&cb=1 (387 bytes of data, 30 frames, compressed)
2024-01-23 09:49:10,641 INFO     emoncmsorg compression ratio: 7%
2024-01-23 09:49:10,697 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (55 ms)
2024-01-23 09:54:10,796 DEBUG    emoncmsorg Buffer size: 31
2024-01-23 09:54:10,799 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706003650&cb=1 (393 bytes of data, 31 frames, compressed)
2024-01-23 09:54:10,800 INFO     emoncmsorg compression ratio: 7%
2024-01-23 09:54:10,847 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (46 ms)
2024-01-23 09:59:10,928 DEBUG    emoncmsorg Buffer size: 31
2024-01-23 09:59:10,931 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706003950&cb=1 (392 bytes of data, 31 frames, compressed)
2024-01-23 09:59:10,932 INFO     emoncmsorg compression ratio: 7%
2024-01-23 09:59:11,161 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (229 ms)
2024-01-23 10:04:11,225 DEBUG    emoncmsorg Buffer size: 30
2024-01-23 10:04:11,231 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706004251&cb=1 (372 bytes of data, 30 frames, compressed)
2024-01-23 10:04:11,232 INFO     emoncmsorg compression ratio: 7%
2024-01-23 10:04:11,282 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (48 ms)
2024-01-23 10:09:11,364 DEBUG    emoncmsorg Buffer size: 31
2024-01-23 10:09:11,369 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706004551&cb=1 (375 bytes of data, 31 frames, compressed)
2024-01-23 10:09:11,370 INFO     emoncmsorg compression ratio: 6%
2024-01-23 10:09:11,410 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (39 ms)
2024-01-23 10:14:11,483 DEBUG    emoncmsorg Buffer size: 31
2024-01-23 10:14:11,485 INFO     emoncmsorg sending: http://emoncms.org/input/bulk.json?sentat=1706004851&cb=1 (390 bytes of data, 31 frames, compressed)
2024-01-23 10:14:11,486 INFO     emoncmsorg compression ratio: 7%
2024-01-23 10:14:11,530 DEBUG    emoncmsorg acknowledged receipt with 'ok' from http://emoncms.org (43 ms)
17

Another test, emonhub disabled and avahi-deamon disabled:

  • TX: 0.15 bytes/s
  • RX: 21.4 bytes/s

Still downloading … avahi-deamon MDNS uses quite a bit but that’s really a needed service I think.

18

Interestingly monitoring traffic with nethogs does not include the DNS traffic. Looking at the difference between HTTPS and HTTP:

  • HTTP RX is about 70% of TX
  • HTTPS: RX is about 400% of TX

A few data points that will hopefully be useful

19

Trystan,

here is what I see when typing correctly :flushed:

pi@emonpi:~ $ sudo crontab -e
GNU nano 5.4 /tmp/crontab.rV9paG/crontab
*/5 * * * * /usr/local/bin/wifi-check > /var/log/emoncms/wificheck.log 2>&1

2 Likes
20

In just under 2 days, I am seeing RX at ~2800% of TX, so something else seems to be going on. I have just changed to HTTP to see what happens.

Uptime 1d 20h 25m 17s
Activity (24hrs) 18.6 MB
Down Pkts/Bytes 3,397,132 / 630 MB
Up Pkts/Bytes 159,498 / 21.8 MB