From emonpi to shared linux emoncms: error 410

Emoncms
1

Hello,

I have installed emoncms on my shared linux server. I can access to this emoncms. I copy the read write api and paste it to my emonpi. I write the right http address of the server. When I look the emonhub log, I have a WARNING with an httperror 410. Someboby can help me? Thank you

News: When I do that with the browser and it function:

https://www.gftechnology.ch/emoncms/input/post.json?node=1&json={power:200}&apikey=xxxxxxxxMyAPIKeyxxxxxxxxx

News2: The log of emonhub:

2017-06-05 13:34:35,351 INFO     gftechnology sending: https://www.gftechnology.ch/emoncms/input/bulk.json?apikey=E-M-O-N-C-M-S-A-P-I-K-E-Y&data=[[1496662449.488229,11,306,151,682,0,226.06,18.7,301,301,301,301,301,0,0,-22],[1496662460.322888,11,304,151,680,0,226.3,18.8,301,301,301,301,301,0,0,-28],[1496662471.166822,11,303,147,680,0,226.45000000000002,18.8,301,301,301,301,301,0,0,-22]]&sentat=1496662475
2017-06-05 13:34:35,574 WARNING  gftechnology gftechnology couldn't send to server, HTTPError: 410
2017-06-05 13:34:35,576 WARNING  gftechnology send failure: wanted 'ok' but got ''

News 3: and that is the error on my linux shared server:

 [Mon Jun 05 14:19:10.232574 2017] [core:info] [pid 797284] [client 83.76.121.9:47856] AH00128: File does not exist: /home/wd1wugvx/public_html/410.shtm

News 4: I try this line (the one with error) in my web browser and it function. I understand nothing…:

https://www.gftechnology.ch/emoncms/input/bulk.json?apikey=******MYAPIKEY*******&data=[[1496662449.488229,11,306,151,682,0,226.06,18.7,301,301,301,301,301,0,0,-22],[1496662460.322888,11,304,151,680,0,226.3,18.8,301,301,301,301,301,0,0,-28],[1496662471.166822,11,303,147,680,0,226.45000000000002,18.8,301,301,301,301,301,0,0,-22]]&sentat=1496662475

So when I try manually to do a bulk in the web browser it function. But when the bulk is done with emonpi it don’z function. It is exactly the same line? When I post on emoncms.or all is ok…???

here my emonhub.conf:

[[emoncmsorg]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://emoncms.org
        apikey = xxxx
        senddata = 1                    # Enable sending data to Emoncms.org
        sendstatus = 1                  # Enable sending WAN IP to Emoncms.org MyIP > https://emoncms.org/myip/list
        sendinterval= 30                # Bulk send interval to Emoncms.org in seconds

[[emoncmsgfte]]
    Type = EmonHubEmoncmsHTTPInterfacer
    [[[init_settings]]]
    [[[runtimesettings]]]
        pubchannels = ToRFM12,
        subchannels = ToEmonCMS,
        url = https://www.gftechnology.ch/emoncms
        apikey = xxxxx
        senddata = 1                    # Enable sending data to Emoncms.org
        sendinterval= 30                # Bulk send interval to Emoncms.org in seconds
2

What you describe sounds much like a dns look up issue if the Pi can deliver to one site and not another, despite the correct url being used. However the “410” error code sounds like the Pi is reaching the shared server but the url is no longer valid. That is what a “410” is, “no longer present” or “gone”. It is not saying the URL never existed, it’s saying it is no longer there and no alternative is known.

Could this be a caching issue? have you tried rebooting the Pi? (although the dns cache maybe retained on the data partition, I’m not sure). Can you try using wget (or curl) from the command line of the Pi to post data?

4

I find the right synthax fur curl to post the json:
curl "https://www.gftechnology.ch/emoncms/input/post?apikey=MYAPIKEY&node=1&json={power:252.4}"

With that in the EmonPi on SSH. It’s ok.

I try now to find the right syntax for a json bulk.

5

I find the right synthax for bulk with curl:
curl -g "https://www.gftechnology.ch/emoncms/input/bulk.json?apikey=MYAPIKEY&data=[[1496662449.488229,11,306,151,682,0,2.06,18.7,301,301,301,301,301,0,0,-22],[1496662460.322888,11,304,151,680,0,226.3,18.8,301,301,301,301,301,0,0,-28],[1496662471.166822,11,303,147,680,0,226.45000000000002,18.8,301,301,301,301,301,0,0,-22]]&sentat=1496662475"

Also it is ok with command mode on the emonpi to send data to my shared server in BULK mode. I use CURL with the synthax above. But with emoncms (emonhub.conf see my last post) there is always the same error.So what is the problem witn emoncms??

6

I find the solution. Not easy but:
My linux shared server don’t like the http default header “Python-urllib/2.7” from the urllib2 that use the Pythton script from emonhub when it send data. So I changed it to ‘Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)’

On the emonpi:

go to /pi/emonhub/src/interfacers/

then open with nano EmonHubEmoncmsHTPPInterfacer.py

$sudo nano EmonHubEmoncmsHTPPInterfacer.py

And add the code betweenn “”“MyCode”“” … “”“EndOfMyCode”“”

 """Send data to server.

        data (list): node and values (eg: '[node,val1,val2,...]')
        time (int): timestamp, time when sample was recorded

        return True if data sent correctly

        """

        reply = ""

        """MyCode"""
        user_agent ='Mozilla/4.0(compatible; MSIE 5.5;Windows NT)'
        headers = {'User-Agent':user_agent}
        

        request = urllib2.Request(post_url, post_body, headers)
        """EndOfMyCode"""
7

Thanks for sharing the solution, it’s interesting that this has never cropped up before.

The very first article I read on this (Urllib2 - User Agent - PythonForBeginners.com) starts

This post will show how to add headers to a HTTP request.

By default urllib2 identifies itself as Python-urllib/2.7 : GET / HTTP/1.1" 200
151 “-” “Python-urllib/2.7”

That can sometimes be confusing for certain sites. With the user_agent header in
Python, it’s possible to alter that and specify any identity you like.

Should the user agent really make much difference when posting data? I understand the importance when using a browser, is this just a case of this server just being particularly picky or is it common place?

Is “Spoofing” the user agent acceptable practice?

Commonly used user agent strings