Linux Memory vs real memory & MQTT version

Jon · 20 October 2016 03:24

Glyn & Andy,

When I was experiencing big slowdowns with my emonPi, one of the items I suspected was low memory. When I looked at the Admin page I saw this:

I went reading about Linux Memory and found this article:
http://www.linuxatemyram.com

It took me a bit of research to learn it wasn’t a memory issue. So I’d like to propose the Administration page display “real” or “available” memory instead of Linux memory in the admin_main_view.php file.

I’m not sure of the right way to offer this via github so I decide to post it here:

$sysRamUsed = $system['mem_info']['MemTotal'] - $system['mem_info']['MemFree'] - $system['mem_info']['Buffers'] - $system['mem_info']['Cached'];
$sysRamPercent = sprintf('%.2f',($sysRamUsed / $system['mem_info']['MemTotal']) * 100);
echo "<tr><td><b>Real Memory</b></td><td>RAM</td><td><div class='progress progress-info' style='margin-bottom: 0;'><div class='bar' style='width: ".$sysRamPercent."%;'>Used&nbsp;".$sysRamPercent."%</div></div>";
echo "<b>Total:</b> ".formatSize($system['mem_info']['MemTotal'])."<b> Used:</b> ".formatSize($sysRamUsed)."<b> Free:</b> ".formatSize(($system['mem_info']['MemFree'] + $system['mem_info']['Buffers'] + $system['mem_info']['Cached']))."</td></tr>\n";

I think this is the correct file…

github.com

emoncms/emoncms/blob/d869c39f82ba743fb61c1d6b49758af1185a041e/Modules/admin/admin_main_view.php#L281-L284

    
      
          $sysRamUsed = $system['mem_info']['MemTotal'] - $system['mem_info']['MemFree'];
          $sysRamPercent = sprintf('%.2f',($sysRamUsed / $system['mem_info']['MemTotal']) * 100);
          echo "<tr><td><b>Memory</b></td><td>RAM</td><td><div class='progress progress-info' style='margin-bottom: 0;'><div class='bar' style='width: ".$sysRamPercent."%;'>Used&nbsp;".$sysRamPercent."%</div></div>";
          echo "<b>Total:</b> ".formatSize($system['mem_info']['MemTotal'])."<b> Used:</b> ".formatSize($sysRamUsed)."<b> Free:</b> ".formatSize($system['mem_info']['MemFree'])."</td></tr>\n";

Jon · 20 October 2016 03:31

Forgot one thing…

Here is some code for the MQTT version - I saw it was displayed as “n/a”. I could not figure out how to plug it into the admin_main_view.php code.

mosquitto -h | grep -oP '(?<=mosquitto\sversion\s)[0-9.]+(?=\s*\(build)'

glyn.hudson · 20 October 2016 13:17

Hi Jon,

Nice fixes. I see you have included Andy in your message text but he does not seem to be included in this PM.

Submitted a file change proposal via Github is not difficult. The easiest way is just to edit the file directly on github: Editing files - GitHub Docs

Please, could you do this to submit a pull-request since there it’s easy for me to change the wrong bit etc. Github is amazingly useful, it’s well worth trying to get acquainted with it.

Mosquitto version in Emoncms admin will be super useful, this has been a recent topic since a mosquitto update broke the emonPi implementation: Mosquitto not starting after emonSD update & upgrade. It would be great if users could easily see which version their running.

Thanks again, really appreciate your help.

Jon · 20 October 2016 13:55

At the bottom of my first post I see this:

And it includes Andy so I think he is part of this PM.

Thanks for the github link, I’ll read thru it!

In a post many month ago you had mentioned the emonPi updates come from the stable version of emoncms. So I normally watch and review the stable version. But I noticed there are four Branches so I wasn’t sure what Branch to modify.

I also looked at a few other repositories and did not see a branch named stable and became confused about the overall process for repositories.

glyn.hudson · 20 October 2016 16:16

Wops, your right. Sorry @Andy_Taylor, I didn’t spot you!

Correct, emonPi pull from stable however all changes should be made to master then I can review and eventually when stable and tested release periodically into stable which usually includes an Emoncms version number bump.

There is only a stable branch on emoncms core. The emonPi pull from the master branch on the other repos e.g dashboard etc. Ideally we should have a stable branch on all repos, however, the core is the most actively developed and has the greatest chance of breaking stuff if a bug gets committed!

Jon · 20 October 2016 16:17

got it! Thank you!

Jon · 20 October 2016 17:57

Glyn - all of the changes and testing were done directly on the emonPi and directly in the admin_main_view.php file.

I’ve done the git fork and made the same changes (I hope) to the forked admin_main_view.php file.

Now I want to test it one more time before doing the PR. How do I get my fork loaded into my emonPi to test my git changes?

glyn.hudson · 20 October 2016 23:36

Nice! I’ve created a PR by viewing your branch on gh.com and clicking create-PR.

The command line instructions at the bottom of the PR page show you how to ‘checkout’ the PR to test:

First make your emonpi RW:

$ rpi-rw

Ensure your running latest master branch (not sure if this is needed but won’t hurt)

$ cd /var/www/emoncms
$ git checkout master 
$ git pull origin master

Then checkout your branch

$ git checkout -b JonMurphy-patch-1 master
$ git pull https://github.com/JonMurphy/emoncms.git patch-1

When your done you can revert back to master (or stable) with:

$ git checkout master

or

$ git checkout stable

$ rpi-ro

Please also make a PR for the Mosquitto version, this will be super useful

Thanks

Jon · 21 October 2016 00:22

Thank you!! I’ve been reading about github much of the day and I could not figure this out.

Let me test with the Mosquito within the PHP file. I created it via the Terminal and I have no clue if it works within PHP.

Jon · 21 October 2016 03:38

Glyn -

The code in the PR is not correct. I forgot to re-calculate the “Free:” memory. Do not commit it.

The current code in my github will display Linux memory and real memory:

I am not sure if this will meet chaveiro concerns. Let me know what you’d prefer.

https://github.com/JonMurphy/emoncms/blob/2569988452cb547b393890e1e3efb8d0ae9d32ba/Modules/admin/admin_main_view.php

Jon · 21 October 2016 03:42

Also I cannot get the MQTT version command to work within PHP. The single quotes are giving me problems. Need to find someone with PHP skills!

@Andy_Taylor help!!

mosquitto -h | grep -oP '(?<=mosquitto\sversion\s)[0-9.]+(?=\s*.build)'

Andy_Taylor · 21 October 2016 07:50

I’ve been deliberately not joining this thread, because if I just solved your problem, you wouldn’t have tried
@Jon - it’s great to see you having a go and sharing back to the community, don’t give in.

My Pi is currently spread all over my desk (been working on another Pi related project this week) - I’ll stick it back together and see if we can’t get this going for you…

Andy_Taylor · 21 October 2016 09:12

@Jon @glyn.hudson

Jon - your version code for Mosquitto was very good - I pretty much used that exactly in the pull request just submitted, it needed wrapping up in PHP - but I used pretty much exactly what you submitted as the basis for the code (you should be able to see my PR on GitHub here: Update admin_main_view.php by AndyTaylorTweet · Pull Request #578 · emoncms/emoncms · GitHub - feel free to take a look.

Also your suggestions about the RAM use are spot on, rather than show RAM used with buffers / cache and without buffers / cache, I believe its more use to all of us to only show the figure without buffers / cache included, since that is what we’re all actually interested in, so I just modified the current graph rather than adding a new one.

Glyn - PR submitted for the changes, due to the use of Grep etc the mosquito versioning is going to be limited to Linux / unix hosts, so there is some detection wrapped around that, Windows hosts will get the old N/A and linux hosts will get the version pulled so long as the binary is where its expected to be (that is also tested).

Happy coding friday everyone…

Jon · 21 October 2016 15:32

Andy - thank you!

So how do I load your code at Update admin_main_view.php by AndyTaylorTweet · Pull Request #578 · emoncms/emoncms · GitHub? Glyn mentioned to do this for my code:

git checkout -b JonMurphy-patch-1 master
git pull https://github.com/JonMurphy/emoncms.git patch-1

but I cannot figure out the checkout and pull for your code (still learning!)

It was fun figuring out the Memory side and I did learn. But the exec command with the mosquitto -h command was giving me fits!

I created a small PHP file and tried any and all exec commands I could find but I kept getting errors like this:

PHP Parse error:  syntax error, unexpected '(' in /home/pi/data/myApps/mqttver.php on line 4
PHP Notice:  Array to string conversion in /home/pi/data/myApps/mqttver.php on line 6
PHP Parse error:  syntax error, unexpected ';' in /home/pi/data/myApps/mqttver.php on line 6
sh: 1: mosquitto -h | grep -oP '(?<=mosquitto\sversion\s)[0-9.]+(?=\s*.build)': not found
PHP Parse error:  syntax error, unexpected 'echo' (T_ECHO) in /home/pi/data/myApps/mqttver.php on line 5
sh: 1: /usr/sbin/mosquitto -h: not found

(no need to explain any of my errors I just included them as an example)

Thank you again! Jon

Andy_Taylor · 21 October 2016 15:40

You can pull my branch from here:

$ git pull https://github.com/AndyTaylorTweet/emoncms.git master

Your issues with executing your command line relate to the use of quotes.

in PHP to run something on the command line and return the results you will use “exec”.

Something like <?php exec ('whoami'); ?> would return the user running the command on the command line.

so since you already have quotes in your command, you would need to “escape” those quotes in order for them to be passed to the command line rather than PHP interpreting them.

Depending on how much programming you have ever done, this will either make complete sense, or you will be left with a feeling of “what the hell does this mean”.

If you are at the latter of those, do a little googling, and then ask any questions you have.

Andy

Jon · 21 October 2016 15:53

Thank you! Yes, I do understand escaping a character or quote (and it took a while to sink in!). I did try escaping quotes but it didn’t work

@exec('mosquitto -h | grep -oP \''(?<=mosquitto\sversion\s)[0-9.]+(?=\s*\(build)'\'', $mqtt_version);

Also - Line 284 needs a quick update (I did the same thing):

from

echo “Total: “.formatSize($system[‘mem_info’][‘MemTotal’]).” Used: “.formatSize($sysRamUsed).” Free: “.formatSize($system[‘mem_info’][‘MemFree’]).”\n”;

to

echo “Total: “.formatSize($system[‘mem_info’][‘MemTotal’]).” Used: “.formatSize($sysRamUsed).” Free: “.formatSize($system[‘mem_info’][‘MemTotal’] - $sysRamUsed).”\n”;

Andy_Taylor · 21 October 2016 16:06

Goot catch - I was so wrapped up in sorting out the graph that I forgot to fix up the text.

Changes applied in my branch and the PR also updated. - Please re-test.

Jon · 21 October 2016 16:10

it is a match and we are good to go! Thank you again!

pi@emonpi:/var/www/emoncms $ free -m
             total       used       free     shared    buffers     cached
Mem:           973        860        112         43         13        602
-/+ buffers/cache:        244        728
Swap:            0          0          0
pi@emonpi:/var/www/emoncms $

glyn.hudson · 21 October 2016 22:59

Nice work guys, thanks for instigating @Jon and thanks for helping @Andy_Taylor.

I have merged @Andy_Taylor PR, since this contained the OS check code:

Ignore the face this PR failed the Travis test, this was due to today’s events in the US. Stay safe out there : Update admin_main_view.php by AndyTaylorTweet · Pull Request #578 · emoncms/emoncms · GitHub

BTW this thread has now been made public for future ref by all