linux - Python program eating up RAM -


i wrote small program collect data on serial port using minimalmodbus. data dumped csv file. have read several posts on , other places. few things mentioned are:

  1. using lazy evaluation wherever possible (xrange instead of range)
  2. deleting large unused objects
  3. use child processes , upon death memory released os

the script on github here. use script periodically upload these files server. both these scripts trivial. nothing else running on system, feel withing these 2 systems memory hogging taking place. best way tackle issue. not willing adopt subprocess route.

some more information:

  1. data collection on raspberry pi (512 mb ram)
  2. python version: 2.7
  3. it takes 3-4 days ram used after raspberrypi freezes

i followed this guide find out top 20 programs eating ram.

$ ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20 12434 2.2 python 12338 1.2 python 2578 0.8 /usr/sbin/console-kit-daemon 30259 0.7 sshd: 30283 0.7 -bash 1772 0.6 /usr/sbin/rsyslogd 2645 0.6 /usr/lib/policykit-1/polkitd 2146 0.5 dhclient 1911 0.4 /usr/sbin/ntpd 12337 0.3 sudo 12433 0.3 sudo 1981 0.3 sudo 30280 0.3 sshd: 154 0.2 udevd 16994 0.2 /usr/sbin/sshd 17006 0.2 ps 1875 0.2 /usr/bin/dbus-daemon 278 0.2 udevd 290 0.2 udevd 1 0.1 init

so 2 python processes eating ram, small when compared overall ram consumed. following output of free command.

pi@raspberrypi ~ $ free -m              total       used       free     shared    buffers     cached mem:           438        414         23          0         45        320 -/+ buffers/cache:         48        389 swap:           99          0         99

the following output of top command.

tasks:  69 total,   1 running,  68 sleeping,   0 stopped,   0 zombie %cpu(s): 66.9 us,  5.0 sy,  0.0 ni, 18.1 id,  0.0 wa,  0.0 hi, 10.0 si,  0.0 st kib mem:    448776 total,   429160 used,    19616 free,    47016 buffers kib swap:   102396 total,        0 used,   102396 free,   332288 cached  pid user      pr  ni  virt  res  shr s  %cpu %mem    time+  command            12338 root    20   0 10156 5644 2384 s  69.8  1.3   3059:31 python             26039 root    20   0     0    0    0 s   1.6  0.0   0:02.71 kworker/0:1        26863 pi      20   0  4664 1356 1028 r   1.3  0.3   0:00.12 top                1982 root     20   0  1752  516  448 s   0.3  0.1   1:08.36 sh                 1985 root     20   0  1692  552  460 s   0.3  0.1   5:15.16 startpar           1 root        20   0  2144  728  620 s   0.0  0.2   0:17.43 init               2 root        20   0     0    0    0 s   0.0  0.0   0:00.14 kthreadd           3 root        20   0     0    0    0 s   0.0  0.0   0:13.20 ksoftirqd/0        5 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 kworker/0:0h       7 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 kworker/u:0h       8 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 khelper            9 root        20   0     0    0    0 s   0.0  0.0   0:00.00 kdevtmpfs          10 root       0 -20     0    0    0 s   0.0  0.0   0:00.00 netns              12 root      20   0     0    0    0 s   0.0  0.0   0:00.06 bdi-default        13 root       0 -20     0    0    0 s   0.0  0.0   0:00.00 kblockd

edit 2

as suggested in first answer, decided log files. had @ syslog , following result of tail on it.

may 19 10:03:26 raspberrypi wpa_supplicant[7065]: wlan0: failed initialize driver    interface may 19 10:03:49 raspberrypi wpa_supplicant[7157]: nl80211: 'nl80211' generic netlink not found may 19 10:03:49 raspberrypi wpa_supplicant[7157]: failed initialize driver 'nl80211' may 19 10:03:49 raspberrypi wpa_supplicant[7157]: rfkill: cannot open rfkill control device may 19 10:03:49 raspberrypi wpa_supplicant[7157]: not read interface wlan0 flags: no such device

these messages filling log files , coming every second. interesting part using ethernet , not wifi.

thus, unclear ram has gone?

most of ram free applications, because it's used buffers , caching. @ "-/+ buffers/cache:" line see amount of ram really used/free. explanation can found here.

to verify wether python leaking memory, monitor python's rss size (or %mem) on time. e.g. write shell-script called cron job every couple of hours append output of ps command chain , output of free command file.

if find python processes are leaking memory there couple of things can do;

  • modify script extis after 24 hours , use e.g. cron job restart (the easy way out.)
  • take in-depth python , expecially extension modules you're using. use gc module monitor , influence memory usage. can e.g. call gc.count() regularly monitor amount of objects marked collection. can call gc.collect() explicitly , see if reduces memory usage. modify collection threshhold.

if python's ram use doesn't increase on time, program of daemon. memory logging script mentioned above should tell 1 is.

there reason computer freezes. @ linux logfiles clues.

edit: since have wpa_supplicant filling log file, should check state of filesystem(s). full filesystem might cause system hang. if aren't using wireless interface, disable it.


Comments

Post a Comment

Popular posts from this blog

SPSS keyboard combination alters encoding -

my question not have programming practical issue on spss. accident keyword combination cannot remember because did not intend press changed encoding in spss , text in data in different language english shown small boxes. nevertheless, when kind of analysis performed different language text shown in output. does know reason? tried change encoding did not produce result. first, @ edit > options , see whether setting unicode or locale. might fix it. if have text in multiple languages, want unicode. otherwise, may have specific locale set wrong. can change set locale, need know locale want use.
Read more

Add new record to the table by click on the button in Microsoft Access -

in ms access form want implement separate button, adds new record table. in order added button , attached button event: private sub btnaddrec_click() refresh codecontextobject on error resume next docmd.gotorecord , , acnewrec if err.number <> 0 btnaddrec.enabled = false end if end end sub everything ok when open window , click btnaddrec button, problem when first of perform navigation through existed records , after click on button. got runtime error: 2105: «you can't go specified record. may @ end of recordset» . how solve issue, need have ability add new record on click on specific button, no matter, have walked or not walked through records before. thanks. i created simple form field call description (and autonumber) , created button code follows click event. filled number of records , navigated through them, clicked addnewrec button. form navigated new record without issues. able click addnewrec button directly after ...
Read more

CSS3 Transition to highlight new elements created in JQuery -

i want use css3 color transition apply highlight-fading color effect (yellow transparent) new elements appended markup using jquery. css #content div { background-color:transparent; -moz-transition:background-color 2s; -webkit-transition:background-color 2s; -o-transition:background-color 2s; transition:background-color 2s; } #content div.new { background-color:yellow; } html <div id="content"></div> <a id="add-element" href="#">add new element</a> js $('#add-element').click(function() { var newelement = $('<div class="new">new element</div>'); $('#content').append(newelement); newelement.removeclass('new'); }); when click link, new element created. class "new" (background color yellow) , it's appended html markup. should able remove "new" class trigger background color transition transparent (or whate...
Read more