linux - Python program eating up RAM -


i wrote small program collect data on serial port using minimalmodbus. data dumped csv file. have read several posts on , other places. few things mentioned are:

  1. using lazy evaluation wherever possible (xrange instead of range)
  2. deleting large unused objects
  3. use child processes , upon death memory released os

the script on github here. use script periodically upload these files server. both these scripts trivial. nothing else running on system, feel withing these 2 systems memory hogging taking place. best way tackle issue. not willing adopt subprocess route.

some more information:

  1. data collection on raspberry pi (512 mb ram)
  2. python version: 2.7
  3. it takes 3-4 days ram used after raspberrypi freezes

i followed this guide find out top 20 programs eating ram.

$ ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20 12434 2.2 python 12338 1.2 python 2578 0.8 /usr/sbin/console-kit-daemon 30259 0.7 sshd: 30283 0.7 -bash 1772 0.6 /usr/sbin/rsyslogd 2645 0.6 /usr/lib/policykit-1/polkitd 2146 0.5 dhclient 1911 0.4 /usr/sbin/ntpd 12337 0.3 sudo 12433 0.3 sudo 1981 0.3 sudo 30280 0.3 sshd: 154 0.2 udevd 16994 0.2 /usr/sbin/sshd 17006 0.2 ps 1875 0.2 /usr/bin/dbus-daemon 278 0.2 udevd 290 0.2 udevd 1 0.1 init

so 2 python processes eating ram, small when compared overall ram consumed. following output of free command.

pi@raspberrypi ~ $ free -m              total       used       free     shared    buffers     cached mem:           438        414         23          0         45        320 -/+ buffers/cache:         48        389 swap:           99          0         99

the following output of top command.

tasks:  69 total,   1 running,  68 sleeping,   0 stopped,   0 zombie %cpu(s): 66.9 us,  5.0 sy,  0.0 ni, 18.1 id,  0.0 wa,  0.0 hi, 10.0 si,  0.0 st kib mem:    448776 total,   429160 used,    19616 free,    47016 buffers kib swap:   102396 total,        0 used,   102396 free,   332288 cached  pid user      pr  ni  virt  res  shr s  %cpu %mem    time+  command            12338 root    20   0 10156 5644 2384 s  69.8  1.3   3059:31 python             26039 root    20   0     0    0    0 s   1.6  0.0   0:02.71 kworker/0:1        26863 pi      20   0  4664 1356 1028 r   1.3  0.3   0:00.12 top                1982 root     20   0  1752  516  448 s   0.3  0.1   1:08.36 sh                 1985 root     20   0  1692  552  460 s   0.3  0.1   5:15.16 startpar           1 root        20   0  2144  728  620 s   0.0  0.2   0:17.43 init               2 root        20   0     0    0    0 s   0.0  0.0   0:00.14 kthreadd           3 root        20   0     0    0    0 s   0.0  0.0   0:13.20 ksoftirqd/0        5 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 kworker/0:0h       7 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 kworker/u:0h       8 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 khelper            9 root        20   0     0    0    0 s   0.0  0.0   0:00.00 kdevtmpfs          10 root       0 -20     0    0    0 s   0.0  0.0   0:00.00 netns              12 root      20   0     0    0    0 s   0.0  0.0   0:00.06 bdi-default        13 root       0 -20     0    0    0 s   0.0  0.0   0:00.00 kblockd

edit 2

as suggested in first answer, decided log files. had @ syslog , following result of tail on it.

may 19 10:03:26 raspberrypi wpa_supplicant[7065]: wlan0: failed initialize driver    interface may 19 10:03:49 raspberrypi wpa_supplicant[7157]: nl80211: 'nl80211' generic netlink not found may 19 10:03:49 raspberrypi wpa_supplicant[7157]: failed initialize driver 'nl80211' may 19 10:03:49 raspberrypi wpa_supplicant[7157]: rfkill: cannot open rfkill control device may 19 10:03:49 raspberrypi wpa_supplicant[7157]: not read interface wlan0 flags: no such device

these messages filling log files , coming every second. interesting part using ethernet , not wifi.

thus, unclear ram has gone?

most of ram free applications, because it's used buffers , caching. @ "-/+ buffers/cache:" line see amount of ram really used/free. explanation can found here.

to verify wether python leaking memory, monitor python's rss size (or %mem) on time. e.g. write shell-script called cron job every couple of hours append output of ps command chain , output of free command file.

if find python processes are leaking memory there couple of things can do;

  • modify script extis after 24 hours , use e.g. cron job restart (the easy way out.)
  • take in-depth python , expecially extension modules you're using. use gc module monitor , influence memory usage. can e.g. call gc.count() regularly monitor amount of objects marked collection. can call gc.collect() explicitly , see if reduces memory usage. modify collection threshhold.

if python's ram use doesn't increase on time, program of daemon. memory logging script mentioned above should tell 1 is.

there reason computer freezes. @ linux logfiles clues.

edit: since have wpa_supplicant filling log file, should check state of filesystem(s). full filesystem might cause system hang. if aren't using wireless interface, disable it.


Comments

Post a Comment

Popular posts from this blog

.htaccess - First slash is removed after domain when entering a webpage in the browser -

sometimes when hit address of webpage in browser, removes first slash after domain. examples: www.mywebsite.com/scripts/script_6.php -> trying load www.mywebsite.comscripts/script_6.php www.mywebsite.com/test.php -> trying load www.mywebsite.comtest.php after entering again, loads page. a .htaccess settings must wrong: rewritebase / errordocument 404 http://www.mywebsite.com/oops.php errordocument 401 /notallowed.php rewriteengine on # turn on rewriting engine #gzip <ifmodule mod_deflate.c> addoutputfilterbytype deflate text/text text/html text/plain text/xml text/css application/x- javascript application/javascript </ifmodule> #end gzip rewritecond %{http_host} ^mywebsite\.com rewriterule ^(.*)$ http://www.mywebsite.com$1 [r=permanent,l] directoryindex index6.php rewriterule ^termsofuse/?$ termsofuse.php [nc,l] # handle requests "adpolicy" rewriterule ^privacy-policy/?$ privacy-policy.php [nc,l] # handle requ...
Read more

Socket.connect doesn't throw exception in Android -

if put nonsense url no exception thrown , none of rest of code executed not rest of asynctask called method connects. try { socketcliente.connect(new inetsocketaddress("f", port), 2000); } catch (unknownhostexception e) { geterror("host doesn't exist"); return -1; } catch (ioexception e) { geterror("could not connect: host down"); return -1; } add catch statement. catch ( exception e ) { log.d(tag, e.getmessage(),e); toast.maketext(getapplicationcontext(), "unexpected error:" + e.getmessage(), toast.length_long).show(); } this write log , pop toast tell what's going on.
Read more

SPSS keyboard combination alters encoding -

my question not have programming practical issue on spss. accident keyword combination cannot remember because did not intend press changed encoding in spss , text in data in different language english shown small boxes. nevertheless, when kind of analysis performed different language text shown in output. does know reason? tried change encoding did not produce result. first, @ edit > options , see whether setting unicode or locale. might fix it. if have text in multiple languages, want unicode. otherwise, may have specific locale set wrong. can change set locale, need know locale want use.
Read more