linux - Python program eating up RAM -

i wrote small program collect data on serial port using minimalmodbus. data dumped csv file. have read several posts on , other places. few things mentioned are:

1. using lazy evaluation wherever possible (xrange instead of range)
2. deleting large unused objects
3. use child processes , upon death memory released os

the script on github here. use script periodically upload these files server. both these scripts trivial. nothing else running on system, feel withing these 2 systems memory hogging taking place. best way tackle issue. not willing adopt subprocess route.

some more information:

1. data collection on raspberry pi (512 mb ram)
2. python version: 2.7
3. it takes 3-4 days ram used after raspberrypi freezes

i followed this guide find out top 20 programs eating ram.

$ ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20 12434 2.2 python 12338 1.2 python 2578 0.8 /usr/sbin/console-kit-daemon 30259 0.7 sshd: 30283 0.7 -bash 1772 0.6 /usr/sbin/rsyslogd 2645 0.6 /usr/lib/policykit-1/polkitd 2146 0.5 dhclient 1911 0.4 /usr/sbin/ntpd 12337 0.3 sudo 12433 0.3 sudo 1981 0.3 sudo 30280 0.3 sshd: 154 0.2 udevd 16994 0.2 /usr/sbin/sshd 17006 0.2 ps 1875 0.2 /usr/bin/dbus-daemon 278 0.2 udevd 290 0.2 udevd 1 0.1 init 

so 2 python processes eating ram, small when compared overall ram consumed. following output of free command.

pi@raspberrypi ~ $ free -m              total       used       free     shared    buffers     cached mem:           438        414         23          0         45        320 -/+ buffers/cache:         48        389 swap:           99          0         99 

the following output of top command.

tasks:  69 total,   1 running,  68 sleeping,   0 stopped,   0 zombie %cpu(s): 66.9 us,  5.0 sy,  0.0 ni, 18.1 id,  0.0 wa,  0.0 hi, 10.0 si,  0.0 st kib mem:    448776 total,   429160 used,    19616 free,    47016 buffers kib swap:   102396 total,        0 used,   102396 free,   332288 cached  pid user      pr  ni  virt  res  shr s  %cpu %mem    time+  command            12338 root    20   0 10156 5644 2384 s  69.8  1.3   3059:31 python             26039 root    20   0     0    0    0 s   1.6  0.0   0:02.71 kworker/0:1        26863 pi      20   0  4664 1356 1028 r   1.3  0.3   0:00.12 top                1982 root     20   0  1752  516  448 s   0.3  0.1   1:08.36 sh                 1985 root     20   0  1692  552  460 s   0.3  0.1   5:15.16 startpar           1 root        20   0  2144  728  620 s   0.0  0.2   0:17.43 init               2 root        20   0     0    0    0 s   0.0  0.0   0:00.14 kthreadd           3 root        20   0     0    0    0 s   0.0  0.0   0:13.20 ksoftirqd/0        5 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 kworker/0:0h       7 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 kworker/u:0h       8 root         0 -20     0    0    0 s   0.0  0.0   0:00.00 khelper            9 root        20   0     0    0    0 s   0.0  0.0   0:00.00 kdevtmpfs          10 root       0 -20     0    0    0 s   0.0  0.0   0:00.00 netns              12 root      20   0     0    0    0 s   0.0  0.0   0:00.06 bdi-default        13 root       0 -20     0    0    0 s   0.0  0.0   0:00.00 kblockd  

edit 2

as suggested in first answer, decided log files. had @ syslog , following result of tail on it.

may 19 10:03:26 raspberrypi wpa_supplicant[7065]: wlan0: failed initialize driver    interface may 19 10:03:49 raspberrypi wpa_supplicant[7157]: nl80211: 'nl80211' generic netlink not found may 19 10:03:49 raspberrypi wpa_supplicant[7157]: failed initialize driver 'nl80211' may 19 10:03:49 raspberrypi wpa_supplicant[7157]: rfkill: cannot open rfkill control device may 19 10:03:49 raspberrypi wpa_supplicant[7157]: not read interface wlan0 flags: no such device 

these messages filling log files , coming every second. interesting part using ethernet , not wifi.

thus, unclear ram has gone?

most of ram free applications, because it's used buffers , caching. @ "-/+ buffers/cache:" line see amount of ram really used/free. explanation can found here.

to verify wether python leaking memory, monitor python's rss size (or %mem) on time. e.g. write shell-script called cron job every couple of hours append output of ps command chain , output of free command file.

if find python processes are leaking memory there couple of things can do;

• modify script extis after 24 hours , use e.g. cron job restart (the easy way out.)
• take in-depth python , expecially extension modules you're using. use gc module monitor , influence memory usage. can e.g. call gc.count() regularly monitor amount of objects marked collection. can call gc.collect() explicitly , see if reduces memory usage. modify collection threshhold.

if python's ram use doesn't increase on time, program of daemon. memory logging script mentioned above should tell 1 is.

there reason computer freezes. @ linux logfiles clues.

edit: since have wpa_supplicant filling log file, should check state of filesystem(s). full filesystem might cause system hang. if aren't using wireless interface, disable it.