YJL

I need to have a counter which allows a few increments within a second. Also the number will be stored in datastore. However, it doesn’t have to be very accurate. The error can be from the eviction on memcache or cron doesn’t work and could get the number in memcache being added to datastore in time.

I came up with a pingpong mechanism. The number will be increased by memcache.incr on two slots: ping slot and pong slot. When increment is on one slot, then a cron job will add the number from another slot to datastore and empty that slot if it successfully adds the number.

The time interval of slot change is 10 minutes, therefore, the number could be missing if memcache suddenly out of work. Every ten minutes, a cron job kicks in. So, normally, if the memcache doesn’t work, the lost number could be from between 0 seconds and 10 minutes ago, if it’s just short glitches from Google App Engine. If memcache is out of work longer than that, Google App Engine is probably down, that it doesn’t matter.

When datastore is down or under the scheduled maintenance, as long as memcache still works, the number still counts and won’t be lost as long as the number isn’t evicted from memcache. Once datastore is back with write capability, cron job can add the number to datastore.

There be no race condition when add the number to datastore because of using pingpong and cron job. I believe there is also no race condition with memcache.incr, if it’s processed on memcache server.

Here is my code, it’s from the source of my project. Unmodified, it’s not generalized for you to use it out-of-the-box. Read it and modify it before you use it.

Recently, I have a chance to use pylint, an analyzer of Python codes. Usually, you would use it's command-line program to generate analysis reports. However, we — people I recently work with for a game development — import pylint and generate in a script, then immediately parse the reports.

That cost me some time. I can't figure out why I open the report (pylint_global.txt), and get no content from it. Finally, I guess pylint doesn't close files, and it doesn't. Here is a simple program to show your:

#!/usr/bin/python


"""Find out how many files that pylint left opend"""

import __builtin__
import sys

from pylint import lint
import time

def main():
    """Main function"""

    # Dirty job to close pylint left opened
    original_open = __builtin__.open

    f_handlers = []
    def count_open(*args, **kwargs):
        """Counts how many file opened"""
        f_handler = original_open(*args, **kwargs)
        f_handlers.append(f_handler)
        return f_handler

    __builtin__.open = count_open

    lint.Run(['-ftext', '--files-output=y', sys.argv[0]])
    # Run lsof here
    time.sleep(30)
    # Restore the original file open
    __builtin__.open = original_open

    opened = len(f_handlers)
    closed = sum([f_handler.closed for f_handler in f_handlers])
    print 'Open: %d, unclosed %d.' % (opened, opened - closed)


if __name__ == '__main__':
    main()

A sample output:

$ python pylint_open.py
No config file found, using default configuration
Open: 9, unclosed 9.

Here is related result from lsof:

python  32310 livibetter    3r   REG  253,0      870 4554767 /home/livibetter/tmp/pylint_open.py
python  32310 livibetter    4w   REG  253,0        0 4554764 /home/livibetter/tmp/pylint_pylint_open.txt
python  32310 livibetter    5r   REG  253,0      870 4554767 /home/livibetter/tmp/pylint_open.py
python  32310 livibetter    6r   REG  253,0      815 1221017 /usr/lib/python2.5/site-packages/pylint-0.15.2-py2.5.egg/pylint/__init__.py
python  32310 livibetter    7r   REG  253,0    38194 1221015 /usr/lib/python2.5/site-packages/pylint-0.15.2-py2.5.egg/pylint/lint.py
python  32310 livibetter    8r   REG  253,0    16986 1221710 /usr/lib/python2.5/site-packages/pylint-0.15.2-py2.5.egg/pylint/checkers/rpython.py
python  32310 livibetter    9r   REG  253,0        0 3768605 /home/livibetter/.pylint.d/pylint_open1.stats
python  32310 livibetter   10w   REG  253,0        0 4554766 /home/livibetter/tmp/pylint_global.txt
python  32310 livibetter   11w   REG  253,0        0 3768605 /home/livibetter/.pylint.d/pylint_open1.stats

You should look at *.txt. That means pylint doesn't close files after it finishes its job. I tried to report but I can't find a link to sign up or to file a report.