I need to have a counter which allows a few increments within a second. Also the number will be stored in datastore. However, it doesn’t have to be very accurate. The error can be from the eviction on memcache or cron doesn’t work and could get the number in memcache being added to datastore in time.

I came up with a pingpong mechanism. The number will be increased by memcache.incr on two slots: ping slot and pong slot. When increment is on one slot, then a cron job will add the number from another slot to datastore and empty that slot if it successfully adds the number.

The time interval of slot change is 10 minutes, therefore, the number could be missing if memcache suddenly out of work. Every ten minutes, a cron job kicks in. So, normally, if the memcache doesn’t work, the lost number could be from between 0 seconds and 10 minutes ago, if it’s just short glitches from Google App Engine. If memcache is out of work longer than that, Google App Engine is probably down, that it doesn’t matter.

When datastore is down or under the scheduled maintenance, as long as memcache still works, the number still counts and won’t be lost as long as the number isn’t evicted from memcache. Once datastore is back with write capability, cron job can add the number to datastore.

There be no race condition when add the number to datastore because of using pingpong and cron job. I believe there is also no race condition with memcache.incr, if it’s processed on memcache server.

Here is my code, it’s from the source of my project. Unmodified, it’s not generalized for you to use it out-of-the-box. Read it and modify it before you use it.

class SimpleCounter(db.Model):

  name = db.StringProperty(required=True)
  count = db.IntegerProperty(required=True)
  added = db.DateTimeProperty(auto_now_add=True)
  updated = db.DateTimeProperty(auto_now=True)


def pingpong_incr(key_name):

  # Changing slot every 10 minutes
  slot = gmtime().tm_min / 10 % 2
  # A bug with initial value: http://code.google.com/p/googleappengine/issues/detail?id=2012
  if memcache.incr('%s_%s' % (key_name, slot), namespace='ia') == None:
    # Can set no such key existed
    memcache.set('%s_%s' % (key_name, slot), 1, namespace='ia')


def pingpong_get(key_name, from_ping=True):

  if from_ping:
    slot = gmtime().tm_min / 10 % 2
  else:
    slot = (gmtime().tm_min + 10) / 10 % 2
  return memcache.get('%s_%s' % (key_name, slot), namespace='ia')


def pingpong_delete(key_name):

  slot = (gmtime().tm_min + 10) / 10 % 2
  memcache.delete('%s_%s' % (key_name, slot), namespace='ia')


class UpdateCron(webapp.RequestHandler):

  def update_counter(self, key_name):

    # Checking memory cache
    count = pingpong_get(key_name, from_ping=False)
    if not count:
      return

    counter = SimpleCounter.get_by_key_name(key_name)
    if not counter:
      counter = SimpleCounter(key_name=key_name, name=key_name, count=count)
    else:
      counter.count += count
    counter.put()
    pingpong_delete(key_name)

  def get(self):

    if CapabilitySet('datastore_v3', capabilities=['write']).is_enabled() \
        and CapabilitySet('memcache').is_enabled():
      self.update_counter('scratches')
      self.update_counter('itch_gets')

1   An old memcache bug

When I was writing this code, I hit a mine with the following code:

count = memcache.incr(key_name, initial_value=0)
real_count = memcache.get(key_name)

There is no such item in memory cache when first run, so we need initial_value to set the number to be increased, or nothing will happen. If you print out the values, count=1L (long type), which is expected, but real_count='1' (str type), which is not we are expecting to have.

If you look into SDK code, line 265 in google/appengine/api/memcache/memcache_stub.py, the end of def _internal_increment(),

new_value = max(old_value + delta, 0) % (2**64)

entry.value = str(new_value)
return new_value

, and the line 245 in the middle,

self._the_cache[namespace][key] = CacheEntry(str(request.initial_value()),

When you uses .incr(), the return value will be new_value, but if you .get(), you get that str type value. The most interesting thing is, if you run

memcache.set('the_key', 0)
count = memcache.incr('the_key')
real_count = memcache.get('the_key')

Both count values will be long 1.

There is a bug report, fired one and a half years ago and has been acknowledged almost a year. I thought maybe it doesn’t affect on production server, but I did a test, this both affect on both production and development servers.

You can always do

count = long(memcache.get(key_name))

But it’s so awkward. Instead, I currently use a more awkward code to resolve this issue,

if memcache.incr(key_name) == None:
  # Can't set, no such key existed
  memcache.set(key_name, 1)