YJL

Last time, I did a test on simplejson 2.1.3, it was actually a post for writing this one, I want to show you using feedparser causes quite some time.

Last month, feedparser 5.0 was released and just few days ago 5.0.1 was released. Before version 5, I was using 4.1, which I found it’s slow. But version 5.0 is even slower in Python 2.5 , much more more slower in Python 2.6. 5.0 claims to support Python 3, but Python 3 says the module has invalid syntax. So, I won’t be knowing if it knows faster in Python 3.

Here is how I test. I downloaded 500 entries of Google Blog using Blogger API for format Atom and RSS, I also downloaded the JSON for comparison:

http://www.blogger.com/feeds/10861780/posts/default?max-results=500&alt=atom
http://www.blogger.com/feeds/10861780/posts/default?max-results=500&alt=rss
http://www.blogger.com/feeds/10861780/posts/default?max-results=500&alt=json

Because the execution/parsing time is really long, I simply used Bash’s time built-in to get the time. The commands are as follow:

time python -c 'import feedparser as fp; fp.parse("test.atom")'
time python -c 'import feedparser as fp; fp.parse("test.rss")'
time python -c 'import simplejson as json; json.load(open("test.json"))'

Here is the result:

I don’t think I need to explain the number, it’s very clear. I listed simplejson because I also did this:

simplejson + YQL for 100 entries: 4.445s

Using a YQL like:

select * from atom where url="http://www.blogger.com/feeds/10861780/posts/default?max-results=100&start-index=1&alt=atom"

and:

time python -c 'import simplejson as json; from urllib2 import urlopen; json.load(urlopen("http://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20atom%20where%20url%3D%22http%3A%2F%2Fwww.blogger.com%2Ffeeds%2F10861780%2Fposts%2Fdefault%3Fmax-results%3D100%26start-index%3D=1%26alt%3Datom%22&format=json&callback="))'

I found if I used 500 entries, YQL returns empty result. 250 seems to be the limit but simplejson shows error in the JSON. I don’t know if YQL causes that or not, but 100 with 5 requests seems fine. The time is around 1 to 5 seconds, so multiply by 5 requests, the estimation will be upto 25 seconds in Python 2.6. And most of time is waiting for response from YQL, which doesn’t utilize much CPU resource.

YQL is a great tool I have known. You can load a cross-domain XML file or get the HTTP error code for cross-domain JSONP. You can convert any format it supports to JSON.

feedparser is a great tool, too, but it’s really slow. I noticed because I have a script which loads many feeds using feedparser every ten minutes and it always is the TOP 1 CPU time user. After this test, I don’t think I will use 5.0 and I will probably go back to use Python 2.5 to run the script. I might even switch to process JSON with YQL’s help since the data is downloaded, it would be just a URL change.

I know it’s unfair to say feedparser is slow, if you only look at the time without considering the formats or the implementation, then it’s slow. And speed is only thing I do care. If there is one library implemented in C, I am sure the time will have huge improvement. I tried to search for one, but I can’t find any.