Good Unix-related blog

Vladimir Melnikoff brought his blog to my attention: "Nothing but Unix". Good resource for Unix enthusiasts, mostly composed of industry-related news.

Python Fuzz Testing Tools

Ian Bicking suggested I create a new category in the Python Testing Tools Taxonomy: Fuzz Testing or Fuzzing. Done. If you're not familiar with the term, see the Wikipedia article which talks about this type of testing. Here's an excerpt: "The basic idea is to attach the inputs of a program to a source of random data ("fuzz"). If the program fails (for example, by crashing, or by failing built-in code assertions), then there are defects to correct. The great advantage of fuzz testing is that the test design is extremely simple, and free of preconceptions about system behavior."

Ian told me about the Peach Fuzzer Framework. I was familiar with Pester (the home page talks about a Java tool called Jester, and it has links to the Python version called Pester); I also googled some more and found other Python fuzzing tools such as antiparser and Taof, which are both geared towards fuzzing network protocols. In fact, many fuzzing tools are used in security testing because they can aid in attacking software via random inputs. See this Hacksafe article on "Fuzz testing tools and techniques" and this PacketStorm list of fuzzing tools. Another good overview is Elliotte Harold's developerWorks article on fuzz testing. Very interesting stuff. If the "Python Testing Tools" tutorial Titus and I proposed for PyCon gets accepted, expect to see some fuzz testing included in our arsenal :-)

I also added Ian's minimock tool to the PTTT page. Very cool minimal approach to mock testing, achieved by embedding mocking constructs in doctests.

In other testing-related blog posts, Titus talks about the difficulty of retrofitting testing to an existing application (even when you wrote the testing tools!), and Max Ischenko presents some uber-cool plugins which integrate nose into vim.

"Swap space management" article at IBM developerWorks

From IBM developerWorks, a very nice summary of the issues involved in setting up and maintaining swap space on *nix systems: "Swap space management and tricks".

Daniel Read on software and Apgar scores

Daniel Read blogs on the topic: "Does software need an Apgar score?". He mentions the fact that a simple metric (the Apgar score for newborns) revolutionized the childbirth process, "through standardization of techniques, training, and regulation of who exactly was allowed to perform certain procedures (based on whether they had the training and experience)". He then talks about how a similar simple score might help the quality of software development, by assessing its "health". Hmmm... all this sounds strangely familiar to me -- Cheesecake anybody? Of course, Daniel accepts that this idea is highly controversial and maybe a bit simplistic. However, I for one am convinced that it would help with improving, if not the quality, then at least the kwalitee of the software packages we see in the wild today.

Got Edge?

I mean Edgy. I mean Edgy Eft. Get it.

Update

I followed the EdgyUpgrades document from the Ubuntu Wiki and all it took to upgrade my Dell laptop from Dapper to Edgy was one command:

gksu "update-manager -c" 

I call this painless. Everything seems to be working just fine after the upgrade. Haven't had time to play with it at all -- in fact, I don't even know how long the upgrade process took, since I left after I started it.

Proposal for Testing Tools Panel at PyCon07

Following Titus's example with his Web Frameworks Panel proposal for PyCon07, I proposed a Testing Tools Panel. And yes, I expect the author of twill to participate and take questions :-)

I created a TestingToolsPanel page on the PyCon07 wiki. Please feel free to add your own testing-related topics of interest and/or questions for the authors. If you are a testing tool author, please consider participating in the panel. You can leave a comment here or send me an email (grig at gheorghiu.net) and let me know if you're interested in participating.

Here's what I have so far on the Wiki page:

I maintain a "Python Testing Tools Taxonomy" (PTTT) Wiki page.

Here are some of the tools listed on the PTTT page:

• unit testing tools (unittest/doctest/py.test/nose/Testoob/testosterone)
• mock/stub testing tools (python-mock/pmock/stubble)
• Web testing tools (twill/webunit/zope.testbrowser/Pamie/paste.test.fixture)
• acceptance testing tools (PyFit/texttest/FitLoader)
• GUI testing tools (pywinauto/WATSUP/winGuiAuto/guitest)
• source code checking tools (pylint/pychecker/pyflakes)
• code coverage tools (coverage/figleaf/Pester)
• other miscellaneous testing tools (pysizer/pymetrics/svnmock/testtools)

I propose to have a panel where authors of some of these tools would discuss and take questions on topics such as:

• what need prompted the creation of the testing tool
• what type of testing does the tool belong to (unit, functional, acceptance, system, performance)
• what specific features does the tool offer that other tools lack
• what are the most common testing scenarios you have seen in your user base
• are there any platform- or OS-specific gotchas related to the tool
• how extensible is the tool (plugins etc.)
• how easy to learn is the tool
• how well tested is the tool
• how well documented is the tool

"Agile in action" photostream

From a blog I read with pleasure, Simon Baker's "Agile in action", here's a link to a Flickr photostream that shows in my opinion what agile is all about: collaboration, camaraderie, storytelling....in short, having great fun and producing great software in the process. Stevey, I can tell you've never been part of an agile team in your life -- otherwise why would you be so bitter and cranky about it?

The 90-9-1 rule and building an open source community

Jakob Nielsen talks about the 90-9-1 rule in his latest Alertbox newsletter: "Participation inequality: encouraging more users to contribute". Simply put, the rule states that in a typical online community, 90% of the users are lurkers, 9% are occasional contributors, and only 1% are active contributors. This should be interesting for people trying to build and grow open source projects. Nielsen has some suggestions to offer on how to overcome this "participation inequality". Read the article for his suggestions.

Here are some of my own observations and lessons learned from various open source efforts I've been part of (many of them are things I've tried to do on the Pybots project):

How to build an open source community

* Blog, blog, blog
* Send messages to mailing lists related to the area of your project
* Write extensive documentation, make it easy for people to join
* Create a project repository (Google Code)
* Get help from early adopters, involve them in the project

How to sustain and grow an open source community

* Blog, blog, blog
* Send messages to individuals who might be interested in contributing
* Acknowledge contributions
* Respond quickly to issues on mailing list
* Demonstrate usefulness of the project, both to contributors, and to any organizations involved (e.g. the PSF)
* Market/promote/evangelize the project tirelessly
* Recommended reading: "Fearless change: Patterns for introducing new ideas" by Mary Lynn Manns and Linda Rising

Comments about your own experience in building an open source community are much appreciated.

Let's celebrate Roundup by turning it into the official Python bug tracker

Richard Jones just posted a note about Roundup turning 5. What better birthday gift than turning it into the official Python bug/issue tracker. Readers of Planet Python certainly know by now that there are 2 issue trackers in contention: JIRA and Roundup. Unless people step up to volunteer as admins for maintaining a Roundup-based Python issue tracker, the PSF will choose JIRA. I walked the walk and volunteered myself. I know there must be other people out there who would like to see a Python-based project be selected over a Java project. Any takers? Send an email by Oct. 16th to infrastructure at python.org stating your intention to volunteer. All it takes is 8 to 10 people.

Pybots news

I'm happy to report that the Pybots project continues to gain momentum. In raw numbers, we have 8 buildslaves running the automated tests for 17 projects, and also testing the installation of 18 other packages. Pretty impressive, if I may say so myself. This table is copied from the main pybots.org page and shows the current setup:

Builder name	Project(s) tested	Pre-requisites installed	Owner
x86 Red Hat 9	Twisted	setuptools, zope.interface, pycrypto, pyOpenSSL	Grig Gheorghiu
x86 Debian Unstable	docutils, roundup	N/A	Seo Sanghyeon
x86 Ubuntu Dapper	parsedatetime	setuptools	Mike Taylor (Bear)
x86 OSX	vobject, zanshin	setuptools, zope.interface, Twisted	Mike Taylor (Bear)
x86 Gentoo	pysqlite, Genshi, Trac, feedvalidator	clearsilver, pysqlite	Manuzhai
amd64 Ubuntu Dapper	MySQLdb, CherryPy	N/A	Elliot Murphy
x86 Windows 2003	lxml (dev, stable) Bazaar (dev, stable)	libxml2, libxslt, zlib, iconv	Sidnei da Silva
x86 Ubuntu Breezy	Cheesecake	setuptools, nose, logilab-astng, pylint	Grig Gheorghiu

Some more projects and buildslaves are in the pipeline, so I hope to be able to announce them soon. I'd like to thank all the contributors so far, in chronological order of their contributions: Seo Sanghyeon, Mike Taylor (Bear), Manuzhai, Elliot Murphy and Sidnei da Silva.

People interested in this project -- whether they'd like their project to be tested on an existing buildslave, or they'd like to contribute a buildslave -- are encouraged to peruse the documentation on the Pybots page, then send a message to the Pybots mailing list.

Notepad++ rocks

I heard about Notepad++ from Michael Carter, who showcased SQLAlchemy at our last SoCal Piggies meeting and used Notepad++ to edit his Python files. I downloaded it this week and all I can say is that it rocks! For Windows users, it's one of the best editors I've ever seen, and of course it's completely free. Notepad++ is based on the Scintilla editor, and it does syntax coloring for a gazillion languages (Python included of course), it includes a huge number of plugins, such as a Hex Editor, etc., etc. Highly recommended!

Buildbot used for continuous integration in the Gnome project

José Dapena Paz has a nice write-up on how the Gnome Build Brigade is using buildbot for continuous integration of all the projects under the Gnome umbrella. Buildbot scores again!

Any projects that need Pybots buildslaves?

Know of any projects that need Pybots buildslaves?

This is a question that's been asked twice already on the Pybots mailing list, or in emails addressed directly to me. Every time I answered that I don't really know any, and I advised the people asking the question to go through the list of their favorite Python projects, and pick some that they'd like to test. Or go through the list of major Web frameworks and pick some.

Anyway -- maybe a better way is to ask all the readers of this blog: do you have a project that you'd like to test in the Pybots buildbot farm, but maybe you don't have the hardware to run the buildslave on? Then let me know, or even better, let the Pybots mailing list know. There are people such as Jeff McNiel and Skip Montanaro who have hardware and resources, and are looking for projects to test.

BTW, the Pybots farm now has 6 buildslaves (the 6th was courtesy of Elliot Murphy, who contributed an AMD-64 Ubuntu Dapper box running the MySQLdb tests), with 3 or 4 buildslaves on the way.

Thanks a lot to all the people who have contributed or will contribute soon. It's very rewarding to see the Python community responding in an enthusiastic manner to a call to do better testing of Python projects.

Titus has a new blog

If you've been reading Titus's advogato blog, you'll be interested in knowing that he has a new blog: "Daily Life in an Ivory Basement". Will Guaraldi will be happy to know the blog is based on pyblosxom.

Pybots project keeps rolling

Some updates on the Pybots project:

• Mike Taylor aka Bear from OSAF contributed 2 buildslaves: an Ubuntu Dapper box for testing his parsedatetime package, and an Intel Mac OSX box for testing two libraries used by OSAF -- vobject and zanshin
• Manuzhai contributed a Gentoo box for testing pysqlite and Trac
• Elliot Murphy from mysql.com will contribute an AMD-64 Ubuntu Dapper box for testing MySQLdb
• In summary, we're up to 5 (soon to be 6) buildslaves, with more on the way
  
• Seo Sanghyeon created a Google Code project for pybots; he and I are the current admins for this project; you can browse the Subversion repository for various scripts and buildbot config. files
• As a cool side note, Sanghyeon rewrote the home page for pybots.org in reST; the page is kept in subversion and the server hosting pybots.org is doing a svn update every hour, followed by a rest2html call

As I said in a previous post, the Pybots setup already proved its usefulness by uncovering issues with new keywords such as 'with' and 'as'. Some of the projects affected by these new keywords are zope.interface, roundup and zanshin. According to the Zope developers, the issue had been fixed a while ago in the svn repository, but no release has happened since. The roundup developers already fixed the issue -- nice to see this.

Also, the Trac unit tests, when running against the latest Python trunk, are failing with an ugly backtrace. If somebody can shed some light, please do.

Again, if you're interested in running a Pybots buildslave, take a look at the various pieces of documentation at pybots.org and send a message to the Pybots mailing list.

Pay attention to the new 'with' and 'as' keywords

As of Sept. 6th (revision 51767), Python 2.6 has two new keywords: with and as. Python code that uses either one of these words as a variable name will be in trouble. How do I know that? Because the Twisted unit tests have been failing in the Twisted Pybots buildslave ever since. Actually the issue is not with Twisted code, but with zope.interface, which is one of the Twisted pre-requisites. Here's the offending code:

Traceback (most recent call last):
File "/tmp/Twisted/bin/trial", line 23, in 
from twisted.scripts.trial import run
File "/tmp/Twisted/twisted/scripts/trial.py", line 10, in 
from twisted.application import app
File "/tmp/Twisted/twisted/application/app.py", line 10, in 
from twisted.application import service
File "/tmp/Twisted/twisted/application/service.py", line 20, in 
from twisted.python import components
File "/tmp/Twisted/twisted/python/components.py", line 37, in 
from zope.interface.adapter import AdapterRegistry
File "/tmp/python-buildbot/local/lib/python2.6/site-packages/zope/interface/adapter.py", line 201
for with, objects in v.iteritems():
       ^
SyntaxError: invalid syntax

It would be great if the Zope folks fixed their code so that the Twisted tests will start passing again. This issue actually impacts all packages that depend on zope.interface -- for example zanshin, which also fails in the Pybots buildslave for the OSAF libraries (graciously set up by Bear from OSAF).

If you're interested in following such issues as they arise in the Pybots buildbot farm, I encourage you to subscribe to the Pybots mailing list.

Update

I mentioned the 'with' keyword already causing problems. As it turns out, Seo Sanghyeon's buildslave, which is running tests for docutils and roundup, uncovered an issue in roundup, related to the 'as' keyword:

Traceback (most recent call last):
 File "run_tests.py", line 889, in 
   process_args()
 File "run_tests.py", line 879, in process_args
   bad = main(module_filter, test_filter, libdir)
 File "run_tests.py", line 671, in main
   runner(files, test_filter, debug)
 File "run_tests.py", line 585, in runner
   s = get_suite(file)
 File "run_tests.py", line 497, in get_suite
   mod = package_import(modname)
 File "run_tests.py", line 489, in package_import
   mod = __import__(modname)
 File "/home/buildslave/pybots/roundup/./test/test_actions.py", line 6, in 
   from roundup import hyperdb
 File "/home/buildslave/pybots/roundup/roundup/hyperdb.py", line 29, in 
   import date, password
 File "/home/buildslave/pybots/roundup/roundup/date.py", line 735
   as = a[0]
    ^
SyntaxError: invalid syntax

Pybots update

I'm happy to report that the Pybots project got its first user other than yours truly. Seo Sanghyeon set up a buildslave running on Debian Unstable which is running the docutils unit tests every time a checkin is made into Python trunk or in the 2.5-maint branch.

Marc-Andre Lemburg also offered to run a buildslave for the egenix-mx-base tests, while Manuzhai offered to run a buildslave for the Trac tests. Skip Montanaro also expressed interest in running a buildslave, but he hasn't decided on a project yet.

You can see the current pybots buildslaves here. Expect to see more active buildslaves in the next few days.

Marc-Andre suggested I write a Pybots FAQ and put some info on the Python wiki, so here they are:

• Pybots page on Python wiki
• Pybots FAQ on Python wiki
  

On the importance of functional testing

I did not need further proof of the fact that functional tests are a vital piece in a project's overall testing strategy. I got that proof anyway last night, while watching the Pybots buildmaster status page. I noticed that the Twisted unit tests were failing, but not because of errors within the Twisted package, but because pre-requisite packages such as ZopeInterface could not be installed anymore. If you followed my post on setting up a Pybots buildslave, you know that before running the Twisted unit tests, I attempt to instal ZopeInterface and other packages using the newly-built python binary, via "/tmp/python-buildbot/local/bin/python setup.py install".

Well, all of a sudden last night this last command was failing with errors such as:

error: invalid Python installation:
unable to open /tmp/python-buildbot/local/lib/python2.6/config/Makefile
(No such file or directory)

This proved to be was a transient error, due to some recent checkins that modified the release numbers in the python svn trunk from 2.5 to 2.6. This issue was fixed within an hour, but the interesting thing to me was that, while this step was failing in the Pybots Twisted tests, the Python builbots running the Python-specific unit tests against the trunk were merrily chugging along, green and happy (at least on Unix platforms). This was of course to be expected, since nothing major changed as far as the internal Python unit tests were concerned. However, when running a functional test involving the newly-built Python binary -- and in my case that functional test consisted simply in running "python setup.py install" on some packages -- things started to break.

Lesson learned? Always make sure you test your application from the outside in, by exercising it as a user would. Unit tests are necessary (indeed, they are essential), but they are not sufficient by any means. A 'holistic' test strategy takes into consideration both white-box-type unit tests, and black-box-type functional tests. Of course, the recommended way of running all these types of tests is via a continuous integration tool such as buildbot.

QA blog at W3C

Karl Dubost sent me a message about some issues he had with running Cheescake on a Mac OS X machine. It turned out he was using an ancient version of Cheesecake, although he ran "easy_install Cheesecake". I told him to upgrade to the latest version via "easy_install Cheesecake==0.6" and his problems dissapeared.

Anyway, this is not what I was trying to blog about. Reading his email signature, I noticed he works as a Conformance Manager at W3C. Karl also mentions a QA blog at W3C in his signature. Very interesting blog, from the little I've seen so far. For example, from the "Meet the Unicorn" post, I found out about a W3C project (code-name Unicorn) which aims to be THE one tool to use when you want to check the quality -- i.e. the W3C conformance I suspect -- of web pages. This tool would "gather observations made on a single document by various validators and quality checkers, and summarize all of that neatly for the user." BTW, here is a list of validators and other test tools that you can currently use to check the conformance of your web pages.

Added the blog to my fluctuating collection of Bloglines feeds...Thanks, Karl!

Setting up a Pybots buildslave

If you're interested in setting up a buildbot buildslave for the Pybots project, here are some instructions:

Step 1

Install buildbot on your machine. Instructions can be found here, here, here and here.

Step 2

Create a user that will run the buildbot slave process. Let's call it buildslave, with a home directory of /home/buildslave. Also create a /home/buildslave/pybot directory.

Step 3

Create the file buildbot.tac in /home/buildslave/pybot, with content similar to this:

from twisted.application import service
from buildbot.slave.bot import BuildSlave

# set your basedir appropriately
basedir = r'/home/buildslave/pybot'
host = 'www.python.org'
port = 9070
slavename = 'abc'
passwd = 'xyz'
keepalive = 600
usepty = 1

application = service.Application('buildslave')
s = BuildSlave(host, port, slavename, passwd, basedir, keepalive, usepty)
s.setServiceParent(application)

Step 4

Create a python-tool directory under /home/buildslave/pybots. You must name this directory python-tool, as the buildmaster will use this name in the build steps.

Step 5

Create a file called run_tests.py under the python-tool directory. This is where you will invoke the automated tests for your projects.

How this all works

The buildmaster will have your buildslave execute the following steps, every time a check-in is made into the python subversion trunk (and also every time a check-in is made in the 2.5 branch):

1. Update step: runs "svn update" from the python svn trunk
2. Configure step: runs "./configure --prefix=/tmp/python-buildbot/local"
3. Make step: runs "make all"
4. Test step: runs "make test" (note: this step runs the python unit tests, not your project's unit tests)
5. Make install step: runs "make install"; this will install the newly-built python binary in /tmp/python-buildbot/local/bin/python
6. Project-specific tests step: this is when your run_tests.py file will be run via "/tmp/python-buildbot/local/bin/python ../../python-tool/run_tests.py"
7. Clean step: runs "make distclean"

Important note: since your automated tests will be run via the newly-built python binary installed in /tmp/python-buildbot/local/bin/python, you need to make sure you install all the pre-requisite packages for your package using this custom python binary, otherwise your unit tests will fail because they will not find these pre-requisites. For example, for the Twisted unit tests, I had to install setuptools, ZopeInterface, pycrypto and pyOpenSSL, before I could actually run the Twisted test suite.

So in my run_tests.py file I first call a prepare_packages.sh shell script, before I launch the actual test suite (I copied the pre-requisite packages in /home/buildslave):

$ cat prepare_packages.sh

#!/bin/bash

cd /tmp

rm -rf setuptools*
cp ~/setuptools-0.6c1.zip .
unzip setuptools-0.6c1.zip
cd setuptools-0.6c1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf ZopeInterface*
cp ~/ZopeInterface-3.1.0c1.tgz .
tar xvfz ZopeInterface-3.1.0c1.tgz
cd ZopeInterface-3.1.0c1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf pycrypto-2.0.1*
cp ~/pycrypto-2.0.1.tar.gz .
tar xvfz pycrypto-2.0.1.tar.gz
cd pycrypto-2.0.1
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf pyOpenSSL-0.6*
cp ~/pyOpenSSL-0.6.tar.gz .
tar xvfz pyOpenSSL-0.6.tar.gz
cd pyOpenSSL-0.6
/tmp/python-buildbot/local/bin/python setup.py install
cd ..

rm -rf Twisted
svn co svn://svn.twistedmatrix.com/svn/Twisted/trunk Twisted

Then I call the actual Twisted test suite, via:

/tmp/python-buildbot/local/bin/python -Wall /tmp/Twisted/bin/trial --reporter=bwverbose --random=0 twisted

You can see the current Pybots status page here.

If you are interested in setting up your own buildslave to participate in the Pybots project, please send a message to the Pybots mailing list. I will send you a slavename and a password, and then we can test the integration of your buildslave with the buildmaster.

Update 10/16/09

I realized that these instructions for setting up a Pybot buildslave are a bit outdated. Discussions on the Pybots mailing list prompted certain changes to run_tests.py, even though you're still OK if you follow the instructions above to the letter.

Here are some enhancements that you can take advantage of:

1. You can test several projects, each in its own build step, simply by having your run_tests.py script be aware of an extra command-line argument, which will be the name of the project under tests. An example of such a script is here: run_tests.py. The script receives a command-line argument (let's call it proj_name) and invokes a shell script called proj_name.sh. The shell script checks out the latest code for project proj_name (or downloads the latest distribution), then runs its unit tests. Here is an example: Cheesecake.sh.

2. You do not have to hardcode the path to the newly built Python binary in your run_tests.py or your shell scripts. You can simply retrieve the path to the binary via sys.executable. This run_tests.py script sets an environment variable called PYTHON via a call to

os.putenv('PYTHON', sys.executable)

Then the variable is used as $PYTHON in the shell scripts invoked by run_tests.py (thanks to Elliot Murphy and Seo Sanghyeon for coming up with this solution.)