The last time I did this, the version of Django was 0.9 and today’s version is 1.3.1. As a I started work on the site I discovered some things here and there that had changed in the years in-between those version. The one that bit me the hardest was this: I could view the main page (/), but any other page (including index) would give me a 404 error.

This was really weird because the mappings existed in the urls.py file, but a 404 error means it couldn’t be found. If it was a programmatic error you would expect a 500 error, but I never saw one.

Slowly but surely I tore the system apart looking to see what could cause mapped urls to disappear in Django. Eventually I discovered the root cause was how static resources are handled.

One of the changes in the new versions of Django was to add functionality for handling static resources. The biggest change (at least as far as my ancient code was concerned) was in introduction of the STATIC_URL variable. According the documentation this is the prefix for static resources (like css, javascript) that are referenced by the web page that is being built by Django.

In my laziness many years ago, I just plopped the main CSS file (and everything else) in the root of the web directory so that in the html pages didn’t have to have paths on them. So when I was adding the STATIC_URL variable to make sure my CSS file would actually load, I set the variable this way:

STATIC_URL = ‘/’

…because it wouldn’t let me put an empty string of ”. Well, it turns out that this was wrong wrong wrong. Django was looking at the incoming requests as I tried to visit the pages of the site and checking the static directory to see if there was anything there (which there wasn’t, just my CSS file).

Once I realized this, I changed it to:

STATIC_URL = ‘/static/’

…which basically forces you to create a directory to hold your static files. Which is right right right right. Basically Django was forcing me to clean up my act and code the site in a much cleaner manner. As soon as I did that all of the 404 errors went away and I was able to hit every page in my urls.py file.

I was pretty pleased with myself that I knew a lot of the things listed, and I was really happy when I ran into some nuggets that I knew nothing about. For example, I did not know that you could do this in Python:

1 < x <10

To me, that is really awesome because not only is what you would write in a mathematical sense, but it also is much more concise and what you would see in a pseudo-code representation of a program. That is the power of Python, taking an idea (a computer program) and presenting it in a readable manner. Being able to use the above type of expression just takes us further down that path of readability and that is a good thing. And yes, you can write crappy code in Python, but at least you have the choice: many languages simply have one way to do things. Choices are good.

And speaking of choices, I thought this feature was very cool, even if I would never use it:

from __future__ import braces

That allows the use of {} instead of white space in a Python program. I know some people can’t stand the indention rules of the language and feel more comfortable when they are “braced” (for impact?), but the fact the language has features to be able to turn this on? That is 1 million pounds of awesome in  a 5 pound sack.

There is a ton of other neat tricks there, go and read up and see what you can do to make your code work more for you! Choice is a great thing. With Python, you have lots of great choices you can make. And if you decide to make no choices, you still wind up with pretty readable code which is always a good thing.

If you are going to go down the Linux route with AWS (and honestly why wouldn’t you?), then you are in luck since python ships on most modern distributions. But, while python is “batteries included” there are some really nice projects out there that it is good to take advantage of. Installing these projects is where you start to find out there are differences in the EC2 AMI’s.

My first attempt was with the default Amazon AMI. It seemed like a good choice, but the more I played with it, the more I realized that it was pretty bare-bones. easy_install is included, but when running it to try and install something you run into permissions problems.

Because this is supposed to be a nice experiment, I only played around with it for a hour or so. My patience exhausted, I went looking for an alternative and ran right into the perfect match: ActiveState has an AMI specifically for python developers

This is awesome because ActiveState puts out a really nice Windows package for Python (and other lesser languages). I fired up their AMI instance and have found it to be really nice. I put my python project out there earlier this month and it ran without a problem… until the code crashed because I forgot to put a try catch block in place.

Seriously though, I really like this AMI. Not only is a nice python distribution there, but they also included gcc so if you need to compile something, you don’t have to go far. The easy_install script runs like a dream, and I haven’t had any problems with it at all. In fact, my whole experiment (which seems to getting attention mostly from blog aggregators at the moment) has only cost me a total of $0.19. Yes, only 19 cents! And 1 cent of that was from the ActiveState Python AMI (the other 18 cents was from my playing around with the Amazon AMI that wasn’t that great).

Now to get back to my ideas, and sling some python code out into the cloud…

Specifically, I’m taking an existing code base (the AIAA SGP4 code which is written in C/C++), wrapping it with python, and then eventually putting a web front end on it. The end goal is that astronomers will be able to use this code/website to check their observations to see if there was a satellite in their field of view when they took their measurements.

Since I’m using a known library in to create something new, this seemed like a really great opportunity to use some Test Driven Development (TDD) to ensure that as I build up the parts of the system, the numbers stay true and that the code stays honest.

This is working out really well. Combining python’s unit test module with nosetests allows me to quickly bang out a test to make sure that the positional numbers that the SGP4 code generates is what the documentation says it should be. By using TDD I can map out where the code needs to go a little better: As I find something that is missing I can write out a quick test that will fail. When I run nosetests, I have that failure glaring at me telling me where more code is needed.

For example, I have one set of test data that gives the observer’s location in standard latitude and longitude coordinates. My code isn’t quite expecting this, the MPC format is a little bit different. This is a problem that is easy to fix, but I have a plethora of problems at the moment and I need to make sure this one gets taken care of properly. So I opened up the python file that tests the Observatory class and added in a failing test that called testLatLongToXYZ(). When I run the tests I now see that this is a failing method which gives me the mental kick in the pants to go and implement that method, even if it is just a stub.

Why do I say just a stub? Because putting in the stub lets  me do two things: 1) I’ve now got something there, and 2) I can now update the test to call that stub. The secret is that the test should now call the stub and expect it to pass, but the stub should always return something (None, False, -1, etc.) to indicate that it failed.

This acts as a pointer to an area that needs improvement in the code. This is what TDD is all about. You keep repeating this process until you converge on a “correct” answer which is a method that does just what it needs to do. In theory, and so far my practice confirms this, the resulting code should be smaller and more accurate.

For me this is a great thing because it encourages code that is more honest. What is honest code? To me, it is code that does what it says and nothing else. The shorter the code is, the more focused it is. The python language is very expressive and allows you to do a lot without saying a lot. This power can lead you to try and do too much in one function. Once you have a function that not only converts Lat and Long, but also converts polar coordinates and writes satellite data to the database, can you really honestly call that function convertLatLongToXYZ()?

This is especially valuable in the scientific context of this astronomy related code. The code is open source, and in the event there is a problem with it, as people dig into the code they need to have confidence that the code is relatively well written. At a high level the tests give them the numerical sense of ease that it is working when the tests run and show the proper numbers being returned. At a low level, seeing that the code is broken up into small logical sections gives the confidence that once  problem is found it can be corrected, tested, and shown that there are minimal side effects in other parts of the program.

As the lone developer on this project, this type of check-and-balance gives me a lot of hope that I’ll be able to produce something that is accurate and useful to the community at large. Openness and honesty help build confidence and trust. If you are interested in the project, be sure to check it out:

ObsSatId: A python astronomy project to wrap the SGP4 code to check for satellites

I love the python language, but as I blogged about a while ago, not having overloaded methods in python is kind of a bummer. But do you have to trade your dynamic freedom in for static types to gain this? It turns out that with a little ingenuity, you can get that functionality using the generic dispatch pattern and python’s decorator syntax.

The example on the above wiki page is nice and concise, but I think that Alex Gaynor’s blog post about multimethods is the best example of how to implement this useful idea. In leiu of an explanation of how it works, Alex gives us a unit test that exercises the mutimethod class and the dispatch/overloading idea that it brings to life. As long as you know what a decorator is the code does make sense.

There are other examples of using a dispatch, but I feel that Alex’s is more clear. These are of course no substitute for having function overloading built into the python language, but having the functionality wrapped up in a nice neat class like this is a very good thing.

Fatal Python error: Interpreter not initialized (version mismatch?)

Abort trap

Fatal Python error: Interpreter not initialized (version mismatch?)

Abort trap

gcc myfile.c -lpython -shared -o mylib.so

gcc myfile.c -framework python -shared -o mylib.so

At the time it seemed like the obvious choice was to get something (hgsubversion) that would let hg talk directly to the svn server, but still allow me to explore DVCS workflows. It turns out that this isn’t the case. Luckily there are several different approaches that can be used.

I coulda been a contender…

The idea behind hgsubversion is to use hg as your version control tool. This extension lets it “talk” to svn servers, so you only need one tool. It sounds like a really great idea, but I ran into some stumbling blocks that while not necessarily show stoppers, they did cause me to look around for alternatives (more on that later). Here’s the issues, good and not-so-good, that I had with this extension:

• Poor support in cygwin – Installing this on cygwin was a bitch-and-half. Getting all of the pieces to compile and install was a major act that I just couldn’t pull off. Which is a bummer, I do love me some command line. However, TortoiseHg to the rescue!
• It works great on the mac – I usually have the most trouble installing new stuff on my mac, but hgsubversion works like a champ with no problems.
• TortoiseHg rocks - This program is amazing. It worked correctly right out of the box, and took no effort to get up and running on my windows machine. It really let me get started on this little experiment.

Overall I really like using hg. Making it work with svn has been a big bonus because now I can get the best of both worlds without having to abandon svn. (That’s a big bonus as most day jobs rely on svn!)

The ability to branch quickly and in a lightweight manner is one of the major selling points of systems like Mercurial and Git. Recently I’ve been thinking about trying out some refactorings and that can be difficult to do when you need to make sure that you can have a build-able version of your code at all times. Some things just take time, and the whole point of refactoring is to improve your code so rushing through it to meet an unrelated code deadline just doesn’t make a lot of sense.

Enter Mercurial.

There’s a project for Mercurial called hgsubversion that will allow you to pull from a subversion repository and make a mercurial repository locally. (Yes, git has something similar) Then you can hack away to your heart’s content using hg to branch and keep track of changes without pushing the changes out globally to the other SVN users.

This is exactly what all of the cool kids using hg, git, and bzr have been doing for years. Now those of us who talk to SVN can leverage this technique to bring a little more awesomeness to our day-to-day work, and no one is the wiser. At least that’s my theory.

Installing hgsubversion on a Mac or in Cygwin is like pulling teeth. I take that back, pulling teeth is not as painful. Plus you can leave your teeth under your pillow for some cash… but I digress. The best way to install it (for me at least) is to do the following:

• Have the following installed: mercurial, easy_install, the XCode tools (command line tools like gcc)
• Get a copy of the python swig bindings for Subversion. Collab has prebuilt binaries that seem to work best. I just copied the bindings from /opt/subversion/lib/svn-python/ to my python site-packages directory. I installed Python 2.6, so for me that path is: /Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/site-packages
• Get hgsubversion by typing at the command line: hg clone http://bitbucket.org/durin42/hgsubversion/ ~/hgsubversion
• cd into ~/hgsubversion/ and type in python setup.py install
• For me, the system had to download a few things and build them, so make sure you have XCode and the 10.4 SDK installed. (I’m running XCode 3.2.1 for this exercise)
• Hopefully everything completes normally. If it does, do a victory shot. If it fails, do 2 shots and then try to figure out what went wrong.
• Next make a file in your home directory called .hgrc
• Inside the file put the following:

[extensions]
rebase=
svn=/Users/<YOUR USERNAME>/hgsubversion/hgsubversion

• This should tell hg that you’ve got something extra for it. I went blind sometime during my attempts to get this working and had a typo in that line that kept me from getting it working for an embarrassingly long time.
• If you have done everything right, the moon is in the proper phase, and the wind is blowing from the NW at 7 mph, then this command should let you pull down a project from Google code:

hg clone svn+http://<A PROJECT OF YOUR CHOICE>.googlecode.com/svn <WHATEVER DIR>

• The end result should be a mercurial repository on your local machine. Refer to the link for hgsubversion for more details about commands and links to other installation instructions. (Some are older than others, and the commands have changed a little bit over time.)

So, that’s what I’m up to. Hopefully this will let me do some interesting experiments locally without sacrificing the safety blanket of using version control.

Developing with small data sets is fine, and most of the time that is what you want to do as you work out the kinks of the code. But when it is time to ship the code, you must test with a large data set.

For rent my resume.com I’ve been testing one portion of it with 4 different size documents. This is a fluke however, I chose the documents based on the contents not the size. Today I made a simple change that turned out to have some unforeseen ripple effects.

When I write unit tests, they seem to fall into two categories: Tests where I just do a simple check on the size of a return (i.e. did I get 21 items in the list?), and tests where I check the contents of the resulting data.

As a side note, you should really do both kinds of tests, not one or the other. Simply checking to see that the right number of things was returned is no substitute for making sure that the correct data was actually returned!

Today I was re-running my pyunit tests and one of the four failed because the size of the returned list wasn’t what was expected. By coincidence this happened to be the biggest data set I was testing with. If I had not had this test, I would have thought everything was ok, but it truth my modified function is returning questionable data!

There are of course other benefits of using larger data sets, mainly seeing how your code works under stress. What works fine for 10 items might not be so great with 1000 items. Testing with a large data set at the end will help catch these problems and will also help you adhere to Knuths’ advice to not optimize code prematurely.

So, now I’m off to learn how to trace through python code to find out how such a seemingly simple “fix” to my code could so subtly break it…

Rate-my-resume.com

Now if you are wondering if your resume is a good match for a particular job posting, you can use my site to find out! At the moment I’m giving the score in terms of 0 (being a total non-match) to 100 (being the absolute perfect match). In this economy, the more your resume reflects the skills listed in a particular job, the more likely your resume will be looked at seriously.

If you run your resume through and it gives you a low score, look at your resume and the job posting and try and figure out what keywords are in the job posting that are not in your resume. Then, assuming you have the necessary experience, put those keywords into your resume! Be sure to add them in a way that makes sense to a person, after all humans (especially HR people) don’t like to read fragments and words peppered into someone’s resume.

Try out the site with your resume and see how you rank!

p.s. Python rocks!