Teaching an old Pythonista new Gopher tricks

I recently got a new job where I need to write a lot of Golang, so needed to learn it.
I figured that you don't really learn a language unless you try and write code that actually does something useful. However having been to a recent Golang meetup where someone had come to a similar conclusion, and had written a full emulator of the Gameboy in Go - I also figured I wanted to do something that was not quite so complex or low level ... ie hopefully, could be done in a week.

So I decided to take the plunge by creating an open source package that does the same job, as a Python one that I released many years ago called django-csvimport. A simple add-on for the Django ORM that caters for loading data to models from CSV files, with the option to generate the model code from scratch for a CSV file by checking the data fields and determining the data type for each column.

Also doing a task where I had solved the problems in another language would mean I could just focus on how Golang might approach the problem, not the problem itself. So this post is about the practical differences between writing a Python and Golang solution. As such it compares the languages as tools for a certain job, which I hope is complementary to the many posts that compare the languages themselves. Suffice is to say, they differ in many ways ... most significantly in static vs. dynamic typing ... whilst being most similar in regarding readable consistent simple syntax as paramount - where other languages have different priorities - hence for both auto-formatting code is good practise, with Go's builtin go format doing the job of Python's black or yapf.

So firstly Django is one of the leading full web frameworks for Python, so what is the equivalent for Go? Gorilla, Gin, Buffalo etc. there are plenty of frameworks but which is the leading one with an ORM? ... I tried out a couple but reading around it, it became apparent that if you choose to develop a web app in Go, then the majority of devs don't use a framework at all!, so already the differences in the languages was becoming apparent. Reasons? If you choose Go for creating a web app then performance may be a significant requirement, even micro frameworks can be slower than raw code. Go is a recent language and as such has lots of web related features built into the core already ...  templating, etc. and even imports are url based so a web framework in Go gives you less than it does in Python.

So instead I checked out Go ORMs and decided to write an extension package for Gorm as one of the leading Go ORMs.

So ditching the Web Framework / UI integration features of django-csvimport as an unnecessary extra, then the problem just consists of two parts, creating ORM model definitions that create relational database tables and parsing the CSV files to import the data to those tables.

From this high level spec. the core functional components that compose the tool that we want to rebuild in Golang are:

1. CLI interface to take arguments specifying source files and actions to perform
2. An ORM to manage vendor independent database schema creation and population
3. Utility to inspect data sources and determine data types
4. Template tool to create ORM models (metaprogramming)
5. CSV parser to read in CSV files - ideally capable of handling various formats and poor or inconsistent formatting - ie real CSV files!

 For all of these we would hope for language level packages are available to do the major lifting. Then the package can just knit them together into a CSV to relational database import utility.

So stepping through these and rating Go vs Python...

CLI framework (draw)

As a minimum, our task requires a command line utility to point to the CSV data files to be imported.

Django comes with a CLI framework in the form of management commands. For our Go CSV import, gormcsv, we just have the ORM so we could roll our own CLI handling, but in this case that is probably not a great idea, since like Python, Go has a dominant CLI framework - Cobra equates to Python's Click. It uses the Viper config framework which is like Python's core configparser lib with extras. Within the gormcsv module I created these CLI command go files as a cmd package via Cobra's autogenerate feature and used them to wrap the importcsv.go and inspectcsv.go source files in the importcsv package that do the real work.

ORM (draw)

Any language's leading ORM's should cope with the database management and data population tasks and GORM is functionally similar in its capabilities to the Django ORM

Data source introspection tool  (Python win)

Messytables is a mature package designed for the task of scraping in data from various heterogenous third party sources - possibly of poor quality. As such it is one of the many utilities created around  Python's well established role in the data analytics realm. Go has no such tool. There is no third party package to cater for inspecting, type checking and cleaning up data sources :-(

So we have to make our own much simpler data inspector that will hopefully cope Ok with the most common data types if they are reasonably consistently formatted.

Templating tool for creating models (Go win)

For GORM and Django the ORM models are implemented directly as classes in the language rather than using an intermediate DSL or XML etc. So to create models based on introspecting source data metaprogramming must be used to generate code.
Templates are available in the core of Go. Also given it is statically typed and has no generics, then for some problems that generics would solve, the best alternative is to use metaprogramming. Hence templated generation of Go code is a normal Go pattern. So arguably this is better (core) supported in Go than Python. For Python code generation is rarely needed, and my original django-csvimport implementation just uses string construction and didn't even employ one of Python's many add on template packages, eg. Django or Jinja2 templates (hmm needs a rewrite!)
Note that both languages have fully functional reflection / introspection libraries in the core.

CSV Parser (Python win)

Most important to this application is the quality of the CSV parser. This is where Go is sadly completely let down. Its CSV parser is frankly inadequate and can only cope with CSV that is strictly formatted according to RFC 4180.

To quote from Python's csv parser library ...

CSV format was used for many years prior to attempts to describe the format in a standardized way in RFC 4180. The lack of a well-defined standard means that subtle differences often exist in the data produced and consumed by different applications. These differences can make it annoying to process CSV files from multiple sources.

TBH Python 3's CSV parser is itself significantly more strict about format than the old Python 2 one and so certain CSV files cannot be parsed that Python 2 happily dealt with - largely due to the switch to unicode resulting in more character encoding related critical fails.  However the Go parser is a whole other level of strict and realistically it can probably handle less than 10% of the real world CSV source files out there that you might want to scrape data from, into a database. Whilst Python 3's can probably cope with over 80%

I also investigated third party Go librarys that cater for parsing a more realistic range of CSV formatting, but found none that did so.

Conclusion

So in conclusion, Python may not be a Gopher Snake but for this task it does rather eat Go for breakfast. There is no ready made third party package to deal with ingesting unknown or badly formatted data like Python's aptly named messytables. Golang may sometimes be used for writing performant concurrent data processing in data science ... but it isn't used for the scraping and cleaning data sources part of the job! However this is a minor issue compared to the major blocker of not having an existing library that can import real world (ie sloppy format) CSV files.

So I have written my Go package for pushing CSV files to databases, gormcsv, and due to Go's great concurrency features it could certainly beat django-csvimport hands down in speed terms where big data quantities of CSV sources need ingesting. But I have yet to release it.  Because with such poor compatibility with real CSV files, there doesn't seem to be much point - however I will hopefully persist in finishing things off, probably as a less performant work around to pre-clean CSV files into strict RFC 4180 prior to parsing. Since implementing my own CSV parser from scratch for Go would likely break my original goal's of coming up with an open source project in the language that would take no longer than a week!

Oh and what do I think of Go? Well I like it, I most like the concept of classes just being data structs with bags of composed methods loosely coupled to them. I least like the error handling unseparated from normal code flow ... since it can lead to poor readability of code due to the excessive error boilerplate stuck within the program flow. It is my new favourite (statically typed) language ... but it hasn't replaced Python as my overall favourite.

Fixing third party Django packages for Python 3

With the release of Django 1.7 it could be argued that the balance has finally tipped towards Python 3 being its preferred platform. Well given Python 2.7 is the last 2.* then its probably time we all thought about moving to Python 3 for our Django deployments.

Problem is those pesky third party package developers, because unless you are determined wheel reinventor (unlikely if you use Django!) - you are bound to have a range of third party eggs in your Django sites. As one of those pesky third party developers myself, it is about time I added Python 3 compatibility to my Django open source packages.

There are a number of resources related to porting Python from 2 to 3, including specifically for Django, but hopefully this post may still prove useful as a summarised approach for doing it for your Django projects or third party packages. Hopefully it isn't too much work and if you have been writing Python as long as me, it may also get you out of any legacy syntax  habits you have.

So lets get started, first thing is to set up Django 1.7 with Python 3
For repeatable builds we want pip and virtualenv - if they are not there.
For a linux platform such as Ubuntu you will have python3 installed as standard (although not yet the default python) so if you just add pip3 that lets you add the rest ...

Install Python 3 and Django for testing

sudo apt-get install python3-pip
(OR sudo easy_install3 pip)
sudo pip3 install virtualenv


So now you can run virtualenv with python3 in addition to the default python (2.*)

virtualenv --python=python3 myenv3
cd myenv3
bin/pip install django


Then add a src directory for putting the egg in you want to make compatible with Python 3 so an example from git (of course you can do this as one pip line if the source is in git)


mkdir src
git clone https://github.com/django-pesky src/django-pesky
bin/pip install -e src/django-pesky


Then run the django-pesky tests (assuming nobody uses an egg without any tests!)
so the command to run pesky's test may be something like the following ...

bin/django-admin.py test pesky.tests --settings=pesky.settings
One rather disconcerting thing that you will notice with tests is that the default assertEqual message is truncated in Python 3 where it wasn't in Python 2 with a count of the missing characters in square brackets, eg.

AssertionError: Lists differ: ['Failed to open file /home/jango/myenv/sr[85 chars]tem'] != []



Common Python 2 to Python 3 errors

And wait for those errors. The most common ones are:

1. print statement without brackets
2. except Error as err (NOT except Error, err)
3. File open and file methods differ.
   Text files require better quality encoding - so more files default to bytes because strings in Python 3 are all stored in unicode
   (On the down side this may need more work for initial encoding clean up *,
   but on the plus side functional errors due to bad encoding are less likely to occur)
4. There is no unicode() method in Python 3 since all strings are now unicode - ie. its become str() and hence strings no longer need the u'string' marker 
5. Since unicode is not available as a method, it is not used for Django models default representation. Hence just using
   def __str__(self):
           return self.name
   is the future proofed method. I actually found that models with __unicode__ and __str__ methods may not return any representation, rather than the __str__ one being used, as one might assume, in Django 1.7 and Python 3
6. dictionary has_key has gone, must use in (if key in dict)

* I found more raw strings were treated as bytes by Python 3 and these then required raw_string.decode(charset) to avoid them going into the database string (eg. varchar) fields as pseudo-bytes, ie. strings that held 'élément' as '\xc3\xa9l\xc3\xa9ment' rather than bytes, ie. b'\xc3\xa9l\xc3\xa9ment'

Ideally you will want to maintain one version but keep it compatible with Python 2 and 3,
since this is both less work and gets you into the habit of writing transitional Python :-)

Test the same code against Python 2 and 3

So to do that you want to be running your tests with builds in both Pythons.
So repeat the above but with virtualenv --python=python2 myenv2
and just symlink the src/django-pesky to the Python 2 src folder.

Now you can run the tests for both versions against the same egg code -
and make sure when you fix for 3 you don't break for 2.

For current Django 1.7 you would just need to support the latest Python 2.7
and so the above changes are all compatible except for use of unicode() and how you call open().

Version specific code

However in some cases you may need to write code that is specific to 2 or 3.
If that occurs you can either use the approach of latest or anything else (cross fingers)

try:
    latest version compatible code (e.g. Python 3 - Django 1.7)
except:
    older version compatible code (e.g. Python 2 - Django < 1.7)

Or you can use specific version targetting ...

import sys, django
django_version = django.get_version().split('.')

if sys.version_info['major'] == 3 and django_version[1] == 7:
    latest version
elif sys.version_info['major'] == 2 and django_version[1] == 6:
    older django version
else:
    older version

where ...

django.get_version() -> '1.6' or '1.7.1'
sys.version_info() -> {'major':3, 'minor':4, 'micro':0, 'releaselevel':'final', 'serial':0}

Summary

So how did I get on with my first egg, django-csvimport ? ... it actually proved quite time consuming since the csv.reader library was far more sensitive to bad character encoding in Python 3 and so a more thorough manual alternative had to be implemented for those important edge cases - which the tests are aimed to cover. After all if a CSV file is really well encoded and you already have a model for it - it hardly needs a pesky third party egg for CSV imports - just a few django shell lines using the csv library will do the job.

WSGI functional benchmark for a Django Survey Application

I am currently involved in the redevelopment of a survey creation tool, that is used by most of the UK University sector. The application is being redeveloped in Django, creating surveys in Postgresql and writing the completed survey data to Cassandra.
The core performance bottleneck is likely to be the number of concurrent users who can simultaneously complete surveys. As part of the test tool suite we have created a custom Django command that uses a browser robot to complete any survey with dummy data.
I realised when commencing this WSGI performance investigation that this functional testing tool could be adapted to act as a load testing tool.
So rather than just getting general request statistics - I could get much more relevant survey completion load data.

There are a number of more thorough benchmark posts of raw pages using a wider range of WSGI servers - eg. http://nichol.as/benchmark-of-python-web-servers , however they do not focus so much on the most common ones that  serve Django applications, or address the configuration details of those servers. So though less thorough, I hope this post is also of use.

The standard configuration to run Django in production is the dual web server set up. In fact Django is pretty much designed to be run that way, with contrib apps such as static files provided to collect images, javascript, etc. for serving separately to the code. Recognizing that in production a web server optimized for serving static files is going to be very different from one optimized for a language runtime environment, even if they are the same web server, eg. Apache. So ideally it would be delivered via two differently configured, separate server Apaches. A fast and light static configured Apache on high I/O hardware, and a mod_wsgi configured Apache on large memory hardware. In practise Nginx may be easier to configure for static serving, or for a larger globally used app, perhaps a CDN.
This is no different from optimising any web application runtime, such as Java Tomcat. Separate static file serving always offers superior performance.

However these survey completion tests, are not testing static serving, simpler load tests suffice for that purpose. They are testing the WSGI runtime performance for a particular Django application.

Conclusions

Well you can draw your own, for what load you require, of a given set hardware resource! You could of course just upgrade your hardware :-)

However clearly uWSGI is best for consistent performance at high loads, but
Apache MPM worker outperforms it when the load is not so high. This is likely to be due to the slightly higher memory per thread that Apache uses compared to uWSGI

Using the default Apache MPM process may be OK, but can make you much more open to DOS attacks, via a nasty performance brick wall. Whilst daemon mode may result in more timeout fails as overloading occurs.

Gunicorn is all Python so easier to set up for multiple django projects on the same hardware, and performs consistently across different loads, if not quite as fast overall.

I also tried a couple of other python web servers, eg. tornado, but the best I could get was over twice as slow as these three servers, they may well have been configured  incorrectly, or be less suited to Django, either way I did not pursue them.

Oh and what will we use?

Well probably Apache MPM worker will do the trick for us, with a separate proxy front-end Apache configured for static file serving.
At least that way, its all the same server that we need to support, and one that we are already well experienced in. Also our static file demands are unlikely to be sufficient to warrant use of Nginx or a CDN.

I hope that these tests may help you, if not make a decision, maybe at least decide to try out testing a few WSGI servers and configs, for yourself. Let me know if your results differ widely from mine. Especially if there are some vital performance related configuration options I missed!

Running the functional load test

To run the survey completion tool via number of concurrent users and collect stat.s on this, I wrapped it up in test scripts for locust.

So each user completes one each of seven test surveys.
The locust server can then be handed the number of concurrent users to test with and the test run fired off for 5 minutes, over which time around 3-4000 surveys are completed.

The number of concurrent users tested with was 10, 50 and 100
With our current traffic peak loads will probably be around the 20 users mark with averages of 5 to 10 users. However there are occasional peaks higher than that. Ideally with the new system we will start to see higher traffic, where the 100 benchmark may be of more relevance.

Fails

A number of bad configs for the servers produced a lot of fails, but with a good config these seem to be very low. So all 3 x 5 minute test runs for each setup created around 10,000 surveys, these are the actual number of fails in 10,000
so insignificant perhaps ...

Apache MPM process = 1
Apache MPM worker = 0
Apache Daemon = 4
uWSGI = 0
Gunicorn = 1

(so the fastest two configs both had no fails, because neither ever timed out)

Configurations

The test servers were run on the same virtual machine, the spec of which was
a 4 x Intel 2.4 GHz CPU machine with  4Gb RAM
So optimum workers / processes = 2 * CPUs + 1= 9

The following configurations were arrived at by tinkering with the settings for each server until optimal speed was achieved for 10 concurrent users.
Clearly this empirical approach may result in very different settings for your hardware, but at least it gives some idea of the appropriate settings - for a certain CPU / memory spec. server.

For Apache I found things such as WSGIApplicationGroup being set or not was important, hence its inclusion, with a 20% improvement when on for MPM prefork or daemon mode, or off for MPM worker mode.

Apache mod_wsgi prefork

WSGIScriptAlias / /virtualenv/bin/django.wsgi
WSGIApplicationGroup %{GLOBAL}

Apache mod_wsgi worker

WSGIScriptAlias / /virtualenv/bin/django.wsgi

<IfModule mpm_worker_module>
#  ThreadLimit    1000
    StartServers         10
    ServerLimit          16
    MaxClients          400
    MinSpareThreads      25
    MaxSpareThreads     375
    ThreadsPerChild      25
    MaxRequestsPerChild   0
</IfModule>

Apache mod_wsgi daemon

WSGIScriptAlias / /virtualenv/bin/django.wsgi
WSGIApplicationGroup %{GLOBAL}

WSGIDaemonProcess testwsgi \
    python-path=/virtualenv/lib/python2.7/site-packages \
    user=testwsgi group=testwsgi \
    processes=9 threads=25 umask=0002 \
    home=/usr/local/projects/testwsgi/WWW \
    maximum-requests=0

WSGIProcessGroup testwsgi

uWSGI

uwsgi --http :8000  --wsgi-file wsgi.py --chdir /virtualenv/bin \
                               --workers=9 --buffer-size=16384 --disable-logging

Gunicorn

django-admin.py run_gunicorn -b :8000 --workers=9 --keep-alive=5

Django Cardiff User Group

Last night I went to the second meeting of the Django Cardiff User Group.

This is a sister group to the DBBUG Bristol based one that I have been attending for the last 5 years. It was organised by Daniele Procida, who started attending DBBUG events a few years ago and has now decided to spread the word over the Severn, in Wales.

He is also organising the first UK Django conference in a couple of months, https://djangoweekend.org/ so its good to see one open source / Python group be inspiration for spawning another, and one that is perhaps more organisationally active than its progenitor.

The evening was fun, and it was good to meet and chat with Djangonauts over the border.

Andrew Godwin, Django core developer / release manager, gave us an update on all the new goodies to be added in Django 1.7
So this release is largely about really sorting out the niggling issues with relational database features, and the low level ORM handling of them.
It sees rationalisation of transaction handling with the use of nestable atomic statements, addition of generic connection pooling, and handling of composite keys.

Daniele demonstrated how to fly a helicopter (a toy one) via the Python command line, although Andrew seemed rather more adept at landing it safely. I gave a little reprise of a talk introducing DBBUG and how a developer can follow the road to their own open source contributions.

Thanks to everyone involved, I hope to get to the Django weekend too.

IT Megameet

Yes MegaMeet may have a slightly cheesey ring to it, but the Bristol IT MegaMeet was a lot of fun, and a great idea for a regional software community event. So unlike most conferences this one is not for a particular company, language, platform or area of software expertise. Instead it brings together all the voluntary community software and technology groups within the region of Bristol, UK.

There are quite a number as it turns out, and so squeezing the conference into a single day resulted in 5 tracks. For a conference organised for the first time last year by a student to save his course - thanks Lyle Hopkins, it rather put our local University's official efforts in software community engagement to shame - however perhaps it might encourage them to rise to the challenge. (Lyle is a student at one, and I work at the other.)

So of the perhaps 30-40 software groups that are based in and around Bristol, over 20 were represented, a good turnout partly due to the efforts of one of Lyle's fellow volunteers, Indu Kaila, to do the leg work of attending all the local events and getting various members (like myself) to volunteer to represent their group at the event. So I am one of the hundred odd members of Bristol and Bath's Django User Group (DBBUG), started by Dan Fairs, and did a presentation about Python, Django, our group, and the process of contributing to open source - so rather a lot to pack into 40 minutes, but it seemed to go down OK.

There was the full range of enthusiast groups present, so I started the day finding out how the four colour theorem from map making applies to optimisation algorithms used in compilers, from the ACCU, who have been around for a very long time, starting out as a C programming community group. Then near the finish saw a good talk from Bristol Web folk reminding me about the core important issues to remember concerning front end web development - as more of a back end developer  it can be easy to label this stuff somebody else's job, but with an ever increasing slice of the stack being client side in web development, these days, that is clearly a bad attitude.

There was more than a smattering of javascript related talks going on, from big data CouchDB and node.js back end use, through to more client side, and a very popular session, flying helicopters via javascript code.

The talks were rounded off with some talks about the charity cause that the day was helping to raise funds for, a cross atlantic row in aid of cervical cancer charity (plus an appeal for graphic design work for another member of the volunteer team from the Ukraine, who is in need of health care).

I then found myself in the rather comical position of receiving two awards from the extensive award ceremony for community involvement, etc. Both really on behalf of other people, but it was fun and lead on to the free bar and barbecue, always a popular way to round off a conference.

So thanks to the Megameet team, if nobody else comes forward, I can always represent DBBUG, South West Big Data or perhaps another new local group, again next year!

Cookie law, Cookieless and django tips.

django-cookieless

Last week I released a new add on for django, django-cookieless, it was a relatively small feature that was required for a current project, and since it was such a generic package seemed ideal for open sourcing as a separate egg. It made me realise that I hadn't released a new open source package for well over a year, and so this one is certainly long over due in that sense.

Cookie Law

It also over due in another sense, EU Cookie law has been in force since May 2011, so legally any sites that are used in Europe, and set cookies which are not strictly necessary for the functioning of the site, must now request permission of the user before doing so. Of course it remains to be seen if any practical enforcement measures will happen, although they were due to this summer in the UK, for example. Hence many of the first rush of JavaScript pop up style solutions, have come and gone, as a result of user confusion. But for public sector clients particularly, it is certainly simpler to just not use cookies, if they are not technically required. Also it may at least, make developers rather less blasé about setting cookies.

Certainly most people would prefer not to see their browsers filled with deliberate user tracking and privacy invasive cookies that are entirely unrelated to the sites functionality. In the same way most of us don't like being tracked by CCTV everywhere we go ... unfortunately, the current Law doesn't have a good technical solution behind it, hence it may well founder over time. This is because cookie control is too esoteric for ordinary users, and even with easy browser based privacy levels configuration, any technical solutions are problematic, because a single cookie can be used to both protect privacy (in terms of security - e.g. a CSRF token) and invade it.  It is entirely down to the specific applications usage of it, where these distinctions lie. Invasive methods can also be implemented via other session maintenance tools, such as URL rewriting, yet because no data is written to the users browser, it is outside the remit of this Law, so the Law makes little sense currently, and may well be unenforceable.

Perhaps it would of been better to aim to set laws related to encouraging adherence to set standards of user tracking, starting with compliance with the browser added 'Do Not Track' header, perhaps adding some more subtle gradations over time. With the targets of the Law, being companies whose core business is user tracking for advertising sales etc., starting with Google and working down. Rather than pushing the least transgressive public service sector, as the most likely to comply, to add a bunch of annoying 'Will you accept our cookies?' pop ups.

However even if  this law dries up and blows away, for our particular purposes, we needed django to cater for any number of sessions per browser (as well as not using cookies for anonymous users).
Django's default session machinery requires cookies, so ties a browser to a single session - request.session set against a cookie. But because django-cookieless provides sessions maintainable by form posts, it automatically delivers multiple sessions per browser.

There are a number of security implications with not using cookies, which revolve around the difficulty of preventing session stealing without them. Given this is the case, django-cookieless has a range of settings to reduce that risk, but even so I wouldn't recommend using it for sessions that are tied to authenticated users, and hence could lead to privilege escalation, if the session were stolen.

Django Tips

I thought the egg would be done in a day, but in reality it took a few days, due to a number of iterations that were necessary as I discovered a series of features around the lesser known (well to me) parts of  django. So I thought I would share these below, in case, any of the tips I gained are useful ...

1. The request object life cycle goes through three main states in django:
   unpopulated - the request that is around at the time of process_request type middleware hooks - before it gets passed by the URL handler to decorators and then views.
   partly populated - the request that has session, user and other data added to it (mainly by decorators) and gets passed to a view
   fully populated - the request that has been passed through the view to add its data, and is used to generate a response - this is the one that process_response sees.
2. I needed to identify requests that were decorated with my no_cookies decorator at the time of process_request. But the flag it sets has not be set yet. However there is a useful utility to work around this, django.core.urlresolvers.resolve, which when passed a path, gives a match object containing the view function to be used, and hence its decorators, if any.
3. Template Tags that use a request get the unpopulated one by default.  I needed request to have the session populated for the option of adding manual session tags - see the tags code, to have the partly populated request in tags django.core.context_processors.request must be added to the TEMPLATE_CONTEXT_PROCESSORS in settings.
   
   
4. The django test framework's test browser is in effect a complex mocking tool to mock up the action of a real browser, however like any mock objects - it may not exactly replicate the behaviour one desires. In my case it only turns on session mocking if it finds the standard django session middleware in settings. In the case of cookieless it isn't there, because cookieless acts as a replacement for it, and a wrapper to use it for views undecorated with no_cookies. Hence I needed to use a trick to set a TESTING flag in settings - to allow for flipping cookieless on and off.

Talks vary, especially mine

This week I had the opportunity to go to a couple of gatherings and deliver two python related talks. The first was at our local Django Bath and Bristol Users Group (#DBBUG). The second was at the Google Apps for EDU European User Group (GEUG12).

I don't do a talk that often, maybe 5 or 6 times a year, and I guess share some of the traits of the stereotyped geek that I am not a natural extrovert who wants to be up on stage. But at the same time, there is a bit of an adrenalin rush and subsequent good feeling (maybe just relief its over!) if a talk seems to go OK.

The first talk went well, although perhaps that was just from my perspective, having consumed a generous quantity of the free beer before hand, provided by the hosters, Potato . It was good to see a lot of new faces at the meeting, and hear about interesting projects from other speakers.
My talk was hardly rocket science, instead just a little session on the basics of python packaging and why its a good thing to do it. But it seemed to me it was pitched at about the right technical level for the newbies to get something from it, and the more experienced to contribute comments. It was paced about right and I pretty much followed the thread of the slides ie. from 'what is an egg' to C.I. and local PyPi, without slavishly reading out from them. The atmosphere was relaxed and people seemed interested. All in all a good session.

The second talk unfortunately did not go well, even though I spent more time preparing it - it even triggered me into upgrading my personal App Engine site (see previous blog posting) - which as described took a good few hours in itself - to make sure my App Engine knowledge was up to date. So what went wrong. Well maybe with the previous talk going well with little preparation and no rehearsal - I had started to get a bit blase. Hey look at me, I can fire off a talk no problem, don't worry about it. However to some extent the quality of any talk depends as much on the audience as the speaker - its an interactive process - and for a few reasons, I didn't feel that I had established that communication - and it threw me, I guess, to the point where I was really stumbling along at the end.

So I thought I would try and analyse some of the reasons, to try to avoid it happening again. These are all common sense and probably in any guide to public speaking - but I thought its worth writing it down - even if its only for my benefit!

The core reason was that I assumed that there was a disconnect between the audience, and what I was talking about. So I wasn't preaching the gospel to the humanist society - or zope at a django conference ;-) I was talking about how you can use App Engine as a CMS, to a group of University educational technology staff - managers, learning support and some developers.

So first mistake was that I had adapted a talk that I had delivered to a group of developers a year before. It had gone well before because the audience, like me, were not interested in using the tools - they were interested in building the tools, how these tools could be used to build others - and what implications that had for how we should approach building tools in the future.

Lesson 1: Try to get an idea of  your audience's background - then write the talk tailored for them from scratch (even if its a previous talk - unless you know the audience profile hasn't changed). Also if a previous demo and talk with questions had been an hour - and now it has to be done in 20 minutes - rewrite it - or at least delete half the slides - but don't expect to talk three times faster!

Lesson 2: If you do feel that you might of pitched a talk at the wrong technical level - and there is no time to rewrite or rethink it, its probably best to just deliver it as it stands. Moderating all the slides with 'sorry too techie' and rephrasing things in layman's terms on the fly - is probably going to be less coherent, and lose the thread of the talk anyhow - unless you are a well experienced teacher.

My first slide was entitled APIs in transition - hmmm that was a mistake, a few people immediately left the room.

Lesson 3: The most interesting initial thing to me coming back to my site, were all the changes that had occurred with the platform. However if you haven't used it before that is irrelevant. So remember don't focus on what you last found interesting about the topic - focus on the general picture for somebody new to it.

Lesson 4: Don't start a talk with the backend technology issues. Start it with an overview of the purpose of the technology and ideally a demo or slide of it in use. However backend your topic, its always best to start with an idea of it from the end user perspective - even when talking to a room full of developers.

When I got to the demo part I skipped it due to feeling time pressure - actually however this would of been best acting as the main initial element of the talk, with all the technical slides skipped and just referred to for those interested. Finishing with the wider implications regarding sites based around mash-ups, driven by a common integration framework. So ditching most of the technical details.

Lesson 5: Don't be scared to reorganise things at the last minute, before starting (see Lesson 2) - if that reorganisation is viable, eg. in terms of pruning and sequence.


There was a minor organisational issue in that I started 5 minutes before I was due to end a 20 minute talk, with no clock to keep track. So there was a feeling of having over run almost from the start, combine that with people leaving or even worse people staying but staring at you with a blank, 'You are really boring' look!

Lesson 6: Check what time you are really expected to end before you start and get your pace right based on this. Keep looking around the audience and try to just find a least some people who look vaguely interested! - ignore the rest - it is unlikely you can ever carry the whole audience's interest - unless you are a speaker god - but you need to feel you have established some communication with at least some members of it - to keep your thread going.


OK well I could go on about a number of other failings ... but hey, I have spent longer writing about it than I did delivering it. So that will do, improve by learning from my mistakes, and move on.

User groups vary too

As a footnote another difference was the nature of the two user groups.

The DBBUG group was established and is entirely organised by its members, like minded open source developers, who take turns organising the meetings, etc. Its really just an excuse for techies to go to the pub together on a regular basis - and not necessarily always chat about techie stuff. Its open to anyone and completely informal.

GEUG is also largely organised by its members taking turns, but was originally established by Google's  HE division for Europe and has a lot of input from their team, it requires attendees to be members of customer institutions. So essentially its a customer group and has much more of that feel. Members attend as a part of their job. Google's purpose is to use it to expand its uptake in HE - by generating a self supporting community that promotes the use of its products, and trials innovative use cases. To some extent feeding back into product development. With a keynote by Google and all other talks by the community members. Lots of coloured cubes, pens, sports jackets - and a perhaps, slightly rehearsed, informality. But interestingly quite open to talks that perhaps didn't praise their products or demonstrated questionable use cases, regarding the usual bugbear of data protection. Something that is a real sore spot within Europe apparently - the main blocker to cloud adoption in HE.

Having once attended a Microsoft user group event in Dublin at the end of the 90s, I would say that this was a long way removed from that. The Microsoft event was strictly controlled, no community speakers, nothing but full sales engineer style talks about 'faultless' products, there was no discussion  of flaws or even of approaches that could generate technical criticisms. Everybody wore suits - maybe that is just the way software sales were way back when Microsoft dominated the browser and desktop.

Where as now community is where it is at. Obviously GEUG felt slightly less genuinely community after DBBUG, but I would praise Google that they are significantly less controlling over shaping a faultless technical suited business face to HE, than some of their competitors. Unfortunately for them non-technical managers with their hands on the purse strings tend to be largely persuaded by the surface froth of suits and traditional commercial software sales - disguise flaws, rather than allow discussion of whether and how they may be addressed.

In essence the Google persona that is carefully crafted to sit more towards an open source one, but as a result may suffer from the same distrust that traditional non-technical clients have for open source over commercial systems. Having said that they are not doing too badly ... dominating cloud use in US HE. Maybe Europe HE is just a tougher old nut to crack.

Upgrading a Google App Engine Django app

In my day to day work, I haven't had an opportunity to use Google App Engine (GAE). So to get up to speed with it, and since it offers free hosting for low usage, I created my home site on the platform a year ago. The site uses App Engine as a base, and integrates in Google Apps for any content other than custom content types.

Recently I have been upgrading the Django infrastructure we use at work, from Python 2.6 to 2.7 and Django 1.3 to 1.4. I thought having been a year, it was probably time to upgrade my home site too, having vaguely registered a few changes in App Engine being announced. On tackling the process, I realised that 'a few changes' is an understatement.

Over the last year GAE has moved from a beta service to a full Google service. Its pricing model has changed, its backend storage has changed, the python deployment environment has changed and the means of integrating the Django ORM has changed. A raft of other features have also been added. So what does an upgrade entail?

Lets start with where we were in Spring 2011

1. Django 1.2 (or 0.96) running on a Python 2.5 single threaded CGI environment
2. The system stores data in the Master/Slave Big Table datastore
3. Django's standard ORM django.db is replaced by the NOSQL google.appengine.ext.db *
4. To retain the majority of forms functionality ext.djangoforms replaces forms *
5. python-gdata is used as the standard means to integrate with Google Apps via common RESTful Atom based APIs
   
   * as an alternative django-norel could of been used to provide full standard ORM integration - but this was overkill for my needs

To configure GAE a typical app.yaml would of been the following:

application: myapp
version: 1-0
runtime: python
api_version: 1

handlers:
- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

- url: /.*
  script: main.py

- url: /media
  static_dir: _generated_media
  secure: optional 

With the main CGI script to run it

import os

from google.appengine.ext.webapp import util
from google.appengine.dist import use_library

use_library('django', '1.2')
os.environ['DJANGO_SETTINGS_MODULE'] = 'edcrewe.settings'

import django.core.handlers.wsgi
from django.conf import settings

# Force Django to reload its settings.
settings._target = None

def main():
  # Create a Django application for WSGI.
  application = django.core.handlers.wsgi.WSGIHandler()

  # Run the WSGI CGI handler with that application.
  util.run_wsgi_app(application)

if __name__ == '__main__':
  main()

What has changed over the last year

Now we have multi-threaded WSGI Python 2.7 with a number of other changes.

1. Django 1.3 (or 1.2) running on a Python 2.7 multi threaded WSGI environment
2. The system stores data in the HRD Big Table datastore
3. For Big Table the NOSQL google.appengine.ext.db is still available, but Django's standard ORM django.db is soon to be available for hosted MySQL
4. google.appengine.ext.djangoforms is not available any more
   Recommendation is either to stop using ModelForms and hand crank data writing from plain Forms - or use django-norel - but it does have a startup overhead *
5. python-gdata is still used but it is being replaced by simpler JSON APIs specific to the App in question, managed by the APIs console and accessible via google-api-python.
   
   * django-norel support has moved from its previous authors - with the Django 1.4 rewrite still a work in progress

Hmmm ... thats a lot of changes - hopefully now we are out of beta - there won't be so many in another year's time! So how do we go about migrating our old GAE Django app.

Migration

Firstly the Python 2.7 WSGI environment requires a different app.yaml and main.py 

Now to configure GAE a typical app.yaml would be:

application: myapp-hrd
version: 2-0
runtime: python27
api_version: 1
threadsafe: true

libraries:
- name: PIL
  version: latest
- name: django
  version: "1.3"

builtins:
- django_wsgi: on
- remote_api: on

handlers:
# Must use threadsafe: false to use remote_api handler script?
#- url: /remote_api
#  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
#  login: admin

- url: /.*
  script: main.app

- url: /media
  static_dir: _generated_media
  secure: optional 

With the main script to run it just needing...

import os
import django.core.handlers.wsgi

os.environ['DJANGO_SETTINGS_MODULE'] = 'edcrewe.settings'
app = django.core.handlers.wsgi.WSGIHandler()

But why is the app-id now myapp-hrd rather than myapp?
In order to use Python 2.7 you have to move to the HRD data store. To migrate the application from the deprecated Master/Slave data store it must be replaced with a new application. New applications now always uses HRD.
Go to the admin console and 'Application Settings' and at the bottom are the migration tools. These wrap up the creation of a new myapp-hrd which you have to upload / update the code for in the usual manner. Once you have fixed your code to work in the environment (see below) - upload it.
The migration tool's main component is for pushing data from the old to the new datastore and locking writes to manage roll over. So assuming all that goes smoothly you now have a new myapp-hrd with data in ready to go, which you can point your domain at.

NB: Or you can just use the remote_api to load data - so for example to download the original data to your local machine for loading into your dev_server:

${GAE_ROOT}/appcfg.py download_data --application=myapp
--url=http://myapp.appspot.com/remote_api --filename=proddata.sql3

${GAE_ROOT}/appcfg.py upload_data --filename=proddata.sql3 
${MYAPP_ROOT}/myapp --url=http://localhost:8080/remote_api 
--email=foo@bar --passin --application=dev~myapp-hrd

Fixing your code for GAE python27 WSGI

Things are not quite as straight forward as you may think from using the dev server to test your application prior to upload. The dev server's CGI environment no longer replicates the deployed WSGI environment quite so well - like the differences between using Django's dev server and running it via Apache mod_wsgi. For one thing any CGI script imports OK as before for the dev server - yet may not work on upload or require config adjustments - e.g. ext.djangoforms is not there, and use of any of the existing utility scripts - such as the remote_api script for data loading, requires disabling of the multi-threaded performance benefits. Probably the workaround here for more production scale sites than mine, is to have a separate app for utility usage than the one that runs the sites.

If you used ext.djangoforms, either you have to move to django-norel or do code writes directly. For my simple use case I wrote a simple pair of utility functions to do data writes for me, and switched my ModelForms to plain Forms.

def get_ext_db_dicts(instance):                                 
    """ Given an appengine ext.db instance return dictionaries
        for its values and types - to use with django forms
    """                                               
    value_dict = {}                                                
    type_dict = {}                                              
    for key, field in instance.fields().items():                                                 
        try:                                                                      
            value_dict[key] = getattr(instance, key, '')                                                
            type_dict[key] = field.data_type                                                          
        except:                                                                   
            pass                                                        
    return value_dict, type_dict      

def write_attributes(request, instance):
    """ Quick fix replacement of ModelForm set attributes
        TODO: add more and better type conversions
    """
    value_dict, type_dict = get_ext_db_dicts(instance)                                                                                                 
    for field, ftype in type_dict.items():                                                                                                             
        if request.POST.has_key(field):                                                                                                                
            if ftype == type(''):                                                                                                                      
                value = str(request.POST.get(field, ''))                                                                                               
            elif ftype == type(1000):                                                                                                                  
                try:                                       
                    value = int(request.POST.get(field, 0))                             
                except:         
                    value = 0                                                                                                                
            elif ftype == type([]):                              
                value = request.POST.getlist(field)                            
            else:                                                              
                value = str(request.POST.get(field, ''))                                                                                                                                                                   
            setattr(instance, field, value)

Crude but it allows one line form population for editing ...
mycontentform = MyContentForm(value_dict)
... and instance population for saving ...
write_attributes(request, instance)

However even after these fixes and data import, I still had another task. Images uploaded as content fields were not transferred - so these had to be manually redone. This is maybe my fault for not using the blobstore for them - ie since they were small images they are were just saved to the Master/slave data store - but pretty annoying even so.

Apps APIs

Finally there is the issue of gdata APIs being in a state of flux. Well currently the new APIs don't provide sufficient functionality and so given that this API move by Google still seems to be in progress - and how many changes the App Engine migration required - I think I will leave things be for the moment and stick with gdata-python ... maybe in a years time!

Django - generic class views, decorator class

I am currently writing a permissions system for a Django based survey application and we wanted a nice clean implementation for testing users for appropriate permissions on the objects displayed on a page.

Django has added class views in addition to the older function based ones. Traditionally tasks such as testing authorisation has been applied via decorator functions.

 @login_required
 def my_view:  
   return response  

The recommended approach to do this for a class view is to apply a method decorator.
There is a utility convertor method_decorator, to do this for function decorators.

 class ProtectedView(TemplateView):
    @method_decorator(login_required)
    def dispatch(self, *args, **kwargs):
        return super(ProtectedView, self).dispatch(*args, **kwargs)

However this isn't ideal since it is less easy to check all the methods for decorators than just look at the top of the view function as before.

So why not use a class decorator instead to make things more clear. Fine, except we do actually want to decorate the dispatch method. But we can add a utility decorator that wraps this up.*

@class_decorator(login_required, 'dispatch')
class ProtectedView(TemplateView):
    def dispatch(self, *args, **kwargs):
        return super(ProtectedView, self).dispatch(*args, **kwargs)

But Django's generic class views contain more than just the TemplateView, they have generic list, detail and update views. All of which use a standard pattern to associate object(s) in the context data. Not only that but the request will also have user data populated if the view requires a login.

What I want to do is have a simple decorator that just takes a list of permissions, then ensures users who access the class view must login and then have each of these object permissions checked for the context data object(s). So my decorator for authorising user object permissions will be @class_permissions('view', 'edit', 'delete')

To do this the class_permissions decorator itself, is best written as a class. The class can then combine the actions of three method decorators on the two Django generic class view methods - dispatch and get_context_data.
Firstly login_required wraps dispatch - then dispatch_setuser wraps this to set the user that login_required delivers as an attribute of the class_permissions class.
These must decorate in the correct order to work.

Finally class_permissions wraps get_context_data to grab the view object(s). The user, permissions and objects can now all be used to test for object level authorisation - before a user is allowed access to the view. The core bits of the final code are below - my class decorator class is done :-)

class class_permissions(object):
    """ Tests the objects associated with class views
        against permissions list. """
    perms = []
    user = None
    view = None
 
    def __init__(self, *args):
        self.perms = args

    def __call__(self, View):
        """ Main decorator method """
        self.view = View

        def _wrap(request=None, *args, **kwargs):
            """ double decorates dispatch 
                decorates get_context_data
                passing itself which has the required data                                              
            """
            setter = getattr(View, 'dispatch', None)
            if setter:
                decorated = method_decorator(
                                   dispatch_setuser(self))(setter)
                setattr(View, setter.__name__,
                        method_decorator(login_required)(decorated))
            getter = getattr(View, 'get_context_data', None)
            if getter:
                setattr(View, getter.__name__,
                        method_decorator(
                               decklass_permissions(self))(getter))
            return View
        return _wrap()

The function decorators and imports that are used by the decorator class above

from functools import wraps
from django.utils.decorators import method_decorator

def decklass_permissions(decklass):
    """ The core decorator that checks permissions """                                                                                                      

    def decorator(view_func):
        """ Wraps get_context_data on generic view classes """ 
        @wraps(view_func, assigned=available_attrs(view_func))
        def _wrapped_view(**kwargs):
            """ Gets objects from get_context_data and runs check """                                          
            context = view_func(**kwargs)
            obj_list = context.get('object_list', [])                                                      
            if not obj_list:     
                obj = context.get('subobject',                                          
                                  context.get('object', None))                          
                if obj:                                                                 
                    obj_list = [obj, ]       
            check_permissions(decklass.perms, decklass.user, obj_list)
            return context 
        return _wrapped_view
    return decorator

def dispatch_setuser(decklass):
    """ Decorate dispatch to add user to decorator class """

    def decorator(view_func):
        @wraps(view_func, assigned=available_attrs(view_func))
        def _wrapped_view(request, *args, **kwargs):
            if request:
                decklass.user = request.user
            return view_func(request, *args, **kwargs)
        return _wrapped_view
    return decorator

Although all this works fine, it does seem overly burdened with syntactic sugar. I imagine there may be a more concise way to achieve the results I want. If anyone can think of one, please comment below.

* I didnt show the code for class_decorator since it is just a simplified version of the class_permissions example above.

KISS Djangocon.eu

Last week I had a great time in Amsterdam at Djangocon.eu, and many thanks to everybody involved in the event, especially the organisers.
Many useful talks and it was energizing to see an event and platform growing as Django is doing, with lots of young new developers coming on board.
I gave a short talk myself, about how I am using Django at the University of Bristol as an ideal lightweight agile framework for glueing together components that have traditionally been labelled as 'Enterprise' architecture. Traditionally SOAP web services and 'Business' objects, are sold as the necessary integration components. Complex environments and problems requiring complex (commercially named) technologies as solutions. The gist of my talk was that just because it is a complex problem it doesn't need a complex solution. Integration technology should be as simple and decoupled as possible to be robust.
This is really just the old engineering principle, KISS (Keep It Simple Stupid). Examples in IT trends are the decline of SOAP in favour of restful approaches, the rise of the cloud - and social media over more workflowed institutional CMS. NoSQL replacing large relational databases. Simple component services perhaps tied together just with javascript and JSON, rather than complex integration APIs and XML, tightly coupled to relational schemas.
Applying the same KISS ideals to the world of python, then it appears similar lessons are being learned. From a recent blog posting, by Mikko Ohtamaa, Plone is attempting to shed complexity, whilst as Martijn Faassen alluded to in his talk - zope found that a full rewrite to add java style interface adaptor patterns via XML, to zope's already overly full basket of patterns, was a step too far.
To finish Russell Keith-Magee gave a Roadmap talk, appealing to developers to get on board, and for Django to not suffer from a clique mentality. The approach of tell us where you want Django to go, and if you help out, it will. In response, I sincerely hope that Django remembers that KISS is its mission, and makes it a point of principle to not increase its core size. If anything components such as the admin should probably be taken out to develop separately - along with any other more CMS specific components. Storage too, could be pluggable as relational or no-rel, etc. with just SQLite by default.

PS: For those that attended, appologies over my talk. My netbook proved to have been rather too bashed around by the kids, and the graphics card couldn't cope with the multi-head projectors. Hence I had to borrow one from the audience, so an already rather ambitiously squeezed talk had to be skipped through in even less time. I hope the result (and diagrams) were not too impenetrable, maybe next time I can deliver something a bit clearer!

Using App Engine and Google Apps as a CMS

I recently submitted a conference talk, and the form asked for the usual home page url. I tend to use my blog, but I also have stuff scattered around in other places and it made me realize it was probably a good idea to stick this stuff together somewhere. Given that I am not Ryan Giggs it is unlikely that this site would attract a vast amount of public interest, so what I wanted was a fairly simple CMS, that was free for low traffic, and could pull in content and documents from other sites or systems.

For the standard page, image and file CMS functionality then there are now a number of free Wikis / simple CMS, such as Wikia or Google Sites. These tend to have sufficient features, usually no workflow, but often content versioning, which is a bonus. So why not just use one of these out of the box, well where's the fun in that? ... but more seriously they don't cater for full design freedom. They often have fixed templates, they also tend to have very limited, if any, custom content types.

I needed a project type for populating with a portfolio of past work, with image fields and categorisation lists. I also want to pull in some of my documents, and drawing and painting is a hobby of mine, so I fancied adding some of my efforts via a photo sharing site.

Finally there was this blog, well currently its in Wordpress hosted at my workplace, but I may not be there forever, so I thought it was time to migrate to an external service - and have the advantage of some extra tools and integration capabilities as well.

Although happy to do some development, given that this site was a spare time task, I didn't want to spend days on integration code. The leading choice that combines all these tools with a standard well documented API is Google Apps - I plumped for that. But the same principles can be applied to any mainstream hosted content service, since they should all have an API if they are any good. For example there is a python library for Flickr.

So to start mashing up all this Google-ness, I created a site in App Engine's adapted Django which holds the aggregation code for the site, and caters for full customisation and complex content types, such as the portfolio projects.

With inspiration from Django's flatpages app I added a middleware handler to check 404's against requests to the site, any content not found in App Engine falls back to Google Sites and retrieves the page from there via the gdata API. So in effect you have an admin interface for editing which is the bare untemplated Google Site, and the public interface for display via App Engine.
Google Sites has the advantage of already being integrated via common authorisation with Google docs,  Picasa, YouTube etc. so dropping in slides shows of photos, or Office documents is trivial. 

The gdata API is ATOM based and extensive with a full python library wrapper. It caters for read or writes of content, ACL, revisions, templates etc. for the various component Apps. Alongside it is the OAuth protocol acting as the single sign on system to bind together the authentication and authorisation.  It raises the potential of a developing much larger scale CMS by developing tools to automate site creation and integration from Google App domains of thousands of users.

But back to my home site. For my limited requirements the job was relatively trivial, so I would recommend the Google approach for simple low traffic CMS.  But before this becomes a pure advert for Google, I did come across a few random gripes along the way.

1. The default gdata API wrapper for sites doesnt return all the content via entity.content.html - you have to pass in a custom handler function.
2. Upgrading from Ubuntu 10 to 11 broke the GAE dev server, due to this issue! 
3. GAE's python is 2.5 (until later this year) and its django uses custom data types and other customisations - hence django-norel for unadulterated Django on App Engine.
4. A custom list widget was needed to handle BigTable's list type.
5. Image uploads from a HTC Desire to Picasa break. The broken images also caused picnik to break until chrome's cache was fully flushed.
6. Distinguishing public links and edit links in docs has bad usability - eg. any share link is not the view link.
7. Quotas are getting lower soon for free AppEngine - but via gdata you can circumvent this with the much larger Apps ones - e.g. send mails via the gdata GMail API, not the GAE mail API.
8. The wake up time for the site can occasionally be pretty slow, ie. 5 seconds or more, to bring up a dynamic instance if traffic is so low, it regularly shuts down.  Of course you can pay 20p a day for always on.
9. Neither Google sites or Blogger's WYSIWYG editors produce clean minimal html - instead they seem to churn out loads of divs filled with embedded styles - that may validate but they are bloated and a designer's nightmare.
   

Any how the site is now up, http://www.edcrewe.com, although it could do with some more work on content and design, when I can squeeze it in. At least all my stuff is in one place now :-)

Django barcamping

I hosted a django barcamp this week, in the midst of a hectic schedule of attending a UKOLN DevSci mobile day and the Plone Conference.

I had offered to host one of the events for the local Bristol and Bath Django users group, DBUG, after attending the last one. So when prodded I suggested lightning talks and set about putting together a couple. Having got the ball rolling I thought I should offer to host to. The beer and pizza came for free via Dan Fairs contact at Austin Fraser IT recruitment agency.

It was good fun and quite successful ~ 40 odd attendees, so aside from ending up tidying all the empty beer bottles ;-) I would recommend it.
I did a talk on using Google App Engine, which I will also have to post on in more detail - once I have finally written the accompanying Android app that goes with the GAE site.

The main positive was just how in demand python and particularly django skills are at the moment, certainly no difficulty in getting a job locally that pays equivalent to a University developer post. The positive buzz in commercial python web development also made a nice contrast to the current cold winds blowing through the public sector.

Finally it was a good social occasion to get to meet, and get a bit drunk, with a bunch of fellow programmers, who were interested in the same coding tools as myself.

EuroPython 2010

Just got back from the EuroPython 2010 conference in Birmingham this week. See my day by day write up.

It was a change for me, my first language conference, as opposed to web platform or more academic one, enlivened by some very entertaining talks, from Martijn Fassen and the Guardian's Michael Brunton-Spall.

Other highlights included the clear educational style of Raymond Hettinger, who made open to me the internals of twisted, generators and many other things. Along with particularly relevant talks such as
Laurence Rowe's one on XDV which I could go back to work with the next day and immediately investigate as an ideal tool for using with a moodle, mahara, drupal, PHP mashup, via Apache mod_transform.

The spectrum of python users was wide including HPC (parallel processing), maths, games, robotics, cloud computing and the more familiar web uses and platforms.

Dominating was probably the more commercial Google App Engine / Django related stuff, followed by the more academic parallel processing / twisted contingent. One thing I felt was surprisingly absent was the software packaging, management area, which python dominates in the linux world. However maybe thats because sys-admins dont attend language conferences.

I gave a very bad lightning talk in a small effort to remedy this. Next time I will have to remember to decide what I am going to say ... rather than just make sure the slides work!

There was a distinct feeling that zope was on the wain, and that zope3 development was now just Grok, with enthusiasts battling on despite their dwindling numbers.

I guess that raised a concern that although Google's wholesale adoption of Python as its language of choice, and django as its web framework, are great for boosting its profile. There is the danger that it could become so Googlized that other python frameworks that compete in the same arena are drowned out, or that python becomes Google's language just like java is Oracle's.

First django egg release

I have just released my first django egg – see django-oraclepool. This is a packaging of some code available on the django site for pooling oracle connections which gave truly radical performance benefits within our production setup.
Hence it seemed of value to package it and add performance tests, etc. to make it easy for others to use.

(Further details on my django experiences).

Getting up to speed with django

I finally tracked down why my experience of django has been really slow, when supposedly one of its qualities is speed. So it is meant to be faster than a lot of LAMP frameworks, eg. Symfony, and also Ruby on the Rails, so certainly better than a non-rdbms centric platform like zope/plone - yet I was getting really terrible performance.

Initially with a standard Apache set up I was getting 10-20 secs/req ! ... gak unusable what was wrong?
Simply adding KeepAlive Off to the apache conf had a radical effect. Now I was getting 3-4 secs/req.
So pretty terrible but not totally unusable. Things were a bit better under mod_wsgi so 2-3 secs/req.

However the real nub of the issue was Oracle, the backend was not using pooled connection so each page was renegotiating an Oracle connection. With our distributed Oracle server network that could take 2-3 secs (ie around 90% of the request time).

I investigated pyorapool and it doubled performance, but had issues and it seemed overkill to run something more suited to pooling connections across servers.

So I went back to a post about a pooled oracle backend by Taras Halturin, which I initially couldnt get working. Having fixed it for current django 1.1.1 I have packaged it up in our repo as ilrtdjango.oracle_pool I guess if I add some extra features to it such as logging etc, I should contact Taras about punting it up on pypi.


This really deals with the problem by wrapping up cx_Oracle's own connection pooling. Now I am seeing 0.4 secs/req on average, with cached pages down to 0.1 secs.
There is a lot more that could be done with better fragment caching and use of memcache rather than a simple file system backend. But for the moment I am happy that django really is fast, ie. the framework can do 10 req/sec out of the box. With good queries and materialised views for remote databases, using a pooled backend a few Oracle queries should at most halve that speed, so a 5 req/sec uncached baseline.

This is a much better performance point to work from for database applications, and if I need the sort of 50 req/sec performance of a high traffic site then I have all the caching headroom to use (e.g. memcache or a web cache frontend).

Check out ILRT PyPi

ILRT now has its own python code repository and general documentation server at http://pypi.ilrt.bris.ac.uk

So some of the more technical posts from this blog have been moved to the HowTos there.

Any ILRT python code should have its packaged documentation this is now checked out from svn to an eggserver folder on devbox and punted up into web pages via a cron job.
I will look at adding the dump of these to the windows file share as well, so code specs etc. can be tied to release tags and round trip within the code to svn to eggs to the repo web pages to windows share text docs (editable in Word or whatever).

Along with that the repo can hold 'manuals' eg. code club presentations etc. and other bits and pieces. Plus FAQs a useful place to add any quick hints or tips - certainly if you ever waste an hour working out some undocumented thing that in itself only takes 5 minutes, please always add it as a FAQ.

Sorry the actual server is a bit bare plone, it could probably do with being made more like http://www.coactivate.org (a free basecamp stylee plone used for open source project planning)
Or possibly use sphinx, or one of the more documentation centric python tools?

NB: The server is only accessible within the University. Internet development team members can log in via their standard zope accounts if anyone has the urge to edit anything (hopefully!)