A website's content journey

I had an email from an old friend a few weeks ago. He had a site with a bunch of work related research pages and documents collaboratively edited by a handful of people, and had let its domain lapse. It was a 6 year old plone site that he and his fellow editors suddenly wanted back up, for another 5 years, but had no cash to do so. It also needed to move hoster. The first thing to do was re-register the domain, which costs around 40 dollars for a cut price option - now I needed the site back to point it to. Initially I assumed a static dump was bound to be the quickest and cheapest option.

I had a look at some static site python tools such as hyde, this is a django based mirroring tool inspired by Ruby's Jekyll. Where simple database apps such as blogs can be dumped to static files for performance, whilst still being editable. However my friend was not a techie so was unlikely to be able to cope with file system editing of text 'content' files. So for the time being I just ran httrack over the site to dump it to the file system. Next I copied it over to a 'free' amazon micro instance. Since this was a static site, using Apache also seemed overkill, and I thought it was long overdue that I tried out nginx.
However there proved to be almost nothing to try out, since the default Amazon AMI comes with an nginx package. All you need do is add a new micro instance, start it up, run
>sudo yum install nginx
and copy the contents of a httrack dump of the site to /usr/share/nginx/html
Thats it. It was very fast and the config was very simple. A big thumbs up for nginx then, and I also quite like its Russian constructivist styled website, especially now the original Russian only documentation has many translations ;-) The final stage was to assign an Amazon elastic ip to the instance and point the domain registration at that ip.

Great the site was back and seemed pretty nippy, however two problems - it was no longer a CMS, and secondly Amazon's free micro instances are actually only available as a month of uptime hours free trial. After that the hosting was a lot more expensive than a standard minimal shared hoster, and neither option were free. So if hosting was to be paid for I might as well do a proper job and upgrade the plone site to current plone 4, so making it a CMS again.

Fortunately I released a tool that does just that, called ilrt.contentmigrator, a year or so ago.
It takes content from old plone (eg 2.0) and exports it to a simple email style format of content and metadata, that can be reimported to a new plone site. Only problem was I hadnt updated all the tests and bits and pieces to release a plone 4 version yet. But since 4 had been out for some months, it was high time that I did, and this was the excuse I needed. I got the upgrade done and exported the site, where it ran happily in plone 4.
So now I had a working CMS back up on the old domain, and could run it up on an Amazon micro as a fully featured CMS again. So email my friend - its back you can edit it, only problem - its going to cost about 20 dollars a month.

Ahhh now of course I should of recalled that one of my friends defining characteristics was being a tight wad - the idea of paying hundreds of dollars over the next 5 years meant the site was effectively down again! So back to the drawing board. Ok so with all these free services / cloud technologies out there these days, there must be a cheaper solution. A quick hunt around and the answer was obvious, the CMS had no sophisticated features, so a free Google site would easily cover my friends requirements whilst not requiring the dropping of the site-like strucure and collaborative document nature that a simpler blog, such as a Wordpress solution might do.

So set up a Google site, now put the content in. Well of course I could just tell my friend to cut and paste it all, but Google has a pretty extensive data and provisioning API. I had already written a content migrator for Plone. Why not make it work between Plone and Google sites API as well. So using the python wrapper for the restful Atom feed based Google data APIs, I added an export tool that writes the basic content types and folders from Plone to a Google site.
The two share in common the storage paradigm of a NOSQL database and a folder like interface to content creation. Plone has an inherent folder like storage paradigm at an internal level implemented via its acquisition mechanism within the ZODB, whilst Google sites have a much thinner skin of folder like behaviour added by parent child node properties to the objects stored in its BigTable hash table cloud (the shared storage behind site, apps, app engine etc.)
As it turned out this meant that the writing of a migration tool to push the more metadata rich content from Plone to Google was quite straight forward. I rewrote the import to Plone script as an import to Google one, using the gdata library. So the site was up as a Google site. Change that domain ip again, and my friend had his site back, for free, hurray job done.

However I couldn't quite leave it there. I had written a tool to move simple Plone sites to Google for free hosting. But there was probably at least as big a use case for moving more limited design, content types and workflow Google sites to Plone, when those sites customisation demands have outgrown their Google site origins. On top of that I should really write some tests and package things up properly to add these features to my new ilrt.contentmigrator release.
As it turned out migrating from Google site to Plone was a little harder, partly because the Google sites restful Atom API, doesn't expose the folder tree layer
of content by default. So all content is available from the root feed, but it misses empty folders out. Also there seemed to be a bug with getting the folder's (or filecabinets as Google sites calls them) summary text. I guess the API is still in Labs status so this may be fixed in time.

Anyhow I have released it as a first version for the standard folder, pages and file attachment types. So I hope somebody has reason to use the tool's new features, and can give me any feedback when they do.

Plone conference keynote

This is an attempt to paraphrase Alexander Limi's keynote speech, rather than reflect any of my own views (added as comments instead).

Intro

Good to see two new plone books are out, e.g. Erik Rose
plone 3 for education -
and contrary to Google trends - usage is still increasing.

plone 4/5 roadmap:

native blobs in 4

performance should be twice as fast as 3 -
largely due to Chameleon and its compiled ZPT template acceleration (also used by Pylons for Genshi)

code base should be 30% smaller

deliverance - more into the core

dexterity - to replace archetypes but will still have AT in 5

deco - UI layout design framework using tiles to replace viewlets and portlets. All of the page elements can be drag and dropped rearranged.
- so edit mode becomes a design mode too! (Live demo of deco drag and dropping layout followed)

Issues about promoting plone

Cant compete against cost and simplicitiy of PHP CMS for web page publishing - so sees goal as the high end of the market - enterprise collaborative CMS, easy to extend with simple apps.

Hence the competition is Alfresco, RedDot, Sharepoint etc - Plone community need to create standard content importers from these.

Fix plone.org to drive plone again. More user groups, etc.

Check out ILRT PyPi

ILRT now has its own python code repository and general documentation server at http://pypi.ilrt.bris.ac.uk

So some of the more technical posts from this blog have been moved to the HowTos there.

Any ILRT python code should have its packaged documentation this is now checked out from svn to an eggserver folder on devbox and punted up into web pages via a cron job.
I will look at adding the dump of these to the windows file share as well, so code specs etc. can be tied to release tags and round trip within the code to svn to eggs to the repo web pages to windows share text docs (editable in Word or whatever).

Along with that the repo can hold 'manuals' eg. code club presentations etc. and other bits and pieces. Plus FAQs a useful place to add any quick hints or tips - certainly if you ever waste an hour working out some undocumented thing that in itself only takes 5 minutes, please always add it as a FAQ.

Sorry the actual server is a bit bare plone, it could probably do with being made more like http://www.coactivate.org (a free basecamp stylee plone used for open source project planning)
Or possibly use sphinx, or one of the more documentation centric python tools?

NB: The server is only accessible within the University. Internet development team members can log in via their standard zope accounts if anyone has the urge to edit anything (hopefully!)

Another ILRT egg released

Finally got around to finishing and releasing the egg for generating content / plone sites matching different profiles - e.g. intranet, public etc.

http://pypi.python.org/pypi/collective.contentgenerator ... our second cheeseshop / plone.org release and the first that I acted as release manager for.

Technical gotchas regarding releasing code

Do all your metadata in the right places first then use setup tools to regenerate the metadata files.

So my recommendation is start at the top with setup.cfg which should release tag changed from dev to nothing and svn from true to false ... or else you get _rSVNnumber appended to your release versions.
Next setup.py add your metadata here (classifiers can be found on the pypi form dropdown) ... then make sure the version you specify in setup.py is copied to the one in egg.name/version.txt
Next get your final text ready in README.txt and HISTORY.txt (using restructured text)
Now run
> python2.4 setup.py egg_info
this regenerates your.eggname.egg-info/PKG-INFO and the other metadata files. So you can check them before uploading.
Finally you cpuld upload PKG-INFO to pypi by hand but is easier to do it all in one fell swoop via setuptools register.

> python2.4 setup.py register sdist bdist_egg upload

(the sdist makes a source tarball and the bdist_egg and egg)

If you really need to do final tweakings ... via the pypi web form ... you can do so but you will then have to go to the generated PKG_INFO link on pypi and copy and paste the results back into README.txt etc., rerun egg-info and re-upload your egg tarball ... I did all that at first and then realised that was a really bad idea!

Lastly all you need to do is go through the whole process again via different web forms on plone.org ;-)
... though I think this is being addressed?

* Oh and one final bugbear ... where do you set platform in the metadata?

Plone performance sprint

I (Ed Crewe) took part in the Bristol Performance sprint - or at least the first two days of it.
The project site details topics explored.

Key areas for performance increase looked at were:

ZODB storage paradigm - Lawrence Rowe et al.
ZCatalog querying (for set ops), splitting and queuing. - Jarn people
Chameleon - This precompiles page templates into python for execution speed improvements ... the guy who wrote it and the ZPT implementaion ... Malthe Borsch was looking at improving its effects.

Benchmarking and load testing

Alongside this there was the need to establish a testing framework and benchmarks to test against. The aim being to have a set of standard content profiles for realworld sites ... along with an easy to install load testing framework. Hence provide a set of standards with which to compare performance over time and code versions.

We decided to go for funkload over jmeter to reduce cross language dependencies and make the load testing code more accessible to plone / python developers. A recipe / buildout and set of test plans is being developed to make the installation of a standard load test benchmark available.

My part of the work was to do the content profiles ... so there is now an egg in the collective that generates content according to various options to provide a set of content profiles ... currently just going with small intranet and large public sites. Details in the README.

Hopefully collective.contentgenerator will be finished off within the next month or so, along with the load test recipes. They could then be used for a standard performance report on plone.org as versions come out.

NB: Also punted up the performance fix for placeful workflow for the latest version in terms of patch files.

World plone day 2008

I am taking part in a meeting of plone developers in Bristol for world plone day.

I will be giving a couple of lightning talks

A glimpse of what we did with plone for the University of Bristol CMS

Egg start - how and why I started contributing back to open source in python and plone

... so please come along ... it is only for an hour or so tommorrow lunchtime.

----------

Here is the video of the event.

General python

Welcome to ILRT Python. This blog is for platform related issues, how tos, gotchas, comments and rants. It was set up after having bothered to blog the plone conference 2008 ... it seemed sensible to keep an ILRT python stream of conscienceness (or stream of something) going.

Note: Any ILRTer is welcome to be added as another editor for the blog, anybody in ID whose is registered with blogs.ilrt is already one ... so please stick something up here!