What is JSON transcoding?

A tool that enables use of gRPC as a single API for microservices and REST web frontend, by automated translation of your gRPC API into JSON RESTful services.

What is gRPC?

gRPC is the name of a language agnostic data transfer framework designed for cloud microservices implemented via HTTP2, created by Google, around the same time they released Kubernetes.
It is to some extent equivalent to the REST standard that was developed over HTTP1.1 and used to transfer data for web browsers. But REST uses plain text JSON, where as gRPC encodes messages as binary protobuf format data.

There are similarities between gRPC development and REST development, hence similar tools.
For scripted API interactions REST has Postman which also has some gRPC support, but there is also Kreya, grpcUI etc.

Why use gRPC rather than REST?

gRPC is 10 x faster than REST and best suited to cross microservice remote procedure calls.

gRPC uses protoc to generate any or all of Go, Python, C#, C++, Java, Node.js, PHP, Ruby, Dart, Kotlin & Rust code out of the box - so allowing your microservice engineers to use native Go, your SREs Python and other integrators their language of choice. Each language can import the autogenerated API libraries for it, which are generated from the *.proto source files that developers create to define the API.

You have a single versioned API defined in one place for all services. This may be a dedicated myservices-proto package with all your source/*.proto files in and a script to run protoc and generate the libraries for each language your company uses. Along with the master API definition file eg. descriptor/proto.pb or proto.pbtext for the bigger human readable version.

The gRPC protocol is strongly typed and allows full validation of data in and out. It is a binary format so more compact. JSON via REST does not allow such control of the data typing and validation. It was designed as a simple serialisation format for dynamically typed HTML page scripting language, Javascript objects. Not as a performant backend RPC protocol for cloud microservices.

Because of the much looser standards around REST and JSON many people adopt the Swagger  framework to help standardize and document their REST API. Whilst gRPC has formal standards in its core protocols.

Why use JSON transcoding?

Why still use REST too?

The first question that occurs is why run a REST API at all? The reason is that whilst it uses large, slow and minimally typed messaging. It is the default standard approach for front end web development. Since web front ends are implemented in Javascript it is natural to build them against a RESTful backend that provides data in the native JSON format.

Added to this, even the latest web browsers have incomplete support for the HTTP2 protocol required by gRPC. This in turn leads to poor support in the Javascript eco-system. Also for devx accessibility, gRPC messages are not immediately human readable.

Perhaps most importantly gRPC was never designed to replace REST. It was designed for fast backend cloud deployed internal service composition.
REST is a web protocol designed for loose coupling services from across the web. Where the loose standards, typing, simplicity and transparency of REST / JSON complement HTML5. Given attempts to impose stricter typing, such as XHMTL were a failure, using gRPC as a front end replacement for REST is asking for trouble. Instead standardizing REST via Swagger, OpenAPI and the like are a more practical approach.

The front-end web world loves REST. But gRPC is far superior for more tightly coupled backend microservices. Given that, JSON transcoding is likely to remain a very useful means of saving on API proliferation and complexity by providing a single API for both, via your edge servers (ie the servers between your cloud deployed services and the internet).

How does JSON transcoding work?

JSON transcoding suits a gRPC centric single master API design. The ideal approach when designing new cloud services built from micro-services. 

It is implemented by using the transcoder plugin which can run in existing proxy servers such as Envoy Proxy The plugin can use gRPC proto source files to autogenerate a JSON REST API without any code generation required.

Alternatively there is grpc-gateway which generates the implementation from the proto files and requires a compilation step. This different implementation under the hood is not strictly transcoding. But in effect it does the same job.

Or for Microsoft language world, there is the JSON transcoder for ASP.net

In all cases you create the REST API based on the google.api.http annotations standard in your *.proto files (add the green text below) to a simple example service ...

syntax = "proto3";

package your.service.v1;

option go_package = "github.com/yourorg/yourprotos/gen/go/your/service/v1";

import "google/api/annotations.proto";

 message StringMessage {

   string value = 1;

 }

service YourService {

   rpc Echo(StringMessage) returns (StringMessage) {

      option (google.api.http) = {

          post: "/v1/example/echo"

          body: "*"

      };

   }

}

The google.api.http set of annotations are used to define how the gRPC method can be called via REST. 

Why use it?

Use a transcoder and you only need to maintain your gRPC and your public REST API in one place - the gRPC proto files - with simple one liner annotations

You no longer need to develop any REST server code. Running up web servers that provide a REST API and have to be manually coded to keep it in synch with the backend gRPC API can be dispensed with. Or if you are running direct REST interfaces from your Golang microservices these can be dropped as less type safe and more error prone. Replacing them with gRPC microservices and replacing data validation code layers in different backend code languages, with proto level data validation in one place. Validation can be your own custom validator code or you can use a full plugin such as buf.build or protoc-gen-validator

Now as you build your gRPC API you also build a JSON RESTful one too. Adding custom data validators at the gRPC level, defining your full API in one place.

The gRPC API annotations give you a performant REST API that is auto generated and run via Envoy Proxy’s fast C++ based edge server - for use by the JSON front end. Automatically transcoding messages back and forth from REST request / responses to gRPC ones.

What about Transcoding other HTTP content types that are not JSON?

It may be called a JSON transcoder but the transcoder can also transcode other content types.

To transcode other http content types, you must use the google.api.HttpBody type. Put the content in the body and set (and call the UI) with the appropriate content-type header. For example for a gRPC CSV file download, eg. getting log files ...

syntax = "proto3";

package your.service.v1;

option go_package = "github.com/yourorg/yourprotos/gen/go/your/service/v1";

import "google/api/annotations.proto";

import "google/api/httpbody.proto";

 message StringMessage {

   string value = 1;

 }

service YourService {

  rpc GetCSVFile(StringMessage) returns (google.api.HttpBody) {

    option (google.api.http) = {

        get : "/v1/example/echo.csv"

    }; 

  }

}

A Go implementation of the method to return the CSV file might be ...

import (

"embed"

"google.golang.org/genproto/googleapis/api/httpbody"

api_v1 "github.com/my-proto-pkg/generated/go/public/v1"

)

//go:embed *.csv

var EmbedFS embed.FS

// GetCSVFile to return a CSV file via gRPC as HttpBody

func (p *Provider) GetCsvFile(ctx context.Context, req api_v1.StringMessage) (*httpbody.HttpBody, error) {

csvData, err := EmbedFS.ReadFile(req.Value)

if err != nil {

return nil, err

}

return &http.HttpBody{

ContentType: "text/csv",

Data: csvData,

}, nil

}

A call to /v1/example/echo.csv?value=smalldata.csv with content-type=text/csv (or application/json) should return that file.

Other content types such as PDF can be similarly handled.

Data streaming large content types

When returning content other than compact gRPC message formats another issue arises.
What if bigdata.csv is 2 Gb in size?  gRPC upload limits are 4 Mb and although download is unlimited it is best to stream anything that may be over that size.

For large messages response streaming content needs to be used.

It is very simple to switch the protocol for gRPC and the transcoded Http REST request and / or responses. If either is prefixed with the word stream then that streaming handling is implemented both in gRPC and for the transcoded REST API. So to stream 2 Gb files from REST change the proto definition as shown in bold red ...

rpc StreamCSVFile(StringMessage) returns (stream google.api.HttpBody) {

Although this is not the only thing to be done. The main work is that the method implementation needs to stream the data too.

// StreamCsvFile to stream large CSV files via HttpBody

func (p *Provider) StreamCsvFile(req *emptypb.Empty, responseStream api_v1.MyProto_StreamCsvFileServer) error {

f, err := os.Open("/tmp/bigdata.csv")

if err != nil {

return nil

}

defer f.Close()

r := bufio.NewReader(f)

buf := make([]byte, 4*1024*1024) // Use 4 MB buffer

for {

n, err := r.Read(buf)

if n > 0 {

resp := &gapi.HttpBody{

ContentType: "text/csv",

Data: buf[:n],

}

if err := responseStream.Send(resp); err != nil {

return nil

}

}

if err == io.EOF {

break

}

if err != nil {

return nil

}

}

return nil

}

When the file is fetched it returns it to the browser as streamed Http - so in chunks.
So if you do this via Envoy and have it on debug mode you can see it being served by a series of responses in chunks via streaming http, in response to a REST get.

1. curl --header 'Content-Type: application/json' http://localhost:8080/v1/example/echo.csv?value=bigdata.csv

2. transcoder translates that request to gRPC http://grpc_host:80/myproto.api.v1.StreamCsvFile

3. The microservice returns a stream of gRPC responses with the file chunked up into them to Envoy

4. Envoy starts serving those chunks as http responses to the curl web client

5. When all the responses are done the task is complete and curl has downloaded the full file and stuck it back together.

When tested out locally on my Mac it downloaded files at over 100Mb/sec so transcoding does not appear to introduce any performance hit.

How to try out Envoy with the transcoder filter

Envoy can be downloaded and installed locally to try out JSON transcoding. You just need to be able to run up the gRPC service for it to talk to and provide the proto.pb API definition to it via the Envoy config.yaml

Envoy is easily installed on Linux or Mac

Once it is installed run up your gRPC service or port-foward it from k8s or kind.

Copy its proto.pb to a local protos/helloworld.pb directory for envoy to access it. Then run up envoy...

envoy --service-node ingress --service-cluster ingress -c envoy-config.yaml --log-level debug

A sample config for running a service is detailed in the Envoy transcoder help page 
Note that the REST path should not be added to the config with the current version 3 transcoder.
The transcoder reads the paths for REST from the proto.pb
It just needs the gRPC dot notation based URL /helloword.Greeter that indicates the gRPC service

15    - filters:
16      - name: envoy.filters.network.http_connection_manager
17        typed_config:
18          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
19          stat_prefix: grpc_json
20          codec_type: AUTO
21          route_config:
22            name: local_route
23            virtual_hosts:
24            - name: local_service
25              domains: ["*"]
26              routes:
27              - match:
28                  prefix: /helloworld.Greeter
29                route:
30                  cluster: grpc_myproto 

When port forwarding from a k8s running service something along the following lines could be used to point Envoy at it. 

clusters: - name: grpc_myproto type: STATIC connect_timeout: 5s http2_protocol_options: {} load_assignment: cluster_name: grpc_myproto endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 port_value: 9090

Note that the sample config from the envoy transcoder help page is missing some useful config that you may want in production, eg.  convert_grpc_status: true, so that if gRPC errors occur, they are transcoded and returned in the body of the http response. By default (== false), the body is blank and the gRPC error is only in the header. There is a full list of config options available.

Testing

If you want to write tests against REST in Go (rather than just testing the gRPC) you will need the Go JSON annotated structs to test with, that marshal JSON back and forth. 

Since the REST API is transcoded on the fly by Envoy then these don't exist in any of your source code. To generate them you need to use command line translation tools.

gRPC proto.pb > buf.build > OpenAPIv2 at this point you have a JSON open API spec which can be used to create your REST tests in Javascript or any language with OpenAPI v2 support.

To do so in Go we can continue the pipeline with OpenAPIv2 > OAPI-codegen > generated_v1.go 

A test using httpexpect (to clean up the raw syntax of http) would be something like ...

resp := generated_v1.GetCsvResponse{}

e := httpexpect.Default(t, restSvcURL)

e.GET("/v1/example/echo.csv").Expect().Status(http.StatusOK).JSON().Decode(&resp)

assert.Equal(t, resp.Data, sampleCSV)

(this assumes our example GetCsvResponse embeds the google.api.HttpBody as Data and includes other fields)

Picking a language for shell scripting based on its framework ecosystem

• Do you work somewhere that has a lot of old shell scripts? 
• Were they all dashed off quickly according to the whims of their original creator - just deploy scripts so maybe they missed some of the care (and documentation) the application code had?
• Whenever a major revision of said shell scripts is needed, do all of them end up in the bin - sorry trash in case you thought I meant the /usr/bin. Because its easier to write such platform and author specific scripts from scratch than maintain them?

Well the team I recently joined does. So as somebody who personally moved from shell scripts to a shell compatible object orientated scripting language some time ago. Then on to a shell framework, I was asked to assess what to move our menagerie of bash, zshell, cshell, powershell etc. scripts to.

My remit was to recommend a language instead of all those developer specific procedural shell scripts. After all the point of a mainstream language is to introduce cross platform code with a huge set of useful libraries, along with hopefully some standard approaches. Hence get above the sea of shells.

However my preference is to move the team to using a shell framework.

Because what a framework does is give you opinions. It chooses how to do stuff, so you ... and all the developers that come after you, don't have to. In my opinion most of the productivity comes from everyone becoming familiar with the same way of doing things ... whether or not its the best way
(and if it isn't the best way the framework developers spend a lot of time fixing that way with hopefully the minimum of disruption to its API for the end users).

I felt it would be useful to see how active the shell framework ecosystem of a language was, in order to assess how suited to shell scripting it is. Since the goal of a shell framework is essentially the same goal, standardisation of shell scripts. So one indicator for the most suited language is to list the mainstream shell frameworks for the contenders and see which comes out top.

It was decided we will choose from Javascript, Python or Perl. So this is for a team that almost exclusively use Java, which due to its pre-compiled statically typed nature is inherently poorly suited to shell scripting.
Unfortunately Ruby was not to be a candidate. However I have added it anyway, because if its good enough for Amazon, and leading configuration management frameworks such as Puppet ... it should be included.

Strangely it is difficult to find definitive lists of shell frameworks ... perhaps whilst I thought the term had been around for 10 years or more, it has not been coined very effectively ... its not even on Wikipedia ... maybe I made it up! Anyway they exist whether or not there is a proper name for them ... they also often cross over the grey area into being full blown configuration management tools.
So to avoid that argument ... I will add those in as well. I have marked entries with a * where I think it is more a config management tool than a shell framework.

For configuration management systems wikipedia does have an entry ... so using it as a similar benchmark it gives a relative language score of:

Python 11, C 5, Perl 5, Java 4, Ruby 3,
and 1 each for C++, Scala, Erlang, Go and PHP  (none for Javascript)

So on the same sort of  basis I have ranked the languages with the most healthy shell framework ecosystem ... in the belief that it is a good indication that they may be the most suited to being used for shell scripting ...

Python 8

Python has the largest number of well established shell frameworks, as it does in terms of configuration management systems. So on that basis I could award it first prize as the most used language for standardising shell scripting. However numbers are not everything. It doesn't necessarily have the leaders of those two categories of software (see Ruby below)

1. fabric
2. cliff
3. python-deploy-framework
4. clustershell
5. builtoncement
6. ansible *
7. salt *
8. bcfg *

Ruby 6

Ruby has capistrano which is possibly the most popular of all shell frameworks, it also has two of the most popular configuration management systems, Puppet and Chef, so on that basis it should perhaps be first or at least joint first with Python. As the most appropriate language for modern shell scripting.

1. capistrano
2. commander
3. mina
4. vlad the deployer
5. puppet *
6. chef *

Perl 3

Perl has things that perhaps the other languages don't in related areas. So a number of shell implementations in Perl, and frameworks for writing custom shells. But in terms of what I mean by a shell framework - ie. a tool to run shell commands in a standard way across different servers, shells and platforms it is surprisingly lacking.

1. Shell::Tools and Shell::Tools::Extra
2. Rex *
3. quattor *

Javascript 2

Ummm well Javascript may have broken free of the browser with node.js and other server side runtime environments. But are there any shell frameworks or configuration management systems built with it? ie. tools designed for running shell commands and doing anything other than Javascript installation. Not much TBH and certainly no configuration management engine.

1. commander.js   (clone of Ruby commander)
2. shelljs is the leading bunch of utilities for node.js that can act as a shell framework

Conclusion

Its a tie between Python and Ruby, so the decision as to what to go for depends on the individual tools that best suit your shell framework requirements, what they may need to integrate with wrt. configuration management and deployment tools ... plus the existing skills of your developers.
Oh and whether diversity is more important than being the market leader ... or vice versa.

For my own case, Ruby isn't in the running, so that makes it Python based on this premise.
I may follow up this blog post with one comparing the languages wrt. a sample shell script - and see which looks the most maintainable. As a second means of deciding between them.

Note: One other factor is related virtual frameworks ( ... yep made that term up too!)
Basically these wrap up configuration and deployment of virtualised software deployments - either hypervisors or containers - to give a full platform application build - usually for development purposes.
So here Ruby is well ahead with Vagrant along with box grinder

Lessons learned from setting up a website on Amazon EC2

I recently got involved with helping someone sort out their website on an Amazon EC2 instance, it had been a few years since I had the need to do anything with EC2, I realised that I was a novice in this world - and it raised a number of issues related to deploying to EC2 and performance.

So I thought it may be useful to run through them for any other EC2 novices who are asked to do something similar, and want to learn from my rather blundering progress through this :-)

Apologies for those of you are already well familiar with EC2 for covering some of the basics.

The system moodpin.co.uk was based on a commercial PHP application, Pintastic.
So this allows you to set up a site like pinterest.com or wanelo.com
These sort of sites are for creating subject specific photo sharing social media systems, so like Instagram, Picassa etc. but focussed around communities of shared (usually commerical) interest. For example buying shoes, interior decor etc.
The common UI that they tend to present are big scrolling pages of submitted images related to topics for sharing, comment and discussion.

So this system sends out a lot of notification emails, involves displaying hundreds of images per page - the visual pin board - and to help with performance has custom caching built in - triggered by cron jobs.

Hence we have a number of cron jobs with the caching ones running every couple of minutes. To me this appeared a pretty crude caching mechanism - but my job was not to rewrite the application, but just tweak the code and get it all running OK.
The code mainly uses a standard MVC approach like everything else these days!

So demonstrating how outdated my knowledge of EC2 or this application were. I thought OK - first of all what platform is it. It was Amazon's own Linux - this uses yum rather than apt for package installs so as distros go its perhaps more Redhat-like than Debian.

For those unfamiliar with the basics - go to Amazon web services and sign up!
You can then choose to add some of the 40 odd different services that are available under the EC2 umbrella.

Once you have signed up to a few of these, you get a management console that links to a control dashboard for each service. The first step usually being - the one with the computer instances on, EC2. From there you can pick an AMI (ie. operating system image), a zone - eg. US West (Oregon) and use it to create a new instance. Add an SSH key pair for shell access and then fire it up and download the pem file so you can ssh into your new Amazon box.

So the client wanted the usual little tweaks to PHP code,  CSS tweaking - so easy stuff its just web development ... done in a jiffy (well after digging through the MVC layers, templating language, cache issues and CSS inheritance etc. for a fairly complex PHP app you have never come across before, when PHP is not exactly your favourite language ... jiffyish maybe)
Then we got to the more SysAdmin related requests ... lets just say I probably shouldn't rush out and buy a DevOps tee-shirt just yet ...

'Get email working'

1. Try to send an email from the web application - write a plain PHP script that just sends a test email - just run mail from the linux command line ... Got it there is no MTA installed! 
2. Install an MTA - sendmail. Go back up that stack of actions and they are all working ... hurray that was easy.
3. A week or so later ... 'emails stopped working'
4. Go back to step 1. and yep - emails stopped working
5. Look at the mail logs and see what the problem is.
6. Realise that there are masses of emails being sent out ... but all of it is bouncing back as unverified.
7. Think ... wow that pintastic site's notifier is busy - must be getting lots of traffic *
8. So why has Amazon started bouncing all the email?
9. Search Amazon's docs. Amazon has a very minimal test quota allowed for email. Once that quota is filled, unverified email will be blocked.
10. Amazon has historically been one of the main sources of SPAM machines, that history means that it has to set up a much more elaborate mechanism for validating email that most hosting companies, and it no longer allows direct emailing from EC2 boxes (apart from minimal test quotas)
11. So what we need to do is set up our mail to be sent via the Amazon SES service - add SES service and enable it
12. So now we need to send authorised emails to the Amazon SES gateway that will then forward them on to the outside world
13. Try to get sendmail to send authenticated emails, follow guide but it continues to bounce with authentication failure, give up and install postfix, follow the 20 steps of setting up the SASL password etc., and eventually it doesn't bounce with authentication errors - hurray!
14. But the email still bounces. So we need to verify all our sending email addresses - managed by the SES console - or use DKIM to get the whole domain verified and signed from which we are sending.
15. Modify the emails used by the sending software to ones which we can receive and validate - send and validate them. Our emails are working again.
16. Leave it a few days, we are not sending email anymore, boooo!
17. Check all the SES documentation, surprise, surprise SES also has quota limits for test level only, and you have to formally apply to get those limits lifted.
18. Contact the client and get him to make a formal request for quota lifting on his account.
19. *As part of the investigation check that email log a little more closely, it seems rather large, and we seem to be using up our quotas really quickly ... ah the default setup for unix cron sends an email for every job that returns text. The pintastic cache job returns text, so we are sending a pointless email every two minutes ... or trying to ... whoops. Make sure no cron or other unix system command is acting as a SPAM bot.
20. A few days later - Amazon say our quota has been lifted
21. Our emails have started sending again ... and they are still sending today !!!

Clients response, OK thanks, by the way since we added all the start up data / ie. uploaded images, the site takes at least two minutes to render the home page - or times out altogether.
Hmmm I did kinda notice that ... but hey he hadn't asked me to make the site actually usable speed wise ... until now!

'Why is the site, really, really slow?'


Hmmm wow it really is slow, lots of the time it just dies, that PHP cache thingy can't be doing much, so whats the problem.

1. Lets look at the web site, wow it takes 5 minutes for the page to come back ... so this isnt exactly Apache bench territory ... run up a few tabs looking at the home page ... and it starts just returning server timeouts.
2. So whats happening on the server ... whats killing the box ... top tells us that its Apache killing us here - with 50 odd processes spawning and sucking up all memory and CPU.
3. So we check out our Apache config and its the usual PHP orientated config of MPM prefork. But what are the values set ... they are for a great big multiprocessor cadillac of a machine, whilst ours is more of a smart car in its scale. 
4. Lesson is that Amazon AMI's are certainly not smart enough to have different image configs for different hardware specs of instances they provide. So it appears they default their configs to suiting the top of the range instances (since I guess they cost the most). If you have a minimal hardware spec box ... you should reconfigure hardware related parameters for the software you run on it ... or potentially it will fail.
5. Slash all those servers, clients etc. values to the number of servers and processes the server can actually deliver. Slightly trial and error here ... but eventually we got MaxClients 30 instead of 500 etc. and give it a huge timeout.
   
   <IfModule prefork.c>
   StartServers       4
   MinSpareServers    2
   MaxSpareServers  10
   ServerLimit      30
   MaxClients       30
   MaxRequestsPerChild  4000
   </IfModule>
6. Now lets hammer our site again ... hurray it doesn't completely fall over ... one day it may return a page, but its horribly horribly slow still ie. 3 minutes absolute top speed - further home page requests the slower they get.
7. So lets get some stat.s, access the page with browser web dev network tools. Whats taking the time here. Hmmm web page a second, not great but acceptable, JS and CSS 0.25 sec, OK. Images hmmm images hmmm for the home page particularly ... 3-6 minutes ... so basically unusable.
8. So time to bite the bullet we know Apache can be slower at serving static pages if its not optimised for it - especially if resources are limited (its processes have a bigger memory overhead), thats why the Apache foundation has another web server, Apache Trafficserver , for that job
9. But whats the standard static server (thats grabbed half of Apache's share of the web in the last few years), yep nginx
10. So lets set up the front end of our site as nginx acting as a reverse proxy to Apache just doing the PHP work, with nginx serving all images. So modify Apache to just serve on 8080 on localhost and flip the site over to an nginx front end, with the following nginx conf ...
    
    server {        listen       80;
            server_name  moodpin.co.uk;
                                                                                                                                                                                                       
            location ^~ /(cache|cms|uploads) {
                     root   /var/www/html/;
                     expires 7d;
                    access_log  /var/log/nginx/d-a.direct.log ;
            }
                                                                                                                                                                                                      
            location ~* \.(css|rdf|xml|ico|txt|gif|jpg|png|jpeg)$ {
                     expires 365d;
                     root  /var/www/html/;
                    access_log  /var/log/nginx/d-a.direct.log ;
            }
    
          location / {
                proxy_pass         http://127.0.0.1:8080/;
     
    Wow, wow, so take that 3-6 minutes and replace it with 1-2 seconds.
11. So how many images on the home page - about 150 plus more with scrolling ... so that means we have a site that is on average under 0.5% dynamic code driven content and 99.5% static content/requests per page.
    That is a very very static site - hence the 100 x faster speed!
12. So there you go client take that souped up smart car and go 
13. Client replies ... ummm sites down - server proxy timeout error
14. Go to Google and check, so we have to make sure that nginx has timeout settings greater than Apache's - and nginx default timeout is 60 seconds
15. Make nginx _timeout settings into 10 minutes ... sounds bad, try the site, and it consistently delivers pages in 3 seconds or so assume that the scrolling request update page nature of the app, makes the timeout required much longer than the apparent time Apache is delivering PHP within?
16. Show the client again, hes happy.
17. Few days later ... this bit of the sites not working now
18. Check the code, discover that there is a handful of javascript files used by the system that are not really static - they are PHP templates generating javascript that appear static. Remove js file types from the list of files above in the nginx config. Hurray generated javascript served from Apache PHP now. Bit of site works again
19. OK we are done ... don't run Apache bench against the site ... if the client actually gets any users and it cant' cope - tell him to upgrade his instance.

I hope my tails of devops debuggery are useful to you, Bye!
   

Finally I have left behind the printed word

The printing press was invented by Gutenberg getting on for 600 years ago, as a technology the printed book had a good innings compared to the cassette tape or the floppy disk. But maybe its time to recognize that at least as far as text only mass paperback printed media goes, its unlikely to reach that 600th birthday. So I finally decided to stop being a Luddite and buy an e-reader this Christmas, along with 1.3 million other people in Britain. I actually bought it for my partner but I have ended up hogging it.

So what's so great about an e-reader?, well its a good excuse to get back into the classics, anything out of copyright is available free. Over the last month I managed to plough through a book a piece by George Orwell, Thomas Hardy, Charles Dickens, Joseph Conrad and Emily Bronte. But in addition to that any book is available instantaneously either from repositories or Amazon and its competitors. Or for the less scrupulous there is the twilight world of file sharing networks - where all e-books are available for free.

Once the habit is formed of reading via a Kindle or other e-reader it quickly becomes just as natural as turning pages - I prefer the cheaper LCD grey scale readers without touch screen or keyboard - they are the simplest direct replacement for a paperback, and if current price wars continue will soon cost about the same as a hardback! For that you get wireless, space for 1500 books and a month battery life.

So you may be thinking - OK big deal - but why are you talking about this on a python blog? Well the reason is I fancied reading other digital text formats on my e-reader - what if I want a copy of a pdf, some sphinx docs or even a whole website for example. I soon came across the open source Calibre software, written in Python and C and available on linux, Mac and Windows. Kovid Goyal may not be Gutenberg - but he has certainly produced a really good e-book management software package.

Once you have installed the software it sets up that machine as your personal e-book library, to which you can add pretty much any text format you wish, along with news feeds or other HTML content. Plug in your e-reader then press the convert e-book button to translate them to .mobi or whatever, and another to send to the device. BeautifulSoup is used to help with HTML conversion and there is an API for creating conversion recipes. The software also includes a server to make your personal e-library available to you over the web, book repository browsers, synchronisation tools, handling of annotations, etc.

Friends talk about missing the tactile experience, the smell, but with a good e-reader and good e-book management software - I can't really justify lugging around those funny old glued together sheets of  bleached wood pulp any more ;-)