I'm a bit skeptical on the fact that such companies do not disclose their custom infrastructure design. So their value is both in hardware optimizations as well as software ones ?

being a grad student myself, I'm wondering if we'll ever see any specific technical aspects on infrastructures that could drive systems research and provide real problems to be attacked...

I'm skeptical that there's anything that innovative going on behind the veil.

I suspect if you had access you would look at a whole bunch of evolutionary improvements and probably trace them back to which papers you've seen it sort of mentioned in.

Probably not as much custom silicon as the article implies either, simply because on the scale of things Amazon is not a semiconductor company and Intel pretty unambiguously own the cutting edge there.

Facebook reveals their infrastructure through the Open Compute Project since it can't compete against their core business.

are there any particular performance improvements over the previous versions ?

I was thinking basically of transaction management, especially the autocommit behavior.

I browsed through the docs and saw that the default python database API requires autocommit to be turned off... but then again django overrides this behavior.

any comments on this ? (or am I completely lost ?)...

> Improved transaction management¶

> Django’s transaction management was overhauled. Database-level autocommit is now turned on by default. This makes transaction handling more explicit and should improve performance. The existing APIs were deprecated, and new APIs were introduced, as described in the transaction management docs.

> Please review carefully the list of known backwards-incompatibilities to determine if you need to make changes in your code.

> Persistent database connections¶

> Django now supports reusing the same database connection for several requests. This avoids the overhead of re-establishing a connection at the beginning of each request. For backwards compatibility, this feature is disabled by default. See Persistent connections for details.

It's hard to imagine using an ORM for anything high volume that just got around to adding (still very limited) connection pooling. I'd tend to think anyone overly concerned with performance would use SQLAlchemy, if they use any ORM at all.

> I'd tend to think anyone overly concerned with performance would use SQLAlchemy, if they use any ORM at all.

Or they did connection pooling outside the development framework.

right tool for the job. Djangoś ORM is great for simple stuff. And it works with the admin interface (which is worth a lot). If You need something complex enough, use raw SQL.

The admin was too much to give up to swap the ORM to SQLAlchemy for me.

I think a lot of people might've been using an external connection pooler, eg pgpool.

Actually I was wondering if there is any experience/benchmarks about Amazon high-performance instances.

anyone already used this ?

I am using a number of high-performance instances - cc2.8xlarge machines for EMR jobs, and cr1.8xlarge machines for analytics databases. From my testing:

- Running EMR jobs on cc2.8xlarge machines as spot instances is a great way to get a LOT of computer power very cheaply. Because our jobs are periodic we run both Core and Task as spots and simply retry the job if our machines get terminated. I did a lot of benchmarking and found that a small number of cc2.8xlarge machines out-performs and is cheaper than a large number of lesser instances (and I tried most of the lesser machines). In us-west-2 it's very uncommon to lose our instances, unlike us-east-1 which has major price fluctuations (this is true for all types of spot instance).

- The cr1.8xlarge has fantastic performance, relative to the rest of the AWS machines. It's also very expensive compared to the cost of hardware or a similar solution on another cloud provider. Since we're fully integrated with AWS and don't want to run our own hardware we're sucking up the cost for now, but it's definitely a sore-point in our budget. The cr1.8xlarge is also all-round a better machine than the hi.4xlarge, which has a lot of disk but is pitiful in terms of CPU.

One thing to note with EMR: you still pay 25% of the ondemand price as overhead to use EMR. If you're bringing up and turning off clusters all the time, it's probably worth it, but you might want to look into using Whirr instead.

You've a good point about the EMR charge - that's easy to overlook.

I took a look at Whirr [1] but I don't see how having a cloud-agnostic platform helps - are there really alternatives to EMR out there? Can they give me 500+ cc2.8xlarge equivalent machines on-demand but at spot prices?

[1] Assuming this is the Whirr to which you refer: https://whirr.apache.org/

Whirr just uses the AWS APIs to provision a cluster for you, but then you're getting a cluster built on EC2 instances rather than EMR, so you don't pay that EMR overhead. You can choose spot or ondemand instances. If your spots get reaped, I think it would fail pretty similarly to an EMR cluster built on spots. I have no idea how quickly it could provision a 500 node cluster, however.

Ahh, I see - I thought it was about being able to move my EMR jobs to other providers. Running on EC2 without the EMR charge would be nice, and assuming that there's no special casing for spot requests for EMR (I don't know of any, the spot instance requests show up like normal ones) then it may be well able to get me that many instances.

Thanks for the tip!

great stuff!

thanks for sharing.

sysrq ftw. Really useful option. AFAIK alt-sysrq-e and -i are disabled by default in new kernels

go buy bitcoins ;-)

Well I guess that's expected behavior. So you may need to explicitly state that the feature works if you save an image on your clipboard -- not just pressing printscreen ;-)

nice work for a weekend app!

why would you say that ? is it a provocative question ?