War in the skies and some other rumors from the performance engineers world

I’ve just returned from ICPE2014 conference taking place in Dublin. Apart from lots of cloud, VM and power scaling related talks there were many that applied to what I do – building and maintaining SW performance models, creating and working with load tests and lower level stuff like niceness in Linux and CPU cache.

Some fancy facts and rumors I’ve picked up:

  • Amazon estimates that 100ms in latency on their pages costs 1% in its sales.
  • Google says 0.5s lag results in 20% drop in their traffic.
  • Bank of America had 2 performance engineers earning 1 million $ a year. New manager came to cut costs and fired them. After a year the performance dropped so badly that Bank had to buy additional servers for 15 million $. Unfortunately after another year it was not enough to support the load and other expenses (and sacking of the said manager) had to follow.
  • One mentioned bottleneck bugs during conference was bad implementation of visitor pattern (we also had issues with those).
  • Lots of attention given to optimal cache utilization (e.g. use structure of arrays rather than arrays of structures).
  • In 2012 already data centers used power equivalent to that of 30 nuclear power plants.
  • Dark shipping – a technique to test your new SW by transparently routing request in parallel to your old and new SW. Customer only gets results from old SW but you observe how new SW works and if it provides same output.
  • New CPU accelerator Intel Xeon Phi (60 CPU ~= 1 TFLOP) still slower than NVIDIA CUDA Kepler GPU due to different cache memory structure – you really need to put a lot of effort in parallelizing your program (like manual vectorization) to utilize such accelerators.

All in all it seems it pays off to be a cloud expert those days as there is a kind of arms race and cloud war going on with Google aggresively dropping its prices. During the conference there was a lot of talks on scaling of the cloud and its power efficiency so there is definitely something in the air ;).

Some photos from my stay in Dublin: