A good post on engineering complex systems. He talks about the components that go into building resiliency

http://www.kitchensoap.com/2012/06/18/resilience-engineering-part-ii-lenses/