Optimizing of simulation computations on the Amazon EC2 spot market

The BidElastic team published an article in a special issue of Simulation Modelling Practice and Theory. The article describes some of the technology and principles used by Bid Server to plan and provision EC2 spot instances. 

Using the Amazon spot price market can significantly lower the execution costs of large-scale simulations that require millions of computations. However, Amazon can interrupt computations on the spot price market when user bids are too low. To complete computations without incurring high costs, a bidding algorithm should be developed that balances costs and completion time of computations.

We have developed such an algorithm by identifying drivers of spot prices on Amazon EC2 and using these insights to propose an adaptive bidding strategy to minimize computation costs and delays due to computation termination simultaneously. It turns out that bidding close to spot prices and dynamically switching between instances is an efficient and simple strategy. To develop and test other bidding strategies on the Amazon spot price market, we have also built a simulator of the EC2 spot pricing mechanism.

The article is available as Open Access on Science Direct.