How to Program AWS Spot Instances to Strategically Bid So the Auction is Never Lost Until a Competitor Beats the Maximum I'm Willing to Pay?

Question

I believe I'm in the right section of stack exchange to ask this. If not, let me know.

I only use Amazon Web Services for temporary type hosting services, so the spot instances are quite valuable to me. I would also just make an instance and start and stop it - but - that doesn't necessarily fit my bootstrapped budget sadly.

Anyways, it really kills me when someone outbids me on a spot instance I have (I tend to go for the larger ones which there are fewer of available) and I get randomly kicked off. I know or at least I believe there is a way to program in something somehow to dynamically change your bidding price to beat a potential competitor's if their's is higher than yours.

Now, I previously believed Amazon would just charge me for the highest price right above the next lowest competitor automatically (eliminating the need for this) - so if I bid too high, then I only pay what I would of needed to in order to win and keep the auction. Essentially, I thought my bid price was my max bid price.

Apparently, according to my bills and several experiments I've done - this is not the case. They charge me for whatever I bid even when I know there is no one else around to counter bid me.

I needed to clarify that, but let me get back to the main point:

Let's say I'm bidding $0.50, competitor comes in and bids 0.55 cents. I get kicked off. I want to have it to where I'd set a maximum I'm willing to pay (let's say $1.00 here), and then when competitor comes in and tries to bid $.55, my bid is dynamically adjusted to beat his at $0.56 up until he breaks my $1.00 threshold.

I've been reading the guides and although they are more or less straightforward, I feel like they leave a few holes in them that end up confusing me. Like, for instance, where do I input said command or when do I do it? Maybe I'm just tech illiterate and need help deciphering these guides.

A good start for someone willing to answer/help me decipher this problem would be here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-as-update-bid.html

Answer 1

There is no secret methodology to be determined here, and no need to do what you are trying to do.

You should bid what the machines are worth to you, because you will not pay any more than the market demands.

Now, I previously believed Amazon would just charge me for the highest price right above the next lowest competitor automatically ...

I'm not sure where you got that idea.

There is a fixed spot price at any moment in time for all instances with all four of the following identical configuration elements:

• same availability zone
• same instance class
• same platform (Linux/Unix, Windows, or SUSE)
• same product class (VPC or EC2 Classic)

“Everyone pays that same Spot Price for that period regardless of whether their maximum bid price was higher”

^{— http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html}

In light of that, you are either mistaken with the following assertion, or you need to contact AWS support, because this is incorrect:

They charge me for whatever I bid even when I know there is no one else around to counter bid me.

That's not correct, on two counts.

“You will often pay less per hour than your maximum bid price. The Spot Price is adjusted periodically as requests come in and the available supply of instances changes. ”

^{— http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html}

My bills are consistent with the documentation in this regard. I am consistently being billed the spot market price as shown in the console, which is most definitely not my max bid.

If you aren't seeing this, then I suspect you are not interpreting market signals correctly.

Importantly, you don't have any way of knowing who else or whether someone else is around to counter bid you, since the current market price does not reflect the current high bid, and in cases where supply exceeds demand, I have found that the presence or absence of my instances does not affect the market price.

The only conclusion I can draw from my experience analyzing pricing history and trends is that EC2 has a minimum price for each combination of the four factors listed above; when more of that particular capacity exists in a particular place than there are total current spot requests, then the spot market price remains at the minimum price for that instance class + availability zone + platform + product. Otherwise, it rises.

Your spot requests are probably shifting the market by their presence. If the market is not at its minimum price, then you can be almost certain that this is what you're seeing -- in which case, it would look like they're charging you what you bid, but that's because your requests are causing the minimum price to rise.

Let's assume -- with numbers purely for the sake of illustration -- the following:

You and I are both bidding for c3.large instances, running Linux/Unix, in Amazon VPC, in the "us-east-1a" availability zone. (Let's further assume, for the sake of discussion, that "us-east-1a" refers to the same availability zone in my AWS account and your AWS account, which is by no means necessarily true -- availability zone to account mappings are pseudo-random among different AWS customers.)

We'll speculate that EC2 has only 10 such instances available for spot allocations, and the minimum market price is $0.032/hour. My max bid is $0.096/hour.

I have 8 of these running, and another customer has 1 instance running, and the other customer has a max bid of $0.035/hour.

The other customer and I are both paying $0.032/hour, since there are still idle machines allocated for servicing the spot market, so our instances have not altered the pricing.

But if you are watching the price, you would have no sense of whether our machines are running or not, or whether we have bid against you, since you don't see the price changing.

Now, you come along, see the price at $0.032/hour, and you bid $0.04 for 2 instances.

There's not enough capacity to serve 8 + 1 + 2 = 11 because only 10 machines are available, so the system decides who gets to run and who has to stop, and the market shifts.

My max bid of $0.096 is the highest, so my 8 machines get to keep running. Your max bid is next in line at $0.04, so your 2 instances gets started, and the other customer, who has a max bid lower than both of us, gets terminated.

And what happens to the market price?

“the last fulfilled bid, sets the Spot Price”

^{— http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-pricing-interruptions.html}

The spot price is now, according to the documentation, $0.04.

Later, I scale back and don't need all of my instances, so I terminate 5 of them. Now there is excess capacity, so the other user (mentioned before, max bid $0.035) will fire back up, your instances will keep running, and the market reverts to the minimum price of $0.032 -- which is what we all pay until the market shifts again.

Any time the prices are not at the baseline, the demand has exceeded the supply, and any successful bid is likely to raise the rate for everyone.

The spot market is a fundamentally fair marketplace. Everyone pays the price bid by the lowest bidder whose request can be fulfilled, except, apparently, in cases where the allocated capacity isn't being fully utilized, in which case, a minimum price established for that particular configuration is what appears to be applied.

The only real "secrets" are finding the particular location/configuration combinations that work for you that are also in the least demand, and bidding what those machines are actually worth to you. You won't pay more than that, and sometimes you will pay less.

It would also, apparently, make sense not to put multiple machines on the same spot requests, but to stagger you bid amounts if you need multiple identical machines, but you don't necessarily need all of them running... because if you are right on the edge of the demand curve, it appears that your lowest fulfilled bid could be, in some cases, the bid that sets the market price... so, in the example above, if you had bid $0.04 and $0.036, the market would have coalesced at $0.036 instead of $0.04, saving us all just a little bit per hour.