Putting in place traffic shaping on winehq

[Jeremy White]

> threshold of 36 hours.  I'm not sure I see the point in traffic shaping
> beyond the initial building of the cache (and total cache invalidation
> should almost never happen going forward).  Additionally, that means
> that even with multiple CDNs making caching queries directly, after a
> full cache of the files, you could safely handle adding 10-20 CDN direct
> mirrors. All of that logic assumes that you bite the bullet and do a
> complete caching at the start.

Note that I think it was a lot more than 10-20 CDNs; that may have been
a bug on Fastly's part, or they just may have a lot more mirrors than
anyone appreciates.

> 
> Does that logic seem correct to you?

There are a lot of good reasons to do traffic shaping; we should have
been doing it all along.  For example, when we do exceed the 20Mbit
limit, our provider just chops us off, and drops arbitrary packets,
which leads to bad user experiences.

With traffic shaping, we can prioritize different traffic, which allows
us to provide better service.  For example, we can tune it so that when
under load, the downloads to the CDNs go more slowly, but all other
functions stay responsive.  Notably, we can now shape it so ssh sessions
stay responsive, so we can get in to the box and figure out the problem
more easily (which we were unable to do during one particularly bad storm).

Also, we don't have great data on the actual use of traffic this past
fall / winter; we mostly have theories.  We have now put in place ntop
so we should be able to better monitor going forward.

For the curious, our 'steady state' seems to be about 3Mbit, with about
two thirds of that being web traffic, and one third being git.  We have
many spikes, though; I need to puzzle out the cause of some of those.  I
also don't have our Fastly stats ready to hand, but I know it's a lot.
We owe Fastly a great debt of gratitude.

I realize it's counter intuitive, but the bottom line is that it's
better for us to decide how to manage traffic than to have it imposed on
us, whether by our provider or by an upstream CDN with a much bigger
pipe than ours.

Again, please let me know of any problems that arise.

Cheers,

Jeremy