gateway: polls sps every 2s rather than every 1s.

[jmcarp]

This patch includes a tiny configuration change, as well as lots of commentary explaining the motivation. To recapitulate some of that commentary: as described in #10552, our test racks drop metrics when oximeter collects a large volume of samples during a short period of time. On average, oximeter drains its database queue faster than it collects samples, but if we collect more than DEFAULT_BATCH_SIZE * MAX_BUFFER_SIZE_MULTIPLIER metrics during a short interval, we overflow the queue and drop the oldest samples. This can happen regularly when multiple high-volume producers are collected from around the same time. The worst offender here is mgs, which currently produces about 40k samples per collection from each gateway; if oximeter is unlucky enough to poll both gateways around the same time, it's immediately at 80% of its maximum queue size.

This patch configures mgs to poll the sps every 2s rather than every 1s, halving the volume of the spikiest oximeter producer. We also add notes to the relevant constants explaining how they should be sized, and what might cause us to change them in the future.

Note: my original plan was to increase the size of the oximeter database queue, but if we don't identify any downsides to polling the sps every 2s rather than every 1s, this change is safer.

Part of #10552.

[bnaecker]

I think this is probably fine, just a few thoughts.

1. For a lot of this data, 1s is already too slow to capture some useful information. 2s isn't categorically worse, since it's also too slow. But it is in the wrong direction, so we might want to find ways to improve the situation that let us reduce the sampling rate.
2. IWe insert data into the database as soon as we get DEFAULT_BATCH_SIZE samples, right? And when we start an insertion, we take the whole buffer. So the relevant burst is: getting more than the queue size new samples, within the time it takes us to insert one batch. Does that actually happen? Looking at dogfood, it takes a few 10s of milliseconds to insert a batch, and we never drop anything. I think you've seen example systems where we have dropped samples in the past, so I'm curious how much we're dropping today.