[helm] scheduling primitives

[fresh-borzoni]

closes #3153

Exposes affinity, nodeSelector, tolerations, and topologySpreadConstraints on both coordinator and tablet. Required to spread replicas across zones/nodes under replication factor 3 - without these, a single node failure can take out all tablet replicas for segments not yet tiered.

[fresh-borzoni]

@morazow @affo @polyzos PTAL 🙏

[affo]

Loving where we are going with the Helm charts, thanks @fresh-borzoni for this one, another step towards full flexibility 🚀

I just have a question, why do you mention "With replication factor 3, a single node failure could take out all replicas simultaneously..."? I mean, this would hold with any replication factor if nodes are co-located, isn't it?

[morazow]

Minor nit from my side in the values file 🤝

Thanks, shaping nicely 🚀

[fresh-borzoni]

I just have a question, why do you mention "With replication factor 3, a single node failure could take out all replicas simultaneously..."? I mean, this would hold with any replication factor if nodes are co-located, isn't it?

Well, it's just me missing "Even" before with, bc we had exactly this scenario and the first reaction was - but how all 3 simultaneously :)