Proposal
Enhance the Kafka Scaler or create a new KEDA Scaler to support scaling Kafka by partitions per pod as opposed to simple increase in pod count.
Use-Case
Say I have 12 partitions for my topic.
At 3 pods I will have four partitions per pod:
4 + 4 + 4 = 12
At 4 pods I will have three partitions per pod:
3 + 3 + 3 + 3 = 12
If I naively scale to 5 pods, I will end up with 3 pods with 2 partitions each and 2 pods with still 3 partitions each:
2 + 2 + 2 + 3 + 3 = 12
To scale correctly I need to jump from 4 pods directly to 6 pods:
2 + 2 + 2 + 2 + 2 + 2 = 12
And instead of scaling to 7, 8, 9, 10, or 11, it should scale directly from 6 to 12.
1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 12
As such the list of valid scale targets is not a simple sequence from 0 to 12. It is rather a sequence of the divisors of the partition count:
0, 1, 2, 3, 4, 6, 12
Anything else?
No response
Proposal
Enhance the Kafka Scaler or create a new KEDA Scaler to support scaling Kafka by partitions per pod as opposed to simple increase in pod count.
Use-Case
Say I have 12 partitions for my topic.
At 3 pods I will have four partitions per pod:
4 + 4 + 4 = 12At 4 pods I will have three partitions per pod:
3 + 3 + 3 + 3 = 12If I naively scale to 5 pods, I will end up with 3 pods with 2 partitions each and 2 pods with still 3 partitions each:
2 + 2 + 2 + 3 + 3 = 12To scale correctly I need to jump from 4 pods directly to 6 pods:
2 + 2 + 2 + 2 + 2 + 2 = 12And instead of scaling to 7, 8, 9, 10, or 11, it should scale directly from 6 to 12.
1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 12As such the list of valid scale targets is not a simple sequence from 0 to 12. It is rather a sequence of the divisors of the partition count:
0, 1, 2, 3, 4, 6, 12Anything else?
No response