After that we checked the Kafka host and inside that we found out that the topics that need to be created with the right configurations were created with default ones because we have the Kafka configuration “auto.create.topics.enable=true”. Instead of the num.partitions=50 that was described in modules configurations (mod-search, mod-inventory-storage) we’ve got the num.partitions = 1 as default values. It means that the re-index with more than 8M record can’t be executed.
To fix that we’ve go gone to MSK and change changed the log.retention.minutes to 1 and then wait until the cluster will apply this configuration. After the cluster is running with a new configuration you need to change configurations back, and then wait again. It needs to reset all the topics. Then connect to the jumphost in the target account and manually reconfigure the numbers of partitions for needed topics. And in addition change the configurations for mod-search, mod-inventory-storage, and okapi by adding the resources and number of containers, because these modules don't have enough resources to work well:
- mod-search 2x400cpu;
- okapi 3x600cpu;
- mod-inventory-storage 2x500cpu.
If you have any questions about working with Kafka you can get directly to Pavel Filippov and Oleksii Kuzminov.