Kafka integration test in CI.

[YooSunYoung]

For now, kafka related test is excluded by a flag kafka-test among pytest test cases.

Those tests should be automated later once there are more complicated kafka integration test cases.

There are a few ideas about how to run kafka integration test

Mock Kafka clients

Unit tests with Mocked kafka objects.
Most of interfaces could be unit-tested with mocked object.
See mocked kafka broker example

Local kafka instance in the CI environment

Similarly what we are doing on the local machine, we can set up a kafka broker in the same machine/virtual environment and use it.
It may require a lot of work to maintain the kafka container/deployment to be consistent with the service version otherwise it may not be sufficient to test with the local kafka broker instance.

Tester credential

We can set up a dummy user in the cluster with a limited access to topics/read-write permissions and use it for testings.
It might be as easy as setting a secret in the github repository but it may also not be possible for security issue...

[MridulS]

Just dropping this here for future use: https://www.confluent.io/blog/apache-kafka-ci-cd-with-github/