Integration tests with kafka

Build Environment
1

Hey all.

I’m trying to setup integration test workflow through circleci for an app dependant on some external sources. One source I need for the integration tests is a running kafka instance.

In my cicleci/config I’m setting up kafka as follows;

      - image: bitnami/zookeeper
        environment:
          ALLOW_ANONYMOUS_LOGIN: yes
      - image: bitnami/kafka:2.7.0
        environment:
          ALLOW_PLAINTEXT_LISTENER: yes
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9093
          KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9093
          KAFKA_CFG_ZOOKEEPER_CONNECT: localhost:2181
          KAFKA_DELETE_TOPIC_ENABLE: 'true'
          KAFKA_BROKER_ID: 1
          KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE: true

Then in the steps I am trying to run consumers which will subscribe to topics on kafka. The issue is it’s timing out.

Broker: Unknown topic or partition

I’ve so far been unable to find an example configuration for a circleci workflow with kafka running. I’m sure someone must have encountered this before I just can’t find it.

Any help much appreciated.