{"action":"created","issue":{"url":"https://api.github.com/repos/SOHU-Co/kafka-node/issues/859","repository_url":"https://api.github.com/repos/SOHU-Co/kafka-node","labels_url":"https://api.github.com/repos/SOHU-Co/kafka-node/issues/859/labels{/name}","comments_url":"https://api.github.com/repos/SOHU-Co/kafka-node/issues/859/comments","events_url":"https://api.github.com/repos/SOHU-Co/kafka-node/issues/859/events","html_url":"https://github.com/SOHU-Co/kafka-node/issues/859","id":293918512,"number":859,"title":"Support separation of Internal and External traffic by Broker (KIP-103)","user":{"login":"PHameete","id":3921280,"avatar_url":"https://avatars0.githubusercontent.com/u/3921280?v=4","gravatar_id":"","url":"https://api.github.com/users/PHameete","html_url":"https://github.com/PHameete","followers_url":"https://api.github.com/users/PHameete/followers","following_url":"https://api.github.com/users/PHameete/following{/other_user}","gists_url":"https://api.github.com/users/PHameete/gists{/gist_id}","starred_url":"https://api.github.com/users/PHameete/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/PHameete/subscriptions","organizations_url":"https://api.github.com/users/PHameete/orgs","repos_url":"https://api.github.com/users/PHameete/repos","events_url":"https://api.github.com/users/PHameete/events{/privacy}","received_events_url":"https://api.github.com/users/PHameete/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-02T15:07:40Z","updated_at":"2018-02-14T14:39:08Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Currently, kafka-node does not have support for brokers that are configured to have separate listeners for internal and external traffic. This was introduced in KIP-103.\r\n\r\nThere are 2 aspects to this KIP that are important:\r\n\r\n1. The protocol of the url's in the endpoints listed in the broker metadata is not necessarily the used protocol. It can instead be the name of the listener which must then be mapped to the protocol via the 'listener_security_protocol_map' metadata that was newly introduced in KIP-103.\r\n2. The kafka-node adapter currently selects the first endpoint that supports the protocol that is selected for usage by the kafka-node adapter. However, there may be multiple endpoints that use this protocol: for example both an INTERNAL as well as an EXTERNAL endpoint. In this case the EXTERNAL endpoint should be chosen as the endpoint that is used because the INTERNAL endpoint can not be reached by the client.\r\n\r\n### Reproduction\r\n\r\nTo reproduce this, set up a Kafka broker that separates INTERNAL and EXTERNAL traffic. You can do this by setting up a Kafka Broker using Docker and using the following configuration:\r\n\r\n```\r\n---\r\nversion: '3'\r\nservices:\r\n  zookeeper:\r\n    image: confluentinc/cp-zookeeper\r\n    hostname: zookeeper\r\n    ports:\r\n      - \"3500:3500\"\r\n    environment:\r\n      ZOOKEEPER_CLIENT_PORT: 3500\r\n      ZOOKEEPER_TICK_TIME: 2000\r\n\r\n  broker:\r\n    image: confluentinc/cp-kafka\r\n    hostname: broker\r\n    depends_on:\r\n      - zookeeper\r\n    ports:\r\n      - \"3501:3501\"\r\n    environment:\r\n      KAFKA_BROKER_ID: 1\r\n      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:3500'\r\n      KAFKA_ADVERTISED_LISTENERS: 'INTERNAL://broker:9092,EXTERNAL://localhost:3501'\r\n      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT'\r\n      KAFKA_LISTENERS: 'PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:3501'\r\n      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1\r\n      KAFKA_DEFAULT_REPLICATION_FACTOR: 1\r\n      KAFKA_MESSAGE_MAX_BYTES: 100000000\r\n      KAFKA_REPLICA_FETCH_MAX_BYTES: 100000000\r\n```\r\n\r\nThe endpoint `broker:9092` is ment to be used used internally between the containers. The endpoint `localhost:3501` is used by external Kafka clients to connect to the broker.\r\n\r\nIf you now run the example consumer.js for `zookeeper:3500` no suitable endpoint can be found because the advertised listeners with names EXTERNAL and INTERNAL are not mapped to the protocol PLAINTEXT. Moreover, if this mapping were in place, it would select the INTERNAL endpoint as a first match (see client.js line 126, function setupBrokerProfiles). However, the INTERNAL endpoint can not be used to access the broker from outside the Docker bridge network. \r\n"},"comment":{"url":"https://api.github.com/repos/SOHU-Co/kafka-node/issues/comments/365626577","html_url":"https://github.com/SOHU-Co/kafka-node/issues/859#issuecomment-365626577","issue_url":"https://api.github.com/repos/SOHU-Co/kafka-node/issues/859","id":365626577,"user":{"login":"hyperlink","id":133309,"avatar_url":"https://avatars3.githubusercontent.com/u/133309?v=4","gravatar_id":"","url":"https://api.github.com/users/hyperlink","html_url":"https://github.com/hyperlink","followers_url":"https://api.github.com/users/hyperlink/followers","following_url":"https://api.github.com/users/hyperlink/following{/other_user}","gists_url":"https://api.github.com/users/hyperlink/gists{/gist_id}","starred_url":"https://api.github.com/users/hyperlink/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hyperlink/subscriptions","organizations_url":"https://api.github.com/users/hyperlink/orgs","repos_url":"https://api.github.com/users/hyperlink/repos","events_url":"https://api.github.com/users/hyperlink/events{/privacy}","received_events_url":"https://api.github.com/users/hyperlink/received_events","type":"User","site_admin":false},"created_at":"2018-02-14T14:39:07Z","updated_at":"2018-02-14T14:39:07Z","author_association":"COLLABORATOR","body":"Did merging your PR resolve this issue?"}}
{"actor":{"display_login":"hyperlink"}}