来自Spark Streaming Programming Guide ：
将逻辑扩展到在集群上运行，分配给Spark Streaming应用程序的核心数必须大于接收器数。 否则系统将接收数据，但无法处理数据。
这是否意味着如果我在整个Spark集群中有16个CPU核心，我无法读取超过15个Kafka主题的数据？
From Spark Streaming Programming Guide:
Extending the logic to running on a cluster, the number of cores allocated to the Spark Streaming application must be more than the number of receivers. Otherwise the system will receive data, but not be able to process it.
Does this mean that if I have 16 CPU cores in the whole Spark cluster I cannot read data from more than 15 Kafka topics?
最满意答案
仅当您使用基于消费者/接收者的API时。 这不适用于直接流。
看看这两者之间的差异
Only if you use the consumer/receiver based API. This does not apply to the Direct Stream one.
Have a look here for the differences between the two
阅读更多Kafka主题而不是CPU核心数(Read more Kafka topics than number of CPU cores)来自Spark Streaming Programming Guide ：
将逻辑扩展到在集群上运行，分配给Spark Streaming应用程序的核心数必须大于接收器数。 否则系统将接收数据，但无法处理数据。
这是否意味着如果我在整个Spark集群中有16个CPU核心，我无法读取超过15个Kafka主题的数据？
From Spark Streaming Programming Guide:
Extending the logic to running on a cluster, the number of cores allocated to the Spark Streaming application must be more than the number of receivers. Otherwise the system will receive data, but not be able to process it.
Does this mean that if I have 16 CPU cores in the whole Spark cluster I cannot read data from more than 15 Kafka topics?
最满意答案
仅当您使用基于消费者/接收者的API时。 这不适用于直接流。
看看这两者之间的差异
Only if you use the consumer/receiver based API. This does not apply to the Direct Stream one.
Have a look here for the differences between the two
发布评论