如何在自定义算法中利用Spark功能？(How to leverage Spark features in a custom algorithm?)

我在Java中开发了一种处理大数据集的算法。使用Eclipse。 现在我想利用Spark Apache。 如何在Spark环境下运行此算法？ 我需要什么工具，如何将Eclipse连接到Spark？ 顺便说一句，我按照Edx网站的课程，在VirtualBox和Vagrant上安装了Spark作为本地服务器。 但我不知道如何将它连接到我用Eclipse开发的算法。 谢谢

I developed an algorithm in Java which treats large data set.Using Eclipse. Now I want to take advantages of Spark Apache . How can I run this algorithm under Spark environment? what tools I need and how can I connect Eclipse to Spark? By the way, I followed a course i Edx web site and installed Spark as a local server on VirtualBox and Vagrant. But I don't know how can I connect it to My algorithm that I developed it by Eclipse. Thank you

最满意答案

你必须使用“spark-submit”命令在spark集群中运行。

./bin/spark-submit --class <main-class> --master <master-url> <application-jar> [application-arguments]

you have to use the "spark-submit" command to run in the spark cluster.

./bin/spark-submit --class <main-class> --master <master-url> <application-jar> [application-arguments]如何在自定义算法中利用Spark功能？(How to leverage Spark features in a custom algorithm?)

我在Java中开发了一种处理大数据集的算法。使用Eclipse。 现在我想利用Spark Apache。 如何在Spark环境下运行此算法？ 我需要什么工具，如何将Eclipse连接到Spark？ 顺便说一句，我按照Edx网站的课程，在VirtualBox和Vagrant上安装了Spark作为本地服务器。 但我不知道如何将它连接到我用Eclipse开发的算法。 谢谢

I developed an algorithm in Java which treats large data set.Using Eclipse. Now I want to take advantages of Spark Apache . How can I run this algorithm under Spark environment? what tools I need and how can I connect Eclipse to Spark? By the way, I followed a course i Edx web site and installed Spark as a local server on VirtualBox and Vagrant. But I don't know how can I connect it to My algorithm that I developed it by Eclipse. Thank you

最满意答案

你必须使用“spark-submit”命令在spark集群中运行。

./bin/spark-submit --class <main-class> --master <master-url> <application-jar> [application-arguments]

you have to use the "spark-submit" command to run in the spark cluster.

./bin/spark-submit --class <main-class> --master <master-url> <application-jar> [application-arguments]