在实时计算框架Storm集群上运行Storm-Starter例子
官方文档待翻译,个人实践在下面
Example Storm Topologies
Learn to use Storm!
Table of Contents
Getting started
Prerequisites
First, you need java
and git
installed and in your user's PATH
. Also, two of the examples in storm-starter
require Python and Ruby.
Next, make sure you have the storm-starter code available on your machine. Git/GitHub beginners may want to use the following command to download the latest storm-starter code and change to the new directory that contains the downloaded code.
$ git clone git://github.com/apache/incubator-storm.git && cd incubator-storm/examples/storm-starter
storm-starter overview
storm-starter contains a variety of examples of using Storm. If this is your first time working with Storm, check out these topologies first:
- ExclamationTopology: Basic topology written in all Java
- WordCountTopology: Basic topology that makes use of multilang by implementing one bolt in Python
- ReachTopology: Example of complex DRPC on top of Storm
After you have familiarized yourself with these topologies, take a look at the other topopologies in src/jvm/storm/starter/ such as RollingTopWords for more advanced implementations.
If you want to learn more about how Storm works, please head over to the Storm project page.
Using storm-starter with Maven
Install Maven
Install Maven (preferably version 3.x) by following the Maven installation instructions.
Build and install Storm jars locally
If you are using the latest development version of Storm, e.g. by having cloned the Storm git repository,
then you must first perform a local build of Storm itself. Otherwise you will run into Maven errors such as
"Could not resolve dependencies for project org.apache.storm:storm-starter:<storm-version>-SNAPSHOT
".
# Must be run from the top-level directory of the Storm code repository
$ mvn clean install -DskipTests=true
This command will build Storm locally and install its jar files to your user's $HOME/.m2/repository/
. When you run
the Maven command to build and run storm-starter (see below), Maven will then be able to find the corresponding version
of Storm in this local Maven repository at $HOME/.m2/repository
.
Running topologies with Maven
storm-starter topologies can be run with the maven-exec-plugin. For example, to
compile and run WordCountTopology
in local mode, use the command:
$ mvn compile exec:java -Dstorm.topology=storm.starter.WordCountTopology
You can also run clojure topologies with Maven:
$ mvn compile exec:java -Dstorm.topology=storm.starter.clj.word_count
In Windows parameter should be quoted, like this:
$ mvn compile exec:java "-Dstorm.topology=storm.starter.clj.word_count"
Packaging storm-starter for use on a Storm cluster
You can package a jar suitable for submitting to a Storm cluster with the command:
$ mvn package
This will package your code and all the non-Storm dependencies into a single "uberjar" at the path
target/storm-starter-{version}-jar-with-dependencies.jar
.
Running unit tests
Use the following Maven command to run the unit tests that ship with storm-starter. Unfortunately lein test
does not
yet run the included unit tests.
$ mvn test
Using storm-starter with IntelliJ IDEA
Importing storm-starter as a project in IDEA
The following instructions will import storm-starter as a new project in IntelliJ IDEA.
- Open File > Import Project... and navigate to the storm-starter directory of your storm clone (e.g.
~/git/incubator-storm/examples/storm-starter
). - Select Import project from external model, select "Maven", and click Next.
- In the following screen, enable the checkbox Import Maven projects automatically. Leave all other values at their defaults. Click Next.
- Click Next on the following screen about selecting Maven projects to import.
- Select the JDK to be used by IDEA for storm-starter, then click Next.
- At the time of this writing you should use JDK 6.
- It is strongly recommended to use Sun/Oracle JDK 6 rather than OpenJDK 6.
- You may now optionally change the name of the project in IDEA. The default name suggested by IDEA is "storm-starter". Click Finish once you are done.
个人实践
默认通过sudo -s 进入root环境
安装环境 git ,maven
# apt-get install git maven
通过git clone 项目
# git clone git://github.com/apache/incubator-storm.git
进入项目目录
# cd incubator-storm/examples/storm-starter
编译
# mvn compile
第一次出现错误,去google了一下,找到一种办法
- 不使用最新的代码而使用apache-storm-0.9.2-incubating中的例子,这是我们在安装storm时下载的 进入apache-storm-0.9.2-incubating/examples/storm-starter
执行
# mvn compile
成功!
打包所有的依赖
# mvn package
进入target目录,可以看到storm-starter-0.9.2-incubating-jar-with-dependencies.jar
提交到storm(以WordCountTopology为例)
在nimbus上运行
# ./bin/storm jar storm-starter-0.9.2-incubating-jar-with-dependencies.jar storm.starter.WordCountTopology test
可以看到console的输出,当然可以在storm ui的http上查看topology状态。例子已经被提交到集群。