]]>
]]>
logstash 鏄竴涓簲鐢ㄧ▼搴忔棩蹇椼佷簨浠剁殑浼犺緭銆佸鐞嗐佺鐞嗗拰鎼滅儲鐨勫鉤鍙般?br />浣犲彲浠ョ敤瀹冩潵緇熶竴瀵瑰簲鐢ㄧ▼搴忔棩蹇楄繘琛屾敹闆嗙鐞嗭紝鎻愪緵 Web 鎺ュ彛鐢ㄤ簬鏌ヨ鍜岀粺璁°?br />
]]>
]]>
]]>
鏌ヨ錛?br /> http://www.elasticsearch.org/blog/your-data-your-search/
JAVA API錛屾敞鎰忕鍙ｆ槸9300錛屼笉鏄?200
http://stackoverflow.com/questions/16670219/why-cant-i-connect-to-elasticsearch-through-java-api
http://www.elasticsearch.org/guide/reference/java-api/client/
涔︾睄
http://fuxiaopang.gitbooks.io/learnelasticsearch/getting_started/README.html
]]>
Kafka閮ㄧ講涓庝唬鐮佸疄渚?br /> http://shift-alt-ctrl.iteye.com/blog/1930791
Flume-ng+Kafka+storm鐨勫涔犵瑪璁?br />http://blog.csdn.net/zxcvg/article/details/18600335
Storm搴旂敤緋誨垪涔?#8212;—闆嗘垚Kafka(0.8鐗堢殑KAFKA)
http://blog.csdn.net/xeseo/article/details/18615761
]]>
zkServer.sh startstorm nimbus &storm supervisor &storm ui &storm jar /opt/hadoop/loganalyst/storm-dependend/data/teststorm-1.0.jar teststorm.TopologyMain /opt/hadoop/loganalyst/storm-dependend/data/words.txtstorm kill {toponame}storm active {toponame}storm deactive {toponame}storm list]]>
http://www.cnblogs.com/kemaswill/archive/2012/10/24/2737833.html
http://abentotoro.blog.sohu.com/197023262.html
http://www.cnblogs.com/panfeng412/archive/2012/11/30/how-to-install-and-deploy-storm-cluster.html
浣跨敤 Twitter Storm 澶勭悊瀹炴椂鐨勫ぇ鏁版嵁
http://www.ibm.com/developerworks/cn/opensource/os-twitterstorm/
Storm鏁版嵁嫻佹ā鍨嬬殑鍒嗘瀽鍙婅璁?br />http://www.cnblogs.com/panfeng412/archive/2012/07/29/storm-stream-model-analysis-and-discussion.html
http://www.cnblogs.com/panfeng412/tag/Storm/
storm-kafka
https://github.com/nathanmarz/storm-contrib/tree/master/storm-kafka
浣跨敤Storm瀹炵幇瀹炴椂澶ф暟鎹垎鏋愶紒
http://www.csdn.net/article/2012-12-24/2813117-storm-realtime-big-data-analysis
storm-deploy-aws
https://github.com/nathanmarz/storm-deploy/wiki
!!!鐭ヤ箮緗戠珯涓婄殑Twitter Storm
http://www.zhihu.com/topic/19673110
storm-elastic-search
https://github.com/hmsonline/storm-elastic-search
storm-examples
https://github.com/stormprocessor/storm-examples
kafka-aws
https://github.com/nathanmarz/kafka-deploy
Next Gen Real-time Streaming with Storm-Kafka Integration
http://blog.infochimps.com/2012/10/30/next-gen-real-time-streaming-storm-kafka-integration/
flume+kafka+storm+mysql 鏁版嵁嫻?
http://blog.csdn.net/baiyangfu/article/details/8096088
http://blog.csdn.net/baiyangfu/article/category/1244640
Kafka瀛︿範絎旇
http://blog.csdn.net/baiyangfu/article/details/8096084
STORM+KAFKA
https://github.com/buildlackey/cep
STORM+KETTLE
https://github.com/buildlackey/kettle-storm
]]>
閫氳繃O(1)鐨勭鐩樻暟鎹粨鏋勬彁渚涙秷鎭殑鎸佷箙鍖栵紝榪欑緇撴瀯瀵逛簬鍗充嬌鏁頒互TB鐨勬秷鎭瓨鍌ㄤ篃鑳藉淇濇寔闀挎椂闂寸殑紼沖畾鎬ц兘銆?br />楂樺悶鍚愰噺錛氬嵆浣挎槸闈炲父鏅氱殑紜歡kafka涔熷彲浠ユ敮鎸佹瘡縐掓暟鍗佷竾鐨勬秷鎭?br />鏀寔閫氳繃kafka鏈嶅姟鍣ㄥ拰娑堣垂鏈洪泦緹ゆ潵鍒嗗尯娑堟伅銆?br />鏀寔Hadoop騫惰鏁版嵁鍔犺澆銆?br />
璁捐渚ч噸楂樺悶鍚愰噺錛岀敤浜庡ソ鍙嬪姩鎬侊紝鐩稿叧鎬х粺璁★紝鎺掕緇熻錛岃闂鐜囨帶鍒訛紝鎵瑰鐞嗙瓑緋葷粺銆傚ぇ閮ㄥ垎鐨勬秷鎭腑闂翠歡鑳藉澶勭悊瀹炴椂鎬ц姹傞珮鐨勬秷鎭?鏁版嵁錛屼絾鏄浜庨槦鍒椾腑澶ч噺鏈鐞嗙殑娑堟伅/鏁版嵁鍦ㄦ寔涔呮ф柟闈㈡瘮杈冨急銆?br />
kakfa鐨刢onsumer浣跨敤鎷夌殑鏂瑰紡宸ヤ綔銆?br />
瀹夎kafka
涓嬭澆錛歨ttp://people.apache.org/~nehanarkhede/kafka-0.7.0-incubating/kafka-0.7.0-incubating-src.tar.gz
> tar xzf kafka-.tgz
> cd kafka-
> ./sbt update
> ./sbt package
鍚姩zkserver:
bin/zookeeper-server-start.sh config/zookeeper.properties
鍚姩server:
bin/kafka-server-start.sh config/server.properties
灝辨槸榪欎箞綆鍗曘?br />
浣跨敤kafka
import java.util.Arrays;
import java.util.List;
import java.util.Properties;
import kafka.javaapi.producer.SyncProducer;
import kafka.javaapi.message.ByteBufferMessageSet;
import kafka.message.Message;
import kafka.producer.SyncProducerConfig;
Properties props = new Properties();
props.put(“zk.connect”, “127.0.0.1:2181”);
props.put("serializer.class", "kafka.serializer.StringEncoder");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
Send a single message
// The message is sent to a randomly selected partition registered in ZK
ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-message");
producer.send(data);
producer.close();
import java.util.List;
import java.util.Properties;
import kafka.javaapi.producer.SyncProducer;
import kafka.javaapi.message.ByteBufferMessageSet;
import kafka.message.Message;
import kafka.producer.SyncProducerConfig;
Properties props = new Properties();
props.put(“zk.connect”, “127.0.0.1:2181”);
props.put("serializer.class", "kafka.serializer.StringEncoder");
ProducerConfig config = new ProducerConfig(props);
Producer<String, String> producer = new Producer<String, String>(config);
Send a single message
// The message is sent to a randomly selected partition registered in ZK
ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-message");
producer.send(data);
producer.close();
榪欐牱灝辨槸涓涓爣鍑嗙殑producer銆?br />
consumer鐨勪唬鐮?br />
// specify some consumer properties
Properties props = new Properties();
props.put("zk.connect", "localhost:2181");
props.put("zk.connectiontimeout.ms", "1000000");
props.put("groupid", "test_group");
// Create the connection to the cluster
ConsumerConfig consumerConfig = new ConsumerConfig(props);
ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
// create 4 partitions of the stream for topic “test”, to allow 4 threads to consume
Map<String, List<KafkaMessageStream<Message>>> topicMessageStreams =
consumerConnector.createMessageStreams(ImmutableMap.of("test", 4));
List<KafkaMessageStream<Message>> streams = topicMessageStreams.get("test");
// create list of 4 threads to consume from each of the partitions
ExecutorService executor = Executors.newFixedThreadPool(4);
// consume the messages in the threads
for(final KafkaMessageStream<Message> stream: streams) {
executor.submit(new Runnable() {
public void run() {
for(Message message: stream) {
// process message
}
}
});
}
Properties props = new Properties();
props.put("zk.connect", "localhost:2181");
props.put("zk.connectiontimeout.ms", "1000000");
props.put("groupid", "test_group");
// Create the connection to the cluster
ConsumerConfig consumerConfig = new ConsumerConfig(props);
ConsumerConnector consumerConnector = Consumer.createJavaConsumerConnector(consumerConfig);
// create 4 partitions of the stream for topic “test”, to allow 4 threads to consume
Map<String, List<KafkaMessageStream<Message>>> topicMessageStreams =
consumerConnector.createMessageStreams(ImmutableMap.of("test", 4));
List<KafkaMessageStream<Message>> streams = topicMessageStreams.get("test");
// create list of 4 threads to consume from each of the partitions
ExecutorService executor = Executors.newFixedThreadPool(4);
// consume the messages in the threads
for(final KafkaMessageStream<Message> stream: streams) {
executor.submit(new Runnable() {
public void run() {
for(Message message: stream) {
// process message
}
}
});
}
]]>
apache kafka鍦ㄦ暟鎹鐞嗕腑鐗瑰埆鏄棩蹇楀拰娑堟伅鐨勫鐞嗕笂浼氭湁寰堝鍑鴻壊鐨勮〃鐜幫紝榪欓噷鍐欎釜绱㈠紩錛屽叧浜巏afka鐨勬枃绔犳殏鏃跺氨鏇存柊鍒拌繖閲岋紝鏈榪戝埄鐢ㄧ┖闂叉椂闂村湪瀵筴afka鍋氫竴浜涘姛鑳芥у寮猴紝騫秊ava鍖栵紝铏界劧鐜板湪宸茬粡鏈夊緢澶氳繖鏍風殑鐗堟湰錛屼絾鏄牴鎹疄闄呴渶姹傛潵鏀瑰彉鎵嶆槸鏈閫傚悎鐨勩?/p>
棣栧厛褰撶劧鎺ㄨ崘鐨勬槸kafka鐨勫畼緗?nbsp;http://kafka.apache.org/
鍦ㄥ畼緗戞渶鍊煎緱鍙傝冪殑鏂囩珷灝辨槸kafka design錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">http://kafka.apache.org/design.html錛屾垜鐨勬枃绔犱篃鍩烘湰閮芥槸鍙傜収榪欓噷鐨勮鏄庯紝澶у瑕佺壒鍒噸瑙嗚繖綃囨枃绔狅紝閲岄潰鏈夊ソ澶氱悊蹇甸兘鐗瑰埆濂斤紝鎺ㄨ崘澶氳鍑犻亶銆?/p>
鍦∣SC鐨勭炕璇戦閬撴湁kafka design鍏ㄤ腑鏂囩殑緲昏瘧錛岀炕寰楁尯濂界殑錛屾帹鑽愪竴涓嬶細http://www.oschina.net/translate/kafka-design
kafka鐨剋iki鏄緢涓嶉敊鐨勫涔犳枃妗ｏ細https://cwiki.apache.org/confluence/display/KAFKA/Index
——————————————————————————————————
鎺ヤ笅鏉ュ氨鏄垜鍐欑殑涓緋誨垪鏂囩珷錛屾枃绔犻兘鏄驚搴忔笎榪涚殑鏂瑰紡甯︿綘浜嗚Вkafka錛?/p>
鍏充簬kafka鐨勫熀鏈煡璇嗭紝鍒嗗竷寮忕殑鍩虹錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">銆婂垎甯冨紡娑堟伅緋葷粺Kafka鍒濇銆?/a>
kafka鐨勫垎甯冨紡鎼緩錛宷uick start錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">銆妅afka鍒嗗竷寮忕幆澧冩惌寤恒?/a>
鍏充簬kafka鐨勫疄鐜扮粏鑺傦紝榪欎富瑕佸氨鏄design鐨勯儴鍒嗭細銆婄粏鑺備笂銆?/a>銆?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">銆婄粏鑺備笅銆?/a>
鍏充簬kafka寮鍙戠幆澧冿紝scala鐜鐨勬惌寤猴細銆婂紑鍙戠幆澧冩惌寤恒?/a>
鏁版嵁鐢熶駭鑰咃紝producer鐨勭敤娉曪細銆妏roducer鐨勭敤娉曘?/a>銆?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">銆妏roducer浣跨敤娉ㄦ剰銆?/a>
鏁版嵁娑堣垂鑰咃紝consumer鐨勭敤娉曪細銆奵onsumer鐨勭敤娉曘?/a>
榪樻湁浜涢浂紕庣殑錛屽叧浜庨氫俊孌電殑婧愮爜瑙ｈ錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">銆妌et鍖呮簮鐮佽В璇匯?/a>銆?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">銆奲roker閰嶇疆銆?/a>
——————————————————————————————————
鎵╁睍鐨勯槄璇昏繕鏈変笅闈㈣繖浜涳細
鎴戠殑濂藉弸鍐欑殑鍏充簬kafka鍜宩afka鐨勭浉鍏沖崥瀹紝鐗瑰埆濂斤紝鎴戞湁寰堝闂涔熼兘鎵句粬瑙ｅ喅鐨勶紝澶х涓鑸殑瀛樺湪錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">http://rockybean.github.com/ @rockybean
kafka鐨刯ava鍖栫増鏈琷afka錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">https://github.com/adyliu/jafka
娣樺疂鐨刴etaQ錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">https://github.com/killme2008/Metamorphosis
鎴戞渶榪戝湪鍐欑殑inforQ錛屽垰寮濮嬪啓錛屾垜涔熺函綺規槸涓轟簡璇諱笅婧愮爜錛屼笉瀹氭湡鏇存柊鍝堬細https://github.com/ielts0909/inforq
鍚庨潰涓闃舵鍙兘鏇存柊鐐瑰効鍏充簬cas鐨勪笢瑗垮惂錛屽叿浣撲篃娌℃兂濂斤紝鏈榪戜竴鐩村嚭宸紝鍐欎唬鐮佺殑鏃墮棿閮藉緢灝?/p>
--------------------------------------------------------------------------------
0.8鐗堟湰鐨勭浉鍏蟲洿鏂板涓嬶細
0.8鏇存柊鍐呭浠嬬粛錛?a target="_blank" rel="nofollow" style="padding: 0px; margin: 0px; color: #0072bb; outline: 0px;">銆妅afka0.8鐗堟湰鐨勪竴浜涙洿鏂般?/a>
]]>