亚洲AV无码专区在线观看成人,亚洲国产成人久久三区,亚洲欧洲国产经精品香蕉网

zh-cnTue, 23 Apr 2013 21:36:25 GMTTue, 23 Apr 2013 21:36:25 GMT60ä¸€ä¸ªPIGè„šæœ¬ä¾‹ååˆ†æžhttp://m.tkk7.com/paulwong/archive/2013/04/13/397791.htmlpaulwongpaulwongSat, 13 Apr 2013 07:21:00 GMThttp://m.tkk7.com/paulwong/archive/2013/04/13/397791.htmlhttp://m.tkk7.com/paulwong/comments/397791.htmlhttp://m.tkk7.com/paulwong/archive/2013/04/13/397791.html#Feedback0http://m.tkk7.com/paulwong/comments/commentRss/397791.htmlhttp://m.tkk7.com/paulwong/services/trackbacks/397791.html

PIGGYBANK_PATH=$PIG_HOME/contrib/piggybank/java/piggybank.jar
INPUT=pig/input/test-pig-full.txt
OUTPUT=pig/output/test-pig-output-$(date +%Y%m%d%H%M%S)
PIGSCRIPT=analyst_status_logs.pig

#analyst_500_404_month.pig
#analyst_500_404_day.pig
#analyst_404_percentage.pig
#analyst_500_percentage.pig
#analyst_unique_path.pig
#analyst_user_logs.pig
#analyst_status_logs.pig

pig -p PIGGYBANK_PATH=$PIGGYBANK_PATH -p INPUT=$INPUT -p OUTPUT=$OUTPUT $PIGSCRIPT

è¦åˆ†æžçš„æ•°æ®æºï¼ŒLOG æ–‡äšg

46.20.45.18 - - [25/Dec/2012:23:00:25 +0100] "GET / HTTP/1.0" 302 - "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" "-" "-" 46.20.45.18 "" 11011AEC9542DB0983093A100E8733F8 0
46.20.45.18 - - [25/Dec/2012:23:00:25 +0100] "GET /sign-in.jspx HTTP/1.0" 200 3926 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" "-" "-" 46.20.45.18 "" 11011AEC9542DB0983093A100E8733F8 0
69.59.28.19 - - [25/Dec/2012:23:01:25 +0100] "GET / HTTP/1.0" 302 - "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)" "-" "-" 69.59.28.19 "" 36D80DE7FE52A2D89A8F53A012307B0A 15

PIGè„šæœ¬åQ?br />

--æ³¨å†ŒJARåŒ…ï¼Œå› äØ“è¦ç”¨åˆ°DateExtractor
register '$PIGGYBANK_PATH';

--å£°æ˜Žä¸€ä¸ªçŸå‡½æ•°å?br />DEFINE DATE_EXTRACT_MM
org.apache.pig.piggybank.evaluation.util.apachelogparser.DateExtractor('yyyy-MM');

DEFINE DATE_EXTRACT_DD
org.apache.pig.piggybank.evaluation.util.apachelogparser.DateExtractor('yyyy-MM-dd');

-- pig/input/test-pig-full.txt
--æŠŠæ•°æ®ä»Žå˜é‡æ‰€æŒ‡çš„æ–‡äšgåŠ è²åˆ°PIGä¸ï¼Œòq¶å®šä¹‰æ•°æ®åˆ—åï¼Œæ¤æ—¶çš„æ•°æ®é›†ä¸ºæ•°¾l?a,b,c)
raw_logs = load '$INPUT' USING org.apache.pig.piggybank.storage.MyRegExLoader('^(\\S+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] "(\\S+) (\\S+) (HTTP[^"]+)" (\\S+) (\\S+) "([^"]*)" "([^"]*)" "(\\S+)" "(\\S+)" (\\S+) "(.*)" (\\S+) (\\S+)')
as (remoteAddr: chararray,
n2: chararray,
n3: chararray,
time: chararray,
method: chararray,
path:chararray,
protocol:chararray,
status: int,
bytes_string: chararray,
referrer: chararray,
browser: chararray,
n10:chararray,
remoteLogname: chararray,
remoteAddr12: chararray,
path2: chararray,
sessionid: chararray,
n15: chararray
);

--˜q‡æ×oæ•°æ®
filter_logs = FILTER raw_logs BY not (browser matches '.*pingdom.*');
--item_logs = FOREACH raw_logs GENERATE browser;

--percent 500 logs
--é‡å®šä¹‰æ•°æ®é¡¹åQŒæ•°æ®é›†åªå–2™å¹status,month
reitem_percent_500_logs = FOREACH filter_logs GENERATE status,DATE_EXTRACT_MM(time) as month;
--åˆ†ç»„æ•°æ®é›†ï¼Œæ¤æ—¶çš„æ•°æ®ç»“æž„äØ“MAP(a{(aa,bb,cc),(dd,ee,ff)},b{(bb,cc,dd),(ff,gg,hh)})
group_month_percent_500_logs = GROUP reitem_percent_500_logs BY (month);
--é‡å®šä¹‰åˆ†¾l„æ•°æ®é›†æ•°æ®™å¹ï¼Œ˜q›è¡Œåˆ†ç»„¾lŸè®¡åQŒæ¤æ—¶è¦è”åˆåˆ†ç»„æ•°æ®é›†å’ŒåŽŸæ•°æ®é›†¾lŸè®¡
final_month_500_logs = FOREACH group_month_percent_500_logs
{
    --å¯¹åŽŸæ•°æ®é›†åšcountåQŒå› ä¸ºæ˜¯åœ¨foreachjé‡Œåšcountçš„ï¼Œå³ä‹Éæ˜¯å¯¹åŽŸæ•°æ®é›†åQŒä¹Ÿä¼šè‡ªåŠ¨ä¼šåŠ month==groupçš„æ¡ä»?br />    --ä»Žè¿™é‡Œå¯ä»¥çœ‹å‡ºå¯¹äºŽgroupé‡Œçš„æ•°æ®é›†ï¼Œå®Œå…¨æ²¡ç”¨åˆ?br />    --˜q™æ—¶æ˜¯ä»¥æ¯ä¸€è¡ŒäØ“å•ä½çš„ï¼Œ¾lŸè®¡MAPä¸çš„KEY-aå¯¹åº”çš„æ•°¾l„åœ¨åŽŸæ•°æ®é›†ä¸çš„ä¸ªæ•°
    total = COUNT(reitem_percent_500_logs);
    --å¯¹åŽŸæ•°æ®é›†åšfilteråQŒå› ä¸ºæ˜¯åœ¨foreachjé‡Œåšcountçš„ï¼Œå³ä‹Éæ˜¯å¯¹åŽŸæ•°æ®é›†åQŒä¹Ÿä¼šè‡ªåŠ¨ä¼šåŠ month==groupçš„æ¡ä»?br />    --é‡æ–°˜q‡æ×oä¸€ä¸‹åŽŸæ•°æ®é›†ï¼Œå¾—åˆ°status==500,month==groupçš„æ•°æ®é›†
    t = filter reitem_percent_500_logs by status== 500; --create a bag which contains only T values
    --é‡å®šä¹‰æ•°æ®é¡¹åQŒå–groupåQŒç»Ÿè®¡ç»“æž?br />    generate flatten(group) as col1, 100*(double)COUNT(t)/(double)total;
}
STORE final_month_500_logs into '$OUTPUT' using PigStorage(',');

paulwong 2013-04-13 15:21 å‘è¡¨è¯„è®º

]]>æŠŠå‘½ä»¤è¡Œä¸çš„å€ég¼ ˜q›PIGä¸?/title><link>http://m.tkk7.com/paulwong/archive/2013/04/10/397645.html</link><dc:creator>paulwong</dc:creator><author>paulwong</author><pubDate>Wed, 10 Apr 2013 07:32:00 GMT</pubDate><guid>http://m.tkk7.com/paulwong/archive/2013/04/10/397645.html</guid><wfw:comment>http://m.tkk7.com/paulwong/comments/397645.html</wfw:comment><comments>http://m.tkk7.com/paulwong/archive/2013/04/10/397645.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://m.tkk7.com/paulwong/comments/commentRss/397645.html</wfw:commentRss><trackback:ping>http://m.tkk7.com/paulwong/services/trackbacks/397645.html</trackback:ping><description><![CDATA[<a target="_blank">http://wiki.apache.org/pig/ParameterSubstitution<br /> <br /> <br /> </a> <div> <div style="background-color:#eeeeee;font-size:13px;border:1px solid #CCCCCC;padding-right: 5px;padding-bottom: 4px;padding-left: 4px;padding-top: 4px;width: 98%;word-break:break-all">%pig -param input=/user/paul/sample.txt -param output=/user/paul/output/</div> </div><br /><br />PIGä¸èŽ·å?br /><div style="background-color:#eeeeee;font-size:13px;border:1px solid #CCCCCC;padding-right: 5px;padding-bottom: 4px;padding-left: 4px;padding-top: 4px;width: 98%;word-break:break-all">records = LOAD <span style="color: #800080; ">$input</span>;</div><img src ="http://m.tkk7.com/paulwong/aggbug/397645.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://m.tkk7.com/paulwong/" target="_blank">paulwong</a> 2013-04-10 15:32 <a href="http://m.tkk7.com/paulwong/archive/2013/04/10/397645.html#Feedback" target="_blank" style="text-decoration:none;">å‘è¡¨è¯„è®º</a></div>]]></description></item><item><title>PIGä¸çš„åˆ†ç»„¾lŸè®¡ç™‘Öˆ†æ¯?/title><link>http://m.tkk7.com/paulwong/archive/2013/04/10/397642.html</link><dc:creator>paulwong</dc:creator><author>paulwong</author><pubDate>Wed, 10 Apr 2013 06:13:00 GMT</pubDate><guid>http://m.tkk7.com/paulwong/archive/2013/04/10/397642.html</guid><wfw:comment>http://m.tkk7.com/paulwong/comments/397642.html</wfw:comment><comments>http://m.tkk7.com/paulwong/archive/2013/04/10/397642.html#Feedback</comments><slash:comments>0</slash:comments><wfw:commentRss>http://m.tkk7.com/paulwong/comments/commentRss/397642.html</wfw:commentRss><trackback:ping>http://m.tkk7.com/paulwong/services/trackbacks/397642.html</trackback:ping><description><![CDATA[<a target="_blank">http://stackoverflow.com/questions/15318785/pig-calculating-percentage-of-total-for-a-field<br /><br /></a><a target="_blank">http://stackoverflow.com/questions/13476642/calculating-percentage-in-a-pig-query</a><img src ="http://m.tkk7.com/paulwong/aggbug/397642.html" width = "1" height = "1" /><br><br><div align=right><a style="text-decoration:none;" href="http://m.tkk7.com/paulwong/" target="_blank">paulwong</a> 2013-04-10 14:13 <a href="http://m.tkk7.com/paulwong/archive/2013/04/10/397642.html#Feedback" target="_blank" style="text-decoration:none;">å‘è¡¨è¯„è®º</a></div>]]></description></item><item><title>CombinedLogLoaderhttp://m.tkk7.com/paulwong/archive/2013/04/08/397510.htmlpaulwongpaulwongMon, 08 Apr 2013 03:28:00 GMThttp://m.tkk7.com/paulwong/archive/2013/04/08/397510.htmlhttp://m.tkk7.com/paulwong/comments/397510.htmlhttp://m.tkk7.com/paulwong/archive/2013/04/08/397510.html#Feedback0http://m.tkk7.com/paulwong/comments/commentRss/397510.htmlhttp://m.tkk7.com/paulwong/services/trackbacks/397510.html

/*
* Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the
* NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF
* licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file
* except in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License is
* distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and limitations under the License.
*/

package org.apache.pig.piggybank.storage.apachelog;

import java.util.regex.Pattern;

import org.apache.pig.piggybank.storage.RegExLoader;

/**
* CombinedLogLoader is used to load logs based on Apache's combined log format, based on a format like
*
* LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
*
* The log filename ends up being access_log from a line like
*
* CustomLog logs/combined_log combined
*
* Example:
*
* raw = LOAD 'combined_log' USING org.apache.pig.piggybank.storage.apachelog.CombinedLogLoader AS
* (remoteAddr, remoteLogname, user, time, method, uri, proto, status, bytes, referer, userAgent);
*
*/

public class CombinedLogLoader extends RegExLoader {
    // 1.2.3.4 - - [30/Sep/2008:15:07:53 -0400] "GET / HTTP/1.1" 200 3190 "-"
    // "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4; en-us) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1"
    private final static Pattern combinedLogPattern = Pattern
        .compile("^(\\S+)\\s+(\\S+)\\s+(\\S+)\\s+.(\\S+\\s+\\S+).\\s+\"(\\S+)\\s+(.+?)\\s+(HTTP[^\"]+)\"\\s+(\\S+)\\s+(\\S+)\\s+\"([^\"]*)\"\\s+\"(.*)\"$");

    public Pattern getPattern() {
        return combinedLogPattern;
    }
}

paulwong 2013-04-08 11:28 å‘è¡¨è¯„è®º

]]>Analyzing Apache logs with Pig http://m.tkk7.com/paulwong/archive/2013/04/08/397489.htmlpaulwongpaulwongSun, 07 Apr 2013 18:06:00 GMThttp://m.tkk7.com/paulwong/archive/2013/04/08/397489.htmlhttp://m.tkk7.com/paulwong/comments/397489.htmlhttp://m.tkk7.com/paulwong/archive/2013/04/08/397489.html#Feedback0http://m.tkk7.com/paulwong/comments/commentRss/397489.htmlhttp://m.tkk7.com/paulwong/services/trackbacks/397489.html

Analyzing log files, churning them and extracting meaningful information is a potential use case in Hadoop. We don’t have to go in for MapReduce programming for these analyses; instead we can go for tools like Pig and Hive for this log analysis. I’d just give you a start off on the analysis part. Let us consider Pig for apache log analysis. Pig has some built in libraries that would help us load the apache log files into pig and also some cleanup operation on string values from crude log files. All the functionalities are available in the piggybank.jar mostly available under pig/contrib/piggybank/java/ directory. As the first step we need to register this jar file with our pig session then only we can use the functionalities in our Pig Latin

1. Register PiggyBank jar

REGISTER /usr/lib/pig/contrib/piggybank/java/piggybank.jar;

Once we have registered the jar file we need to define a few functionalities to be used in our Pig Latin. For any basic apache log analysis we need a loader to load the log files in a column oriented format in pig, we can create a apache log loader as

2. Define a log loader

DEFINE ApacheCommonLogLoader org.apache.pig.piggybank.storage.apachelog.CommonLogLoader();

(Piggy Bank has other log loaders as well)

In apache log files the default format of date is ‘dd/MMM/yyyy:HH:mm:ss Z’ . But such a date won’t help us much in case of log analysis we may have to extract date without time stamp. For that we use DateExtractor()

3. Define Date Extractor

DEFINE DayExtractor org.apache.pig.piggybank.evaluation.util.apachelogparser.DateExtractor('yyyy-MM-dd');

Once we have the required functionalities with us we need to first load the log file into pig

4. Load apachelog file into pig

--load the log files from hdfs into pig using CommonLogLoader

logs = LOAD '/userdata/bejoys/pig/p01/access.log.2011-01-01' USING ApacheCommonLogLoader AS (ip_address, rfc, userId, dt, request, serverstatus, returnobject, referersite, clientbrowser);

Now we are ready to dive in for the actual log analysis. There would be multiple information you need to extract out of a log; we’d see a few of those common requirements out here

Note: you need to first register the jar, define the classes to be used and load the log files into pig before trying out any of the pig latin below

Requirement 1: Find unique hits per day

PIG Latin

--Extracting the day alone and grouping records based on days

grpd = GROUP logs BY DayExtractor(dt) as day;

--looping through each group to get the unique no of userIds

cntd = FOREACH grpd

{

tempId = logs.userId;

uniqueUserId = DISTINCT tempId;

GENERATE group AS day,COUNT(uniqueUserId) AS cnt;

}

--sorting the processed records based on no of unique user ids in descending order

srtd = ORDER cntd BY cnt desc;

--storing the final result into a hdfs directory

STORE srtd INTO '/userdata/bejoys/pig/ApacheLogResult1';

Requirement 1: Find unique hits to websites (IPs) per day

PIG Latin

--Extracting the day alone and grouping records based on days and ip address

grpd = GROUP logs BY (DayExtractor(dt) as day,ip_address);

--looping through each group to get the unique no of userIds

cntd = FOREACH grpd

{

tempId = logs.userId;

uniqueUserId = DISTINCT tempId;

GENERATE group AS day,COUNT(uniqueUserId) AS cnt;

}

--sorting the processed records based on no of unique user ids in descending order

srtd = ORDER cntd BY cnt desc;

--storing the final result into a hdfs directory

STORE srtd INTO '/userdata/bejoys/pig/ ApacheLogResult2 ';

Note: When you use pig latin in grunt shell we need to know a few factors

1. When we issue a pig statement in grunt and press enter only the semantic check is being done, no execution is triggered.

2. All the pig statements are executed only after the STORE command is submitted, ie map reduce programs would be triggered only after STORE is submitted

3. Also in this case you don’t have to load the log files again and again to pig once it is loaded we can use the same for all related operations in that session. Once you are out of the grunt shell the loaded files are lost, you’d have to perform the register and log file loading steps all over again.

paulwong 2013-04-08 02:06 å‘è¡¨è¯„è®º

]]>PIGž®è®®http://m.tkk7.com/paulwong/archive/2013/04/05/397411.htmlpaulwongpaulwongFri, 05 Apr 2013 13:33:00 GMThttp://m.tkk7.com/paulwong/archive/2013/04/05/397411.htmlhttp://m.tkk7.com/paulwong/comments/397411.htmlhttp://m.tkk7.com/paulwong/archive/2013/04/05/397411.html#Feedback0http://m.tkk7.com/paulwong/comments/commentRss/397411.htmlhttp://m.tkk7.com/paulwong/services/trackbacks/397411.htmlä»€ä¹ˆæ˜¯PIG

æ˜¯ä¸€¿Uè®¾è®¡è¯a€åQŒé€šè¿‡è®¾è®¡æ•°æ®æ€Žä¹ˆ‹¹åŠ¨åQŒç„¶åŽç”±ç›¸åº”çš„å¼•æ“Žå°†æ¤å˜æˆMAPREDUCE JOBåŽ»HADOOPä¸è¿è¡Œã€?/div>

PIGä¸ŽSQL

ä¸¤è€…æœ‰ç›¸åŒä¹‹å¤„åQŒæ‰§è¡Œä¸€ä¸ªæˆ–å¤šä¸ªè¯å¥åQŒç„¶åŽå‡ºæ¥ä¸€äº›ç»“æžœã€?/div>

ä½†ä¸åŒçš„æ˜¯ï¼ŒSQLè¦å…ˆæŠŠæ•°æ®å¯¼åˆ°è¡¨ä¸æ‰èƒ½æ‰§è¡Œï¼ŒSQLä¸å…³å¿ƒä¸é—´å¦‚ä½•åšåQŒå³å‘ä¸€ä¸ªSQLè¯å¥˜q‡åŽ»åQŒå°±æœ‰ç»“æžœå‡ºæ¥ã€?/div>

PIGåQŒæ— ™åÕd¯¼æ•°æ®åˆ°è¡¨ä¸ï¼Œä½†è¦è®¾è®¡ç›´åˆ°å‡ºç»“æžœçš„ä¸é—´˜q‡ç¨‹åQŒæ¥éª¤å¦‚ä½•ç‰½{‰ã€?/div>

paulwong 2013-04-05 21:33 å‘è¡¨è¯„è®º

]]>PIGèµ„æºhttp://m.tkk7.com/paulwong/archive/2013/04/05/397406.htmlpaulwongpaulwongFri, 05 Apr 2013 10:19:00 GMThttp://m.tkk7.com/paulwong/archive/2013/04/05/397406.htmlhttp://m.tkk7.com/paulwong/comments/397406.htmlhttp://m.tkk7.com/paulwong/archive/2013/04/05/397406.html#Feedback0http://m.tkk7.com/paulwong/comments/commentRss/397406.htmlhttp://m.tkk7.com/paulwong/services/trackbacks/397406.html http://guoyunsky.iteye.com/blog/1317084

http://guoyunsky.iteye.com/category/196632

Hadoopå¦ä¹ ½W”è®°(9) Pig½Ž€ä»?br /> http://www.distream.org/?p=385

[hadoop¾pÕdˆ—]Pigçš„å®‰è£…å’Œ½Ž€å•ç¤ºä¾?br /> http://blog.csdn.net/inkfish/article/details/5205999

Hadoop and Pig for Large-Scale Web Log Analysis
http://www.devx.com/Java/Article/48063

Pigå®žæˆ˜
http://www.cnblogs.com/xuqiang/archive/2011/06/06/2073601.html

[åŽŸåˆ›]Apache Pigä¸æ–‡æ•™ç¨‹åQˆè¿›é˜Óž¼‰
http://www.codelast.com/?p=4249

åŸÞZºŽhadoopòq›_°çš„pigè¯è¨€å¯¹apacheæ—¥å¿—¾pÈ»Ÿçš„åˆ†æž?br /> http://goodluck-wgw.iteye.com/blog/1107503

!!Pigè¯è¨€
http://hi.baidu.com/cpuramdisk/item/a2980b78caacfa3d71442318

Embedding Pig In Java Programs
http://wiki.apache.org/pig/EmbeddedPig

ä¸€ä¸ªpigäº‹ä¾‹(REGEX_EXTRACT_ALL, DBStorageåQŒç»“æžœå˜˜q›æ•°æ®åº“)
http://www.myexception.cn/database/1256233.html

Programming Pig
http://ofps.oreilly.com/titles/9781449302641/index.html

[åŽŸåˆ›]Apache Pigçš„ä¸€äº›åŸº¼‹€æ¦‚å¿µåŠç”¨æ³•æ€È»“åQ?åQ?br /> http://www.codelast.com/?p=3621

!PIGæ‰‹å†Œ
http://pig.apache.org/docs/r0.11.1/func.html#built-in-functions

paulwong 2013-04-05 18:19 å‘è¡¨è¯„è®º