<rt id="bn8ez"></rt>
<label id="bn8ez"></label>

  • <span id="bn8ez"></span>

    <label id="bn8ez"><meter id="bn8ez"></meter></label>

    paulwong

    Analyzing Apache logs with Pig



    Analyzing log files, churning them and extracting meaningful information is a potential use case in Hadoop. We don’t have to go in for MapReduce programming for these analyses; instead we can go for tools like Pig and Hive for this log analysis. I’d just give you a start off on the analysis part. Let us consider Pig for apache log analysis. Pig has some built in libraries that would help us load the apache log files into pig and also some cleanup operation on string values from crude log files. All the functionalities are available in the piggybank.jar mostly available under pig/contrib/piggybank/java/ directory. As the first step we need to register this jar file with our pig session then only we can use the functionalities in our Pig Latin
    1.       Register PiggyBank jar
    REGISTER /usr/lib/pig/contrib/piggybank/java/piggybank.jar;
    Once we have registered the jar file we need to define a few functionalities to be used in our Pig Latin. For any basic apache log analysis we need a loader to load the log files in a column oriented format in pig, we can create a apache log loader as
    2.       Define a log loader
    DEFINE ApacheCommonLogLoader org.apache.pig.piggybank.storage.apachelog.CommonLogLoader();
    (Piggy Bank has other log loaders as well)
    In apache log files the default format of date is ‘dd/MMM/yyyy:HH:mm:ss Z’ . But such a date won’t help us much in case of log analysis we may have to extract date without time stamp. For that we use DateExtractor()
    3.       Define Date Extractor
    DEFINE DayExtractor org.apache.pig.piggybank.evaluation.util.apachelogparser.DateExtractor('yyyy-MM-dd');
    Once we have the required functionalities with us we need to first load the log file into pig
    4.       Load apachelog file into pig
    --load the log files from hdfs into pig using CommonLogLoader
    logs = LOAD '/userdata/bejoys/pig/p01/access.log.2011-01-01' USING ApacheCommonLogLoader AS (ip_address, rfc, userId, dt, request, serverstatus, returnobject, referersite, clientbrowser);
    Now we are ready to dive in for the actual log analysis. There would be multiple information you need to extract out of a log; we’d see a few of those common requirements out here
    Note: you need to first register the jar, define the classes to be used and load the log files into pig before trying out any of the pig latin below
    Requirement 1: Find unique hits per day
    PIG Latin
    --Extracting the day alone and grouping records based on days
    grpd = GROUP logs BY DayExtractor(dt) as day;
    --looping through each group to get the unique no of userIds
    cntd = FOREACH grpd
    {
                    tempId =  logs.userId;
                    uniqueUserId = DISTINCT tempId;
                    GENERATE group AS day,COUNT(uniqueUserId) AS cnt;
    }
    --sorting the processed records based on no of unique user ids in descending order
    srtd = ORDER cntd BY cnt desc;
    --storing the final result into a hdfs directory
    STORE srtd INTO '/userdata/bejoys/pig/ApacheLogResult1';
    Requirement 1: Find unique hits to websites (IPs) per day
    PIG Latin
    --Extracting the day alone and grouping records based on days and ip address
    grpd = GROUP logs BY (DayExtractor(dt) as day,ip_address);
    --looping through each group to get the unique no of userIds
    cntd = FOREACH grpd
    {
                    tempId =  logs.userId;
                    uniqueUserId = DISTINCT tempId;
                    GENERATE group AS day,COUNT(uniqueUserId) AS cnt;
    }
    --sorting the processed records based on no of unique user ids in descending order
    srtd = ORDER cntd BY cnt desc;
    --storing the final result into a hdfs directory
    STORE srtd INTO '/userdata/bejoys/pig/ ApacheLogResult2 ';
    Note: When you use pig latin in grunt shell we need to know a few factors
    1.       When we issue a pig statement in grunt and press enter only the semantic check is being done, no execution is triggered.
    2.       All the pig statements are executed only after the STORE command is submitted, ie map reduce programs would be triggered only after STORE is submitted
    3.       Also in this case you don’t have to load the log files again and again to pig once it is loaded we can use the same for all related operations in that session. Once you are out of the grunt shell the loaded files are lost, you’d have to perform the register and log file loading steps all over again.

    posted on 2013-04-08 02:06 paulwong 閱讀(357) 評論(0)  編輯  收藏 所屬分類: PIG

    主站蜘蛛池模板: 亚洲国产精品一区二区成人片国内| 一本无码人妻在中文字幕免费| 国产男女猛烈无遮挡免费视频网站| 亚洲欧洲国产视频| 亚洲毛片免费观看| 亚洲春黄在线观看| 台湾一级毛片永久免费| 亚洲日韩国产精品无码av| 久久综合给合久久国产免费 | 亚洲伦乱亚洲h视频| 精品一区二区三区免费毛片| 国产免费观看视频| 污污的视频在线免费观看| 亚洲午夜爱爱香蕉片| 精品一区二区三区免费视频| 亚洲日韩激情无码一区| 久久99免费视频| 亚洲福利视频网址| 天天操夜夜操免费视频| 人禽伦免费交视频播放| 亚洲精品午夜国产VA久久成人| 日韩免费无码一区二区三区 | 亚洲成av人片天堂网老年人| 一级**爱片免费视频| 亚洲制服中文字幕第一区| A在线观看免费网站大全| 男男gay做爽爽的视频免费| 国产偷国产偷亚洲高清日韩| 日韩免费视频一区二区| tom影院亚洲国产一区二区| 免费观看国产精品| 在线观看人成视频免费无遮挡| 亚洲网址在线观看你懂的| 免费av欧美国产在钱| 黄页网站在线视频免费| 久久青青草原亚洲AV无码麻豆 | 日韩免费观看的一级毛片| yy一级毛片免费视频| 亚洲毛片免费视频| 婷婷亚洲天堂影院| 国产成人精品免费视频大|