|络上面介绍q个配置的比较多Q我׃重复力_了?/p>
推荐文如下QNutch Version 0.8x tutorial Q还有就是这里的?a target="_blank">日志?/p>
我在q里记录一下遇到的几个错误和解军_法，大家可能有用?/p>
如执行如下命令：
./nutch crawl ../urls.txt -dir ../ihooyo -depth 5 -topN 100
参数说明Q?br />
-url 是刚才我们创徏的url文g,存放我们要抓取的|址
-dir 指定抓取内容所存放的目录，如上存在mydir?br />
-threads 指定q发的线E数
-depth 表示以要抓取|站|址v点的爬行深度
-topN 表示获取前多条记录Q可?/p>
可能错误1Q?/p>
Generator: jobtracker is 'local', generating exactly one partition.
Generator: 0 records selected for fetching, exiting ...
Stopping at depth=0 - no more URLs to fetch.
No URLs to fetch - check your seed list and URL filters.
crawl finished: sina5
说明Q指定要抓取的网址(url.txt)l过(crawl-urlfilters.xml)qo后，已经没有可抓取对象了Q检查两者的匚w卛_?/p>
可能错误2Q?/p>
Dedup: starting
Dedup: adding indexes in: ../ihooyo/indexes
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
at org.apache.nutch.indexer.DeleteDuplicates.dedup(DeleteDuplicates.java:439)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:135)
说明Q一般ؓ./conf/nutch-site.xml文g配置有错误。请参考如下配|修攏V?br />
[xml]
<property>
<name>http.agent.name</name>
<value>ihooyo</value>
<description></description>
</property>
<property>
<name>http.agent.description</name>
<value>apersonblog</value>
<description></description>
</property>
<property>
<name>http.agent.url</name>
<value>www.ihooyo.com</value>
<description></description>
</property>
<property>
<name>http.agent.email</name>
<value>pjuneye@qq.com</value>
<description></description>
</property>
[/xml]
q种配置错误Q在log日志中可扑ֈ提示?/p>
可能错误3Q?/p>
Injector: Converting injected urls to crawl db entries.
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:604)
at org.apache.nutch.crawl.Injector.inject(Injector.java:162)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:115)
说明Q一般ؓcrawl-urlfilters.txt中配|问题，比如qo条g应ؓ
+^http://www.ihooyo.com ,而配|成?http://www.ihooyo.com q样的情况就引v如上错误?/p>
好了写完了?/p>
基本思想Q?/strong> 堆的定义: N个元素的序列K1,K2,K3,...,Kn.UCؓ堆，当且仅当该序列满特性： 排序q程Q?br> 根堆排序实玎ͼ HeapSort.java >>>数组排序?lt;<<
堆排序是一树Ş选择排序Q在排序q程中，R[1..N]看成是一颗完全二叉树的顺序存储结构，利用完全二叉树中双亲l点和孩子结点之间的内在关系来选择最的元素?/p>
Ki≤K2i Ki ≤K2i+1(1?I?[N/2])
堆实质上是满_下性质的完全二叉树Q树中Q一非叶子结点的关键字均大于{于其孩子结点的关键字。例如序?0,15,56,25,30,70是一个堆Q它对应的完全二叉树如上图所C。这U堆中根l点Q称为堆Ӟ的关键字最，我们把它UCؓ根堆。反之，若完全二叉树中Q一非叶子结点的关键字均大于{于其孩子的关键字，则称之ؓ大根堆?/p>
堆排序正是利用小根堆Q或大根堆）来选取当前无序Z关键字小Q或最大）的记录实现排序的。我们不妨利用大根堆来排序。每一排序的基本操作是：当前无序区调整Z个大根堆Q选取关键字最大的堆顶记录Q将它和无序Z的最后一个记录交换。这P正好和直接选择排序相反Q有序区是在原记录区的尾部Ş成ƈ逐步向前扩大到整个记录区?package javasort;
// 参考：http://blog.csdn.net/EmaYoung/archive/2007/09/29/1806228.aspx
public class HeapSort {
//调整无序序列为大根堆 s 为数l的起始下标Qm为终止下?/span>
public void HeapAdjust(int[] arr, int s, int m) {
int temp = arr[s];
for (int j = 2 * s + 1; j < m; j = j * 2 + 1) {
if (j + 1 < m && arr[j] > arr[j + 1]) {
++j;
}
if (temp < arr[j]) {
break;
}
arr[s] = arr[j];
s = j;
arr[s] = temp;
}
}
//Ҏ大根堆，对堆排序
public void HeapSorting(int[] arr) {
//把顺序表构徏成ؓ一个大根堆
for (int i = arr.length / 2 - 1; i >= 0; --i) {
HeapAdjust(arr, i, arr.length);
}
for (int j = arr.length - 1; j > 0; --j) {
int temp = arr[0];
arr[0] = arr[j];
arr[j] = temp;
HeapAdjust(arr, 0, j);
}
}
}
Main.java
package javasort;
public class Main {
public static void main(String[] args) {
int[] arry_int = {49, 38, 65, 97, 76, 13, 27, 55};
show("数组排序?, arry_int);
System.out.println();
HeapSort hsort = new HeapSort();
hsort.HeapSorting(arry_int);
show("数组排序?, arry_int);
}
private static void show(String message, int[] array) {
System.out.println(">>>" + message + "<<<");
for (int i = 0; i < array.length; i++) {
System.out.print(" " + array[i]);
}
}
}
执行l果Q?/pre>
49 38 65 97 76 13 27 55
>>>数组排序?lt;<<
97 76 65 55 49 38 27 13
]]>
- AXIS1.4 DOC 阅读W记
- 安装介绍
- 作ؓ单独WebApplication安装
- Step1:拯AXIS工程/webapps/axis到相应web服务器的部v目录卛_
- Step2:启动web服务?讉K: http://127.0.0.1:8080/axis/ ?http://localhost:8080/axis/happyaxis.jsp ;如不正常Ҏ错误卛_
- Step3:Test a SOAP Endpoint:http://localhost:8080/axis/services/Version?method=getVersion
- Step4:Test a JWS Endpoint http://localhost:8080/axis/EchoHeaders.jws?method=list .
- 集成AXIS
- 1.Add axis.jar, wsdl.jar, saaj.jar, jaxrpc.jar and the other dependent libraries to your WAR file.
- 2.Copy all the Axis Servlet declarations and mappings from axis/WEB-INF/web.xml and add them to your own web.xml
- 3.Build and deploy your webapp.
- 4.Run the Axis AdminClient against your own webapp, instead of Axis, by changing the URL you invoke it with
- 遗留问题(没有配置成功)
-
如何配置使用SOAPMonitor?
-
- 注意事项
- 配置Classpath一定要加入所有的Jar包，一个都会错
- 作ؓ单独WebApplication安装
- 用户向导
- 什么是AXIS
- AXIS: Apache EXtensible Interaction System
- AXIS包括什?
- a SOAP engine -- a framework for constructing SOAP processors such as clients, servers, gateways
- a simple stand-alone server
- a server which plugs into servlet engines such as Tomcat
- extensive support for the Web Service Description Language (WSDL)
- emitter tooling that generates Java classes from WSDL
- some sample programs,anda tool for monitoring TCP/IP packets
- 使用AXIS
- 部vWebService
- 方式1QJWSQ将需要部|的WebService的源文gҎ展名.java?jws,其至于工程的根目录下面Q发布即告完成。需要注意的是：要部|的文g只能使用默认包?/span>
- 方式2QWSDDQ编写WSDD文gQ通过执行org.apache.axis.client.AdminClient来部|Ӏ也可以通过此类撤销已部|的WebService服务?/span>
- 配置WSDDQ通过org.apache.axis.client.AdminClient来部|?/span>
- WSDD: Web Service Deployment Descriptor
- 基本配置Q?
<deployment xmlns="http://xml.apache.org/axis/wsdd/" xmlns:java="http://xml.apache.org/axis/wsdd/providers/java"> <service name="MyService" provider="java:RPC"> <parameter name="className" value="samples.userguide.example3.MyService"/> <parameter name="allowedMethods" value="*"/> </service> </deployment>
- 常用配置Q配|JavaBean 客户端代码：
WSDD文gQ?QName qn = new QName( "urn:BeanService", "Order" ); call.registerTypeMapping(Order.class, qn, new org.apache.axis.encoding.ser.BeanSerializerFactory(Order.class, qn), new org.apache.axis.encoding.ser.BeanDeserializerFactory(Order.class, qn)); 扩展配置Q参照《Reference Material ?/span><beanMapping qname="myNS:Order" xmlns:myNS="urn:BeanService" languageSpecificType="java:samples.userguide.example5.Order"/>
- 方式2QWSDDQ编写WSDD文gQ通过执行org.apache.axis.client.AdminClient来部|Ӏ也可以通过此类撤销已部|的WebService服务?/span>
- 方式1QJWSQ将需要部|的WebService的源文gҎ展名.java?jws,其至于工程的根目录下面Q发布即告完成。需要注意的是：要部|的文g只能使用默认包?/span>
- 讉KWebService
- ~写服务讉K客户端，q执行之Q必要步骤，example3ZQ?/span>
- Step1Q获取服务的讉K地址Q?
String endpointURL Q?nbsp;“http://localhost:8080/axis/services/MyService";
- Step2Q构建Service ?Call对象Q?
Service service = new Service(); Call call = (Call) service.createCall();
- Step3Q确定访问目标：
call.setTargetEndpointAddress( new java.net.URL(endpointURL) ); call.setOperationName( new QName("http://example3.userguide.samples", "serviceMethod") );
- Step4Q?可? 配置参数cdQ?
call.addParameter( "arg1", XMLType.XSD_STRING, ParameterMode.IN); call.setReturnType( org.apache.axis.encoding.XMLType.XSD_STRING );
- Step5Q发赯问，q获取反?
String ret = (String) call.invoke( new Object[] { textToSend } ); - Step6Q处理异?
try {//上面那些
} catch (AxisFault fault) {et = "Error : " + fault.toString();}
- Step1Q获取服务的讉K地址Q?
- ~写服务讉K客户端，q执行之Q必要步骤，example3ZQ?/span>
- 通过WSDL来访问WebServiceQ必要步骤，以Example6ZQ?/span>
- WSDLQ?Web Service Description Language Web服务器描q语a是用XML文档来描qWeb服务的标准，是Web服务的接口定义语aQ由Ariba、Intel、IBM、MS{共同提出，通过WSDLQ可描述Web服务的三个基本属性：? 服务做些什么——服务所提供的操作（ҎQ??#22914;何访问服务——和服务交互的数据格式以及必要协??#26381;务位于何处——协议相关的地址Q如URL
- Step1Q获得WSDL文g
- 方式1Q?WSDL http://<host>/axis/services/<service-name>?wsdl http://<host>/axis/*.jws?wsdl
- 方式2QJAVA2WSDL工具 % java org.apache.axis.wsdl.Java2WSDL -o wp.wsdl -l"http://localhost:8080/axis/services/WidgetPrice" -n "urn:Example6" -p"samples.userguide.example6" "urn:Example6" samples.userguide.example6.WidgetPrice
Where: -o indicates the name of the output WSDL file -l indicates the location of the service -n is the target namespace of the WSDL file -p indicates a mapping from the package to a namespace. There may be multiple mappings. the class specified contains the interface of the webservice.
- Step2Q通过WSDL2JAVA工具获得辅助c?% java org.apache.axis.wsdl.WSDL2Java -o . -d Session -s -S true -N urn:Example6 samples.userguide.example6 wp.wsdl
WidgetPriceSoapBindingImpl.java : Java file containing the default server implementation of the WidgetPrice web service. You will need to modify the *SoapBindingImpl file to add your implementation WidgetPrice.java: 定义了Web服务接口 WidgetPriceService.java: 定义了用于获取Web服务接口的方法?WidgetPriceServiceLocator.java: 定义了用于获取Web服务接口的方法?WidgetPriceSoapBindingStub.java:Web服务客户端桩Q通过该类与服务器交互?WidgetPriceSoapBindingSkeleton.java: Server side skeleton. deploy.wsdd: Deployment descriptor undeploy.wsdd: Undeployment descriptor q几个JAVAcd我们处理了大部分的逻辑Q我们需要的仅仅是把q些cd到我们的目然后创徏一个我们自qcL调用它们卛_ - Step3Q编写客L代码: 通过***ServiceLocator构?**ServiceҎQ通过***Service对象获得提供服务的类对象Q进而调用提供服务类对象上的ҎQ提供服务?
- 部vWebService
- 工具使用
- the Axis TCP Monitor :java org.apache.axis.utils.tcpmon [listenPort targetHost targetPort]
- the SOAP Monitor :
- 什么是AXIS
- 安装介绍
- WebService安全
- 常见d方式
- Denial of Service to a server
- Interception and manipulation of messages
- Forged client requests or Forged server responses
- attempts to read the server file system/database
- Attempts to write to the server file system/database
- 判断来访?/span>
- AXIS不支持判断请求服务的是谁.可以使用 xmlsec.jar来支援它
- AXIS推荐使用HTTPS来加U安全?/span>
- Axis 不支持HTTP1.1 Digest Authentication,需要the HttpClient libraries配合工作
- 可考虑的安全措?/span>
- Disguise:不要让h知道你运行了AXIS
- Cut down the build: 仅仅保留你需要的部分
- Rename things:换掉默认的名U?如The AxisServlet, the AdminService, even happyaxis.jsp
- Stop AxisServlet listing services :axis.enableListQuery =false
- Keep stack traces out of the responses :axis.development.system =true
- Edit the .wsdd configuration file, as described in the reference, to return a WSDL resource which is simply an empty <wsdl/> tag.
- Servlets2.3: use filters for extra authentication
- Log things
- Run Axis with reduced Java rights
- Run the web server with reduced rights
- Monitor Load
- Consider 'tripwire' and 'honeypot' endpoints(没明?
- 常见d方式
- AXIS ANT Task
- 在axis-ant.jar中定?/span>
- 主要d
- Creating Java files from WSDL
- Creating WSDL files from Java
- Talking to the admin service
- 具体配置看文?/span>
- 主要d
- 在axis-ant.jar中定?/span>
- 《Reference Material?
- 其中注意包括了WSDL2JAVAQJAVA2WSDL的具体用，以及WSDD的配|信息等技术细?/span>
]]>