I have encountered sereval performance issues during my projects in the past several years and now I try to list them so that I will not forget them or sometimes I can pick it up for a reference.1. During the idol/CFS/lua development, once, a client asked me why there are some documents which they changed is lost in idol database, and we try to troubleshoot it and found the processing time for some lua script are taking days, so we try to optimize the lua query by adding fields to matchindex which is sent to idol to reduce response time.2. There is a program which needs to sync all the data from one idol database to another idol database, and the developers trying to add thousands of xml documents to the ResultList and we found the speed is becoming slow and slow and with more time, we found the equal/hashcode affecting the performance a lot and finally, we try to remove the function and found they are working well(temp solution).
public class ResultListCustom extends ResultList {
private ArrayList _alDocuments = new ArrayList();
/**
* remove the duplicates condition
*/
@Override
public void addDocument(ResultDocument document) {
if (document != null) {
_alDocuments.add(document);
}
}
@Override
public void removeDocument(
com.autonomy.aci.businessobjects.Document document) {
if (document != null) {
int nDocIdx = this._alDocuments.indexOf(document);
if (nDocIdx != -1) {
this._alDocuments.remove(nDocIdx);
}
}
}
@Override
public void clearDocuments() {
this._alDocuments.clear();
}
@Override
public ArrayList getDocuments() {
return (ArrayList) this._alDocuments.clone();
}
@Override
public Iterator iterator() {
return this._alDocuments.iterator();
}
@Override
public int getDocumentCount() {
return this._alDocuments.size();
}
}
3. Once for our enovia system, a client try to search for data with description * in search field and then we found the java process memory increased all the time and finally the application crashed. and we have analyzed this problem, but could not resolve it and finally, this problem is resolved by ds (a company) by adding index to the field description.4.For all the fields which we need to query with match in idol, we need to add this field to match index, and for all the parametric fields for refinement, we need to add to the parametric index and this way will optimize all the queries and ensure the less response time. also when you idol database deletes document too often, you need to do compact operations to optimize the idol database performance also.5. We have some lua scripts which read a excel file and build a record for each of line and then send to idol, this works well in our old cfs version, but when we upgraded to 11.0 version, we found the CFS queue is becoming slow and slow and the lua script will also take days to finish. and we raised this problem to idol team and finally they said we have to send the documents with lua to idol in batches. and we did that and performances changed a lot. but they still could not answer why sometimes the performance is slow when we send document one by one to cfs.6. Usually, connectors crawls data from different repository and then ingest the messages into CFS in batches and this improved performance , during our java developement we should also use this strategy, sending documents to idol in batches, including dreadd, drereplace, and dredelete operations.For example, the format below to support batches of idx files for drereplace#DREDOCREF 30036.20391.35739.7275
#DREFIELDNAME SEARCH-RESULTS/DOCUMENT/CSI_RATING_FORECAST
#DREFIELDVALUE ANVGRP:1,WRLS:1
#DREFIELDNAME SEARCH-RESULTS/DOCUMENT/COMCODEID
#DREFIELDVALUE SRDE10:3596831,ANVGRP:3596831
#DREDOCREF 30036.20391.35739.7274
#DREFIELDNAME SEARCH-RESULTS/DOCUMENT/CSI_RATING_FORECAST
#DREFIELDVALUE ANVGRP:1,WRLS:1
#DREFIELDNAME SEARCH-RESULTS/DOCUMENT/COMCODEID
#DREFIELDVALUE SRDE10:3596831,ANVGRP:3596831
#DREENDDATA
7. when we upgraded enovia connector to version v11.2.0.1279850 and for one of our enovia connector the memory is alway increasing with the time passing, we also raised this problem to idol team and also install valgrind tool to help analyze the memroy problem and then send the logs to idol team, but still they could not ensure whether they can solve the problem or not as this is not reproduced in their application.