LDI Docs – 2 Installing and Testing

2 Installing and Testing

2.1 Requirements

• JDeveloper 11g (optional) only if you want to edit the Java code
• Ant 1.7.0
• Sun JDK 1.5.0_05/1.4.2 ($ORACLE_HOME/jdk directory works fine as Java Home for compiling on 10g and 11g)
• Linux/Windows Database Oracle 10g 10.2/11g production

2.2 Install binary distributions

Binary distributions are available at SourceForge.net and provides a very straightforward installation.

2.2.1 11g Binary Distribution

Edit your ~/build.properties file with your Database values (Windows users can find build.properties file at C:\Documents and Settings\username folder):

  db.str=test
  db.usr=LUCENE
  db.pwd=LUCENE
  dba.usr=sys
  dba.pwd=change_on_install
  javac.debug=true
  javac.source=1.5
  javac.target=1.5

db.str is your SQLNet connect string for your target database, check first with tnsping. This is an example environment setting before installing on 11g database:

  MAVEN_HOME=/usr/local/maven
  ORACLE_BASE=/u01/app/oracle
  ORACLE_HOME=$ORACLE_BASE/product/11.1.0.6.0/db_1
  ORACLE_SID=test
  JAVA_HOME=$ORACLE_HOME/jdk
  PATH=$MAVEN_HOME/bin:$HOME/bin:$ORACLE_HOME/bin:$JAVA_HOME/bin:/usr/local/bin:$PATH
  LD_LIBRARY_PATH=$ORACLE_HOME/lib:/usr/local/lib
  CVS_RSH=ssh
  umask 022
  export PATH LD_LIBRARY_PATH ORACLE_HOME ORACLE_BASE ORACLE_SID JAVA_HOME CVS_RSH NLS_LANG

Upload, install and test your code into the database:

  ant install-odi
  ant test-odi-[clob|dicc|emails|iot|master-detail|similarity|source-small|term-vector|xmltype]

For Oracle 11g you can perform a post-installation step:

  ant jit-lucene-classes

This target force to translate all Lucene, Snowball, etc. and ODI classes to assembler. Instead of waiting that the database compile it by detecting most used classes or method.

2.2.2 10g Binary Distribution

First edit your ~/build.properties with something like this:

  db.str=orcl
  db.usr=LUCENE
  db.pwd=LUCENE
  dba.usr=sys
  dba.pwd=change_on_install
  javac.debug=true
  javac.source=1.4
  javac.target=1.4

db.str property is a SQLNet connect string for the target database. ORACLE_HOME environment setting is required and properly configured to an Oracle 10g database layout, finally execute ant without arguments. Here an example of environment setting on 10g database:

  MAVEN_HOME=/usr/local/maven
  ORACLE_BASE=/u01/app/oracle
  ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
  ORACLE_SID=orcl
  JAVA_HOME=$ORACLE_HOME/jdk
  PATH=$MAVEN_HOME/bin:$HOME/bin:$ORACLE_HOME/bin:$JAVA_HOME/bin:/usr/local/bin:$PATH
  LD_LIBRARY_PATH=$ORACLE_HOME/lib:/usr/local/lib
  CVS_RSH=ssh
  umask 022
  export PATH LD_LIBRARY_PATH ORACLE_HOME ORACLE_BASE ORACLE_SID JAVA_HOME CVS_RSH NLS_LANG

If you are re-installing Oracle Lucene Domain Index (ODI) first drop any Lucene Domain Index. Default target will drop first Lucene schema if exists, example install action:

  ant install-odi-10g
ant test-odi-[clob|dicc|emails|iot|master-detail|similarity|source-small|term-vector|xmltype]

Additionally to installation and testing steps you can perform a NCOMP translation of Lucene ODI and related classes from Java to assembler using:

  ant ncomp-runtime-retrotranslator-sys-code
  ant ncomp-lucene-all

2.3 Install Instructions to compile from sources

• Unpack or checkout Lucene sources
• Checkout ODI sources, by now only Anonymous CVS access is provided you can download from Source Forge servers with
  

    cd /tmp
    svn co http://ldi.svn.sourceforge.net/svnroot/ldi/odi/trunk odi
  

• Copy to $LUCENE_ROOT/contrib
  

    cd $LUCENE_ROOT/contrib
    cp -rp /tmp/odi .
  

• Edit $LUCENE_ROOT/common-build.xml adding a target for creating a jar file with test sources
  

  ....
    <property name="dev.version" value="3.4.0"/>
    <property name="tests.luceneMatchVersion" value="3.4.0"/>
  ....
    <target name="jar-test" depends="compile-test">
      <jar destfile="${build.dir}/${final.name}-test.jar" basedir="${build.dir}/classes/test" excludes="**/*.java"/>
    </target>
  

(OPTIONAL) Update Lucene’s BufferedIndexInput.BUFFER_SIZE according to your db_block_size init.ora parameter.
Before compile and upload Lucene core library you can change org.apache.lucene.store.BufferedIndexInput.BUFFER_SIZE constant to the value of your db_block_size init parameter, this change will improve reading performance by using same block size as the physical block size that your database use.

Compile ODI Directory sources and tests, these targets automatically copies all Lucene Domain Index required libraries from your $ORACLE_HOME and Internet. Starting with ODI 2.4.0.1.x build.xml file automatically compiles all Lucene contrib modules dependency.

  cd $LUCENE_ROOT/contrib/odi
  ant jar-core
  ant jar-test
  ant package-zip

Edit your ~/build.properties file with your Database values:

  db.str=orcl
  db.usr=LUCENE
  db.pwd=LUCENE
  dba.usr=sys
  dba.pwd=change_on_install
  javac.debug=true
  javac.source=1.4
  javac.target=1.4

db.str is your SQLNet connect string for your target database, check first with tnsping utility, also note that for 11g database user and password are case sensitive, so leave LUCENE in uppercase. Upload your code to the database:

  ant install-odi

2.3.1 Generating Maven’s artifacts

You can generate Lucene and ODI Maven’s artifacts following previous one steps, then execute:

   ant generate-maven-artifacts

2.4 Optimizations

Discussion of Optimizations.

2.4.1 Using NCOMP on 10g

Is strongly recommended before going in production that install Oracle Lucene Domain Index NCOMPed in 10g databases. NCOMP automatically translate Lucene and ODI Java code to assembler and finally install it as dynamic link library (.so/.dll) in your Oracle home. To do this simply execute this Ant task instead of install-odi target:

  ant ncomp-lucene-all

2.4.2 Using JIT on 11g

First verify that your database parameter java_jit_enabled is TRUE. Oracle 11g includes a JIT technology which automatically translates most used Java methods to assembler. If you want to pre-compile all Lucene Java code to assembler and not wait for Oracle database detects common used code you can execute this target:

  ant jit-lucene-classes
  ant jit-oracle-classes

2.5 Testing Lucene Domain Index

Required grants for regular Oracle users. IMPORTANT: Before start using Lucene Domain Index grant this to any Oracle user rather than LUCENE:

  -- connected as sysdba
  grant LUCENEUSER to scott;

Lucene Domain Index have two kinds of test suites to check that everything is OK after installation. First test suite which can be launched using Ant and is implemented using SQLUnit, to launch it simply execute (first checking values of SQLNet connection options):

  [mochoa@mochoa odi]$ cat db/sqlunit.properties
  # SQLUnit test
  sqlunit.driver = oracle.jdbc.driver.OracleDriver
  sqlunit.url = jdbc:oracle:oci:@orcl
  sqlunit.user = LUCENE
  sqlunit.password = LUCENE
[mochoa@localhost lucene-odi]$ ant test-odi-clob
Buildfile: build.xml
test-odi-clob:

Second test suite is a set of JUnit tests to simulate middle tier environments, it also use a connection pool. To start these suites run:

[mochoa@localhost lucene-odi]$ ant test-parallel
Buildfile: build.xml

test-parallel:
    [junit] Running org.apache.lucene.index.TestDBIndexParallel
    [junit] Table created: T1
    [junit] Index altered: LIT1
    [junit] OnLine: true
    [junit] No Row updated at: 4030 to: 4039 elapsed time: 22 ms.
    [junit] Inserted rows: 10 time: 193 avg time: 19
      ...
    [junit] Index droped: LIT1
    [junit] Table droped: T1

Next test suite is JUnit tests to simulate middle tier environments, it also use a connection pool. To start these suites run:

[mochoa@localhost lucene-odi]$ ant test-queryhits
Buildfile: build.xml

test-queryhits:
    [junit] Running org.apache.lucene.indexer.TestQueryHits
    [junit] Hits: 35602
      ...
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 63.306 sec
BUILD SUCCESSFUL
Total time: 1 minute 4 seconds

Class org.apache.lucene.indexer.TestQueryHits requires a table which is very big to create and destroy at setup() and tearDown() methods. Before run this test create the table with:

    create table test_source_big as (select * from all_source);

And the index on 10g with:

create index source_big_lidx on test_source_big(text)
indextype is lucene.luceneindex
parameters('AutoTuneMemory:true;IndexOnRam:true;BatchCount:3000;ParallelDegree:2;SyncMode:OnLine;LogLevel:INFO;AutoTuneMemory:true;PerFieldAnalyzer:line(org.apache.lucene.analysis.KeywordAnalyzer),TEXT(org.apache.lucene.analysis.SimpleAnalyzer);FormatCols:line(00000);ExtraCols:line "line";MergeFactor:500');

Or in 11g with:

create index source_big_lidx on test_source_big(text)
indextype is lucene.luceneindex
parameters('AutoTuneMemory:true;IndexOnRam:true;BatchCount:3000;ParallelDegree:2;SyncMode:OnLine;LogLevel:INFO;AutoTuneMemory:true;PerFieldAnalyzer:line(org.apache.lucene.analysis.KeywordAnalyzer),TEXT(org.apache.lucene.analysis.SimpleAnalyzer);FormatCols:line(00000);ExtraCols:line "line";MergeFactor:500;LobStorageParameters:CACHE READS FILESYSTEM_LIKE_LOGGING');

Doc Links

Previous / LDI Docs – 1 Introduction
Next / LDI Docs – 3 Procedures, Functions, Operators and Examples