3.5.0.1.0 Second Release based on Lucene 3 (3.5.0, 23/Jan/12) core base

• Fully customizable Searcher and Updater process to enable JMX monitoring and time outs
• Updated to latest spellchecker interfaces
• Fixed missing changes to work with backported version on 10g

3.4.0.1.0 Second Release based on Lucene 3 (3.4.0, 25/Nov/11) core base

• Use latest merge policy implementation TieredMergePolicy
• Use total RAM reported by getJavaPoolSize() when setting MaxBufferedDocs
• Better error reporting when an Analyzer is not found
• Replaced execute immediate with open-fech-close functionality to avoid core dump on 10g when double check for deleted rowid
• Included a backported version of JUnit4 to jdk1.4 version for 10g releases
• Added a parallel updater process, when working in OnLine mode this process do write operations on LDI structure on behalf of the AQ process
• Delete do not longer required a write exclusive lock on index storage, now deletes are also enqueued as inserts or updates
• Updated source to Lucene 3.4.0 code, removed some deprecated API

3.0.2.1.0 Initial Release based on Lucene 3 (3.0.2, 14/Sep/10) core base

• Added a long awaited functionality, a parallel/shared/slave search process used during a start-fetch-close and CountHits function
• Added lfreqterms ancillary operator returning the freq terms array of rows visited
• Added lsimilarity ancillary operator returning a computed Levenshtein distance of the row visited
• Added a ldidyoumean pipeline table function using DidYouMean.indexDictionary storage
• Added test using SQLUnit

3.0.1.1.0 Initial Release based on Lucene 3 (3.0.1) core base

• Added IndexOnRam functionality by using RAMDirectory for building intermediate index after merging a new set of rows
• RewriteScore and SimilarityMethod can be used to get better result using wildcard operator
• Added auto complete functionality using lautocomplete pipe-line table function
• Removed all deprecated method usage and implementations
• New CVS repository only compatible with 11g, 10g version is implemented by using a retro-translator

2.9.2.1.0 Production Release based on Lucene 2.9 (2.9.2) core base

• Added elapsed time information when log level is INFO
• Removed deprecated usage of LUCENE_CURRENT constant
• Fixed facets inconsitence due ignore internal parameter ColName
• Initial implementation of DidYouMean functionality contributed by Pedro Pinheiro
• Temporary fix until Lucene defines clear semantics for Directory.fileLength (see Lucene issue 2316)

2.9.1.1.0 Production Release based on Lucene 2.9 (2.9.1) core base

• New Lucene Core base libraries
• Full Lucene Test Suites certified
• Fixed bug enqueue more rowids than required when using OnLine mode and ExtraTabs, WhereCondition parameters
• Fixed operator priority when WhereCondition have OR operator
• DefaultUserDataStore now uses an array of cached fields to improve performance
• Spanish Analyzer use latest ASCIIFoldingFilter
• high_freq_terms(idx_name,term,max_num_term) pipeline table function was added to return high frequent terms and the associated docFreq value
• index_terms(idx_name,term) pipeline table function was added to return a list of terms and their associated frequency
• DefaultUserDataStore now have support for ANALYZED, ANALYZED_WITH_VECTORS, ANALYZED_WITH_OFFSETS, ANALYZED_WITH_POSITIONS and ANALYZED_WITH_POSITIONS_OFFSETS Lucene Field option values
• OJVMLock was replaced by SingleInstanceLockFactory for per instance locking, cross sessions lockings are implemented by select for update functionality
• an automatic upgrade from 2.9.0 is possible without Index deletions or rebuild, you have to execute:
  

  ant upgrade-domain-index
  ant ncomp-lucene-ojvm (10g only)
  ant jit-lucene-classes (11g only)
  

2.9.0.1.0 Production release based on Lucene 2.9.0 core base, 29/Sep/09

• Tested with Oracle 11gR2, 11gR1 and 10.2 databases
• DefaultUserDataStore do a SAX parsing to get text nodes and attributes from an XMLType value.
• A SimpleLRUCache is used to load rowids and his associated Lucene doc id, this reduce memory consumption when querying very big tables. A new parameters has been added, CachedRowIdSize by default 10000 to control the size of the LRU cache.
• Lucene Domain Index core was updated to use TopFieldCollector and to avoid computation time when lscore() is not used.
• Two new parameter has been added NormalizeScore which control when to track the Max Score and PreserveDocIdOrder when querying, both parameters are consequence of new Lucene Collector API and boost performance when querying.
• A table alias L$MT is defined for the master table associated to the index to be used in complex queries to associate columns from master tables and columns from dependent tables

2.4.1.1.0 (maintenance release based on Lucene 2.4.1, 27/Mar/09)

• Do not store internal parameters into system’s views and force to PopulateIndex:false
• After every sync, now files marked as deleted are purged to free BLOB storage
• Added lfacets aggregated function for doing facets
• CountHits function no longer requires sort argument
• Filter are stored/retrived only using QueryParser.toString() key
• UN_TOKENIZED format string at DefaultUserDataStore class was replaced by NOT_ANALYZED or NOT_ANALYZED_STORED according to new Lucene definitions.
• Fix bug when sync try to process more than 32767 rowids enqueued.
• Added parameters for highlighting functions Formatter, MaxNumFragmentsRequired, FragmentSeparator and FragmentSize.
• Added PerFieldAnalyzer parameter to use independent Analyzer for each columns.
• Added sample of a custom Formatter org.apache.lucene.search.highlight.MyHTMLFormatter

2.4.1.0.0 (first release based on Lucene 2.4.1, 9/Mar/09)

• Fix compatibility problem between 10g/11g SQL Date representation on pipeline table function.

2.4.0.1.0 (maintenance release based on Lucene 2.4.0, 10/Jan/09)

• Added Rhighlight(index_name VARCHAR2, qry VARCHAR2, cols VARCHAR2, rType IN VARCHAR2, rws IN SYS_REFCURSOR) RETURN ANYDATASET pipeline table function
• Added Phighlight(index_name VARCHAR2, qry VARCHAR2, cols VARCHAR2, stmt IN VARCHAR2) RETURN ANYDATASET pipeline table function
• Added lhighlight(NUMBER):VARCHAR2 ancilliary operator
• Removed usage of Lucene deprecated API (Hits and IndexWriter for example)
• Usage of FIRST_ROWS optimizer hits to decide how many rows load at first time
• sync, optimize and rebuild interfaces now use index_name or [owner,index_name] arguments
• A better build system to build Lucene Domain Index from sources
• More tests
• Tested against 11.1.0.7 and 10.2.0.3
• See online docs to see usage of FIRST_ROWS and lhighlight() operator

2.4.0.0.0 (production release based on Lucene 2.4.0, 10/10/08)

• Added parameter for CLOB enconding
• More Like this function
• NGram analyzer
• EnglishWikipediaAnalyzer
• DataStore interface include API for setting current connection
• Now analyzers, queries, snowball and WikiPedia contrib packages are required

2.3.2.0.0 (binary release based on Lucene 2.3.2, 1/Jun/08)

• Compiled against Lucene 2.3.2 production release
• Used latest API for merging based on RAM usage
• Use Writer for deleting during Sync
• Confirm 4x improvement during indexing reported by Lucene dev group
• Fix workaround which changes order of the rowids in ODCRIDList
• Added an Spanish WikiPedia Analyzer for testing
• Reports IOException instead of RunTimeException to signal EOF or File Not Found
• Decouple Flush functionality from TableIndexer

2.2.0.2.2 (fixpack for 2.2.0.2.0 release, 5/Apr/08)

• Added Rowid to lucene doc id caching.
• Usage of LoadFirstFieldSelector during Document loading to only load rowid field.
• Added a test suite which index a wikipedia dump inside the OJVM.

2.2.0.2.1 (fixpack for 2.2.0.2.0 release, 12/Dec/07)

• DefaultUserDataStore requires usage of XPath text() expresion for getting only textual value
• Added logging info SQL being executed at table indexer
• Change document logging to FINER level
• More pre-defined mapping at DefaultUserDataStore for NUMBER, BINARY_FLOAT, BINARY_DOUBLE, TIMESTAMP, TIMESTAMPTZ and TIMESTAMPLTZ Oracle types.
• New parameter PopulateIndex:[true|false] for populating or not Lucene Index at creation time.
• New parameter IncludeMasterColumn:[true|false], to choose whether or not index master column, useful with Virtual Columns and XMLType.
• New parameter BatchCount:integer, to choose how many rows count are enqueued for indexing using create … index … parameters(‘SyncMode:OnLine’);
• Creating an index with SyncMode:OnLine causes that LuceneDomain index will enqueue batchs of “BatchCount” rows for index by AQ PLSQL callback in background. Lucene Domain Index is intermediately ready for querying after create.
• Batch rowid indexing is doing using a pipeline function.

2.2.0.2.0 (third major release synchronized with Lucene 2.2.0, 12/Dec/07)

http://sourceforge.net/project/showfiles.php?group_id=56183

# CVS access:
cvs -d:pserver:anonymous@dbprism.cvs.sourceforge.net:/cvsroot/dbprism login
cvs -z3 -d:pserver:anonymous@dbprism.cvs.sourceforge.net:/cvsroot/dbprism co -P ojvm

• sort by column passed at lcontains(col,query_parser_str,sort_str,corr_id) syntax
• Logging support using Java Util Logging package
• JUnit test suites emulating middle tier environment
• Support for rebuild and optimize online for SyncMode:OnLine index
• XMLDB Export
• AutoTuneMemory parameter for replacing MaxBufferedDocs parameter
• Functional column support

2.2.0.1.1 (second release, 27/Sep/07 05:39 AM)

https://issues.apache.org/jira/secure/attachment/12366661/ojvm-09-27-07.tar.gz

# CVS access:
cvs -d:pserver:anonymous@dbprism.cvs.sourceforge.net:/cvsroot/dbprism login
cvs -z3 -d:pserver:anonymous@dbprism.cvs.sourceforge.net:/cvsroot/dbprism co -P ojvm

• LuceneDomainIndex.countHits() function to replace select count from .. where lcontains(..)>0 syntax.
• support inline pagination at lcontains(col,’rownum:[n TO m] AND …”) function
• rounding and padding support for columns date, timestamp, mumber, float, varchar2 and char
• ODCI API array DML support
• BLOB parameter support

2.2.0.1.0 (first release synchronized with lucene 2.2.0, 14/Sep/07 06:44 AM)

# CVS access:
cvs -d:pserver:anonymous@dbprism.cvs.sourceforge.net:/cvsroot/dbprism login
cvs -z3 -d:pserver:anonymous@dbprism.cvs.sourceforge.net:/cvsroot/dbprism co -P ojvm

• Synchronized with latest Lucene 2.2.0 production
• Replaced in memory storage using Vector based implementation by direct BLOB IO, reducing memory usage for large index.
• Support for user data stores, it means you can not only index one column at time (limited by Data Cartridge API on 10g), now you can index multiples columns at base table and columns on related tabled joined together.
• User Data Stores can be customized by the user, it means writing a simple Java Class users can control which column are indexed, padding used or any other functionality previous to document adding step.
• There is a DefaultUserDataStore which gets all columns of the query and built a Lucene Document with Fields representing each database columns these fields are automatically padded if they have NUMBER or rounded if they have DATE data, for example.
• lcontains() SQL operator support full Lucene’s QueryParser syntax to provide access to all columns indexed, see examples below.
• Support for DOMAIN_INDEX_SORT and FIRST_ROWS hint, it means that if you want to get rows order by lscore() operator (ascending,descending) the optimizer hint will assume that Lucene Domain Index will returns rowids in proper order avoided an inline-view to sort it.
• Automatic index synchronization by using AQ’s Call Back.
• Lucene Domain Index creates extra tables named IndexName$T and an Oracle AQ named IndexName$Q with his storage table IndexName$QT at user’s schema, so you can alter storage’s preference if you want.
• ojvm project is at SourceForge.net CVS, so anybody can get it and collaborate
• Tested against 10gR2 and 11gR1 database.

2.0.0.1.3 (third release, 09/Jan/07 11:40 AM)

https://issues.apache.org/jira/secure/attachment/12348574/ojvm-01-09-07.tar.gz

• The Data Cartridge API is used without column data to reduce the data stored on the queue of changes and speedup the operation of the synchronize method.
• Query Hits are cached associated to the index search and the string returned by the QueryParser.toString() method.
• If no ancillary operator is used in the select, do not store the score list.
• The “Stemmer” argument is recognized as parameter given the argument for the SnowBall analyzer, for example:
  

  create index it1 on t1(f2) indextype is lucene.LuceneIndex parameters('Stemmer:English');
  

• Before installing the ojvm extension is necessary to execute “ant jar-core” on the snowball directory.
• The IndexWriter.setUseCompoundFile(false) is called to use multi file storage (faster than the compound file) because there is no file descriptor limitation inside the OJVM, BLOBs are used instead of File.
• Files are marked for deletion and they are purged when calling to Sync or Optimize methods.
• Blob are created and populated in one call using Oracle SQL RETURNING information.
• A testing script for using OE sample schema, with query comparisons against Oracle Text ctxsys.context index.

2.0.0.1.2 (second release, 20/Dec/06 02:03 PM)

https://issues.apache.org/jira/secure/attachment/12347614/ojvm-12-20-06.tar.gz

• This new release of the OJVMDirectory Lucene Store includes a fully functional Oracle Domain Index with a queue for update/insert massive operations and a lot of performance improvement.

2.0.0.1.1 (first release, 28/Nov/06 01:04 PM)

https://issues.apache.org/jira/secure/attachment/12345967/ojvm-11-28-06.tar.gz

• The complet API for the Oracle Domain index was completed, but the solution for the operator contains outside the where clause is not good.
• I will implement a singleton solution for the OJVMDirectory object when is used in read only mode, typically when user performs select operations against tables which have columns indexed with Lucene. This implementation will increase a lot the final performance because the index reader will be ready for each select operation. Obviously I will check if another user or thread makes a write operation on the index to reload the read-only singleton.
• The queue for storing the changes on the index is not implemented yet, I’ll add it in a short time.

2.0.0.1.0 (initial implementation, 22/Nov/06 03:45 PM)

https://issues.apache.org/jira/secure/attachment/12345516/ojvm.tar.gz

Doc Links

Previous / LDI Docs – Appendix D (Functions, operators and utilities)

Previous / LDI Docs – Appendix C (JUnit test suites explained)
Next / LDI Docs – Appendix E (Project Change Log)

C.1 DBTestCase base class

This is base class for most of the test suites includes. It provides a connection pool using OracleDataSource with a minimum of two ready to use connection and growing to 5, after this it will wait up to 20 seconds for free connection. This connection pool is created at the class constructor. Utility methods provided by this class, each method use is own SQLConnection, so they are autonomous transactions:

• createTable(), create a test table as follow, (T1 is a constant value defined as TABLE):
  

  create table T1 (
    f1 number primary key,
    f2 varchar2(200),
    f3 varchar2(200),
    f4 number);
  

• dropTable(), drop table created above.
• createIndex(), add a Lucene Domain Index to previous one created table as follow, (LogLevel,Analyzer,MergeFactor,ExtraCols and FormatCols are customizable at class level, after index creation MergeFactor is reduced to 2):
  

  create index IT1 on T1(f2)
    indextype is lucene.LuceneIndex
      parameters('LogLevel:WARNING;
        Analyzer:org.apache.lucene.analysis.StopAnalyzer;
        MergeFactor:500;
        ExtraCols:F1;
        FormatCols:F1(0000)')
  

• dropIndex(), drop previous one index.
• int insertRows(int startIndex, int endIndex), insert a set of rows at above table with F1 column varying from startIndex to endIndex. F2 column is an english text representation of F1, F4 is F1*10 and F3 is an english text representation of F1*10. Return a number of rows inserted. If there are problems such as primary key violation it rollback the transaction.
• int deleteRows(int startIndex, int endIndex), delete a set of rows where F1 between startIndex and endIndex. Return a number of rows deleted. If there are problems rollback the transaction. Note that deleting rows automatically update Lucene Index.
• int updateRows(int startIndex, int endIndex), update F2 column with his own value to fire ODCI update method on each row between startIndex and endIndex.
  Return a number of rows updated.
• findRows(int n), find rows which F2 match again a text representation of n using lcontains operator. It only test for a result having 0 or more rows.
• long syncIndex(), perform a sync operation at Lucene Domain Index applying pending changes (inserts, updates). If there are errors, usually caused by another transaction having an exclusive lock in a row being indexed, it rollback the operation. Next successful sync will apply pending changes of failed operations. Return a long value with the amount of milliseconds spent during sync.
• long optimizeIndex(), perform an optimize operation at Lucene Domain Index merging segments in a new one. If there are errors, usually caused by another transaction having an exclusive lock on the index, it rollback the operation. Return a long value with the amount of milliseconds spent during optimize.

C.2 TestDBIndex

Simple test which create a table his index and performs insertions, sync, optimize and deletions, finally drop index and table. His output look like:

[junit] Testsuite: org.apache.lucene.index.TestDBIndex
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.836 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Inserted rows: 40 total char inserted: 415 avg text length: 10
[junit] Index synced: IT1 elapsed time: 265 ms.
[junit] Avg Sync time: 6
[junit] Index optimized: IT1 elapsed time: 40 ms.
[junit] Avg Optimize time: 1
[junit] Row deleted 40, from: 10 to: 49 elapsed time: 1303 ms. Avg time: 32 ms.
[junit] Index droped: IT1
[junit] Table droped: T1

C.3 TestDBIndexAddDoc

Performs several insertions and sync, starting with 10 rows, then 90 and so on, ending with 3.000 insertions using insertRow method of DBTestCase base class. After each batch of insertions calls to syncIndex method calculating average time of sync method for each row inserted. His output look like:

[junit] Testsuite: org.apache.lucene.index.TestDBIndexAddDoc
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 64.696 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Index synced: IT1 elapsed time: 126 ms.
[junit] Inserted rows: 10 total char inserted: 49 avg text length: 4
[junit] Index synced: IT1 elapsed time: 142 ms.
[junit] Avg Sync time: 14
[junit] Inserted rows: 90 total char inserted: 988 avg text length: 10
[junit] Index synced: IT1 elapsed time: 374 ms.
[junit] Avg Sync time: 4
[junit] Inserted rows: 400 total char inserted: 9201 avg text length: 23
[junit] Index synced: IT1 elapsed time: 1276 ms.
[junit] Avg Sync time: 3
[junit] Inserted rows: 500 total char inserted: 11726 avg text length: 23
[junit] Index synced: IT1 elapsed time: 1601 ms.
[junit] Avg Sync time: 3
[junit] Inserted rows: 1000 total char inserted: 35950 avg text length: 35
[junit] Index synced: IT1 elapsed time: 4675 ms.
[junit] Avg Sync time: 4
[junit] Inserted rows: 3000 total char inserted: 110851 avg text length: 36
[junit] Index synced: IT1 elapsed time: 25480 ms.
[junit] Avg Sync time: 8
[junit] Index droped: IT1
[junit] Table droped: T1

C.4 TestDBIndexDelDoc

At setup method this test case a create a table and fill it with 500 rows. Then performs deletions batch of 10, 90 and 400 rows each calculating average time for each row deleted. His output look like:

[junit] Testsuite: org.apache.lucene.index.TestDBIndexDelDoc
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 20.543 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Inserted rows: 500 total char inserted: 10238 avg text length: 20
[junit] Index synced: IT1 elapsed time: 1643 ms.
[junit] Row deleted 10, from: 1 to: 10 elapsed time: 356 ms. Avg time: 35 ms.
[junit] Row deleted 90, from: 11 to: 100 elapsed time: 2535 ms. Avg time: 28 ms.
[junit] Row deleted 400, from: 101 to: 500 elapsed time: 11526 ms. Avg time: 28 ms.
[junit] Index droped: IT1
[junit] Table droped: T1

C.5 TestDBIndexParallel

This is more complex test case to check concurrent access to Lucene Domain Index. To do this creates several threads, some for simulating batch insertions of 10 rows, others for simulating batch deletions of 10 rows, another for simulating batch updates of 10 rows and finally many threads searching for rows each 0.5 seconds. By default creates 3 threads for each kind of operations and each thread perform:

• 20 inserts
• 5 deletes
• 5 update
• 100 search

Each thread takes his own connection from the connection pool and do his job, if fastSync constant is true after each successful insert and update it calls to syncIndex method to update Lucene Index, if fastSync is false another thread is started performing sync index each 1 second. It end when all threads (inserts, deletes, updates) finish. Here some part of his output:

[junit] Testsuite: org.apache.lucene.index.TestDBIndexParallel
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 97.7 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] FastSync: true
[junit] Deleter 1 deleting at block 70
[junit] Updater 1 updating at block 70
[junit] Inserter 2 inserting at block 90
[junit] No Row deleted at: 70 to: 79 elapsed time: 131 ms.
[junit] No Row updated at: 70 to: 79 elapsed time: 12 ms.
[junit] Searcher 2 searching row 30
[junit] Searcher 1 searching row 77
[junit] Not Found rows with: thirty  elapsed time: 211 ms.
[junit] Not Found rows with: seventy-seven  elapsed time: 170 ms.
[junit] Inserted rows: 10 total char inserted: 115 avg text length: 11
[junit] Searcher 2 searching row 62
[junit] Searcher 0 searching row 63
[junit] Searcher 1 searching row 49
[junit] Not Found rows with: sixty-two  elapsed time: 64 ms.
[junit] Index synced: IT1 elapsed time: 283 ms.
[junit] Not Found rows with: sixty-three  elapsed time: 215 ms.
[junit] Searcher 2 searching row 74
[junit] Not Found rows with: seventy-four  elapsed time: 39 ms.
[junit] Not Found rows with: forty-nine  elapsed time: 137 ms.
[junit] Searcher 1 searching row 95
[junit] Searcher 2 searching row 46
[junit] Found rows with: ninety-five  elapsed time: 103 ms.
....
[junit] Updater 2 updating at block 20
[junit] No Row updated at: 20 to: 29 elapsed time: 3 ms.
[junit] Inserted rows: 10 total char inserted: 80 avg text length: 8
[junit] Searcher 0 searching row 97
[junit] Found rows with: ninety-seven  elapsed time: 60 ms.
[junit] Index synced: IT1 elapsed time: 147 ms.
.....
[junit] Searcher 2 searching row 39
[junit] Searcher 1 searching row 84
[junit] Not Found rows with: thirty-nine  elapsed time: 33 ms.
[junit] Not Found rows with: eighty-four  elapsed time: 38 ms.
[junit] Updater 0 updating at block 90
[junit] Row updated 10, from: 90 to: 99 elapsed time: 16 ms. Avg time: 1 ms.
[junit] Index synced: IT1 elapsed time: 162 ms.
......
[junit] Inserted rows: 10 total char inserted: 125 avg text length: 12
[junit] Searcher 0 searching row 57
[junit] Searcher 1 searching row 28
[junit] Deleter 1 deleting at block 80
[junit] Searcher 2 searching row 64
[junit] No Row deleted at: 80 to: 89 elapsed time: 58 ms.
[junit] Not Found rows with: twenty-eight  elapsed time: 112 ms.
[junit] Not Found rows with: fifty-seven  elapsed time: 155 ms.
[junit] Index synced: IT1 elapsed time: 242 ms.
[junit] Searcher 0 searching row 98
[junit] Found rows with: ninety-eight  elapsed time: 72 ms.
[junit] Not Found rows with: sixty-four  elapsed time: 175 ms.
[junit] Searcher 0 searching row 27
[junit] Not Found rows with: twenty-seven  elapsed time: 75 ms.
[junit] Searcher 1 searching row 5
[junit] Deleter 2 deleting at block 50
[junit] Searcher 2 searching row 84
[junit] Not Found rows with: eighty-four  elapsed time: 20 ms.
[junit] Updater 2 updating at block 10
[junit] No Row deleted at: 50 to: 59 elapsed time: 28 ms.
[junit] Row updated 10, from: 10 to: 19 elapsed time: 36 ms. Avg time: 3 ms.
[junit] Found rows with: five  elapsed time: 216 ms.
.................
[junit] Inserter 1 inserting at block 50
[junit] Found rows at: 50 position, ignoring insertions
[junit] Index droped: IT1
[junit] Table droped: T1

C.6 TestDBIndexSearchDoc

This test check some special features of lcontains operator such as in-line pagination, sort by and filter by expressions. First create a table with 200 rows and then query them, his output look like:

[junit] Testsuite: org.apache.lucene.index.TestDBIndexSearchDoc
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 14.001 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Inserted rows: 200 total char inserted: 3262 avg text length: 16
[junit] Index synced: IT1 elapsed time: 746 ms.
[junit] testFilterAll()
[junit] Excecution time: 129 ms.
[junit] 120 Score: 0.9606395 str: one hundred twenty
[junit] 119 Score: 0.25453204 str: one hundred nineteen
[junit] 118 Score: 0.25453204 str: one hundred eighteen
[junit] 117 Score: 0.25453204 str: one hundred seventeen
[junit] 116 Score: 0.25453204 str: one hundred sixteen
[junit] 115 Score: 0.25453204 str: one hundred fifteen
[junit] 114 Score: 0.25453204 str: one hundred fourteen
[junit] 113 Score: 0.25453204 str: one hundred thirteen
[junit] 112 Score: 0.25453204 str: one hundred twelve
[junit] 111 Score: 0.25453204 str: one hundred eleven
[junit] Index droped: IT1
[junit] Table droped: T1
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Inserted rows: 200 total char inserted: 3262 avg text length: 16
[junit] Index synced: IT1 elapsed time: 721 ms.
[junit] testFilterBy()
[junit] Excecution time: 162 ms.
[junit] 103 Score: 1.0 str: one hundred three
[junit] 120 Score: 0.9606395 str: one hundred twenty
[junit] 101 Score: 0.28600293 str: one hundred one
[junit] 100 Score: 0.27352643 str: one hundred
....
[junit] 115 Score: 0.25453204 str: one hundred fifteen
[junit] 116 Score: 0.25453204 str: one hundred sixteen
[junit] Index droped: IT1
[junit] Table droped: T1
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Inserted rows: 200 total char inserted: 3262 avg text length: 16
[junit] Index synced: IT1 elapsed time: 751 ms.
[junit] testFilterByOrderBy()
[junit] Excecution time: 138 ms.
[junit] 120 Score: 0.9606395 str: one hundred twenty
[junit] 119 Score: 0.25453204 str: one hundred nineteen
....
[junit] 103 Score: 1.0 str: one hundred three
[junit] 102 Score: 0.25453204 str: one hundred two
[junit] 101 Score: 0.28600293 str: one hundred one
[junit] 100 Score: 0.27352643 str: one hundred
[junit] Index droped: IT1
[junit] Table droped: T1
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Inserted rows: 200 total char inserted: 3262 avg text length: 16
[junit] Index synced: IT1 elapsed time: 761 ms.
[junit] testPagination()
[junit] Excecution time: 193 ms.
[junit] 117 Score: 0.03489425 str: one hundred seventeen
[junit] 118 Score: 0.03489425 str: one hundred eighteen
....
[junit] 132 Score: 0.03489425 str: one hundred thirty-two
[junit] 134 Score: 0.03489425 str: one hundred thirty-four
[junit] Index droped: IT1
[junit] Table droped: T1
[junit] Table created: T1
[junit] Index created: IT1
[junit] Index altered: IT1
[junit] Inserted rows: 200 total char inserted: 3262 avg text length: 16
[junit] Index synced: IT1 elapsed time: 743 ms.
[junit] testCountHits()
[junit] Excecution time: 53 ms.
[junit] Hits: 126
[junit] Index droped: IT1
[junit] Table droped: T1

C.7 TestQueryHits

This test is not autonomous because requires an additional step to run. Before run it create a table and his Lucene Index with:

create table test_source_big as (select * from all_source);
create index source_big_lidx on test_source_big(text)
  indextype is lucene.LuceneIndex
    parameters('AutoTuneMemory:true;
      MergeFactor:500;
      FormatCols:line(0000);
      ExtraCols:line "line"');

For 11g databases you can create a best optimize Lucene Index using some new Secure LOB features:

create index source_big_lidx on test_source_big(text)
  indextype is lucene.LuceneIndex
    parameters('FormatCols:line(0000);
      ExtraCols:line "line";
      Analyzer:org.apache.lucene.analysis.StopAnalyzer;
      MergeFactor:500;
      LobStorageParameters:PCTVERSION 0 ENABLE STORAGE IN ROW CHUNK 32768 CACHE READS FILESYSTEM_LIKE_LOGGING');

On 10g running it as SCOTT, TEST_SOURCE_BIG table will have 220731 rows using a typical installation based on database templates. Using above table two test checks performance with a query which returns 18387 hits, once call to LuceneDomainIndex.countHits function and another iterate over the result in pages of ten rows, typical scenario of web applications. His output look like:

[junit] Testsuite: org.apache.lucene.indexer.TestQueryHits
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 2.656 sec
[junit]
[junit] ------------- Standard Output ---------------
[junit] iteration from: 13775 to: 13785
[junit] Step time: 791 ms.
[junit] iteration from: 13785 to: 13795
[junit] Step time: 49 ms.
[junit] iteration from: 13795 to: 13805
[junit] Step time: 40 ms.
[junit] iteration from: 13805 to: 13815
[junit] Step time: 44 ms.
[junit] iteration from: 13815 to: 13825
[junit] Step time: 40 ms.
[junit] iteration from: 13825 to: 13835
[junit] Step time: 42 ms.
[junit] iteration from: 13835 to: 13845
[junit] Step time: 41 ms.
[junit] iteration from: 13845 to: 13855
[junit] Step time: 50 ms.
[junit] iteration from: 13855 to: 13865
[junit] Step time: 41 ms.
[junit] iteration from: 13865 to: 13875
[junit] Step time: 41 ms.
[junit] Elapsed time: 1877
[junit] Hits: 18387
[junit] Elapsed time: 564

Note that first iteration took more time because it includes parsing time and caching, also to simulate a real word web application an SQLConnection is take and returned to the pool on each iteration.

Doc Links

Previous / LDI Docs – Appendix B (Lucene Domain Index Storage)
Next / LDI Docs – Appendix D (Functions, operators and utilities)

OJVMDirectory class creates a set of Oracle objects to represent Lucene Inverted Index and Domain Index functionality. First it creates a table named IDX_NAME$T (IDX_NAME is your Lucene Domain Index used at create index DDL statement) with this structure:

Also have and index based on IDX_NAME$T.DELETED column to speedy up purge operations. To enqueue operation at the index it defines a DBMS_AQ Queue IDX_NAME$Q with his storage table IDX_NAME$QT. IDX_NAME$Q queue have payload defined as LUCENE_MSG_TYP object. This object type is defined as:

SYS.ODCIRIDLIST is an special structure defined by ODCI API to hold a list of rowid changed by an DML operation. OPERATION is one of insert, delete, update, rebuild, optimize, insert-ram or insert-disk reserved keyword. rebuild and optimize operations are used with SyncMode:OnLine to perform these tasks automatically using a background process. insert-ram and insert-disk are messages enqueued internally by LDI when ParallelDegree is enabled.
When using ParallelDegree grater than 1 the structure showed above is replicated for N parallel storages used when indexing, it means that, for example, with ParallelDegree:2 for an index name SOURCE_BIG_LIDX there will be two extra structures named SOURCE_BIG_LIDX$0{$Q|$QT|$T} and SOURCE_BIG_LIDX$1{$Q|$QT|$T}.

Doc Links

Previous / LDI Docs – Appendix A (Parameter reference and syntax)
Next / LDI Docs – Appendix C (JUnit test suites explained)

Lucene Domain Index accept several parameters which can be passed using create index or alter index DDL commands. This parameters are divided into four categories, Index Writer, Analyzer, User Data Store and General parameters.

A.1 Lucene Index Writer parameters

This section covers Lucene Index Writer parameters for more information about this parameter see Lucene docs and Wiki.

A.1.1 MergeFactor

Determines how often segment indices are merged by addDocument(). If you are creating a new index over a table with thousands of rows a value of 100 to 500 is good value.

A.1.2 MaxBufferedDocs

Determines the minimal number of documents required before the buffered in-memory documents are merged and a new Segment is created. This value can cause an out of memory exception you provide a value larger than user space available. A typical SGA configuration can accept values of 4000 or 5000 depending how big are your rows being indexed. If you are not sure of how megabytes can consume your rows you can use AutoTuneMemory:true parameter which is a default value, so you choose true MaxBufferedDocs will be ignored and Lucene Domain Index will try to uso 90% of Oracle Java Pool Size value.

A.1.3 MaxMergeDocs

Determines the largest number of documents ever merged by addDocument().

A.1.4 MaxBufferedDeleteTerms

Determines the minimal number of delete terms required before the buffered in-memory delete terms are applied and flushed.

A.1.5 UseCompoundFile

Setting to turn on usage of a compound file. When on, multiple files for each segment are merged into a single file once the segment creation is finished. This is done regardless of what directory is in use. By default Lucene Domain Index do not use compound file format because its not affected by max open file descriptors.

A.1.6 MaxFieldLength

Determines the maximum number of char indexed for any column of this index, default value is 10000.

A.1.7 AutoTuneMemory

AutoTuneMemory:true (default) overrides MaxBufferedDocs parameter, it defines dynamically MaxBufferedDocs based on how much memory is reported by OracleRuntime.getJavaPoolSize() method.
After each document is added to the index it calls to writer.ramSizeInBytes() and test that is not over a 50% of the ram free.
This parameter works in most of the common cases, but you can get a Java out of memory error in multiuser environments because Java Pool Size is common parameter for all the sessions. If you get an exception during index creation time set AutoTuneMemory:false and adjust MaxBufferedDocs to a value which not raise an out of memory exception.

A.2 Analyzer parameters

An Analyzer builds TokenStreams, which analyze text. It thus represents a policy for extracting index terms from text.

Typical implementations first build a Tokenizer, which breaks the stream of characters from the Reader into raw Tokens. One or more TokenFilters may then be applied to the output of the Tokenizer.
Analyzer, PerFieldAnalyzer or Stemmer parameter affects indexing and query expressions, so if you want to change this parameter on a exists index you to must rebuild it, the priority of these three parameters is first check for the Stemmer if its not present check for PerFieldAnalyzer if its not present checks for Analyzer parameter, finally if none of them are defined will use SimpleAnalyzer.

A.2.1 Analyzer

This parameter is fully qualified Java class name which extends org.apache.lucene.analysis.Analyzer. For example:

• BrazilianAnalyzer
• ChineseAnalyzer
• CJKAnalyzer
• CzechAnalyzer
• DutchAnalyzer
• FrenchAnalyzer
• GermanAnalyzer
• GreekAnalyzer
• KeywordAnalyzer
• PatternAnalyzer
• RussianAnalyzer
• SimpleAnalyzer
• StandardAnalyzer
• StopAnalyzer
• ThaiAnalyzer
• WhitespaceAnalyzer

See Lucene Java Docs for more details. A default analyzer is SimpleAnalyzer.

A.2.2 Stemmer

Stemmer is another kind of analyzer which divides words, stop words and another term related object based on an specific language. Stemmer parameter use Snowball Analyzer, possible values for Stemmer parameter using Lucene 2.2.0 distribution are:

• Danish
• Dutch
• English
• Finnish
• French
• German
• German2
• Italian
• Kp
• Lovins
• Norwegian
• Porter
• Portuguese
• Russian
• Spanish
• Swedish

Stemmer parameter override Analyzer parameter.

A.2.3 PerFieldAnalyzer

PerFieldAnalyzer is a wrapper of other analyzers which provides an independent analyzer for each column being indexed, see PerFieldAnalyzerWrapper class in Lucene documentation. Each column could have his own analyzer which extends org.apache.lucene.analysis.Analyzer. If a column is not in the list StandardAnalyzer will be used as default. For example:

create table t1 (f1 VARCHAR2(10), f2 XMLType);
insert into t1 values ('1', XMLType('<emp id="1"><name>ravi</name></emp>'));
insert into t1 values ('3', XMLType('<emp id="3"><name>murthy</name></emp>'));

create index it1 on t1(f2) indextype is lucene.LuceneIndex
 parameters('IncludeMasterColumn:false;
 ExtraCols:F1,extractValue(F2,''/emp/name/text()'') "name",extractValue(F2,''/emp/@id'') "id";
 FormatCols:F1(000),id(00)');

alter index it1 rebuild
 parameters('PerFieldAnalyzer:F1(org.apache.lucene.analysis.KeywordAnalyzer),id(org.apache.lucene.analysis.KeywordAnalyzer)');

In the above example four columns are being indexed by Lucene Domain Index rowid (added by default) using KeywordAnalyzer, F1 and id (added by ExtraCols parameter) using KeywordAnalyzer too, and finally name which is not included into PerFieldParameter and then using StandardAnalyzer.

A.3 User Data Store parameters

Lucene Domain Index implements a User Data Store functionality, this functionality provides many parameters to control which column will be included into a Lucene Document which is inserted into the index.
and First three parameters are used to choose which columns will added to the index in addition to the master column. Oracle Domain Index are bound to a single column, this is a limitation with Oracle 10g version. To avoid this problem passing ExtraCols, ExtraTabsWhereCondition you can easily build a set of new column from the master table and others. Basically a select DML statement is built using these parameters. To clarify this Lucene Domain Index will performs a query like:

– Full table scan (create index statement):
SELECT rowid, MasterTable.MasterColumn, ExtraCols
FROM MasterTable,ExtraTabs
where WhereCondition;

– Find a particular rowid (insert,update operations):
SELECT MasterTable.MasterColumn, ExtraCols
FROM MasterTable,ExtraTabs
where MasterTable.rowid=:rowid AND WhereCondition;

Text in italic are injected by Lucene Domain Index and text in bold are user defined.

A.3.1 ExtraCols

A coma separated list of columns of the Master table of table being indexed or the tables defined into ExtraTabs parameter. Note that if you don’t define columns alias column name are capitalized by default on Oracle databases. For example ‘ExtraCols:F2 “f2″,T2.F3 “f3″‘ note that you can omit master table name if there is no collisions

A.3.2 ExtraTabs

A coma separated list of table name and alias for this tables. For example ‘ExtraTabs:T2 aliasT2,T3 aliasT3′. Note that ODCI API only will detect changes at index master column, to notify changes based on ExtraCols list you need to attach triggers, see section examples above for more detail.

A.3.3 WhereCondition

An SQL where condition used to join index’s master table with ExtraTabs tables. For example: ‘WhereCondition:T1.f1=T2.f2(+) AND T1.F1=aliasT3.f3′. Be careful to produce a correct join condition to guaranty single row result; multiple or zero row result based on the master table values are not allowed.

Note: Up to Lucene Domain Index 2.9.0, if you use a WhereCondition which have an OR operator put this where condition enclosed with () because the precedence of the OR over the AND operator makes that some queries returns more rows that the correct behavior, for example instead of:
WhereCondition:T1.F1=’AA’ OR T1.F1=’BB’
put:
WhereCondition:(T1.F1=’AA’ OR T1.F1=’BB’)
this workaround fix some problems when working in OnLine mode. Starting with 2.9.1 version this extra () are not required.

A.3.4 UserDataStore

This is a fully Java Class name which implements org.apache.lucene.indexer.UserDataStore interface, you can create your own Data Store class implementing this interface. By default Lucene Domain Index provides an implementation which covers most of the typical scenarios, this class is org.apache.lucene.indexer.DefaultUserDataStore and use FormatCols parameter to create Lucene Fields.

A.3.5 FormatCols

A coma separated list of column(format) strings interpreted by User Data Store class to control how an specific database column will be transformed in a Lucene Field. For example you can choose padding, un-tokenized values and so on.
Supported formats by Default Data Store class are:

• Number padding for numeric columns using java.text.DecimalFormat class syntax, default is 0000000000.
• Date rounding for timestamp and date columns using org.apache.lucene.document.DateTools, default is day.
• Character left padding for VARCHAR2 or CHAR columns using org.apache.lucene.util.StringUtils class (leftPad method), default is no left char padding. Any char can be used for left padding.
• XPath expression for XMLType columns, this XPath string will be passed to XMLType.extract(“format”,”") method, the result of the XPath extraction will be a new XMLType object over getStringVal() will executed. If you want to perform more user defined XMLType to Field extraction extend DefaultUserDataStore class or use virtual column indexing.
• For columns of type VARCHAR2 or CHAR you can use an special string NOT_ANALYZED or NOT_ANALYZED_STORED as format which tell to Default User Data Store class that this column will be indexed but un-tokenized, this is useful with columns which will be used for sorting.

A.3.6 LockMasterTable

When table indexer is getting the row which will be indexed it can use either FOR UPDATE NOWAIT SQL construction or not, setting this parameter to true cause that the row is acquired with a lock.

A.4 General parameters

This set of parameters are Lucene Domain Index specific parameters.

A.4.1 SyncMode

SyncMode tells to Lucene Domain Index which strategy is used to update the index. SyncMode:Deferred (default) left to the application when the index is synced either by calling LuceneDomainIndex.sync procedure after a set of changes pending or by DBMS_SCHEDULER process at an specific time. With SyncMode:Deferred update and insert operations are queued using DBMS_AQ package. Delete operations are never enqueued because require an update on Lucene Index to not return rowid of deleted rows.
SyncMode:OnLine is implemented by using DBMS_AQ PLSQL callback, so immediately after a commit operation which involves insert or update rows a parallel process dbms_j* is automatically started by DBMS_AQ package to applied pending changes. SyncMode:OnLine should be reserved for index which update, insert or delete operations are much lower than select, AQ callbacks can not handle very well exceptions during sync time, for example when a row being index is locked by another session, so some changes can be lost with this scenario.

A.4.2 Updater, Searcher

Lucene Domain Index can be configured to start several parallel shared process which do reader and writer operations on LDI storage on behalf of the user connected session, you can configure multiple searcher process selected randomly using the syntax host1@port1,host2@port2 and one updater process using similar syntax. By default these parameters are defined with the value local which means not using parallel shared servers. Two parallel server are configured and started during database startup process, a searcher process listen at SYS_CONTEXT(‘USERENV’,'SERVER_HOST’)@1099 which usually is localhost@1099 and the updater process at localhost@1098, you can register multiples searcher/updater processes editing the properties db.searcher.job/db.searcher.port,db.updater.job/db.updater.port at build.xml file and calling to the targets create-searcher-job and create-updater-job respectively.
Updater and Searcher processes can be stopped, started using Ant’s targets disable-jobs and enable-jobs.

A.4.3 LobStorageParameters

Lucene Domain Index uses a BLOB column named “data” for storing Lucene Inverted index files. You can control any LOB storage parameter with this parameter during index creation time, his default value is ‘LobStorageParameters:PCTVERSION 0 ENABLE STORAGE IN ROW CACHE READS NOLOGGING’ for 11g databases you can use a better optimize storage by using newest Secure LOB parameter, for example: ‘LobStorageParameters:PCTVERSION 0 ENABLE STORAGE IN ROW CHUNK 32768 CACHE READS FILESYSTEM_LIKE_LOGGING’

A.4.4 LogLevel

Lucene Domain Index uses JDK Java Util Logging package, LogLevel parameter is any of the string defined by Level.parse() method, for example: LogLevel:ALL. By default logging level is defined to WARNING.
Lucene Domain Index uses:

• SEVERE for non recoverable error conditions
• FINER for debugging purpose such as ODCI API arguments
• INFO for checking index operations such as value being indexed
• WARNING for error messages which are reported as ERROR through ODCI API
• CONFIG to see user parameters changed by users

Logging information is sent by default to Oracle .trc files, but you can redirect this output using dbms_java.set_output procedure for example.
If you are not sure which field and how these fields are added to the index change LogLevel to INFO and check for lines starting with: “INFO: Document<”
exiting and throwing methods does not print messages also with log level defined to ALL. This is because logging level used by these methods are controlled by ConsoleHandler level.
To get these methods work copy logging.properties file from your JAVA_HOME/jre/lib to ORACLE_HOME/javavm/lib directory and edit the line which includes level property:

# Limit the message that are printed on the console to INFO and above.
java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

Then shutdown and startup your Oracle database.

A.4.5 CachedRowIdSize

CachedRowIdSize is used by an LRU cached used to maintain the association between Lucene Doc ID and a particular Oracle ROWID. For very big table using an array to store this association can consume a lot of SGA RAM, starting with Lucene Domain Index 2.9.0.1.0 only 10.000 ROWID are stored in this cache, tables with high frequency of updates can use this LRU small due every caused that LRU is completed flushed, but tables with low frequency of updates/deletes can get a lot of performance improvement by using larger LRU cached size.

A.4.6 BatchCount, IndexOnRam and ParallelDegree

These three parameters control parallel index operations (inserts) when OnLine mode is enabled, ParalellelDegree defines how many slave index storage will be created to hold temporary parallel index operations when news rows are inserted or the index is created or rebuild. During index creation or rebuild time BatchCount defines how many rows will processed in batch and parallel with another set of rows. IndexOnRam defines when the new set of rows is indexed in a temporary index in RAM or disk, prior to Lucene Domain Index 2.9.2.1.0 a batch of new rows where processes in temporary index stored in disk, using IndexOnRam:true tells to Lucene Domain Index that the new rows will be indexed in RAM and finally merged into the main index stored in disk.

A.5 Query parameters

This set of parameters which affects QueryParser and search functionality.

A.5.1 DefaultColumn

DefaultColumn defines which columns is used as default column in QueryParser syntax, if this parameter is not set master column of the index is used, this name is a Lucene Field name. Here an example:

create index pages_lidx_all on pages p (value(p))
  indextype is Lucene.LuceneIndex
  parameters('PopulateIndex:false;
    DefaultColumn:text;
    SyncMode:Deferred;
    LogLevel:WARNING;
    Analyzer:org.apache.lucene.analysis.SpanishWikipediaAnalyzer;
    ExtraCols:extractValue(object_value,''/page/title'') "title", extractValue(object_value,''/page/revision/comment'') "comment", extract(object_value,''/page/revision/text/text()'') "text", extractValue(object_value,''/page/revision/timestamp'') "revisionDate";
    FormatCols:revisionDate(day);
    IncludeMasterColumn:false;
    LobStorageParameters:PCTVERSION 0 ENABLE STORAGE IN ROW CHUNK 32768 CACHE READS FILESYSTEM_LIKE_LOGGING');

Note the correlation between DefaultColumn and ExtraCols. ExtraCols defines a Lucene Field named “text” with a value calculated by the SQL expression extract(object_value,”/page/revision/text/text()”), then you can use a Lucene Field text as default Field in QueryParser syntax.

A.5.2 DefaultOperator

DefaultOperator defines which Boolean operator is used in QueryParser syntax, if this parameter is not set OR operator is his default value.

A.5.3 NormalizeScore

NormalizeScore is used during Lucene Index scan to know if they need to track the maximum score, the maximum score then used to normalize the result of lscore() operator to return only values between 0 to 1. If you don’t need a normalized range of the score you can avoid this computation and your query will be fast. Note that a not normalized score not implied that the document are not in order of relevance.

A.5.4 PreserveDocIdOrder

PreserveDocIdOrder is an internal parameter which is used by Lucene in some kind of operator, if you don’t need that result preserve Lucene Doc ID in order rather than the relevance, you can put this value to false (default) and some operator will be fast.

A.5.5 RewriteScore and SimilarityMethod

RewriteScore (true or false) and SimilarityMethod (fully class name) are used when query using wildcard operator (*) these parameters produces better recall values, for example:

create table t1 (f1 number primary key, f2 varchar2(2000), f3 number(5,3));
insert into t1 values (1, 'Cefaleias', 1);
insert into t1 values (2, 'Cefaleia', 1);
insert into t1 values (3, 'Cefaleia em salva', 0.625);
insert into t1 values (4, 'Cefaleias de tensão', 0.625);
insert into t1 values (5, 'Cefaleias / enxaquecas', 0.625);
insert into t1 values (6, 'Desproporção céfalo-pélvica', 0.5);
insert into t1 values (7, 'Deformidade por redução cefálica congénita', 15.87);
insert into t1 values (8, 'Intoxicação por antibióticos do grupo das cefalosporinas', 0.5);
commit;

create index it1 on t1(f2)
  indextype is lucene.luceneindex 
  parameters('LogLevel:ALL;
    Analyzer:org.apache.lucene.analysis.PortugueseAnalyzer;
    FormatCols:F3(00.000);
    ExtraCols:F3;
    RewriteScore:true;
    SimilarityMethod:org.apache.lucene.search.WildcardSimilarity');

select /*+ DOMAIN_INDEX_SORT */ lscore(1) f1, f2 from t1
  where lcontains(f2, 'cefa cefa*',1) > 0

 F1 F2 
 1 Cefaleias 
 1 Cefaleia 
 0.625 Cefaleia em salva 
 0.625 Cefaleias de tensão 
 0.625 Cefaleias / enxaquecas 
 0.5 Desproporção céfalo-pélvica 
 0.5 Deformidade por redução cefálica congénita 
 0.5 Intoxicação por antibióticos do grupo das cefalosporinas 
 8 rows selected

alter index it1 
  parameters('LogLevel:ALL;
    SimilarityMethod:org.apache.lucene.search.DefaultSimilarity');

select /*+ DOMAIN_INDEX_SORT */ lscore(1) f1,f2 from t1
  where lcontains(f2, 'cefa cefa*',1) > 0

 F1 F2 
 0.3539437353610992431640625 Intoxicação por antibióticos do grupo das cefalosporinas 
 0.12431289255619049072265625 Cefaleias 
 0.12431289255619049072265625 Cefaleia 
 0.077695555984973907470703125 Cefaleia em salva 
 0.077695555984973907470703125 Cefaleias de tensão 
 0.077695555984973907470703125 Cefaleias / enxaquecas 
 0.062156446278095245361328125 Desproporção céfalo-pélvica 
 0.062156446278095245361328125 Deformidade por redução cefálica congénita 
 8 rows selected

alter index it1
  parameters('LogLevel:ALL;
    RewriteScore:false');

select /*+ DOMAIN_INDEX_SORT */ lscore(1) f1, f2 from t1
  where lcontains(f2, 'cefa cefa*',1) > 0

 F1 F2 0.15442870557308197021484375 Cefaleias
 0.15442870557308197021484375 Cefaleia
 0.15442870557308197021484375 Cefaleia em salva 
 0.15442870557308197021484375 Cefaleias de tensão 
 0.15442870557308197021484375 Cefaleias / enxaquecas 
 0.15442870557308197021484375 Desproporção céfalo-pélvica 
 0.15442870557308197021484375 Deformidade por redução cefálica congénita 
 0.15442870557308197021484375 Intoxicação por antibióticos do grupo das cefalosporinas
 8 rows selected

A.6 Highlight parameters

This set of parameters which affects lhighlight, phighlight and rhighlight functionality.

A.6.1 Formatter

Formatter defines a valid class name which implements Lucene Interface Formatter and with a constructor with no arguments, default value org.apache.lucene.search.highlight.SimpleHTMLFormatter.

A.6.2 MaxNumFragmentsRequired

MaxNumFragmentsRequired defines a number of text fragments returned by Highlight function, default value is 4.

A.6.3 FragmentSize

FragmentSize defines the size of each fragment returned in characters of each fragment, default value is 100.

A.6.4 FragmentSeparator

FragmentSeparator defines a String used as fragment separator, default value is “…”. Note that you can not use “;” or “:” as fragment separator because are used as parameter and value delimiters into alter index … parameters(..) DDL statement.

Doc Links

Previous / LDI Docs – 4 Locking and Performance
Next / LDI Docs – Appendix B (Lucene Domain Index Storage)

4.1 Locking used by Lucene Domain Index

1. X = Row exclusive lock at the row being inserted, RX = Table row exclusive lock – if index parameter LockMasterTable=true.
2. X = Row exclusive lock at the row being indexed, RS = Table row share lock. A select … for update no wait is performed at all rows being added to Lucene Index.
3. X/T this is a writer lock semaphore of Lucene Index and provide serialize write operations, the write lock is performed using lock table $SCHEMA.IDX$T. X/RX is performed at many rows of this table because Lucene creates and deletes many files.
4. To perform massive dequeue operations at DBMS AQ queue Sync scan this queue with DBMS_AQ.BLOCKED option.

4.2 Performance tips

4.2.1 Index Writer parameters

Lucene Index Writer class uses several parameters to control his index structure. Lucene Domain Index pass to Index Writer several parameters such as MergeFactor, MaxBufferedDocs among others. As best practice if you want to index thousands of rows you can override default Lucene parameters for other which speed up indexing time. With create index or alter index rebuild you can set MergeFactor to 100 and MaxBufferedDocs to 4000. This parameters increase index performance but then DML operations at the base table will batch small set of rows, so after DDL commands change MergeFactor to 2 and MaxBufferedDocs to 100. A good place to start knowing these parameters behavior is the Wiki page Improving Indexing Speed.

4.2.2 Auto Tune Memory functionality

Lucene Domain Index have a parameter called AutoTuneMemory a true value means that for Index Writer operations it will try to use up to 50% of the Java Pool Size configured at the Oracle SGA to adjust how many documents are buffered (MaxBufferedDocs) before call IndexWritter.flush().
With AutoTuneMemory:true MergeFactor,MaxBufferedDocs,MaxMergeDocs are not required, its calculated using free RAM at the SGA, but you has to set MergeFactor. Due Java Pool Size is global parameter the rule is not valid if you want to create many index with parallel connexions, two connections will try to use 50% of the SGA, so one of them will ran out of memory.

4.2.3 Keep Index on RAM

OJVMDirectory replaces Lucene file system storage by a table storage with BLOBs. For every Lucene Domain Index created there is a new table which stores every Lucene file as a row with a BLOB column, see section 6 for more detail, using similar strategy as Oracle Text you can keep this table in RAM. Unlike Oracle Text which uses multiples tables for storing the inverted index, Lucene Domain Index use one table, execute this DDL command to keep Lucene Index on RAM:

create index source_small_lidx on test_source_small(text)
indextype is lucene.LuceneIndex parameters(
  'FormatCols:line(0000); ExtraCols:line "line"; Analyzer:org.apache.lucene.analysis.StopAnalyzer; MergeFactor:500');
alter index source_small_lidx parameters('MergeFactor:100');
alter table source_small_lidx$t storage (buffer_pool keep) modify lob (data) (storage (buffer_pool keep));

During Index creation use AutoTuneMemory:true (default value). Finally change OJVMDirectory storage table and LOB to keep them in RAM. Be sure that your SGA has a enough RAM to keep it. To know how big your index you can query the table:

SQL> select sum(file_size) from source_small_lidx$t where deleted='N';
SUM(FILE_SIZE)
--------------
        147444

Finally as Tom Kyte say, tkprof, tkprof, …. . You can see Lucene Domain Index IO operations with an “alter session set events ’10046 trace name context forever, level 12′; then you can find operations at Lucene Domain Index table SCHEMA.IDX_NAME$T. Using TKPROF information you can alter table and lob storage parameters manually.

4.2.4 Compare your execution plan

To be sure that your Lucene Domain Index is properly used compare your executions plans and try to avoid non necessary filter by or sort order by predicates by using in-line sort or multiples field Query Parser conditions. Here examples of sorting using emails table created in section 3.1.4:

SQL> explain plan for
  2  SELECT subject FROM emails where lcontains(bodytext,'security',1)>0
  3  order by subject ASC;

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------
Plan hash value: 1542204867
Id   Operation                    Name           Rows  Bytes  Cost (%CPU)  Time
0    SELECT STATEMENT                            1     4016   3 (34)       00:00:01
1    SORT ORDER BY                               1     4016   3 (34)       00:00:01
2    TABLE ACCESS BY INDEX ROWID  EMAILS         1     4016   2  (0)       00:00:01
* 3  DOMAIN INDEX                 EMAILBODYTEXT

Predicate Information (identified by operation id):
-----------------------------------------------------------------------------------
3 - access("LUCENE"."LCONTAINS"("BODYTEXT",'security',1)>0)

Above execution plan tells that you are using Lucene Domain Index but you can get a better optimizer plan by using lcontains sort:

SQL> explain plan for
  2  SELECT /*+ DOMAIN_INDEX_SORT */ subject FROM emails
  3  where lcontains(bodytext,'security','subject:ASC',1)>0;

PLAN_TABLE_OUTPUT
-----------------------------------------------------------------------------------
Plan hash value: 1450245214
Id   Operation                    Name           Rows  Bytes  Cost (%CPU)  Time
0    SELECT STATEMENT                            1     4016   2 (0)        00:00:01
1    TABLE ACCESS BY INDEX ROWID  EMAILS         1     4016   2 (0)        00:00:01
* 2  DOMAIN INDEX                 EMAILBODYTEXT

Predicate Information (identified by operation id):
-----------------------------------------------------------------------------------
2 - access("LUCENE"."LCONTAINS"("BODYTEXT",'security','subject:ASC',1)>0)

Here we have a better optimizer plan and lower cost.

4.2.5 Filtering and sorting at index level

This functionality only available on Oracle 11g is valid for Lucene Domain Index in 10g/11g databases and also for standard edition version. The performance improvement is done when most of the rows can be filtered and sorted at index level, to do that you have to push the value of the column(s) involved in filter by or order by at index level during index creation. The syntax differs from Oracle Text but the performance improve is similar, let see an example:

-- Oracle Text 11g syntax
create index source_big_idx on test_source_big(text) indextype is ctxsys.context
  filter by line
  order by line;
-- Lucene Domain Index syntax
create index source_big_lidx on test_source_big(text) indextype is lucene.luceneindex parameters(
  'PerFieldAnalyzer:line(org.apache.lucene.analysis.KeywordAnalyzer),TEXT(org.apache.lucene.analysis.SimpleAnalyzer);
  FormatCols:line(0000);
  ExtraCols:line "line"');

Note that in both cases we choose line as the filter/order by column. Now let see the execution plan and auto trace for an equivalent query using 11g syntax and Lucene Domain Index syntax.









To see the real impact on the performance of using a filter by at index level let see the time involved of an equivalent query:

select count(line) from test_source_big
  where lcontains(text,'varchar2 AND line:[2600 TO 9000]')>0;
  2
COUNT(LINE)
-----------
587
Elapsed: 00:00:00.03

select count(line) from test_source_big
  where lcontains(text,'varchar2')>0 and line>=2600;
  2
COUNT(LINE)
-----------
587
Elapsed: 00:00:00.89

The point here is that into the first example Lucene Domain Index performs the two operations:

1. find all the rows which contains the word varchar2
2. filter the rows that only have line in a range 2600 to 9000

returning only the rows (587) that match both sentence, for the second example the RDBMS:

1. look for the rowid that contains the word varchar2 (19963),
2. visit above rows looking for the value of the column line and filter all that are >=2600

the difference between the rows visited by the RDBMS is the difference on the performance.

4.2.6 OnLine mode, ParallelDegree and IndexOnRam

Starting with 2.9.2.1.1 and 3.0.1.1.0 version inserts are performed in parallel if ParallelDegree is greater than 1 and SyncMode:OnLine, in that case an AQ slave process will create temporary Lucene index adding the rows being indexed, this index is created in RAM if IndexOnRam:true or in disk otherwise, once the index contains the batch of rows added the temporary slave index is merged with the master storage. This speed up massive index additions such as index rebuild, index creation or insert .. into .. select .. from DML operations. Parallel index operations are important in servers which have multiples cores or RAC installations because Oracle AQ starts parallel process doing the job, an SQL trace of WikiPedia dump indexing shows that mostly of time is involved in the scan of table loading the data which is indexed, so using parallel indexing increase the throughput of Lucene Domain Index in multi-core chips.
Oracle 11g AQ implementation checks how many milliseconds are consumed by an AQ Callback, so choosing a BatchCount too bigger causes that no other slave process is started by Oracle AQ engine, the experience with BatchCount values in a range starting with 100 to 500 is good value to guaranty a correct parallel operation. Following screenshot shows multiple AQ process indexing WikiPedia dump:



4.2.7 Parallel Shared Slave Index Scan (available since 3.0.2.1.0)

Starting with 3.0.2.1.0 version a parallel shared slave index scan process is started automatically when your DB start, this process is accepting RMI connections from the other Oracle internal process, that is, once a connection from a client is accepted by the RDBMS a dedicated or shared server is started to performs the SQL operations, this process have the internal OJVM associated to execute the LDI operations. Previous to 3.0.2.1.0 each OJVM have his internal Lucene structures to query the inverted index, due each OJVM is isolated from another if two concurrent connections executes the same SQL operations on LDI each process will load the inverted index structure on RAM and performs the Hit collector operation. The new process now do the same operations but is shared by all the OJVM process which connect to him using RMI, the consequence is that only the first query will load the inverted index structure on RAM, next queries coming from the same or different OJVM process will re-use these structures on RAM. This architectural change increase a lot the Cache Hit rate decreasing the time to performs lcontains operations and reducing the RAM usage by LDI. The screenshot below shows this concept in action:



There are several process named oracletest (LOCAL=NO) these are dedicated process associated to each client connections, there is one process named ora_j009_test, this process is the slave shared server which is performing the search operations on Lucene Index on behalf of the others. The parallel slave search process is started and stopped automatically by two instance triggers registered at SYS schema, these triggers are after startup and before shutdown events. If you disable these triggers Lucene Domain Index back to previous functionality which means every OJVM process has his own memory structures and do the index scan without dispatching RMI calls.

4.3 Know caveats

1. Lucene Domain Index uses Java Util Logging API and RMI to connect to the search process, it means that a grant is required to create and operate on LDI, for example:
   

   grant LUCENEUSER to scott;
   

2. SyncMode:OnLine should be reserved only for index which a number of update/insert/delete operation are too small compared to select operations, because each message process requires almost open an IndexWriter/IndexReader on the associated Lucene Index by a background process, except for bulk collect operation or “insert into … select … from” which are processed in batch off 150 rows. Tables with many insert/update operations by seconds should use LuceneDomainIndex.sync(idx) procedure called by DBMS_JOB periodically or by the application.
3. Syntax for Inline pagination is only supported at the beginning of the Query, it means that if you want to perform pagination using lcontains() query syntax it must start with “rownum:[n TO m] AND” note that this syntax is case sensitive. Also this extraction is performed by splitting the query by position and does not take into account grouping operator, so this query “rownum:[1 TO 10] AND word1 OR word2″ will be passed to Lucene’s Query Parser as “word1 OR word2″ which is not semantically the original one if you look to the precedence operator. We can try to modify Query Parser class in a future to solve this semantic issues.
4. Columns name are case sensitive in ExtraCols and FormatCols parameters using traditional SQL behavior, it means that for this DDL index creation:
   

   create index it1 on t1(f2)
     indextype is lucene.LuceneIndex
       parameters('Stemmer:English;FormatCols:F2(zzzzzzzzzzzzzzz),F3(00.00);ExtraCols:F3');
   

   
   You can use ExtraCols with f3 or F3 but FormatCols should be F3 because f3 is returned by the SQL select operation as F3 during the table full scan, also Lucene Index will have a document with a Field F3 instead of f3. If you want to use f3 as is you can re-write DDL index creation with:
   

   create index it1 on t1(f2)
     indextype is lucene.LuceneIndex
       parameters('Stemmer:English;FormatCols:F2(zzzzzzzzzzzzzzz),f3(00.00);ExtraCols:F3 "f3"');
   

   
   With this sentence Lucene will create documents with two field F2 and f3, F2 is uppercase because is the master column of the index and his passed as “F2″ by ODCI API but, due is the default Field of the query, you can omit his name at lcontains syntax, F3 now is lowercase and will be indexed as a Field “f3″.
5. Index parameters are pre-cached in memory for faster response. Due isolation behaviour of Oracle JVM sessions, if you call to alter index or re-create a new one in another session you need to close all SQL session that have a pre-load index parameter storage.
   Calling to LuceneDomainIndex.getParameter(‘owner.index_name’,'parameter_name’) you can see the values of any parameter passed to the ODCI API either by calling create index or alter index. Otherwise you can call to LuceneDomainIndex.refreshParameterCache stored procedure.
6. If you re-install Lucene Domain Index without deleting existing indexes you can manually drop resources associated to and old index. For example:
   

   drop index source_big_lidx force;
   Index dropped.
   select table_name from tabs;
   
   TABLE_NAME
   ------------------------------
   DEPT
   EMP
   BONUS
   SALGRADE
   SOURCE_BIG_LIDX$QT
   DR$SOURCE_BIG_IDX$I
   DR$SOURCE_BIG_IDX$R
   SOURCE_BIG_LIDX$T
   TEST_SOURCE_BIG
   DR$SOURCE_BIG_IDX$N
   DR$SOURCE_BIG_IDX$K
   
   11 rows selected.
   
   drop table SOURCE_BIG_LIDX$T;
   Table dropped.
   
   conn / as sysdba
   connected.
   
   exec DBMS_AQADM.STOP_QUEUE ('SCOTT.SOURCE_BIG_LIDX$Q');
   PL/SQL procedure successfully completed.
   
   exec DBMS_AQADM.DROP_QUEUE ('SCOTT.SOURCE_BIG_LIDX$Q');
   PL/SQL procedure successfully completed.
   
   exec DBMS_AQADM.DROP_QUEUE_TABLE(queue_table  => 'SCOTT.SOURCE_BIG_LIDX$QT', force=>true);
   PL/SQL procedure successfully completed.
   
   exit
   

   
   Note that “drop index … force” will de-register Lucene Domain Index from Oracle’s system views, then Lucene Domain Index storage’s table is manually dropped, finally connected as SYS Lucene Domain Index AQ’s table is dropped.
7. Oracle 11g have a know bug “6445561 – ORA-00600 [26599] [62] DUE TO INCORRECT PERSISTENCE OF BY INVOKER PIN” please apply patch number p6445561_111060_LINUX.zip available at Metalink, this bug affects select count(*) with a large results.
8. Up to Lucene Domain Index 2.9.0 there is known problem with the WhereCondition parameter using OR SQL operator, see section A.3.3 to see the workaround.

Doc Links

Previous / LDI Docs – 3 Procedures, Functions, Operators and Examples
Next / LDI Docs – Appendix A (Parameter reference and syntax)


Filed under: Documentation ]]> http://ludoix.wordpress.com/2011/03/11/ldi-docs-4-locking-and-performance/feed/ 0 ludoix ddgw7sjp_575ghw55ps9_b ddgw7sjp_576cdhjshc7_b ddgw7sjp_577g89chnfv_b ddgw7sjp_578cbx9m5gq_b ddgw7sjp_574gbjpjhcb_b ddgw7sjp_581gtrxtcdb_b http://ludoix.wordpress.com/2011/03/11/ldi-docs-3-procedures-functions-operators-and-examples/ http://ludoix.wordpress.com/2011/03/11/ldi-docs-3-procedures-functions-operators-and-examples/#comments Fri, 11 Mar 2011 17:34:49 +0000 ludoix http://ludoix.wordpress.com/?p=47 ]]> 3. Procedures, Functions, Operators and Examples

Before you start to work on through the examples below, do grant the LUCENEUSER role to any dedicated Oracle user/schema, who has become selected to run the Lucene Domain Index (LDI). Remember, that you must not run a index within user/schema LUCENE. For example:

 -- connected as sysdba
  grant LUCENEUSER to scott;

3.1 Create a Lucene Domain Index

3.1.1 Single column index

The first example creates a domain index on table t1, column f2 using Lucene’s SimpleAnalyzer along with the Analyzer parameter. After execution, a new index, T1.IT1, and two new LDI-tables, the index storage table, IT1$T, and the index queue table (AQ), IT1$QT, will have been added to the user’s schema. Because of the generated objects, can no Lucene Domain Index name be longer than 21 characters! This is due the secondary generation of the Oracle AQ table name.

  create table t1 (
    f1 number,
    f2 varchar2(200),
    f3 varchar2(200),
    f4 number unique);

  create index it1 on t1(f2) indextype is lucene.LuceneIndex
    parameters('Analyzer:org.apache.lucene.analysis.SimpleAnalyzer');

Another simple example may employ a stemmer instead of an analyzer. A stemmer is kind of a language specific analyzer and behaves comparable to Lucene’s StandardAnalyzer. The Lucene stemming approach used here, the SnowballAnalyzer, is based on the Snowball code stack (snowball.tartarus.org). The employment of a stemmer requires setting the Stemmer parameter. Since a stemmer is somehow an extended analyzer, any Stemmer parameter in a parameter list will override any Analyzer parameter given as well. See Appendix A of the LDI docs for more information about analyzing and stemming.

  create index it1 on t1(f2) indextype is lucene.LuceneIndex
    parameters('Stemmer:English');

3.1.2 Multiple columns

Extending the previous example, you can also index additional columns of the same base table, in sum called compound columns. This requires the usage of the ExtraCols parameter, handing in the column identifier, that can be modified with an alias name, iff required. Note that the parameter list provided for a create or alter index needs to be a string without any line breaks and without any dispensible whitespace characters. Here is the code:

  create index it1 on t1(f2) indextype is lucene.LuceneIndex
    parameters('Stemmer:English;ExtraCols:F1 "f1"');

Note that employing an alias name also modifies the syntax of field searches with lcontains. That is, executing the following search will not result in any rows returned:

  select * from t1 where lcontains(f2, 'F1:xyz', 1) > 0;

You have stick to the exact alias field name, with an appropriate character case instead:

  select * from t1 where lcontains(f2, 'f1:xyz', 1) > 0;

However, the column actually indexed can be searched using an all uppercase field identifier or just no field identifier at all:

  select * from t1 where lcontains(f2, 'xyz', 1) > 0;
  select * from t1 where lcontains(f2, 'F2:xyz', 1) > 0;

gennff: the following statement and example may be misleading because oracle is sometimes smart enough to detect blind updates and does not fire any update trigger under that circumstance. this technique needs to be checked and proved in detail.

Because Oracle ODCI API will not detect changes on other columns than the indexed master column, you need to create a trigger that will fire on any update on columns in the ExtraCols list to rewrite the master column. Such a way, any changes on f1 will also force to change f2. ODCI will notify LDI that an specific rowid was updated and LDI, being based on the specific parameter definition of this index, will update the index tables to reflect the changes. Here is an example:

  CREATE OR REPLACE TRIGGER L$IT1 BEFORE UPDATE OF f1 ON t1 FOR EACH ROW
  BEGIN
    :new.f2 := :new.f2;
  END;
  /

3.1.3 Multiple tables

LDI also supports indexing multiple columns over multiple tables, that can be joined in a natural form. All to be done ist to define a list of tables with the ExtraTabs parameter as well as to specify a where condition with the WhereCondition parameter. The master table alias L$MT is automatically introduced by LDI and can be applied without any preparations. This alias is furthermore important, for example, to create complex joins with xml tables, that imply the use of the existsNode or extracValue operators. This functionality has been added starting with the 2.9.0.1.0 release of LDI. Here is an example:

  create table t2 (
    f4 number primary key,
    f5 VARCHAR2(200));

  create table t1 (
    f1 number,
    f2 VARCHAR2(4000),
    f3 number,
    CONSTRAINT t1_t2_fk FOREIGN KEY (f3)
      REFERENCES t2(f4) ON DELETE cascade);

  create index it1 on t1(f3) indextype is lucene.LuceneIndex
    parameters('ExtraCols:L$MT.f2 "f2",t2.f5 "f5";ExtraTabs:t2;WhereCondition:L$MT.f3=t2.f4');

Note that the tables t1 and t2 are joined directly by a foreign key, such that t1 could be considered a child table of t2. Using this set of parameters, when the ODCI API detects a change on the index master column t1.f3, a select like this will be executed:

  select L$MT.f3, L$MT.f2 "f2", t2.f5 "f5"
  from t1 L$MT, t2
    where L$MT.rowid=? and L$MT.f3=t2.f4;

gennff: the following example may be wrong in terms of the t2 trigger. that is, the it1 index will never see any inserts and/or deletes on table t2 and will therefore hardly stay in sync with the actual table data. in general, i don’t think it is an easy to follow approach to use an index on a child table in this example (t1 is a child of t2 since it references it by fk).

Keeping the LDI index in sync with the changes upon any columns (and tables) defined with ExtraCols (and ExtraTabs) parameters is no more complex than the compound columns on a single table example above. It just requires a combination of two or more triggers, according to the number of additional tables in the index definition. The additional trigger determines all rowids at the master table who have a reference to the row in change and then employs the LuceneDomainIndex.enqueueChange procedure to just notify LDI about the changes, while sys.ODCIRidList is a special ODCI structure to hold a group of rowids. Here you go:

  CREATE OR REPLACE TRIGGER L$IT1 BEFORE UPDATE OF f2 ON t1 FOR EACH ROW
  BEGIN
    :new.f3 := :new.f3;
  END;
  /
  CREATE OR REPLACE TRIGGER LT$IT1 AFTER UPDATE OF f5 ON t2 FOR EACH ROW
    DECLARE
        ridlist sys.ODCIRidList;
    BEGIN
      SELECT ROWID BULK COLLECT INTO ridlist FROM T1 WHERE F3=:NEW.f4;
      LuceneDomainIndex.enqueueChange(USER || '.IT1', ridlist, 'update');
    END;
  /

3.1.4 Padding and formatting

Lucene Domain Index can be customized with a parameter named UserDataStore, that defines which class is responsible for creating Lucene documents. A Lucene document is a list of fields for each column indexed, plus an extra field named rowid, being stored compressed and untokenized. By default UserDataStore is defined to be implemented by org.apache.lucene.indexer.DefaultUserDataStore.

Default UserDataStore supports left padding for NUMBER or FLOAT columns as well as left character padding for VARCHAR2 or CHAR columns. To define padding, use the FormatCols parameter within the create or alter index DDL command. Here is an example that automatically pads all F2 column values to 15 characters and formats all F3 column values to comply to 00.00:

  create table t1 (
    f1 number primary key,
    f2 varchar2(200),
    f3 number(4,2))
  ORGANIZATION INDEX;

  insert into t1 values (1, 'ravi', 3.46);
  insert into t1 values (3, 'murthy', 15.87);
  commit;

  create index it1 on t1(f2) indextype is lucene.LuceneIndex
    parameters('Stemmer:English;ExtraCols:F3;FormatCols:F2(zzzzzzzzzzzzzzz),F3(00.00)');

then these rows will be indexed as Lucene documents like:

FINE: Document<stored,indexed<rowid:*BAGAGfsCwQL+> indexed,tokenized<F2:zzzzzzzzzzzravi> indexed<F3:03,46>>
FINE: Document<stored,indexed<rowid:*BAGAGfsCwQT+> indexed,tokenized<F2:zzzzzzzzzmurthy> indexed<F3:15,87>>

For columns based on Oracle XMLType, FormatCols parameter can be used to define an XPath expression which controls a subset of XML nodes to be indexed.

  create table t1 (
    f1 VARCHAR2(10),
    f2 XMLType);
  insert into t1 values ('1', XMLType('<emp id="1"><name>ravi</name></emp>'));
  insert into t1 values ('3', XMLType('<emp id="3"><name>murthy</name></emp>'));
  commit;

  create index it1 on t1(f1) indextype is lucene.LuceneIndex
    parameters('Analyzer:org.apache.lucene.analysis.WhitespaceAnalyzer;ExtraCols:F2;FormatCols:F1(000),F2(/emp/name)');

Inspecting the trace output of the create index statement, above rows will be indexed as:

FINE: Document<stored,indexed<rowid:AAAWqqAAGAAABodAAA> indexed,tokenized<F1:001> indexed,tokenized<F2:ravi >>
FINE: Document<stored,indexed<rowid:AAAWqqAAGAAABodAAB> indexed,tokenized<F1:003> indexed,tokenized<F2:murthy >>

Columns of type VARCHAR2, CHAR and CLOB allow for some special formatting options, namely NOT_ANALYZED, NOT_ANALYZED_STORED, ANALYZED_WITH_OFFSETS, ANALYZED_WITH_POSITIONS and ANALYZED_WITH_POSITIONS_OFFSETS. These options control the way UserDataStore chooses to analyze/index, store and vectorize the field values. For example, a field value that only holds one token per document may be analyzed/indexed without tokenization and also not be stored with the index to be employed as a sort field:

gennff: the example should also include sorting for numbers, probably using the NumericField class as of Lucene >= 2.9 .

  create table emails (
    emailFrom VARCHAR2(256),
    emailTo VARCHAR2(256),
    subject VARCHAR2(4000),
    emailDate DATE,
    bodyText CLOB);

  INSERT INTO EMAILS (EMAILFROM, EMAILTO, SUBJECT, EMAILDATE, BODYTEXT)
    VALUES ('EMAILFROM', 'EMAILTO', 'SUBJECT', sysdate, 'BODYTEXT');
  commit;

  create index emailbodyText on emails(bodyText) indextype is lucene.LuceneIndex
    parameters('Analyzer:org.apache.lucene.analysis.StopAnalyzer;ExtraCols:emailDate "emailDate",subject "subject",emailFrom "emailFrom",emailTo "emailTo";FormatCols:subject(NOT_ANALYZED),emailFrom(NOT_ANALYZED),emailTo(NOT_ANALYZED);LogLevel:ALL');

Then the trace output for the row processed looks like this, where the fields subject, emailFrom and emailTo are only indexed but not tokenized by the analyzer (obviously, emailDate is also only indexed because the analyzer is smart enough to deduce this approach from the datatype of the value):

FINE: Document<stored,indexed<rowid:AAAWvbAAGAAABrrAAA> indexed,tokenized<BODYTEXT:BODYTEXT> indexed<emailDate:20110617> indexed<subject:SUBJECT> indexed<emailFrom:EMAILFROM> indexed<emailTo:EMAILTO>>

Another example, using the ANALYZED_WITH_VECTORS option of the FormatCols parameter prints:

FINE: Document<stored,indexed<rowid:AAAW5IAAGAAABqOAAG> stored,indexed,tokenized,termVector<STR1:20.12.2010> stored,indexed,tokenized,termVector<STR2:20.12.2010>>

Iff you go and read the Lucene documentation, you won’t find the options introduced above. They have been set up by LDI to define a mapping to Lucene’s concept of field, that is:

• analysis/indexing: (how will the value be searchable via the inverted index
• storage: is there any need to preserve the original value with the index as well, for display purposes probably and
• term vectorization: used for highlightning, categrization etc.

Here is the mapping table where the first row depicts the default value, i.e. when no FormatCols parameter is given:

LDI format option	Lucene field analysis/indexing (Field.Index.*)	Lucene field storage (Field.Store.*)	Lucene field term vectorization (TermVector.*)
default	ANALYZED	NO	NO
ANALYZED_\ STORED	ANALYZED	YES	NO
ANALYZED_\ WITH_VECTORS	ANALYZED	YES	YES
ANALYZED_\ WITH_OFFSETS	ANALYZED	YES	WITH_OFFSETS
ANALYZED_\ WITH_POSITIONS	ANALYZED	YES	WITH_POSITIONS
ANALYZED_\ WITH_POSITIONS_\ OFFSETS	ANALYZED	YES	WITH_POSITIONS_\ OFFSETS
NOT_ANALYZED	NOT_ANALYZED	NO	NO
NOT_ANALYZED_\ STORED	NOT_ANALYZED	YES	NO

The following table depicts mappings that regard the *_NORMS options to describe index time boosting of terms. It is just an extension of the preceding table, this time incorporation the *_NORMS option.

LDI format option	Lucene field analysis/indexing (Field.Index.*)	Lucene field storage (Field.Store.*)	Lucene field term vectorization (TermVector.*)
ANALYZED_\ NO_NORMS	ANALYZED_\ NO_NORMS	NO	NO
ANALYZED_\ NO_NORMS_\ STORED	ANALYZED_\ NO_NORMS	YES	NO
ANALYZED_\ NO_NORMS_\ WITH_VECTORS	ANALYZED_\ NO_NORMS	YES	YES
ANALYZED_\ NO_NORMS_\ WITH_OFFSETS	ANALYZED_\ NO_NORMS	YES	WITH_OFFSETS
ANALYZED_\ NO_NORMS_\ WITH_POSITIONS	ANALYZED_\ NO_NORMS	YES	WITH_POSITIONS
ANALYZED_\ NO_NORMS_\ WITH_POSITIONS_\ OFFSETS	ANALYZED_\ NO_NORMS	YES	WITH_POSITIONS_\ OFFSETS
NOT_ANALYZED_\ NO_NORMS	NOT_ANALYZED_\ NO_NORMS	NO	NO
NOT_ANALYZED_\ NO_NORMS_\ STORED	NOT_ANALYZED_\ NO_NORMS	YES	NO

Concluding, the following table introduces the possibility to directly index numeric values as numbers, dates and times as of Lucene’s release >= 2.9 NumericField class. Since LDI currently only supports sorting by int and float (see § 3.4.4), the basic types long and double are not yet accepted. Specifying NUMERIC_DATETIME should be accompanied with an according FormatCols format to set the desired datetime resolution from year down to second, e.g. FormatCols:revisionDate(day).

LDI format option	Lucene field analysis/indexing (Field.Index.*)	Lucene field storage (Field.Store.*)	Lucene field term vectorization (TermVector.*)
NUMERIC_INT	n/a (NOT_ANALYZED)	NO	n/a (NO)
NUMERIC_FLOAT	n/a (NOT_ANALYZED)	NO	n/a (NO)
NUMERIC_DATETIME	n/a (NOT_ANALYZED)	NO	n/a (NO)

3.1.5 Functional columns

Since some ExtraCols parameter list will just be added to the selected columns of the base table for indexing, one can also define additional or specialized fields for a Lucene index. Actually, any valid SQL expression in a select section is allowed here. For example, using the above table definition, one can introduce another Lucene field named id, that can even be a subject to formatting with the same statement. Again, be careful to use the same character case when writing down database column and Lucene field identifiers:

  create index it1 on t1(f1) indextype is lucene.LuceneIndex
    parameters('Analyzer:org.apache.lucene.analysis.standard.StandardAnalyzer;ExtraCols:F2,extractValue(F2,''/emp/@id'') "id";FormatCols:F1(000),F2(/emp/name),id(00)');

The LDI indexer will use a statement roughly comparable to the following:

  select f1, F2, extractValue(F2,''/emp/@id'') "id" from t1;

Appropriate selects may look like this:

  select * from t1 where lcontains(f1, '001') > 0;
  select * from t1 where lcontains(f1, 'id:01') > 0;
  select * from t1 where lcontains(f1, 'F2:ravi') > 0;

3.1.6 Online synchronization

If you put SyncMode:OnLine, in contrast to SyncMode:Defered, in the parameter list of your create index statement, LDI will set up a PLSQL AQ Callback to the indexes queue it1$q in it1$qt, see above, that immediately synchronizes all changes of insert or update statements with the Lucene index structure in the background. LDI will enqueue and process affected rowids of the master table in batches of BatchCount rows, where the default batch size is 115. For example:

  create index pages_lidx_all on pages p (value(p)) indextype is Lucene.LuceneIndex
    parameters('SyncMode:OnLine;LogLevel:WARNING;Stemmer:Spanish;ExtraCols:extractValue(object_value,''/page/title'') "title",extractValue(object_value,''/page/revision/comment'') "comment",extract(object_value,''/page/revision/text/text()'') "text",extractValue(object_value,''/page/revision/timestamp'') "revisionDate";IncludeMasterColumn:false;LobStorageParameters:PCTVERSION 0 ENABLE STORAGE IN ROW CHUNK 32768 CACHE READS FILESYSTEM_LIKE_LOGGING');

3.1.7 Populate Index

Using PopulateIndex:false with an index create statement will build an empty Lucene index structure that is ready for use but will of course not return any results on a select statement. You may execute an alter index rebuild statement afterwards to actually populate the index. Here is an example:

  create index it1 on t1(f2) indextype is lucene.LuceneIndex
    parameters('PopulateIndex:false;LogLevel:ALL;IncludeMasterColumn:false;ExtraCols:F1,extractValue(F2,''/emp/name/text()'') "name",extractValue(F2,''/emp/@id'') "id";FormatCols:F1(000),id(00)');
  -- At this point the index is set up but not populated, so no rows will be returned
  select lscore(1),f2 from t1 where lcontains(f2, 'name:ravi',1) > 0;
  -- Populate the index
  alter index it1 rebuild parameters('Analyzer:org.apache.lucene.analysis.WhitespaceAnalyzer');
  -- Now, upon a query, rows will be returned (iff matched)
  select lscore(1), f2 from t1 where lcontains(f2, 'name:ravi',1) > 0;

3.1.8 Parallel Index Operations

Starting with Lucene Domain Index 2.9.1.1.0, you can enable parallel operations with the ParallelDegree parameter that may be explicitely set to 0, the default, or to a range of 2 to 9. Parallel operations is implemented by using multiple UserDataStore segements and is most useful on multi core chip boxes or in a RAC environment. Up to now, only insert statemets upon the base table is parallelized and the index must be configured in SyncMode:OnLine mode. This is an example of an index that enables parallel inserts:

  create index source_big_lidx on test_source_big(text)
    indextype is lucene.luceneindex
parameters('BatchCount:1000;ParallelDegree:4;SyncMode:OnLine;LogLevel:INFO;AutoTuneMemory:true;PerFieldAnalyzer:line(org.apache.lucene.analysis.KeywordAnalyzer),TEXT(org.apache.lucene.analysis.SimpleAnalyzer);FormatCols:line(0000);ExtraCols:line "line"');

After execution of the statement, ten new tables will be visible on the user’s schema: SOURCE_BIG_LIDX$T (master index storage) and SOURCE_BIG_LIDX$[0..3]$T (slave index storages) as well as SOURCE_BIG_LIDX$QT (master index queue) and SOURCE_BIG_LIDX$[0..3]$QT (slave index queues). A sequence SOURCE_BIG_LIDX$S is also created and serves generating the numbers 0 to 3.

The parallel implementation will enqueue batches of BatchCount:1000 rows in the master queue SOURCE_BIG_LIDX$Q of the index. Then the PLSQL AQ Callback, which is enabled for this queue by SyncMode:OnLine will dequeue each batch and re-enqueue in the slaves queues SOURCE_BIG_LIDX$[0..3]$Q. As a result, Oracle AQ will execute multiple AQ server processes that you can see as multiple ora_j00x_sid processes on a *U*X box.

With Oracle 11g we saw that that AQ may not always start another slave process if one callback is getting a lot of (too much) CPU usage. Experience shows, that a BatchCount parameter setting around 250 always leaves enough machine ressources for other slaves processes to be started successfully and setting parallelizing to real work.

gennff: the following section is quite hard to understand and should be rewritten, pointing out the several keywords in a more declarative way.

Lucene Domain Index 2.9.2.1.1+ and 3.0.1.1.0+ also includes a new parameter IndexOnRam, default true, that executes indexing in RAM, using the Lucene RAMDirectory implementation. RAMDirectory is around 40% faster than indexing a similar batch by means of the slave storages and parallel index operations, as introduced before. Obviously each slave process will consume more RAM compared to the same batch of rows using a disk storage based on OJVMDirectory. The rule of thumb is to enable LogLevel:INFO and check for a certain BatchCount value, what time it takes to index a new batch on RAM and what time it takes to merge the slave directory with the main directory storage. Here is an example:

  INFO: .addDocToIdx - start indexing on SCOTT.SOURCE_BIG_LIDX numRows= 500
  INFO: .addDocToIdx - indexing done SCOTT.SOURCE_BIG_LIDX elapsedTime: 200 ms.
  INFO: .addDocToIdx - addIndexesNoOptimize merge done SCOTT.SOURCE_BIG_LIDX elapsedTime: 317 ms.

Choosing a greater BatchCount value here will promote an indexing time similar or greater than the merge time and will, in sum, decrease the overall processing time, for example:

  INFO: .addDocToIdx - start indexing on SCOTT.SOURCE_BIG_LIDX numRows= 700
  INFO: .addDocToIdx - indexing done SCOTT.SOURCE_BIG_LIDX elapsedTime: 312 ms.
  INFO: .addDocToIdx - addIndexesNoOptimize merge done SCOTT.SOURCE_BIG_LIDX elapsedTime: 74 ms.

Parallel indexing is are also used for insert .. into .. select .. from DML operations, however, the BatchCount parameter is not used in this case because Oracle automatically chooses the number of rows that are inserted in a batch, usually around 115 rows.

A Lucene Domain Index working in SyncMode:OnLine and using a ParallelDegree greater than 1 will be populated in parallel using RAMDirectory or OJVMDirectory slave storage depending on IndexOnRam parameter. Note that indexing is always done in parallel with other indexing operation because it doesn’t require a write lock on the master indexing.

3.2 Alter index

The Lucene Domain Index alter index command can be used to change any parameter after index creation time. LDI parameters are a simple list of name:value pairs being stored into LDI’s OJVMDirectory storage. If you want to remove any parameter from the index definition, do prepend the parameter name with a "~". Here are some examples of alter index.

Change the Lucene index writer parameter MaxBufferedDocs to 500 and disable auto tuning of indexing memory:

  alter index it1
    parameters('MaxBufferedDocs:500;AutoTuneMemory:false');

Similar to the previous one example but enabling online synchronization:

  alter index it1
    parameters('MaxBufferedDocs:500;AutoTuneMemory:false;SyncMode:OnLine');

This disables online synchronization from the above example. You can get a similar effect by setting SyncMode:Deferred, which is the default value for SyncMode, to overwrite any previous SyncMode setting.

  alter index it1 parameters('~SyncMode:OnLine');

3.2 Alter index rebuild

The alter index rebuild statement rebuilds an index from scratch. This is useful when a LDI instance is damaged, corrupted or you need to change some parameter setting that affects field value preprocessing and indexing. An example is the Lucene Analyzer parameter.

3.2.1 Manual

The following example shows how to change Lucene index analyzer. If you change your index analyzer it is necessary to rebuild the complete index because you should not query an index with an analyzer that is different from the index time.

  alter index it1 rebuild
    parameters('Analyzer:org.apache.lucene.analysis.StopAnalyzer;MaxBufferedDocs:500;AutoTuneMemory:false);

3.2.2 On Line

gennff: it is not comprehensible to the reader what an oracle index online rebuild has in common with a ldi online sync.

Alter index rebuild will not return up to the complete operation is finished. Rebuild On Line is a functionality for Oracle Index available in enterprise edition databases, but with a little trick you can rebuild Lucene Domain Index On Line too. If you are working with SyncMode:Deferred you need to change to SyncMode:OnLine, then you can rebuild the index by using:

  alter index it1 rebuild
    parameters('SyncMode:OnLine;MergeFactor:100;BatchCount:1000');
  commit; -- notify change to AQ Callback

Rebuild command enqueues batchs of 1000 rowids of the master table (it1) for addition to Lucene Index structure then Lucene Domain Index AQ Callback will process these messages using background database process and automatically commit changes when it finish.

3.3 Drop

Dropping a LDI instance is no way different from dropping any other index in an Oracle database. Under the covers, this operation also drops the index’ storage table, IT1$T for the above example, and the index’ AQ IT1$Q with its storage IT1$QT. If the index is configured with SyncMode:OnLine, the PL/SQL AQ Callback is disabled first.

  drop index it1;

If something goes wrong during index drop command you can add a force at the end of the command. That will clear any system views from any stale references to the index.

  drop index it1 force;

3.4 Querying

Lucene Domain Index introduces a new SQL operator named lcontains() with its ancillary operators lscore() and lhighlight() (see below). The functionality of lcontains() and lscore() is comparable to the Oracle Text operators contains() and score().

3.4.1 Simple columns

Based on the xml indexing by FormatCols example from § 3.1.4, some simple statement employs contains() in the where clause and lscore() with the select list. In that order, the operators serve a parameterized text query and a normalized hit scoring. The first parameter to lcontains() denotes the column on which the Lucene Domain Index resides, the second one is the actual string (of tokens) to be searched. Both parameters are mandatory. The third parameter of lcontains() as well as the first parameter of lscore() is a correlation id that establishes a connection between a dedicated lcontains() and a dedicated lscore() in some SQL statement. That is, every call to Lucene by means of lcontains() delivers its own scoring! Note that lcontains() must always return a value > 0 to some consumer (a boolean expression here) to identify a successful match.

Since the Lucene index resides on column f1, this column is called the master column of the index and makes up the default search field of a query expression (see the Lucene Query Parser syntax). That is, leaving out any field qualifier in front of a query expression term, executes a search on column f1.

  select lscore(1) as sc, f1 from t1 where lcontains(f1, '001', 1) > 0;

    SC|F1
------|---
 1.000|1

The approach is different when searching an ExtraCols column, here the search string has to be prefixed by “F2:”.

  select lscore(1) as sc, f1 from t1 where lcontains(f1, 'F2:ravi', 1) > 0;

    SC|F2
------|----------------------------------------
 1.000|<emp id="1"><name>ravi</name></emp>

3.4.2 Multiple columns

Lucene query parser syntax provides a rich query language that comprises logical operators, term modifiers, grouping and stuff. You can apply any of them to each column indexed as long as the resulting expression conforms to the query language. Here is a some more complex example, using the xml indexing by FormatCols example from § 3.1.5 . Note that first row matches against the extra column F2:ravi and the functional column id:01, the second row matches with F1 equal to 003 (where some qualifier its not necessary for F1 because it is the master column of the index).

  select f1, lscore(1) sc, extractValue(f2, '/emp/@id') id from t1
    where lcontains(f1, '003 OR (F2:(ravi OR ravie) AND id:01)', 1) > 0;

  F1 |    SC|ID
  ---|------|---
  1  |  .577|1
  3  |  .206|3

3.4.3 Pagination

The lcontains() operator has an extension to the Lucene query parser syntax that includes in-line pagination information for the Lucene Domain Index result set. You can select a specific window (pagination) of your sorted query results, note the DOMAIN_INDEX_SORT optimizer hint and see next section, by injecting a query qarser like range term inside the query expression. For example (from § 3.1.5):

  select /*+ DOMAIN_INDEX_SORT */
      f1, lscore(1) sc, extractValue(f2, '/emp/@id') id from t1
    where lcontains(f1, 'rownum:[2 TO 2] AND (003 OR (F2:(ravi OR ravie) AND id:01))', 1) > 0
  order by lscore(1) desc;

  F1 |    SC|ID
  ---|------|---
  3  |  .206|3

Lucene Domain Index implementation automatically extracts pagination information rownum:[n TO m] AND from the beginning of the query expression and only returns the required subset of n rowids to the Oracle optimizer. This extension provides a lot of performance gain by eliminating the outer statement of Oracle’s Top-N syntax that, in a worst case, collects all resulting rowids to filter the result set window. Because inline pagination is an home brew extension to the standard Lucene query parser syntax, there are some home brew rules also:

• The rownum:[n TO m] AND term must start the query expression and, as is, we simply use positioned string matching of the rownum .. AND keywords to extract the start and the stop index of the window. Such a way it is your responsibility to provide a well formatted term.
• Pagination information is concatenated to the actual query expression using the AND boolean operator. However, this operator does not have any sense concerning the grouping logic of terms in a search expression, it is a placeholder, like rownum. For example, rownum:[n TO m] AND xx OR bb will be evaluated as ((rownum:[n TO m] AND xx) OR bb) but of course only seached as (xx OR bb).

Although all the pagination functionality looks really easy to use and fast executing yould should always keep the following in mind. Query result sets may change over time, they may change in value, in lenght and in order. For example, a select like the one above may return a completely different result set iff some new row has been added to the base table. The row will, iff matched, also move the page window around. A common pifall is that you inspect some search results on page 123, then a new row comes in and matches right at the top, and all of a sudden when you switch to page 123+1, the last result from page 123 reappears as the first result from page 123+1.

3.4.4 Sort

Pure Lucene provides sorting over the result of a particular query, Lucene Domain Index goes further and provides sorting by using an extra argument to the lcontains() operator (see § 3.4.1). The sort parameter syntax is a coma separated string of field[:ORDER[:TYPE] values, where the fields being included in the sorting spec should be NOT_ANALYZED or NOT_ANALYZED_STORED (see the FormatCols parameter in § 3.1.4). The ORDER can be set to ASC or DESC, default value is ASC. The TYPE key can be string, float or int, starting with Lucene 2.9.0 the default value is string.

Note that if you are using lcontains() to sort anything within the index, you have to add the DOMAIN_INDEX_SORT optimizer hint. This hint tells the Oracle optimizer that the order of the rows will be dictated by that Lucene Domain Index. Also note that the usage of lscore() in conjunction with a non-scored sort does not make any sense at all and will, due to the score computation on the index engine, only produce an overhead to the execution time of some query.

Here are some examples of sorted queries against the emails table created in § 3.1.4 and on that data:

  INSERT INTO EMAILS (EMAILFROM, EMAILTO, SUBJECT, EMAILDATE, BODYTEXT)
    VALUES ('arthur@schop.de', 'friedrich@nietz.de', 'Pessimismus', sysdate-12, 'Denn alles Streben entspringt aus Mangel');
  INSERT INTO EMAILS (EMAILFROM, EMAILTO, SUBJECT, EMAILDATE, BODYTEXT)
    VALUES ('friedrich@nietz.de', 'arthur@schop.de', 'Deine Philosophie', sysdate-7, 'wie beim Eintritt in den Hochwald');
  INSERT INTO EMAILS (EMAILFROM, EMAILTO, SUBJECT, EMAILDATE, BODYTEXT)
    VALUES ('irgendwer@schop.de', 'arthur@schop.de', 'Metaphysik ist pessimistisch', sysdate-3, 'Die Welt als Wille und Vorstellung');
  INSERT INTO EMAILS (EMAILFROM, EMAILTO, SUBJECT, EMAILDATE, BODYTEXT)
    VALUES ('friedrich@nietz.de', 'arthur@schop.de', 'Der neue Pessimismus', sysdate, 'Vitalismus ist doch schicker!');
  commit;

SELECT /*+ DOMAIN_INDEX_SORT */ subject, emailfrom as src, emaildate ts
FROM emails
  where lcontains(bodytext, 'streben || wille || ruhe', 'subject', 1) > 0;

SUBJECT                       |SRC                           |TS
------------------------------|------------------------------|--------
Metaphysik ist pessimistisch  |irgendwer@schop.de            |18.06.11
Metaphysik ist pessimistisch  |arthur@schop.de               |18.06.11
Metaphysik und Pessimismus    |arthur@schop.de               |09.06.11

SELECT /*+ DOMAIN_INDEX_SORT */ subject, emailfrom as src, emaildate ts
FROM emails
 where lcontains(bodytext, 'streben || wille || ruhe', 'subject:DESC', 1) > 0;

SUBJECT                       |SRC                           |TS
------------------------------|------------------------------|--------
Metaphysik und Pessimismus    |arthur@schop.de               |09.06.11
Metaphysik ist pessimistisch  |irgendwer@schop.de            |18.06.11
Metaphysik ist pessimistisch  |arthur@schop.de               |18.06.11

SELECT /*+ DOMAIN_INDEX_SORT */ subject, emailfrom as src, emaildate ts
FROM emails
  where lcontains(bodytext, 'streben || wille || ruhe', 'subject:DESC,emailFrom', 1) > 0;

SUBJECT                       |SRC                           |TS
------------------------------|------------------------------|--------
Metaphysik und Pessimismus    |arthur@schop.de               |09.06.11
Metaphysik ist pessimistisch  |arthur@schop.de               |18.06.11
Metaphysik ist pessimistisch  |irgendwer@schop.de            |18.06.11

SELECT /*+ DOMAIN_INDEX_SORT */ subject, emailfrom as src, emaildate ts
FROM emails
  where lcontains(bodytext, 'streben || wille || ruhe', 'subject:DESC,emailFrom:DESC', 1) > 0;

SUBJECT                       |SRC                           |TS
------------------------------|------------------------------|--------
Metaphysik und Pessimismus    |arthur@schop.de               |09.06.11
Metaphysik ist pessimistisch  |irgendwer@schop.de            |18.06.11
Metaphysik ist pessimistisch  |arthur@schop.de               |18.06.11

Again, be very careful to not change the case of any identifier or add redundant whitespace with the sort specification, Lucene Domain Index is very sloppy as a query parser. For example, writing emailFrom as emailfrom (no capital F) changes the semantics of the result set without notice:

SELECT /*+ DOMAIN_INDEX_SORT */ subject, emailfrom as src, emaildate ts
FROM emails
 where lcontains(bodytext, 'streben || wille || ruhe', 'emailFrom:DESC', 1) > 0;

SUBJECT                       |SRC                           |TS
------------------------------|------------------------------|--------
Metaphysik ist pessimistisch  |irgendwer@schop.de            |18.06.11
Metaphysik und Pessimismus    |arthur@schop.de               |09.06.11
Metaphysik ist pessimistisch  |arthur@schop.de               |18.06.11

SELECT /*+ DOMAIN_INDEX_SORT */ subject, emailfrom as src, emaildate ts
FROM emails
 where lcontains(bodytext, 'streben || wille || ruhe', 'emailfrom:DESC', 1) > 0;

SUBJECT                       |SRC                           |TS
------------------------------|------------------------------|--------
Metaphysik und Pessimismus    |arthur@schop.de               |09.06.11
Metaphysik ist pessimistisch  |irgendwer@schop.de            |18.06.11
Metaphysik ist pessimistisch  |arthur@schop.de               |18.06.11

The following query doesn’t include any sort specification but the DOMAIN_INDEX_SORT hint such that the result set will by default be sorted by score descending. This is an abbreviated syntax for the probably most widely used application of sorts with text searches. You can, of course, exchange it to the classic syntax of order by lscore(1) desc. Yes, ok, the result set are not identical, however, this is because two rows have, to a mantissa of 15 numbers, ichecked it, the same score, so Oracle / LDI has no other choice than returning the rows by accident.

SELECT /*+ DOMAIN_INDEX_SORT */ subject, emailfrom as src, lscore(1) sc
FROM emails
  where lcontains(bodytext, 'streben || wille || ruhe', 1) > 0;

SUBJECT                       |SRC                           |    SC
------------------------------|------------------------------|------
Metaphysik ist pessimistisch  |arthur@schop.de               |  .184
Metaphysik und Pessimismus    |arthur@schop.de               |  .138
Metaphysik ist pessimistisch  |irgendwer@schop.de            |  .138

SELECT subject, emailfrom as src, lscore(1) sc
FROM emails
  where lcontains(bodytext, 'streben || wille || ruhe', 1) > 0 order by lscore(1) desc;

SUBJECT                       |SRC                           |    SC
------------------------------|------------------------------|------
Metaphysik ist pessimistisch  |arthur@schop.de               |  .184
Metaphysik ist pessimistisch  |irgendwer@schop.de            |  .138
Metaphysik und Pessimismus    |arthur@schop.de               |  .138

gennff: the reader gained no understanding how to index and sort by date or timestamp values. one may of course convert dates to well formatted strings or numbers using formatcols. However, per chance ldi may already offer some automated process that, for whatever reason, did not make it to the docs yet.

3.4.5 Count Hits

The countHits() function is a Lucene Domain Index optimization to replace the regular SQL count(*) functionality. By comparison, countHits() is extremely fast because there is no need to pass rowid information from the Lucene Data Cartridge to the Oracle Engine to count matching rows. Here is an example:

  select LuceneDomainIndex.countHits('EMAILBODYTEXT','security') hits from dual;

  HITS
  ----
  5

The first argument of countHits() is the Lucene Domain Index name, the second argument is the search expression in question. You can optionally use a three argument version of countHits() to check some index in another schema. countHits() does not only serve the functionality explained above, a call to countHits() will also cache the rowid information along with the query expression. This is useful for lcontains() queries executed immediately afterwards, because the LDI can shortcut to this cached information. Under the covers, LDI keyes to the cache by sort_string(QueryParser.toString()) such that at least the query expressions of countHits() and lcontains() must match exactly. Following is an example of countHits() in correlation with lcontains() where emailFrom:(security) ist the match key:

  select LuceneDomainIndex.countHits('EMAILBODYTEXT','security') from dual;

  LUCENEDOMAININDEX.COUNTHITS('EMAILBODYTEXT','SECURITY')
  -------------------------------------------------------
  5

  Elapsed: 00:00:00.02

  select emailFrom FROM emails
    where lcontains(bodytext,'security','emailFrom:ASC',1)>0;

  EMAILFROM
  ---------
  codeshepherd@gmail.com
  codeshepherd@gmail.com
  erik@ehatchersolutions.com
  lucenelist2005@danielnaber.de
  lucenelist2005@danielnaber.de

  Elapsed: 00:00:00.04

3.4.6 First Rows Hint

Starting with the 2.4.0.1.0 release of LDI we have replaced deprecated Lucene hit classes by the TopDocs class. If you use the FIRST_ROWS optimizer hint in conjuction with the lcontains() inline pagination, Lucene Domain Index will execute a call to TopDocs to get the first N hits only. For example:

  select /*+ FIRST_ROWS DOMAIN_INDEX_SORT */ lhighlight(1), extractValue(object_value,'/page/title')
  from pages
    where lcontains(object_value, 'rownum:[1 TO 10] AND (musica tango rock)', 1)>0;

FIRST_ROWS and rownum:[1 TO 10] being used together performs a Lucene Query for the first 10 hits only. However, the next query along rownum:[10 TO 20] will find most of the Lucene structures, like the Searcher class instance and the ROWIDLucene to DocID association, already cached in memory. The Lucene index will be nevertheless be re-queried to get first 20 Hits (1..20) again. On the other hand, if you omit FIRST_ROWS, Oracle will by default switch to ALL_ROWS mode which means, if you are using a pagination of (rownum:[n TO > 2000]), Lucene Domain Index will fetch m first hits, but if m is lower than 2000, Lucene Domain Index will try to fetch by default 2000 hits. The magic number of 2000 is due Oracle ODCI API calls to the ODCIFetch routine in batches of 2000 rowids. If FIRST_ROWS and in-line pagination are not included in query, Lucene Domain Index alwasy works in batches of 2000 hits causing several cache misses in a full scan mode. For example, given a query:

  select count(*) from pages where lcontains(object_value, 'musica tango rock')>0;

Lucene Domain Index fetches the first 2000 hits, finally with the information that the hit amount is 2736 and it re-fetches (cache miss) the 2736 hits. Obviously you can use LuceneDomainIndex.countHits() to estimate the hit count in advance being faster than the previous query.

3.4.7 Highlighting

The lhighlight() ancillary operator works just as lscore(), remember the corellation id to lcontains(), but instead returns a VARCHAR2 text with the hit words (tokens) highlightened. The HTML tag used to remark hit words denotes <B></B> and is, as well as the fragment separator (…) and the maximum number of fragments (4), not yet customizable per call. However, starting with the 2.4.1.1.0 release of LDI, these parameters have become statically customizable through alter index ... parameters();, see below. here is some highlighting example:

  SELECT /*+ DOMAIN_INDEX_SORT */ subject, lscore(1) sc, lhighlight(1) txt
  FROM emails
    where lcontains(bodytext, 'security OR mysql', 'subject:ASC', 1)>0;

  SUBJECT
  SC
  TXT                                                                              
  ---
  Re: lucene injection
  .27477634
  On Dec 21, 2006, at 4:56 AM, Deepan wrote: I am bothered about <strong>security</strong> problems with lucene. Is it vulnerable to any kind of injection like <strong>mysql</strong> injection? many times the query from user is passed to lucene for search without validating. Rest easy. There are no known <strong>security</strong> issues with Lucene, and it has even undergone a recent static code analysis by Fortify (see the lucene-dev e-mail list).

Highlighting only works with columns of types VARCHAR2, CLOB and XMLType. You can perform highlighting operation even if your master columns is not indexed/stored. So far, the index creation DDL below features a IncludeMasterColumn:false, which means the actual XMLType representation of the Spanish Wikipedia page dump is not rawly indexed. Only the virtual columns title, comment, text and revisionDate become processed by Lucene. However, the LDI TextHighlight Java method must receive the full value(p) XMLType from the RDBMS engine to feed the Lucene Highlighter class with all the information that is not necessarily contained in the Lucene index.

  create index pages_lidx_all on pages p (value(p))
    indextype is Lucene.LuceneIndex
    parameters('PopulateIndex:false;
      DefaultColumn:text;
      SyncMode:Deferred;
      LogLevel:INFO;
      Analyzer:org.apache.lucene.analysis.SpanishWikipediaAnalyzer;
      ExtraCols:extractValue(object_value,''/page/title'') "title",
        extractValue(object_value,''/page/revision/comment'') "comment",
        extract(object_value,''/page/revision/text/text()'') "text",
        extractValue(object_value,''/page/revision/timestamp'') "revisionDate";
      FormatCols:revisionDate(day);
      IncludeMasterColumn:false;
      LobStorageParameters:PCTVERSION 0 ENABLE STORAGE IN ROW CHUNK 32768 CACHE READS FILESYSTEM_LIKE_LOGGING');

  select /*+ DOMAIN_INDEX_SORT */ lhighlight(1), extractValue(object_value,'/page/title')
  from pages
    where lcontains(object_value, 'rownum:[1 TO 10] AND (musica tango rock)', 1)>0;

  <strong>Música</strong> de Argentina... [[Latinoamérica|latinoamericanos]] con más desarrollo en su [[<strong>música</strong>]]. Se encuentra una gran... argentinos, un instrumento tradicional andino]] Aún se mantiene la <strong>música</strong> de los [[Indígenas_en_Argentina... de grandes corrientes de [[inmigración|inmigrantes]] europeos, la <strong>música</strong> argentina se enriqueció

Música de Argentina

musical emparentado con la [[habanera]] y el [[<strong>tango</strong> (<strong>música</strong>)|<strong>tango</strong>]].
  ==Diferencias con el <strong>tango</strong>==
  Aunque tanto la milonga como el <strong>tango</strong> están en [[compás]] de 2/4, las 8 [[semicorchea]]s de la milonga están distribuidas en 3 + 3 +  2 en cambio el <strong>tango</strong> posee un ritmo más «cuadrado». Las letras...]] criticó en algún momento el <strong>tango</strong> y prefirió la  milonga, que no trasmite la melancolía

Milonga (género musical)

Parameters supported by highlighting functions are:

• Formatter, a valid class name that implements Lucene’s Formatter interface and a constructor with no arguments. The default value is org.apache.lucene.search.highlight.SimpleHTMLFormatter.
• MaxNumFragmentsRequired, number of text fragments returned by the lhighlight() function, the default value is 4.
• FragmentSize, the size of each fragment returned, the default value is 100.
• FragmentSeparator, the string used as fragment separator, the default is “…”. Note that you can not use “;” or “:” as fragment separator because these tokens are used as parameter and value delimiters with the create or alter index ... parameters(); statements.

So far there is no customization allowed by passing any constructor arguments to the Formatter class, but you can easily create your own formatter to call SimpleHTMLFormatter with arguments like this:

  create or replace and compile java source named "org.apache.lucene.search.highlight.MyHTMLFormatter"
  as
  package org.apache.lucene.search.highlight;

  public class MyHTMLFormatter extends SimpleHTMLFormatter {
    public MyHTMLFormatter() {
      super("<span class=\"myhighlightclass\">","</span>");
    }
  }
  /

  alter index emailbodyText
    parameters('Formatter:org.apache.lucene.search.highlight.MyHTMLFormatter;
      MaxNumFragmentsRequired:3;
      FragmentSeparator:...;
      FragmentSize:50');

3.4.8 Highlighting using pipeline table functions

gennff: the author should at least give a short application scenario where the pipeline functions outperform the classis approach od § 3.4.7 .

phighlight() and rhighlight() provide a more general usage pattern for Lucene’s highlighting functionality. phighlight() receives an SQL query as string and performs highlighting according to a set of user defined columns on the query result. rhighlight() receives a SYS_REFCURSOR argument and adain performs highlighting on a set of user defined query columns but, unlike phighlight(), rhighlight() requires that the user defines a return type of the query, usually a TABLE OF collection, because with a SYS_REFCURSOR argument there is no option to know the return type of the query at compilation time. Both functions support the highlighting parameters introduced in § 3.4.7 . Here are two examples of highlighting by pipeline table functions:

  SELECT * FROM
  TABLE(phighlight(
          'EMAILBODYTEXT',
          'lucene OR mysql',
          'SUBJECT,BODYTEXT',
          'select /*+ DOMAIN_INDEX_SORT FIRST_ROW */ lscore(1) sc,e.*
           from eMails e where lcontains(bodytext,''security OR mysql'',''subject:ASC'',1)>0'
      ));

  SELECT * FROM
  TABLE(rhighlight(
          'EMAILBODYTEXT',
          'lucene OR mysql',
          'SUBJECT,BODYTEXT',
          'EMAILRSET',
          CURSOR(select /*+ DOMAIN_INDEX_SORT FIRST_ROW */ lscore(1) sc,e.*
          from eMails e where lcontains(bodytext,'security OR mysql','subject:ASC',1)>0)
      ));

The first three arguments of both pipeline functions read the same: the LDI index used, the Lucene query expression (that should match the lcontains() argument in the query) and finally the column list to be highlightened. The last argument for phighlight() is a VARCHAR2 type that transfers the SQL query to be executed by DBMS_SQL package. Note the additional single quotes used as an escape character. For rhighlight() two further arguments are required. The type returned by the cursor, EMAILRSET, that is a collection of the EMAILR record which holds all columns of the table EMAILS plus the score returned by the lscore() function (see the example below). And finally, the last argument is of CURSOR type which means any SQL query.

  CREATE TYPE EMAILR AS OBJECT (
    sc NUMBER,
    emailFrom VARCHAR2(256),
    emailTo VARCHAR2(256),
    subject VARCHAR2(4000),
    emailDate DATE,
    bodyText CLOB
  );

  CREATE OR REPLACE TYPE EMAILRSET AS TABLE OF EMAILR;

3.4.9 More like this functionality

The more like this functionality of Lucene is provided in the LDI package MoreLike, funtion this (again overloaded to allow for an additional owner name parameter) as follows.

  FUNCTION this(index_name IN VARCHAR2,
    x IN ROWID,
    f IN NUMBER DEFAULT 1,
    t IN NUMBER DEFAULT 10,
    minTermFreq IN NUMBER DEFAULT 2,
    minDocFreq IN NUMBER DEFAULT 5) RETURN sys.odciridlist;

A typical use case may look like this, where the anonymous PL/SQL block gets the first ROWID returned from the first query as pivot element and then expands the result set with other rows that also include terms like “procedure (C, Java or PL/SQL), optionally qualified”. Note that the “C” token will not be taken into account because it is regarded a stop word. Refer to the Appendix D.6 for a full explanation of each parameter.

  select rowid,lscore(1),text from test_source_big
    where lcontains(text,'"procedure java"~10',1)>0 order by lscore(1) desc;

  AAAOaPAAEAAAAnnABV 1.00000003 procedure (C, Java or PL/SQL), optionally qualified
  AAAOaPAAEAAAA0aAAV  .84852819  STATIC PROCEDURE refreshParameterCache as LANGUAGE JAVA NAME
  ...

  declare
    ridlist sys.odciridlist;
  begin
    ridlist := MoreLike.this(index_name=>'SOURCE_BIG_LIDX',x=>'AAAOaPAAEAAAAnnABV',minTermFreq=>1);
    FOR i IN (
      select rowid,text from test_source_big
        where rowid in (select * from table(ridlist_table(ridlist)))
    ) LOOP
      dbms_output.put_line('rowid: '||i.rowid||' text: '||i.text);
    END LOOP;
  end;
  /

  rowid: AAAOaPAAEAAAAhLAAc text: after issuing insert, update, delete or anonymous PL/SQL calls
  rowid: AAAOaPAAEAAAAjrAAo  text: QUALIFIED_SQL_NAME
  rowid: AAAOaPAAEAAAAk5AAe text: ORA-06502: PL/SQL: numeric or value error: character string buffer
  ...
  rowid: AAAOaPAAEAAAAtXAAb text: The name of the Java class, PL/SQL package or object type implementing

3.4.10 Facets

Starting with Lucene Domain Index release 2.4.1.1.0, Lucene’s facets functionality is available through a SQL aggregate function lfacets(). The input parameter is an encoded string containing the LDI (schema.)index name and a list of categories. The aggregated function only accepts a simple scalar value as an input argument so we need to encode the index name and categories list in a comma separated value. Using the index created in § 2.5, some categories in Lucene query Syntax are prefixed by TEXT:, according to the actually indexed column, and carry “procedure” as the main and “java” as the sub category, respectively.

  CREATE OR REPLACE function lfacets(input varchar2)
  return agg_tbl
  parallel_enable aggregate using facets_agg_type;
  /

  select lfacets('SOURCE_BIG_LIDX,TEXT:procedure,TEXT:java') from dual;

Creating a table with categories and linking the rows in a parent-child-relationship is an option to automatically generate facets, for example:

  create table source_categories (
    cat_code    number(4),
    cat_name    varchar2(256),
    cat_parent  number(4),
    CONSTRAINT PK_SOURCE_CATEGORIES PRIMARY KEY (cat_code),
    CONSTRAINT FK_CAT_PARENT FOREIGN KEY (cat_parent)
      REFERENCES source_categories (cat_code)
  );

  insert into source_categories values (1,'TEXT:procedure',null);
  insert into source_categories values (2,'TEXT:function',null);
  ...
  insert into source_categories values (6,'TEXT:java',1);
  insert into source_categories values (7,'TEXT:(pl sql)',1);
  insert into source_categories values (8,'TEXT:wrapped',1);
  ...
  insert into source_categories values (21,'line:[1 TO 1000]',1);
  insert into source_categories values (22,'line:[1001 TO 2000]',1);
  insert into source_categories values (23,'line:[2001 TO 3000]',1);

Now we can query the above table by executing a call to lfacets(), passing a category and sub a category. Note that we are using the ljoin() function which will convert the agg_tbl type to a comma separated string current plus hit cardinality. The first row returned does not have a sub category because the parent column value is null. The trailing 5116, in parantheses, is the number of rows that match the token “procedure”. The TEXT:procedure,line:[1001 TO 2000] result implies an logical intersection between the set of rows that include the token “procedure” against the set of rows that match “line[1001 TO 2000]“. The group by cat_code causes the Oracle ODCI API to first calculate the bit set for “procedure” and then iterate over all the sub categories “java”, “pl sql”, “wrapped”, doing bit calculations. This is fast and once the facets is computed it is stored as a filter in the Lucene Domain Index memory structures.

  select ljoin(lfacets('SOURCE_BIG_LIDX,' ||
     case level when (1) then cat_name
     else prior cat_name || ',' || cat_name end)) facet,
       cat_code, level
     from source_categories
     start with cat_parent is null
     connect by prior cat_code = cat_parent
     group by cat_code,level;

  FACET                                   CAT_CODE   LEVEL
  --------------------------------------------------------
  TEXT:procedure(5116)                    1          1
  TEXT:function(5574)                     2          1
  TEXT:trigger(96)                        3          1
  TEXT:package(860)                       4          1
  TEXT:(object type)(5140)                5          1
  TEXT:procedure,TEXT:java(9)             6          2
  .....
  TEXT:procedure,line:[1 TO1000](3)       21         2
  TEXT:procedure,line:[1001 TO2000](615)  22         2
  ...

When a number of rows or the amount of categories is quite large, one can use a materialized view to work as cache for the facets computation. Such a materialized view can be queried as any other table and the access will be too fast (and can be indexed as well).

  CREATE MATERIALIZED VIEW source_facets
  AS
  select ljoin(lfacets('SOURCE_BIG_LIDX,' ||
     case level when (1) then cat_name
     else prior cat_name || ',' || cat_name end)) facet,
       cat_code, level
     from source_categories
     start with cat_parent is null
     connect by prior cat_code = cat_parent
     group by cat_code,level;

gennff: __rework marker__

3.4.11 Terms pipeline table functions

Starting with Lucene Domain Index 2.9.1.1.0, two pipeline table functions has been included to iterate over terms of Lucene Index structure, high_freq_terms():

  FUNCTION high_freq_terms(index_name VARCHAR2,
    term_name  VARCHAR2,
    num_terms  NUMBER) RETURN term_info_set

is available for getting the Top-N (num_terms) most used terms on the whole index or in a particular field. term_info_set is defined as:

  TYPE term_info AS OBJECT (
    term     VARCHAR2(4000),
    docFreq  NUMBER(10)
  );
  TYPE term_info_set AS TABLE OF term_info;

You can query your index by using:

  select * from table(high_freq_terms('SOURCE_BIG_LIDX','TEXT',10));
  select * from table(high_freq_terms('SOURCE_BIG_LIDX',null,10));
  select * from table(high_freq_terms('SOURCE_BIG_LIDX','line',100));

and, index_terms():

  FUNCTION index_terms(index_name VARCHAR2,
    term_name  VARCHAR2) RETURN term_info_set

  select * from table(index_terms('SOURCE_BIG_LIDX','TEXT')) order by docFreq desc;
  select * from table(index_terms('SOURCE_BIG_LIDX','TEXT'));
  select * from table(index_terms('SOURCE_BIG_LIDX',null)) where rownum  select * from (select * from table(index_terms('SOURCE_BIG_LIDX','line')) order by docFreq desc) where rownum

on both functions if argument term is NULL, these functions will iterate over all index terms. The natural order for high_freq_terms() is descendent by docFreq, but index_terms() is ordered by term_name:term_value ascending. Note that if you pass a non NULL value to term to starts with the first value for the specific term index_terms() do not stop when all the values of this term are completed, this functionality is similar to Lucene Java method reader.terms(new Term(term)). Here example if you want only iterate on an specific term name:

  BEGIN
     FOR term_rec IN (SELECT * FROM table(index_terms('SOURCE_BIG_LIDX','line'))) LOOP
        /* Fetch from cursor variable. */
        EXIT WHEN substr(term_rec.term,1,length('line'))<>'line'; -- exit when last row is fetched
        -- process data record
        dbms_output.put_line('Name = ' || term_rec.term || ' ' || term_rec.docFreq);
     END LOOP;
  END;
  /

You can use index_terms() to get the Top-N terms order by docFreq, for example:

  select * from (select * from table(index_terms('SOURCE_BIG_LIDX',null))
  order by docFreq desc) where rownumselect * from table(high_freq_terms('SOURCE_BIG_LIDX',null,10));   TEXT:in             24952   TEXT:varchar     16996   ...   TEXT:return         6241   Elapsed: 00:00:00.02 

Two queries are equivalent semantically but high_freq_terms() is more efficient because it uses TermInfoQueue structure for sorting, caches his computation one is executed and do not creates a lot of term_info objects which then are sorted by the RDBMS engine. lfreqterms ancillary operator is a complimentary function which requires a column indexed with ANALYZED_WITH_VECTORS option in FormatCols index parameter. Example usage:

   select TERMVECTOR TXT, LFREQTERMS(1) FQTERMS from t1 where lcontains(termvector,'two',1)>0;
  TXT                                                           FQTERMS
  ---                                                           -------
  one two two three three three                       LUCENE.TERM_INFO_SET('LUCENE.TERM_INFO(one,1)','LUCENE.TERM_INFO(three,3)','LUCENE.TERM_INFO(two,2)')
  one two two three three three                       LUCENE.TERM_INFO_SET('LUCENE.TERM_INFO(one,1)','LUCENE.TERM_INFO(three,3)','LUCENE.TERM_INFO(two,2)')
  one two two three three three                       LUCENE.TERM_INFO_SET('LUCENE.TERM_INFO(one,1)','LUCENE.TERM_INFO(three,3)','LUCENE.TERM_INFO(two,2)')
  two two three three three four four four four    LUCENE.TERM_INFO_SET('LUCENE.TERM_INFO(four,4)','LUCENE.TERM_INFO(three,3)','LUCENE.TERM_INFO(two,2)')

3.4.12 Did You Mean functionality

Starting with Lucene Domain Index 2.9.2.1.0, Did You Mean Lucene functionality was added as an extended LDI property using the Lucene SpellChecker library to create the dictionary index from the main index. Finaly, the dictionary index will be merged to the main index.

  PROCEDURE indexDictionary(
    index_name   IN VARCHAR2,
    spellColumns IN VARCHAR2 DEFAULT null,
    distancealg  IN VARCHAR2 DEFAULT 'Levenstein')

is available to create the dictionary index to be merged with main index.

You can create the dictionary by using:

  call didyoumean.indexdictionary('SOURCE_BIG_LIDX');
  Call completed.
  Elapsed: 00:01:11.61

  exec didyoumean.indexdictionary('EMAILBODYTEXT','BODYTEXT,subject,emailFrom,emailTo','NGram');
  PL/SQL procedure successfully completed.
  Elapsed: 00:00:01.62

Only index_name is mandatory. If spellColumns parameter is NULL, the master column of the main index will be used. By default Levenstein Distance Algorithm (a.k.a. edit distance) is applied (other options are Jaro – Jaro Winkler metric – and Ngram distance).

Note: The dictionary structure create the “word”, “gramN”, “startN” and “endN” Lucene fields, so be carefull if you have this fieds in the main index. The structure of this index is (for a 3-4 gram) this:

  Index Structure
    Example
  word
    kings
  gram3
    kin, ing, ngs
  gram4
    king, ings
  start3
    kin
  start4
    king
  end3
    ngs
  end4
    ings

and,

  FUNCTION suggest (
    index_name  IN VARCHAR2,
    cmpval      IN VARCHAR2,
    highlight   IN VARCHAR2 DEFAULT null,
    distancealg IN VARCHAR2 DEFAULT 'Levenstein'
  ) RETURN VARCHAR2;

is available to query the dictionary index. You can query the dictionary by using:

  select didyoumean.suggest('SOURCE_BIG_LIDX','sorce') suggestion from dual;

  SUGGESTION
  ----------
  source
  Elapsed: 00:00:00.31

  select didyoumean.suggest('SOURCE_BIG_LIDX','sorce','b') suggestion from dual;
  SUGGESTION
  ----------
  <strong>source</strong>
  Elapsed: 00:00:00.09

  select didyoumean.suggest('SOURCE_BIG_LIDX','sorce','b','Jaro') suggestion from dual;
  SUGGESTION
  ----------
  <strong>source</strong>
  Elapsed: 00:00:00.07

  select didyoumean.suggest('EMAILBODYTEXT','lucene searhc','i') suggestion from dual;
  SUGGESTION
  ----------
  lucene <em>search</em>
  Elapsed: 00:00:00.06

  select didyoumean.suggest('EMAILBODYTEXT','lucine injetion','b','Levenstein') suggestion from dual;
  SUGGESTION
  ----------
  <strong>lucene</strong> <strong>injection</strong>
  Elapsed: 00:00:00.06

The index_name parameter and the word to respell (cmpval) parameter are mandatory. You can define, optionaly, the highlight to be used (e.g. b for bold, i for italic, etc.) and define the distance algorithm to apply.

  Pipeline table lautocomplete (
    index_name  IN VARCHAR2,
    term_name  VARCHAR2 DEFAULT NULL,
    term_value VARCHAR2 DEFAULT '__ALL__',
    num_terms  NUMBER DEFAULT 10
  ) RETURN term_info_set;

is available to query the dictionary index. You can query the dictionary by using:

  select * from table(lautocomplete('DICC_LIDX','TERM','th',15)) t;

  TERM                            DOCFREQ
  -----------------------------   ------
  there                           3
  theory                          2
  thaw                            2
  then                            2
  therefore                       2
  thence                          2
  ....
  thank                           1




  Pipeline table ldidyoumean (
    index_name  IN VARCHAR2,
    cmpval  VARCHAR2,
    numSug NUMBER DEFAULT 10,
    highlight  VARCHAR2 DEFAULT null,
    distancealg  VARCHAR2 DEFAULT 'Levenstein') RETURN term_info_set;

is available to query the dictionary index. You can query the dictionary by using:

  select * from table(ldidyoumean('DICC_LIDX', 'atention', 5, 'b', 'Levenstein')) t;

  TERM                   DOCFREQ
  --------------------   -------
  <strong>attention</strong>       2
  <strong>intention</strong>       1
  <strong>detention</strong>       1
  <strong>mention</strong>         3
  <strong>attenuation</strong>     1

lsimilarity ancillary operator computes the similarity between the indexed column and the query passed to lcontains using Levenstein Distance algorithm. Example:

  select lsimilarity(1) sim1, term t from dicc where lcontains(term, 'there~0.7', 1) > 0;

  SIM1                          T
  ----                          _
  1                             there
  1                             there
  1                             there
  0.800000011920928955078125    where
  0.800000011920928955078125    theme
  0.800000011920928955078125    here
  0.800000011920928955078125    here

  select lsimilarity(1) sim1, term t from dicc where lcontains(term, 'there*', 1) > 0

  SIM1                          T
  ----                          -
  1                             there
  0.5555555820465087890625      therefore
  1                             there
  1                             there
  0.5555555820465087890625      therefore

3.5 Synchronize

Working with SyncMode:Deferred you has to manually synchronize your index, it means update Lucene Domain Index structure applying pending changes such as insert and update. Deletes operations are always applied due ODCI Api do not accept rowid of deleted rows.
Here an example:

  begin
    LuceneDomainIndex.sync('IT1');
    commit; -- release locks
  end;
  /

LuceneDomainIndex.sync procedure requires an argument of type VARCHAR2 with the index object name, index object name are usually capitalized and have the syntax SCHEMA_OWNER.IDX_NAME.

Synchronize operation could raise an exception if some rows being indexed are locked for update, in that case you have release first locked rows and re-sync the index.

An exclusive lock at Lucene Index storage is obtained during index synchronization, so you has to commit or rollback the connection immediately after this operation to release exclusive lock.

Since Lucene Domain Index 2.4.0.1.0 you can use LuceneDomainIndex.sync(‘IT1′) or LuceneDomainIndex.sync(USER,’IT1′), both procedure are equivalent.

Note: Due a limitation on SYS.ODCIRidList() array you can enqueue more than 32767 additions or deletions, an update is counted as one deletion plus one addition by Lucene implementation code. This limitation will be removed in future releases of Lucene Domain Index.

3.6 Optimize

Optionally you can optimize Lucene Index storage, for doing that execute:

  begin
    LuceneDomainIndex.optimize('IT1');
    commit; -- release locks
  end;
  /

Like sync operation this procedure get an exclusive lock at Lucene Index storage table and perform an optimization of Lucene Index merging multiples segment in new one for example. You can still performing select operation (read-only) using Lucene Domain Index during optimization time, Oracle concurrency system (redo logs) provides you this functionality, once you perform a commit operation any other concurrent session will automatically see index changes.

3.7 XMLDB Export

Lucene Domain Index provides for a raw dump of the Lucene index directory files. You can perform this task by means of an Oracle XMLDB export operation that will create xdb resources, which in turn can be accessed by a variety of methods such as FTP, HTTP or WebDAV.

Before writing the raw dump to XMLDB, go and check the accessibilty of the access method in question. For FTP in Oracle 11g, for example, access to the protocol is switched off by default, having the FTP port initially at value of 0. You can check the current setting of the FTP port using this statement:

  -- SYSDBA
  select extractValue(DBMS_XDB.cfg_get(),
    '/xdbconfig/sysconfig/protocolconfig/ftpconfig/ftp-port/text()')
  from dual;

Resetting the port to an appropriate value goes like this. Test your FTP immediately after resetting the port! You should at least be able to read/see the ftp://thathost:1531/public directory.

  -- SYSDBA
  DECLARE
    newconfig XMLType;
  BEGIN
    SELECT
      updateXML(DBMS_XDB.cfg_get(),
        '/xdbconfig/sysconfig/protocolconfig/ftpconfig/ftp-port/text()', 1531)
      INTO newconfig
      FROM DUAL;
    DBMS_XDB.cfg_update(newconfig);
    COMMIT;
  END;
  /

  -- or for short in 11g
  exec dbms_xdb.setFtpPort(1531);

gennff: it appears nondistinctive to the reader why the method does not comprise the commit with its body. is there a special reason that necessitated this design decision?

The next step is to execute the utility method LuceneDomainIndex.xdbExport() that may be called with parameters (schemaname, indexname) or just (indexname) (the latter will derive the schema name from the index object). It is essential to execute the commit; afterwards because the method itself will not commit and will keep holding all incoming locks until commiting (or rolling back):

  begin
    LuceneDomainIndex.xdbExport('IT1');
    commit; -- makes change visible to Ftp or WebDAV
  end;
  /

For an index IT1 of user SCOTT, the file resources will be available at this xdb directory: ftp://thathost:1531/public/lucene/SCOTT.IT1. Once you copied the files to some public file system, you can open the Lucene index with any Lucene compatible application like, for example, Luke. Here are some screen shots of Luke, analyzing several properties of a Lucene index.









3.8 Exporting/Importing functional index with exp/imp Oracle tools

You can perform an Oracle exp operation for your Lucene Domain Index. Oracle exp tool performs by default functional index for every table being exported during the backup process. As I mention early Lucene Domain Index creates a table named IDX_NAME$T which have Lucene file storage replaced by BLOB, also a DBMS AQ is created during the index creation time, this queue is associated to a table IDX_NAME$QT, both tables have a flag marked as SECONDARY, which means that you can not export these tables alone, but they are automatically includes when Lucene Domain Index is included into the export.

During import operation Oracle re-create the index using a create index … parameters(‘your lucene parameters’) DML statement, all Lucene Domain Index parameters are included except for the parameter PopulateIndex which always is stored as false into Oracle System’s views. This parameter is altered intentional by Lucene Domain Index because if its set to true, during import operation Lucene Domain Index will try to re-create the Lucene Index structure instead of using the information restored into IDX_NAME$T table.

Alternative to XMLDB Export or Oracle exp tool you can also exports your Lucene Domain Index storage using a create table as … DML statement. For example:

  create table SOURCE_BIG_LIDX$T$BK as (select * from SOURCE_BIG_LIDX$T);

You can export now using exp tool SOURCE_BIG_LIDX$T$BK because is regular table:

  -bash-3.2$  exp
  Export: Release 10.2.0.3.0 - Production on Fri Mar 27 02:46:18 2009
  Copyright (c) 1982, 2005, Oracle.  All rights reserved.

  Username: scott/tiger

  Connected to: Oracle Database 10g Release 10.2.0.3.0 - Production
  Enter array fetch buffer size: 4096 >

  Export file: expdat.dmp > SOURCE_BIG_LIDX_BK.dmp

  (2)U(sers), or (3)T(ables): (2)U > 3

  Export table data (yes/no): yes > yes

  Compress extents (yes/no): yes >

  Export done in US7ASCII character set and AL16UTF16 NCHAR character set
  server uses AL32UTF8 character set (possible charset conversion)

  About to export specified tables via Conventional Path ...
  Table(T) or Partition(T:P) to be exported: (RETURN to quit) > SOURCE_BIG_LIDX$T$BK

  . . exporting table           SOURCE_BIG_LIDX$T$BK         19 rows exported
  Table(T) or Partition(T:P) to be exported: (RETURN to quit) >

  Export terminated successfully without warnings.

Now you can drop your index and re-create again without populating it:

  select count(*) from test_source_big where lcontains(text,'function')>0;

  COUNT(*)
  --------
  6167

  drop index SOURCE_BIG_LIDX;

  Index dropped.

  create index source_big_lidx on test_source_big(text)
         indextype is lucene.LuceneIndex
         parameters('PopulateIndex:false;
           AutoTuneMemory:true;
           Analyzer:org.apache.lucene.analysis.SimpleAnalyzer;
           MergeFactor:500;
           FormatCols:line(0000);
           ExtraCols:line "line"');

  Index created.

  drop table SOURCE_BIG_LIDX$T$BK;

  Table dropped.

Restore your .dmp now and check again if your index returns a correct result:

  -bash-3.2$ imp scott/tiger

  Import: Release 10.2.0.3.0 - Production on Fri Mar 27 02:49:40 2009

  Copyright (c) 1982, 2005, Oracle.  All rights reserved.

  Connected to: Oracle Database 10g Release 10.2.0.3.0 - Production

  Import file: expdat.dmp > SOURCE_BIG_LIDX_BK.dmp

  Enter insert buffer size (minimum is 8192) 30720>

  Export file created by EXPORT:V10.02.01 via conventional path
  import done in US7ASCII character set and AL16UTF16 NCHAR character set
  import server uses AL32UTF8 character set (possible charset conversion)
  List contents of import file only (yes/no): no >

  Ignore create error due to object existence (yes/no): no >

  Import grants (yes/no): yes >

  Import table data (yes/no): yes >

  Import entire export file (yes/no): no > yes

  . importing SCOTT's objects into SCOTT
  . importing SCOTT's objects into SCOTT
  . . importing table         "SOURCE_BIG_LIDX$T$BK"         19 rows imported
  Import terminated successfully without warnings.

Check first that your index do not have information and populate them with Lucene Index information:

  conn scott/tiger
  Connected.
  select count(*) from test_source_big where lcontains(text,'function')>0;

  COUNT(*)
  --------
  0

  truncate table SOURCE_BIG_LIDX$T;

  Table truncated.

  insert into SOURCE_BIG_LIDX$T (select * from SOURCE_BIG_LIDX$T$BK);

  19 rows created.
  exit
  ..... and connect again to refresh Lucene Domain Index in memory structures ....
  conn scott/tiger
  Connected.
  select count(*) from test_source_big where lcontains(text,'function')>0;

    COUNT(*)
  ----------
        6167

As you can see the Lucene Domain Index structure can be export alone without exporting the master table, this is useful when you are upgrading Lucene Domain Index that requires that all index need to be dropped first and you don’t want to re-create a very big index.

Doc Links

Previous / LDI Docs – 2 Installing and Testing
Next / LDI Docs – 4 Locking and Performance


Filed under: Documentation Tagged: example, export, function, operator, performance, procedure ]]> http://ludoix.wordpress.com/2011/03/11/ldi-docs-3-procedures-functions-operators-and-examples/feed/ 0 ludoix ddgw7sjp_58g7qgh8fm ddgw7sjp_57fq744zg8 ddgw7sjp_59fk7cwvgg ddgw7sjp_60fgkrbgdd http://ludoix.wordpress.com/2011/03/11/lucene-domain-index-installing-and-testing/ http://ludoix.wordpress.com/2011/03/11/lucene-domain-index-installing-and-testing/#comments Fri, 11 Mar 2011 15:09:14 +0000 ludoix http://ludoix.wordpress.com/?p=26 ]]> 2 Installing and Testing

2.1 Requirements

• JDeveloper 11g (optional) only if you want to edit the Java code
• Ant 1.7.0
• Sun JDK 1.5.0_05/1.4.2 ($ORACLE_HOME/jdk directory works fine as Java Home for compiling on 10g and 11g)
• Linux/Windows Database Oracle 10g 10.2/11g production

2.2 Install binary distributions

Binary distributions are available at SourceForge.net and provides a very straightforward installation.

2.2.1 11g Binary Distribution

Edit your ~/build.properties file with your Database values (Windows users can find build.properties file at C:\Documents and Settings\username folder):

  db.str=test
  db.usr=LUCENE
  db.pwd=LUCENE
  dba.usr=sys
  dba.pwd=change_on_install
  javac.debug=true
  javac.source=1.5
  javac.target=1.5

db.str is your SQLNet connect string for your target database, check first with tnsping. This is an example environment setting before installing on 11g database:

  MAVEN_HOME=/usr/local/maven
  ORACLE_BASE=/u01/app/oracle
  ORACLE_HOME=$ORACLE_BASE/product/11.1.0.6.0/db_1
  ORACLE_SID=test
  JAVA_HOME=$ORACLE_HOME/jdk
  PATH=$MAVEN_HOME/bin:$HOME/bin:$ORACLE_HOME/bin:$JAVA_HOME/bin:/usr/local/bin:$PATH
  LD_LIBRARY_PATH=$ORACLE_HOME/lib:/usr/local/lib
  CVS_RSH=ssh
  umask 022
  export PATH LD_LIBRARY_PATH ORACLE_HOME ORACLE_BASE ORACLE_SID JAVA_HOME CVS_RSH NLS_LANG

Upload, install and test your code into the database:

  ant install-odi
  ant test-odi-[clob|dicc|emails|iot|master-detail|similarity|source-small|term-vector|xmltype]

For Oracle 11g you can perform a post-installation step:

  ant jit-lucene-classes

This target force to translate all Lucene, Snowball, etc. and ODI classes to assembler. Instead of waiting that the database compile it by detecting most used classes or method.

2.2.2 10g Binary Distribution

First edit your ~/build.properties with something like this:

  db.str=orcl
  db.usr=LUCENE
  db.pwd=LUCENE
  dba.usr=sys
  dba.pwd=change_on_install
  javac.debug=true
  javac.source=1.4
  javac.target=1.4

db.str property is a SQLNet connect string for the target database. ORACLE_HOME environment setting is required and properly configured to an Oracle 10g database layout, finally execute ant without arguments. Here an example of environment setting on 10g database:

  MAVEN_HOME=/usr/local/maven
  ORACLE_BASE=/u01/app/oracle
  ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
  ORACLE_SID=orcl
  JAVA_HOME=$ORACLE_HOME/jdk
  PATH=$MAVEN_HOME/bin:$HOME/bin:$ORACLE_HOME/bin:$JAVA_HOME/bin:/usr/local/bin:$PATH
  LD_LIBRARY_PATH=$ORACLE_HOME/lib:/usr/local/lib
  CVS_RSH=ssh
  umask 022
  export PATH LD_LIBRARY_PATH ORACLE_HOME ORACLE_BASE ORACLE_SID JAVA_HOME CVS_RSH NLS_LANG

If you are re-installing Oracle Lucene Domain Index (ODI) first drop any Lucene Domain Index. Default target will drop first Lucene schema if exists, example install action:

  ant install-odi-10g
ant test-odi-[clob|dicc|emails|iot|master-detail|similarity|source-small|term-vector|xmltype]

Additionally to installation and testing steps you can perform a NCOMP translation of Lucene ODI and related classes from Java to assembler using:

  ant ncomp-runtime-retrotranslator-sys-code
  ant ncomp-lucene-all

2.3 Install Instructions to compile from sources

• Unpack or checkout Lucene sources
• Checkout ODI sources, by now only Anonymous CVS access is provided you can download from Source Forge servers with
  

    cd /tmp
    svn co http://ldi.svn.sourceforge.net/svnroot/ldi/odi/trunk odi
  

• Copy to $LUCENE_ROOT/contrib
  

    cd $LUCENE_ROOT/contrib
    cp -rp /tmp/odi .
  

• Edit $LUCENE_ROOT/common-build.xml adding a target for creating a jar file with test sources
  

  ....
    <property name="dev.version" value="3.4.0"/>
    <property name="tests.luceneMatchVersion" value="3.4.0"/>
  ....
    <target name="jar-test" depends="compile-test">
      <jar destfile="${build.dir}/${final.name}-test.jar" basedir="${build.dir}/classes/test" excludes="**/*.java"/>
    </target>
  

(OPTIONAL) Update Lucene’s BufferedIndexInput.BUFFER_SIZE according to your db_block_size init.ora parameter.
Before compile and upload Lucene core library you can change org.apache.lucene.store.BufferedIndexInput.BUFFER_SIZE constant to the value of your db_block_size init parameter, this change will improve reading performance by using same block size as the physical block size that your database use.

Compile ODI Directory sources and tests, these targets automatically copies all Lucene Domain Index required libraries from your $ORACLE_HOME and Internet. Starting with ODI 2.4.0.1.x build.xml file automatically compiles all Lucene contrib modules dependency.

  cd $LUCENE_ROOT/contrib/odi
  ant jar-core
  ant jar-test
  ant package-zip

Edit your ~/build.properties file with your Database values:

  db.str=orcl
  db.usr=LUCENE
  db.pwd=LUCENE
  dba.usr=sys
  dba.pwd=change_on_install
  javac.debug=true
  javac.source=1.4
  javac.target=1.4

db.str is your SQLNet connect string for your target database, check first with tnsping utility, also note that for 11g database user and password are case sensitive, so leave LUCENE in uppercase. Upload your code to the database:

  ant install-odi

2.3.1 Generating Maven’s artifacts

You can generate Lucene and ODI Maven’s artifacts following previous one steps, then execute:

   ant generate-maven-artifacts

2.4 Optimizations

Discussion of Optimizations.

2.4.1 Using NCOMP on 10g

Is strongly recommended before going in production that install Oracle Lucene Domain Index NCOMPed in 10g databases. NCOMP automatically translate Lucene and ODI Java code to assembler and finally install it as dynamic link library (.so/.dll) in your Oracle home. To do this simply execute this Ant task instead of install-odi target:

  ant ncomp-lucene-all

2.4.2 Using JIT on 11g

First verify that your database parameter java_jit_enabled is TRUE. Oracle 11g includes a JIT technology which automatically translates most used Java methods to assembler. If you want to pre-compile all Lucene Java code to assembler and not wait for Oracle database detects common used code you can execute this target:

  ant jit-lucene-classes
  ant jit-oracle-classes

2.5 Testing Lucene Domain Index

Required grants for regular Oracle users. IMPORTANT: Before start using Lucene Domain Index grant this to any Oracle user rather than LUCENE:

  -- connected as sysdba
  grant LUCENEUSER to scott;

Lucene Domain Index have two kinds of test suites to check that everything is OK after installation. First test suite which can be launched using Ant and is implemented using SQLUnit, to launch it simply execute (first checking values of SQLNet connection options):

  [mochoa@mochoa odi]$ cat db/sqlunit.properties
  # SQLUnit test
  sqlunit.driver = oracle.jdbc.driver.OracleDriver
  sqlunit.url = jdbc:oracle:oci:@orcl
  sqlunit.user = LUCENE
  sqlunit.password = LUCENE
[mochoa@localhost lucene-odi]$ ant test-odi-clob
Buildfile: build.xml
test-odi-clob:

Second test suite is a set of JUnit tests to simulate middle tier environments, it also use a connection pool. To start these suites run:

[mochoa@localhost lucene-odi]$ ant test-parallel
Buildfile: build.xml

test-parallel:
    [junit] Running org.apache.lucene.index.TestDBIndexParallel
    [junit] Table created: T1
    [junit] Index altered: LIT1
    [junit] OnLine: true
    [junit] No Row updated at: 4030 to: 4039 elapsed time: 22 ms.
    [junit] Inserted rows: 10 time: 193 avg time: 19
      ...
    [junit] Index droped: LIT1
    [junit] Table droped: T1

Next test suite is JUnit tests to simulate middle tier environments, it also use a connection pool. To start these suites run:

[mochoa@localhost lucene-odi]$ ant test-queryhits
Buildfile: build.xml

test-queryhits:
    [junit] Running org.apache.lucene.indexer.TestQueryHits
    [junit] Hits: 35602
      ...
    [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 63.306 sec
BUILD SUCCESSFUL
Total time: 1 minute 4 seconds

Class org.apache.lucene.indexer.TestQueryHits requires a table which is very big to create and destroy at setup() and tearDown() methods. Before run this test create the table with:

    create table test_source_big as (select * from all_source);

And the index on 10g with:

create index source_big_lidx on test_source_big(text)
indextype is lucene.luceneindex
parameters('AutoTuneMemory:true;IndexOnRam:true;BatchCount:3000;ParallelDegree:2;SyncMode:OnLine;LogLevel:INFO;AutoTuneMemory:true;PerFieldAnalyzer:line(org.apache.lucene.analysis.KeywordAnalyzer),TEXT(org.apache.lucene.analysis.SimpleAnalyzer);FormatCols:line(00000);ExtraCols:line "line";MergeFactor:500');

Or in 11g with:

create index source_big_lidx on test_source_big(text)
indextype is lucene.luceneindex
parameters('AutoTuneMemory:true;IndexOnRam:true;BatchCount:3000;ParallelDegree:2;SyncMode:OnLine;LogLevel:INFO;AutoTuneMemory:true;PerFieldAnalyzer:line(org.apache.lucene.analysis.KeywordAnalyzer),TEXT(org.apache.lucene.analysis.SimpleAnalyzer);FormatCols:line(00000);ExtraCols:line "line";MergeFactor:500;LobStorageParameters:CACHE READS FILESYSTEM_LIKE_LOGGING');

Doc Links

Previous / LDI Docs – 1 Introduction
Next / LDI Docs – 3 Procedures, Functions, Operators and Examples


Filed under: Documentation Tagged: docs, install, test ]]> http://ludoix.wordpress.com/2011/03/11/lucene-domain-index-installing-and-testing/feed/ 0 ludoix http://ludoix.wordpress.com/2011/03/06/lucene-domain-index-introduction/ http://ludoix.wordpress.com/2011/03/06/lucene-domain-index-introduction/#comments Sun, 06 Mar 2011 22:30:54 +0000 ludoix http://ludoix.wordpress.com/?p=10 ]]> 1 Introduction

General introduction, features, benefits and comparison with Lucene standalone implementation and Oracle Text.

1.1 What is Lucene

Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform.
Apache Lucene is an open source project available for free download.
If Lucene is a pure Java framework why not use it inside Oracle Database JVM environment?

1.2 What is Lucene Domain Index

Lucene Domain Index is full integration of Lucene project running inside the Oracle database using Oracle JVM. Oracle provides a full featured JVM inside your Oracle Database compliant with JDK 1.4 in 10g release and 1.5 in 11g.
OJVMDirectory is a replacement for Lucene’s file system storage by a BLOB based storage, the name is related to the class which overrides (Directory.java)

Here a simple list of points take into account to choose this storage:

• Using traditional File System for storing the inverted index is not a good option for some users, you don’t have commit or rollback behavior, backup, etc.
• Using BLOB for storing the inverted index running Lucene outside the Oracle database has a bad performance because there are a lot of network round trips and data marshaling.
• Indexing relational data such as tables with VARCHAR2, CLOB or XMLType with Lucene running outside the database has the same problem as the previous point.
• By using Secure BLOB on Oracle 11g you can choose to encrypt and compress Lucene Index storage transparently reducing disk usage and not exposing your relational data outside the DB increasing risk or violating SOX company regulations.
• The JVM included inside the Oracle database can scale up to 10.000+ concurrent sessions without memory leaks or deadlock and all the operations on tables are in the same memory space!!

More on this, Oracle provides a Data Cartridge API (ODCI), also called Extensible Indexing mechanism because you can write your own Domain Index and integrate it with the Oracle Engine and optimizer.

There are some important points integrating Lucene by using ODCI:

• Changes on rows are automatically notified to Lucene, now these changes are en-queued using Oracle AQ. User can control if these changes are applied OnLine (immediately after commit) or Deferred (application Sync).
• Oracle optimizer can choose a proper execution plan if there is a Domain Index created.
• You can mix lcontains(), lhighlight(), lscore() and many other operators, procedures or functions in your queries.

1.3 Why do I use Lucene Domain Index?

Oracle includes a full featured and enterprise dedicated text search engine named Oracle Text, being coded in C and fully integrated into the Oracle kernel, but:

• on Oracle Text you can not:
  • control which functionality will be included into next release
  • easily customize it for your needs
  • index Index Organized Tables (IOT)
  • index joined tables
  • index unlimited extra columns
  • easily highlight text
  • index NCLOB and NVARCHAR data types
• on Oracle 10g you can not:
  • index multiple columns in a same index
  • sort and filter by using indexed columns at index level
• on Oracle 11g you can not:
  • filter by / sort by on columns of timestamp with TZ, commonly
    used in XMLDB because is the official data type for xsd:date type
• using Lucene Domain Index you can:
  • usually indexes are smaller because Lucene Domain Index do not store any column, except the rowid, inside Lucene’s inverted index structure. By using a rowid Oracle can lookup any column value faster than retrieve it from Lucene inverted index
  • Support padding for Text columns
  • Support formatting (rounding/padding) for Number and Date/Time columns
  • You can create index on-line even in a standard edition databases (feature available en EE for Text)
  • Extending DefaultUserDataStore class an application can implement any data type mapping, specially BLOB which in common cases have non standard encoding
  • An experimental native REST WS can be used to query the index
  • Lucene inverted index is transactional, if a SQL operation is rolled back, the index will be consistent too, avoiding phantom reads or negative hits (rows which should be included as hit but was not included in Lucene index)
  • is a ready to use uptodate solution for any programming language, for example Ruby, .Net, Phyton or PHP
  • an elegant solution for highlighting text use pipeline table functions
  • a high level abstraction layer for Lucene IR library, developers only deal with SQL
  • transparent compression and encryption of Lucene storage if you enable Oracle Transparent Data Encryption and Secure File compression

Doc Links

Next / LDI Docs – 2 Installing and Testing


Filed under: Documentation Tagged: docs, overwiew ]]> http://ludoix.wordpress.com/2011/03/06/lucene-domain-index-introduction/feed/ 0 ludoix