Module org.elasticsearch.server
Package org.elasticsearch.common.lucene.search

Class XMoreLikeThis

java.lang.Object
org.elasticsearch.common.lucene.search.XMoreLikeThis
public final class XMoreLikeThis extends Object
Generate "more like this" similarity queries. Based on this mail: 
 Lucene does let you access the document frequency of terms, with IndexReader.docFreq().
 Term frequencies can be computed by re-tokenizing the text, which, for a single document,
 is usually fast enough.  But looking up the docFreq() of every term in the document is
 probably too slow.
 You can use some heuristics to prune the set of terms, to avoid calling docFreq() too much,
 or at all.  Since you're trying to maximize a tf*idf score, you're probably most interested
 in terms with a high tf. Choosing a tf threshold even as low as two or three will radically
 reduce the number of terms under consideration.  Another heuristic is that terms with a
 high idf (i.e., a low df) tend to be longer.  So you could threshold the terms by the
 number of characters, not selecting anything less than, e.g., six or seven characters.
 With these sorts of heuristics you can usually find small set of, e.g., ten or fewer terms
 that do a pretty good job of characterizing a document.
 It all depends on what you're trying to do.  If you're trying to eek out that last percent
 of precision and recall regardless of computational difficulty so that you can win a TREC
 competition, then the techniques I mention above are useless.  But if you're trying to
 provide a "more like this" button on a search results page that does a decent job and has
 good performance, such techniques might be useful.
 An efficient, effective "more-like-this" query generator would be a great contribution, if
 anyone's interested.  I'd imagine that it would take a Reader or a String (the document's
 text), analyzer Analyzer, and return a set of representative terms using heuristics like those
 above.  The frequency and length thresholds could be parameters, etc.
 Doug

Initial Usage

This class has lots of options to try to make it efficient and flexible. The simplest possible usage is as follows. The bold fragment is specific to this class. 

 IndexReader ir = ...
 IndexSearcher is = ...

 MoreLikeThis mlt = new MoreLikeThis(ir);
 Reader target = ... // orig source of doc you want to find similarities to
 Query query = mlt.like( target);

 Hits hits = is.search(query);
 // now the usual iteration thru 'hits' - the only thing to watch for is to make sure
 //you ignore the doc if it matches your 'target' document, as it should be similar to itself

Thus you:

  1. do your normal, Lucene setup for searching,
  2. create a MoreLikeThis,
  3. get the text of the doc you want to find similarities to
  4. then call one of the like() calls to generate a similarity query
  5. call the searcher to find the similar docs

More Advanced Usage

You may want to use setFieldNames(...) so you can examine multiple fields (e.g. body and title) for similarity.

Depending on the size of your index and the size and makeup of your documents you may want to call the other set methods to control how the similarity queries are generated: 

 Changes: Mark Harwood 29/02/04
 Some bugfixing, some refactoring, some optimisation.
 - bugfix: retrieveTerms(int docNum) was not working for indexes without a termvector -added missing code
 - bugfix: No significant terms being created for fields with a termvector - because
 was only counting one occurrence per term/field pair in calculations(ie not including frequency info from TermVector)
 - refactor: moved common code into isNoiseWord()
 - optimise: when no termvector support available - used maxNumTermsParsed to limit amount of tokenization

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final boolean
    DEFAULT_BOOST
    Boost terms in query based on score.
    static final String[]
    DEFAULT_FIELD_NAMES
    Default field names.
    static final int
    DEFAULT_MAX_DOC_FREQ
    Ignore words which occur in more than this many docs.
    static final int
    DEFAULT_MAX_NUM_TOKENS_PARSED
    Default maximum number of tokens to parse in each example doc field that is not stored with TermVector support.
    static final int
    DEFAULT_MAX_QUERY_TERMS
    Return a Query with no more than this many terms.
    static final int
    DEFAULT_MAX_WORD_LENGTH
    Ignore words greater than this length or if 0 then this has no effect.
    static final int
    DEFAULT_MIN_DOC_FREQ
    Ignore words which do not occur in at least this many docs.
    static final int
    DEFAULT_MIN_TERM_FREQ
    Ignore terms with less than this frequency in the source doc.
    static final int
    DEFAULT_MIN_WORD_LENGTH
    Ignore words less than this length or if 0 then this has no effect.
    static final Set<?>
    DEFAULT_STOP_WORDS
    Default set of stopwords.

  • Constructor Summary

    Constructors
    Constructor
    Description
    XMoreLikeThis(org.apache.lucene.index.IndexReader ir, org.apache.lucene.search.similarities.TFIDFSimilarity sim)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.lucene.search.Query
    like(String fieldName, Reader... readers)
    Return a query that will return docs like the passed Readers.
    org.apache.lucene.search.Query
    like(org.apache.lucene.index.Fields... likeFields)
    Return a query that will return docs like the passed Fields.
    void
    setAnalyzer(org.apache.lucene.analysis.Analyzer analyzer)
    Sets the analyzer to use.
    void
    setBoost(boolean boost)
    Sets whether to boost terms in query based on "score" or not.
    void
    setBoostFactor(float boostFactor)
    Sets the boost factor to use when boosting terms
    void
    setFieldNames(String[] fieldNames)
    Sets the field names that will be used when generating the 'More Like This' query.
    void
    setMaxDocFreq(int maxFreq)
    Set the maximum frequency in which words may still appear.
    void
    setMaxQueryTerms(int maxQueryTerms)
    Sets the maximum number of query terms that will be included in any generated query.
    void
    setMaxWordLen(int maxWordLen)
    Sets the maximum word length above which words will be ignored.
    void
    setMinDocFreq(int minDocFreq)
    Sets the frequency at which words will be ignored which do not occur in at least this many docs.
    void
    setMinTermFreq(int minTermFreq)
    Sets the frequency below which terms will be ignored in the source doc.
    void
    setMinWordLen(int minWordLen)
    Sets the minimum word length below which words will be ignored.
    void
    setSkipTerms(Set<org.apache.lucene.index.Term> skipTerms)
    Sets a list of terms to never select from
    void
    setStopWords(Set<?> stopWords)
    Set the set of stopwords.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • DEFAULT_MAX_NUM_TOKENS_PARSED

      public static final int DEFAULT_MAX_NUM_TOKENS_PARSED
      Default maximum number of tokens to parse in each example doc field that is not stored with TermVector support.
      See Also:

    • DEFAULT_MIN_TERM_FREQ

      public static final int DEFAULT_MIN_TERM_FREQ
      Ignore terms with less than this frequency in the source doc.
      See Also:

    • DEFAULT_MIN_DOC_FREQ

      public static final int DEFAULT_MIN_DOC_FREQ
      Ignore words which do not occur in at least this many docs.
      See Also:

    • DEFAULT_MAX_DOC_FREQ

      public static final int DEFAULT_MAX_DOC_FREQ
      Ignore words which occur in more than this many docs.
      See Also:

    • DEFAULT_BOOST

      public static final boolean DEFAULT_BOOST
      Boost terms in query based on score.
      See Also:

    • DEFAULT_FIELD_NAMES

      public static final String[] DEFAULT_FIELD_NAMES
      Default field names. Null is used to specify that the field names should be looked up at runtime from the provided reader.

    • DEFAULT_MIN_WORD_LENGTH

      public static final int DEFAULT_MIN_WORD_LENGTH
      Ignore words less than this length or if 0 then this has no effect.
      See Also:

    • DEFAULT_MAX_WORD_LENGTH

      public static final int DEFAULT_MAX_WORD_LENGTH
      Ignore words greater than this length or if 0 then this has no effect.
      See Also:

    • DEFAULT_STOP_WORDS

      public static final Set<?> DEFAULT_STOP_WORDS
      Default set of stopwords. If null means to allow stop words.
      See Also:

    • DEFAULT_MAX_QUERY_TERMS

      public static final int DEFAULT_MAX_QUERY_TERMS
      Return a Query with no more than this many terms.
      See Also:

  • Constructor Details

    • XMoreLikeThis

      public XMoreLikeThis(org.apache.lucene.index.IndexReader ir, org.apache.lucene.search.similarities.TFIDFSimilarity sim)

  • Method Details

    • setBoostFactor

      public void setBoostFactor(float boostFactor)
      Sets the boost factor to use when boosting terms

    • setSkipTerms

      public void setSkipTerms(Set<org.apache.lucene.index.Term> skipTerms)
      Sets a list of terms to never select from

    • setAnalyzer

      public void setAnalyzer(org.apache.lucene.analysis.Analyzer analyzer)
      Sets the analyzer to use. All 'like' methods require an analyzer.
      Parameters:
      analyzer - the analyzer to use to tokenize text.

    • setMinTermFreq

      public void setMinTermFreq(int minTermFreq)
      Sets the frequency below which terms will be ignored in the source doc.
      Parameters:
      minTermFreq - the frequency below which terms will be ignored in the source doc.

    • setMinDocFreq

      public void setMinDocFreq(int minDocFreq)
      Sets the frequency at which words will be ignored which do not occur in at least this many docs.
      Parameters:
      minDocFreq - the frequency at which words will be ignored which do not occur in at least this many docs.

    • setMaxDocFreq

      public void setMaxDocFreq(int maxFreq)
      Set the maximum frequency in which words may still appear. Words that appear in more than this many docs will be ignored.
      Parameters:
      maxFreq - the maximum count of documents that a term may appear in to be still considered relevant

    • setBoost

      public void setBoost(boolean boost)
      Sets whether to boost terms in query based on "score" or not.
      Parameters:
      boost - true to boost terms in query based on "score", false otherwise.

    • setFieldNames

      public void setFieldNames(String[] fieldNames)
      Sets the field names that will be used when generating the 'More Like This' query. Set this to null for the field names to be determined at runtime from the IndexReader provided in the constructor.
      Parameters:
      fieldNames - the field names that will be used when generating the 'More Like This' query.

    • setMinWordLen

      public void setMinWordLen(int minWordLen)
      Sets the minimum word length below which words will be ignored.
      Parameters:
      minWordLen - the minimum word length below which words will be ignored.

    • setMaxWordLen

      public void setMaxWordLen(int maxWordLen)
      Sets the maximum word length above which words will be ignored.
      Parameters:
      maxWordLen - the maximum word length above which words will be ignored.

    • setStopWords

      public void setStopWords(Set<?> stopWords)
      Set the set of stopwords. Any word in this set is considered "uninteresting" and ignored. Even if your Analyzer allows stopwords, you might want to tell the MoreLikeThis code to ignore them, as for the purposes of document similarity it seems reasonable to assume that "a stop word is never interesting".
      Parameters:
      stopWords - set of stopwords, if null it means to allow stop words

    • setMaxQueryTerms

      public void setMaxQueryTerms(int maxQueryTerms)
      Sets the maximum number of query terms that will be included in any generated query.
      Parameters:
      maxQueryTerms - the maximum number of query terms that will be included in any generated query.

    • like

      public org.apache.lucene.search.Query like(String fieldName, Reader... readers) throws IOException
      Return a query that will return docs like the passed Readers. This was added in order to treat multi-value fields.
      Returns:
      a query that will return docs like the passed Readers.
      Throws:
      IOException

    • like

      public org.apache.lucene.search.Query like(org.apache.lucene.index.Fields... likeFields) throws IOException
      Return a query that will return docs like the passed Fields.
      Returns:
      a query that will return docs like the passed Fields.
      Throws:
      IOException