org.elasticsearch.index.engine

Class Engine

java.lang.Object

org.elasticsearch.index.engine.Engine

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

Direct Known Subclasses:

InternalEngine, ShadowEngine

public abstract class Engine
extends java.lang.Object
implements java.io.Closeable

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Engine.CommitId
static class	Engine.Delete
static class	Engine.DeleteResult
static interface	Engine.EventListener
static class	Engine.Get
static class	Engine.GetResult
static class	Engine.Index
static class	Engine.IndexResult
protected static class	Engine.IndexThrottle A throttling class that can be activated, causing the acquireThrottle method to block on a lock when throttling is enabled
protected static class	Engine.NoOpLock A Lock implementation that always allows the lock to be acquired
static class	Engine.Operation
static class	Engine.Result Base class for index and delete operation results Holds result meta data (e.g.
static class	Engine.Searcher
static class	Engine.SyncedFlushResult
static interface	Engine.Warmer Called for each new opened engine searcher to warm new segments

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.lucene.index.SnapshotDeletionPolicy	deletionPolicy
protected EngineConfig	engineConfig
protected Engine.EventListener	eventListener
protected org.apache.lucene.util.SetOnce<java.lang.Exception>	failedEngine
protected java.util.concurrent.locks.ReentrantLock	failEngineLock
protected java.util.concurrent.atomic.AtomicBoolean	isClosed
protected long	lastWriteNanos
protected org.apache.logging.log4j.Logger	logger
protected ReleasableLock	readLock
protected java.util.concurrent.locks.ReentrantReadWriteLock	rwl
protected ShardId	shardId
protected Store	store
static java.lang.String	SYNC_COMMIT_ID
protected ReleasableLock	writeLock

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	Engine(EngineConfig engineConfig)

Method Summary

Modifier and Type	Method and Description
abstract org.apache.lucene.index.IndexCommit	acquireIndexCommit(boolean flushFirst) Snapshots the index and returns a handle to it.
Engine.Searcher	acquireSearcher(java.lang.String source) Returns a new searcher instance.
abstract void	activateThrottling() Request that this engine throttle incoming indexing requests to one thread.
void	close()
protected abstract void	closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch) Method to close the engine while the write lock is held.
CommitStats	commitStats() get commits stats for the last commit
EngineConfig	config()
abstract void	deactivateThrottling() Reverses a previous activateThrottling() call.
abstract Engine.DeleteResult	delete(Engine.Delete delete) Perform document delete operation on the engine
protected void	ensureOpen()
void	failEngine(java.lang.String reason, java.lang.Exception failure) fail engine due to some error.
abstract Engine.CommitId	flush() Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation.
abstract Engine.CommitId	flush(boolean force, boolean waitIfOngoing) Flushes the state of the engine including the transaction log, clearing memory.
void	flushAndClose() Flush the engine (committing segments to disk and truncating the translog) and close it.
void	forceMerge(boolean flush) Force merges to 1 segment
abstract void	forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments) Triggers a forced merge on this engine
abstract Engine.GetResult	get(Engine.Get get, java.util.function.Function<java.lang.String,Engine.Searcher> searcherFactory)
protected Engine.GetResult	getFromSearcher(Engine.Get get, java.util.function.Function<java.lang.String,Engine.Searcher> searcherFactory)
abstract long	getIndexBufferRAMBytesUsed() How much heap is used that would be freed by a refresh.
abstract long	getIndexThrottleTimeInMillis() Returns the number of milliseconds this engine was under index throttling.
protected abstract org.apache.lucene.index.SegmentInfos	getLastCommittedSegmentInfos()
long	getLastWriteNanos() Returns the timestamp of the last write in nanoseconds.
MergeStats	getMergeStats()
protected abstract org.apache.lucene.search.SearcherManager	getSearcherManager()
protected Segment[]	getSegmentInfo(org.apache.lucene.index.SegmentInfos lastCommittedSegmentInfos, boolean verbose)
abstract Translog	getTranslog() returns the translog for this engine
protected static long	guardedRamBytesUsed(org.apache.lucene.util.Accountable a) Returns 0 in the case where accountable is null, otherwise returns ramBytesUsed()
abstract Engine.IndexResult	index(Engine.Index index) Perform document index operation on the engine
protected static boolean	isMergedSegment(org.apache.lucene.index.LeafReader reader) Returns whether a leaf reader comes from a merge (versus flush or addIndexes).
boolean	isRecovering() Returns true iff this engine is currently recovering from translog.
abstract boolean	isThrottled() Returns the true iff this engine is currently under index throttling.
protected boolean	maybeFailEngine(java.lang.String source, java.lang.Exception e) Check whether the engine should be failed
protected Engine.Searcher	newSearcher(java.lang.String source, org.apache.lucene.search.IndexSearcher searcher, org.apache.lucene.search.SearcherManager manager)
void	onSettingsChanged()
protected static org.apache.lucene.index.SegmentInfos	readLastCommittedSegmentInfos(org.apache.lucene.search.SearcherManager sm, Store store) Read the last segments info from the commit pointed to by the searcher manager
abstract Engine	recoverFromTranslog() Performs recovery from the transaction log.
abstract void	refresh(java.lang.String source) Synchronously refreshes the engine for new search operations to reflect the latest changes.
boolean	refreshNeeded()
protected static org.apache.lucene.index.SegmentReader	segmentReader(org.apache.lucene.index.LeafReader reader) Tries to extract a segment reader from the given index reader.
abstract java.util.List<Segment>	segments(boolean verbose) The list of segments in the engine.
SegmentsStats	segmentsStats(boolean includeSegmentFileSizes) Global stats on segments.
abstract Engine.SyncedFlushResult	syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId) Attempts to do a special commit where the given syncID is put into the commit data.
abstract void	writeIndexingBuffer() Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.
protected void	writerSegmentStats(SegmentsStats stats)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SYNC_COMMIT_ID

public static final java.lang.String SYNC_COMMIT_ID

See Also:

Constant Field Values

shardId

protected final ShardId shardId

logger

protected final org.apache.logging.log4j.Logger logger

engineConfig

protected final EngineConfig engineConfig

store

protected final Store store

isClosed

protected final java.util.concurrent.atomic.AtomicBoolean isClosed

eventListener

protected final Engine.EventListener eventListener

deletionPolicy

protected final org.apache.lucene.index.SnapshotDeletionPolicy deletionPolicy

failEngineLock

protected final java.util.concurrent.locks.ReentrantLock failEngineLock

rwl

protected final java.util.concurrent.locks.ReentrantReadWriteLock rwl

readLock

protected final ReleasableLock readLock

writeLock

protected final ReleasableLock writeLock

failedEngine

protected final org.apache.lucene.util.SetOnce<java.lang.Exception> failedEngine

lastWriteNanos

protected volatile long lastWriteNanos

Constructor Detail

Engine

protected Engine(EngineConfig engineConfig)

Method Detail

guardedRamBytesUsed

protected static long guardedRamBytesUsed(org.apache.lucene.util.Accountable a)

Returns 0 in the case where accountable is null, otherwise returns ramBytesUsed()

segmentReader

protected static org.apache.lucene.index.SegmentReader segmentReader(org.apache.lucene.index.LeafReader reader)

Tries to extract a segment reader from the given index reader. If no SegmentReader can be extracted an IllegalStateException is thrown.

isMergedSegment

protected static boolean isMergedSegment(org.apache.lucene.index.LeafReader reader)

Returns whether a leaf reader comes from a merge (versus flush or addIndexes).

newSearcher

protected Engine.Searcher newSearcher(java.lang.String source,
                                      org.apache.lucene.search.IndexSearcher searcher,
                                      org.apache.lucene.search.SearcherManager manager)

config

public final EngineConfig config()

getLastCommittedSegmentInfos

protected abstract org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()

getMergeStats

public MergeStats getMergeStats()

getIndexThrottleTimeInMillis

public abstract long getIndexThrottleTimeInMillis()

Returns the number of milliseconds this engine was under index throttling.

isThrottled

public abstract boolean isThrottled()

Returns the true iff this engine is currently under index throttling.

See Also:

getIndexThrottleTimeInMillis()

index

public abstract Engine.IndexResult index(Engine.Index index)
                                  throws java.io.IOException

Perform document index operation on the engine

Parameters:

index - operation to perform

Returns:

Engine.IndexResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

Throws:

java.io.IOException

delete

public abstract Engine.DeleteResult delete(Engine.Delete delete)
                                    throws java.io.IOException

Perform document delete operation on the engine

Parameters:

delete - operation to perform

Returns:

Engine.DeleteResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

Throws:

java.io.IOException

syncFlush

public abstract Engine.SyncedFlushResult syncFlush(java.lang.String syncId,
                                                   Engine.CommitId expectedCommitId)
                                            throws EngineException

Attempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.

Parameters:

syncId - id of this sync

expectedCommitId - the expected value of

Returns:

true if the sync commit was made, false o.w.

Throws:

EngineException

getFromSearcher

protected final Engine.GetResult getFromSearcher(Engine.Get get,
                                                 java.util.function.Function<java.lang.String,Engine.Searcher> searcherFactory)
                                          throws EngineException

Throws:

EngineException

get

public abstract Engine.GetResult get(Engine.Get get,
                                     java.util.function.Function<java.lang.String,Engine.Searcher> searcherFactory)
                              throws EngineException

Throws:

EngineException

acquireSearcher

public final Engine.Searcher acquireSearcher(java.lang.String source)
                                      throws EngineException

Returns a new searcher instance. The consumer of this API is responsible for releasing the returned seacher in a safe manner, preferably in a try/finally block.

Throws:

EngineException

See Also:

Engine.Searcher.close()

getTranslog

public abstract Translog getTranslog()

returns the translog for this engine

ensureOpen

protected void ensureOpen()

commitStats

public CommitStats commitStats()

get commits stats for the last commit

readLastCommittedSegmentInfos

protected static org.apache.lucene.index.SegmentInfos readLastCommittedSegmentInfos(org.apache.lucene.search.SearcherManager sm,
                                                                                    Store store)
                                                                             throws java.io.IOException

Read the last segments info from the commit pointed to by the searcher manager

Throws:

java.io.IOException

segmentsStats

public final SegmentsStats segmentsStats(boolean includeSegmentFileSizes)

Global stats on segments.

writerSegmentStats

protected void writerSegmentStats(SegmentsStats stats)

getIndexBufferRAMBytesUsed

public abstract long getIndexBufferRAMBytesUsed()

How much heap is used that would be freed by a refresh. Note that this may throw AlreadyClosedException.

getSegmentInfo

protected Segment[] getSegmentInfo(org.apache.lucene.index.SegmentInfos lastCommittedSegmentInfos,
                                   boolean verbose)

segments

public abstract java.util.List<Segment> segments(boolean verbose)

The list of segments in the engine.

refreshNeeded

public final boolean refreshNeeded()

refresh

@Nullable
public abstract void refresh(java.lang.String source)
                                throws EngineException

Synchronously refreshes the engine for new search operations to reflect the latest changes.

Throws:

EngineException

writeIndexingBuffer

public abstract void writeIndexingBuffer()
                                  throws EngineException

Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.

Throws:

EngineException

flush

public abstract Engine.CommitId flush(boolean force,
                                      boolean waitIfOngoing)
                               throws EngineException

Flushes the state of the engine including the transaction log, clearing memory.

Parameters:

force - if true a lucene commit is executed even if no changes need to be committed.

waitIfOngoing - if true this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.

Returns:

the commit Id for the resulting commit

Throws:

EngineException

flush

public abstract Engine.CommitId flush()
                               throws EngineException

Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation. This operation is not going to block if another flush operation is currently running and won't write a lucene commit if nothing needs to be committed.

Returns:

the commit Id for the resulting commit

Throws:

EngineException

forceMerge

public void forceMerge(boolean flush)
                throws java.io.IOException

Force merges to 1 segment

Throws:

java.io.IOException

forceMerge

public abstract void forceMerge(boolean flush,
                                int maxNumSegments,
                                boolean onlyExpungeDeletes,
                                boolean upgrade,
                                boolean upgradeOnlyAncientSegments)
                         throws EngineException,
                                java.io.IOException

Triggers a forced merge on this engine

Throws:

EngineException

java.io.IOException

acquireIndexCommit

public abstract org.apache.lucene.index.IndexCommit acquireIndexCommit(boolean flushFirst)
                                                                throws EngineException

Snapshots the index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.

Parameters:

flushFirst - indicates whether the engine should flush before returning the snapshot

Throws:

EngineException

failEngine

public void failEngine(java.lang.String reason,
                       @Nullable
                       java.lang.Exception failure)

fail engine due to some error. the engine will also be closed. The underlying store is marked corrupted iff failure is caused by index corruption

maybeFailEngine

protected boolean maybeFailEngine(java.lang.String source,
                                  java.lang.Exception e)

Check whether the engine should be failed

getSearcherManager

protected abstract org.apache.lucene.search.SearcherManager getSearcherManager()

closeNoLock

protected abstract void closeNoLock(java.lang.String reason,
                                    java.util.concurrent.CountDownLatch closedLatch)

Method to close the engine while the write lock is held. Must decrement the supplied when closing work is done and resources are freed.

flushAndClose

public void flushAndClose()
                   throws java.io.IOException

Flush the engine (committing segments to disk and truncating the translog) and close it.

Throws:

java.io.IOException

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

onSettingsChanged

public void onSettingsChanged()

getLastWriteNanos

public long getLastWriteNanos()

Returns the timestamp of the last write in nanoseconds. Note: this time might not be absolutely accurate since the Engine.Operation.startTime() is used which might be slightly inaccurate.

See Also:

System.nanoTime(), Engine.Operation.startTime()

activateThrottling

public abstract void activateThrottling()

Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call to deactivateThrottling().

deactivateThrottling

public abstract void deactivateThrottling()

Reverses a previous activateThrottling() call.

recoverFromTranslog

public abstract Engine recoverFromTranslog()
                                    throws java.io.IOException

Performs recovery from the transaction log. This operation will close the engine if the recovery fails.

Throws:

java.io.IOException

isRecovering

public boolean isRecovering()

Returns true iff this engine is currently recovering from translog.