Package org.elasticsearch.index.engine

Class InternalEngine

java.lang.Object

org.elasticsearch.index.engine.Engine

org.elasticsearch.index.engine.InternalEngine

public class InternalEngine
extends Engine

Nested Class Summary

Nested classes/interfaces inherited from class org.elasticsearch.index.engine.Engine

Engine.CommitId, Engine.Delete, Engine.DeleteResult, Engine.EventListener, Engine.Get, Engine.GetResult, Engine.Index, Engine.IndexCommitRef, Engine.IndexResult, Engine.IndexThrottle, Engine.NoOp, Engine.NoOpLock, Engine.NoOpResult, Engine.Operation, Engine.Result, Engine.Searcher, Engine.SearcherScope, Engine.SyncedFlushResult, Engine.Warmer

Field Summary

Fields

Modifier and Type	Field	Description
static java.lang.String	MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID

Fields inherited from class org.elasticsearch.index.engine.Engine

allocationId, engineConfig, eventListener, failedEngine, failEngineLock, HISTORY_UUID_KEY, isClosed, lastWriteNanos, logger, readLock, rwl, shardId, store, SYNC_COMMIT_ID, writeLock

Constructor Summary

Constructors

Constructor	Description
InternalEngine(EngineConfig engineConfig)

Method Summary

Modifier and Type	Method	Description
Engine.IndexCommitRef	acquireIndexCommit(boolean safeCommit, boolean flushFirst)	Snapshots the index and returns a handle to it.
void	activateThrottling()	Request that this engine throttle incoming indexing requests to one thread.
protected boolean	assertOriginPrimarySequenceNumber(long seqNo)
protected void	closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch)	Closes the engine without acquiring the write lock.
protected void	commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog, java.lang.String syncId)	Commits the specified index writer.
void	deactivateThrottling()	Reverses a previous Engine.activateThrottling() call.
Engine.DeleteResult	delete(Engine.Delete delete)	Perform document delete operation on the engine
protected long	doGenerateSeqNoForOperation(Engine.Operation operation)	Generate the sequence number for the specified operation.
int	fillSeqNoGaps(long primaryTerm)	Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.
Engine.CommitId	flush()	Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation.
Engine.CommitId	flush(boolean force, boolean waitIfOngoing)	Flushes the state of the engine including the transaction log, clearing memory.
void	forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments)	Triggers a forced merge on this engine
Engine.GetResult	get(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory)
java.lang.String	getHistoryUUID()	returns the history uuid for the engine
long	getIndexBufferRAMBytesUsed()	How much heap is used that would be freed by a refresh.
long	getIndexThrottleTimeInMillis()	Returns the number of milliseconds this engine was under index throttling.
protected org.apache.lucene.index.SegmentInfos	getLastCommittedSegmentInfos()
LocalCheckpointTracker	getLocalCheckpointTracker()	The sequence number service for this engine.
MergeStats	getMergeStats()
protected org.apache.lucene.search.ReferenceManager<org.apache.lucene.search.IndexSearcher>	getSearcherManager(java.lang.String source, Engine.SearcherScope scope)
Translog	getTranslog()	returns the translog for this engine
long	getWritingBytes()	Returns how many bytes we are currently moving from indexing buffer to segments on disk
Engine.IndexResult	index(Engine.Index index)	Perform document index operation on the engine
boolean	isRecovering()	Returns true iff this engine is currently recovering from translog.
boolean	isThrottled()	Returns the true iff this engine is currently under index throttling.
protected boolean	maybeFailEngine(java.lang.String source, java.lang.Exception e)	Check whether the engine should be failed
Engine.NoOpResult	noOp(Engine.NoOp noOp)
void	onSettingsChanged()
InternalEngine	recoverFromTranslog()	Performs recovery from the transaction log.
void	refresh(java.lang.String source)	Synchronously refreshes the engine for new search operations to reflect the latest changes.
void	restoreLocalCheckpointFromTranslog()	Marks operations in the translog as completed.
void	rollTranslogGeneration()	Rolls the translog generation and cleans unneeded.
java.util.List<Segment>	segments(boolean verbose)	The list of segments in the engine.
boolean	shouldPeriodicallyFlush()	Checks if this engine should be flushed periodically.
void	skipTranslogRecovery()	Do not replay translog operations, but make the engine be ready.
Engine.SyncedFlushResult	syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId)	Attempts to do a special commit where the given syncID is put into the commit data.
void	trimTranslog()	checks and removes translog files that no longer need to be retained.
void	writeIndexingBuffer()	Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.
protected void	writerSegmentStats(SegmentsStats stats)

Methods inherited from class org.elasticsearch.index.engine.Engine

acquireSearcher, acquireSearcher, close, commitStats, config, ensureOpen, failEngine, flushAndClose, forceMerge, getFromSearcher, getLastWriteNanos, getSegmentInfo, guardedRamBytesUsed, isMergedSegment, newSearcher, refreshNeeded, segmentsStats

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID

public static final java.lang.String MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID

See Also:

Constant Field Values

Constructor Detail

InternalEngine

public InternalEngine(EngineConfig engineConfig)

Method Detail

restoreLocalCheckpointFromTranslog

public void restoreLocalCheckpointFromTranslog()
                                        throws java.io.IOException

Description copied from class: Engine

Marks operations in the translog as completed. This is used to restore the state of the local checkpoint tracker on primary promotion.

Specified by:

restoreLocalCheckpointFromTranslog in class Engine

Throws:

java.io.IOException - if an I/O exception occurred reading the translog

fillSeqNoGaps

public int fillSeqNoGaps(long primaryTerm)
                  throws java.io.IOException

Description copied from class: Engine

Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.

Specified by:

fillSeqNoGaps in class Engine

Parameters:

primaryTerm - the shards primary term this engine was created for

Returns:

the number of no-ops added

Throws:

java.io.IOException

recoverFromTranslog

public InternalEngine recoverFromTranslog()
                                   throws java.io.IOException

Description copied from class: Engine

Performs recovery from the transaction log. This operation will close the engine if the recovery fails.

Specified by:

recoverFromTranslog in class Engine

Throws:

java.io.IOException

skipTranslogRecovery

public void skipTranslogRecovery()

Description copied from class: Engine

Do not replay translog operations, but make the engine be ready.

Specified by:

skipTranslogRecovery in class Engine

getTranslog

public Translog getTranslog()

Description copied from class: Engine

returns the translog for this engine

Specified by:

getTranslog in class Engine

getHistoryUUID

public java.lang.String getHistoryUUID()

Description copied from class: Engine

returns the history uuid for the engine

Specified by:

getHistoryUUID in class Engine

getWritingBytes

public long getWritingBytes()

Returns how many bytes we are currently moving from indexing buffer to segments on disk

Specified by:

getWritingBytes in class Engine

get

public Engine.GetResult get(Engine.Get get,
                            java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory)
                     throws EngineException

Specified by:

get in class Engine

Throws:

EngineException

assertOriginPrimarySequenceNumber

protected boolean assertOriginPrimarySequenceNumber(long seqNo)

doGenerateSeqNoForOperation

protected long doGenerateSeqNoForOperation(Engine.Operation operation)

Generate the sequence number for the specified operation.

Parameters:

operation - the operation

Returns:

the sequence number

index

public Engine.IndexResult index(Engine.Index index)
                         throws java.io.IOException

Description copied from class: Engine

Perform document index operation on the engine

Specified by:

index in class Engine

Parameters:

index - operation to perform

Returns:

Engine.IndexResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

Throws:

java.io.IOException

delete

public Engine.DeleteResult delete(Engine.Delete delete)
                           throws java.io.IOException

Description copied from class: Engine

Perform document delete operation on the engine

Specified by:

delete in class Engine

Parameters:

delete - operation to perform

Returns:

Engine.DeleteResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

Throws:

java.io.IOException

noOp

public Engine.NoOpResult noOp(Engine.NoOp noOp)

Specified by:

noOp in class Engine

refresh

public void refresh(java.lang.String source)
             throws EngineException

Description copied from class: Engine

Synchronously refreshes the engine for new search operations to reflect the latest changes.

Specified by:

refresh in class Engine

Throws:

EngineException

writeIndexingBuffer

public void writeIndexingBuffer()
                         throws EngineException

Description copied from class: Engine

Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.

Specified by:

writeIndexingBuffer in class Engine

Throws:

EngineException

syncFlush

public Engine.SyncedFlushResult syncFlush(java.lang.String syncId,
                                          Engine.CommitId expectedCommitId)
                                   throws EngineException

Description copied from class: Engine

Attempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.

Specified by:

syncFlush in class Engine

Parameters:

syncId - id of this sync

expectedCommitId - the expected value of

Returns:

true if the sync commit was made, false o.w.

Throws:

EngineException

shouldPeriodicallyFlush

public boolean shouldPeriodicallyFlush()

Description copied from class: Engine

Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.

Specified by:

shouldPeriodicallyFlush in class Engine

flush

public Engine.CommitId flush()
                      throws EngineException

Description copied from class: Engine

Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation. This operation is not going to block if another flush operation is currently running and won't write a lucene commit if nothing needs to be committed.

Specified by:

flush in class Engine

Returns:

the commit Id for the resulting commit

Throws:

EngineException

flush

public Engine.CommitId flush(boolean force,
                             boolean waitIfOngoing)
                      throws EngineException

Description copied from class: Engine

Flushes the state of the engine including the transaction log, clearing memory.

Specified by:

flush in class Engine

Parameters:

force - if true a lucene commit is executed even if no changes need to be committed.

waitIfOngoing - if true this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.

Returns:

the commit Id for the resulting commit

Throws:

EngineException

rollTranslogGeneration

public void rollTranslogGeneration()
                            throws EngineException

Description copied from class: Engine

Rolls the translog generation and cleans unneeded.

Specified by:

rollTranslogGeneration in class Engine

Throws:

EngineException

trimTranslog

public void trimTranslog()
                  throws EngineException

Description copied from class: Engine

checks and removes translog files that no longer need to be retained. See TranslogDeletionPolicy for details

Specified by:

trimTranslog in class Engine

Throws:

EngineException

forceMerge

public void forceMerge(boolean flush,
                       int maxNumSegments,
                       boolean onlyExpungeDeletes,
                       boolean upgrade,
                       boolean upgradeOnlyAncientSegments)
                throws EngineException,
                       java.io.IOException

Description copied from class: Engine

Triggers a forced merge on this engine

Specified by:

forceMerge in class Engine

Throws:

EngineException

java.io.IOException

acquireIndexCommit

public Engine.IndexCommitRef acquireIndexCommit(boolean safeCommit,
                                                boolean flushFirst)
                                         throws EngineException

Description copied from class: Engine

Snapshots the index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.

Specified by:

acquireIndexCommit in class Engine

Parameters:

safeCommit - indicates whether the engine should acquire the most recent safe commit, or the most recent commit.

flushFirst - indicates whether the engine should flush before returning the snapshot

Throws:

EngineException

maybeFailEngine

protected boolean maybeFailEngine(java.lang.String source,
                                  java.lang.Exception e)

Description copied from class: Engine

Check whether the engine should be failed

Overrides:

maybeFailEngine in class Engine

getLastCommittedSegmentInfos

protected org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()

Specified by:

getLastCommittedSegmentInfos in class Engine

writerSegmentStats

protected final void writerSegmentStats(SegmentsStats stats)

Overrides:

writerSegmentStats in class Engine

getIndexBufferRAMBytesUsed

public long getIndexBufferRAMBytesUsed()

Description copied from class: Engine

How much heap is used that would be freed by a refresh. Note that this may throw AlreadyClosedException.

Specified by:

getIndexBufferRAMBytesUsed in class Engine

segments

public java.util.List<Segment> segments(boolean verbose)

Description copied from class: Engine

The list of segments in the engine.

Specified by:

segments in class Engine

closeNoLock

protected final void closeNoLock(java.lang.String reason,
                                 java.util.concurrent.CountDownLatch closedLatch)

Closes the engine without acquiring the write lock. This should only be called while the write lock is hold or in a disaster condition ie. if the engine is failed.

Specified by:

closeNoLock in class Engine

getSearcherManager

protected org.apache.lucene.search.ReferenceManager<org.apache.lucene.search.IndexSearcher> getSearcherManager(java.lang.String source,
                                                                                                               Engine.SearcherScope scope)

Specified by:

getSearcherManager in class Engine

activateThrottling

public void activateThrottling()

Description copied from class: Engine

Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call to Engine.deactivateThrottling().

Specified by:

activateThrottling in class Engine

deactivateThrottling

public void deactivateThrottling()

Description copied from class: Engine

Reverses a previous Engine.activateThrottling() call.

Specified by:

deactivateThrottling in class Engine

isThrottled

public boolean isThrottled()

Description copied from class: Engine

Returns the true iff this engine is currently under index throttling.

Specified by:

isThrottled in class Engine

See Also:

Engine.getIndexThrottleTimeInMillis()

getIndexThrottleTimeInMillis

public long getIndexThrottleTimeInMillis()

Description copied from class: Engine

Returns the number of milliseconds this engine was under index throttling.

Specified by:

getIndexThrottleTimeInMillis in class Engine

commitIndexWriter

protected void commitIndexWriter(org.apache.lucene.index.IndexWriter writer,
                                 Translog translog,
                                 @Nullable
                                 java.lang.String syncId)
                          throws java.io.IOException

Commits the specified index writer.

Parameters:

writer - the index writer to commit

translog - the translog

syncId - the sync flush ID (null if not committing a synced flush)

Throws:

java.io.IOException - if an I/O exception occurs committing the specfied writer

onSettingsChanged

public void onSettingsChanged()

Overrides:

onSettingsChanged in class Engine

getMergeStats

public MergeStats getMergeStats()

Overrides:

getMergeStats in class Engine

getLocalCheckpointTracker

public final LocalCheckpointTracker getLocalCheckpointTracker()

Description copied from class: Engine

The sequence number service for this engine.

Specified by:

getLocalCheckpointTracker in class Engine

Returns:

the sequence number service

isRecovering

public boolean isRecovering()

Description copied from class: Engine

Returns true iff this engine is currently recovering from translog.

Overrides:

isRecovering in class Engine