Package org.elasticsearch.index.engine

Class InternalEngine

java.lang.Object

org.elasticsearch.index.engine.Engine

org.elasticsearch.index.engine.InternalEngine

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class InternalEngine
extends Engine

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
protected static class	InternalEngine.DeletionStrategy
protected static class	InternalEngine.IndexingStrategy

Nested classes/interfaces inherited from class org.elasticsearch.index.engine.Engine

Engine.CommitId, Engine.Delete, Engine.DeleteResult, Engine.EventListener, Engine.Get, Engine.GetResult, Engine.HistorySource, Engine.Index, Engine.IndexCommitRef, Engine.IndexResult, Engine.IndexThrottle, Engine.NoOp, Engine.NoOpLock, Engine.NoOpResult, Engine.Operation, Engine.Result, Engine.Searcher, Engine.SearcherScope, Engine.SyncedFlushResult, Engine.TranslogRecoveryRunner, Engine.Warmer

Field Summary

Fields inherited from class org.elasticsearch.index.engine.Engine

allocationId, engineConfig, eventListener, failedEngine, failEngineLock, HISTORY_UUID_KEY, isClosed, lastWriteNanos, logger, MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID, MIN_RETAINED_SEQNO, readLock, rwl, shardId, store, SYNC_COMMIT_ID, writeLock

Constructor Summary

Constructors

Constructor	Description
InternalEngine(EngineConfig engineConfig)

Method Summary

Modifier and Type	Method	Description
java.io.Closeable	acquireHistoryRetentionLock(Engine.HistorySource historySource)	Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed
Engine.IndexCommitRef	acquireLastIndexCommit(boolean flushFirst)	Snapshots the most recent index and returns a handle to it.
Engine.IndexCommitRef	acquireSafeIndexCommit()	Snapshots the most recent safe index commit from the engine.
void	activateThrottling()	Request that this engine throttle incoming indexing requests to one thread.
void	advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary)	A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.
protected void	advanceMaxSeqNoOfUpdatesOrDeletesOnPrimary(long seqNo)
protected boolean	assertNonPrimaryOrigin(Engine.Operation operation)
protected boolean	assertPrimaryCanOptimizeAddDocument(Engine.Index index)
protected boolean	assertPrimaryIncomingSequenceNumber(Engine.Operation.Origin origin, long seqNo)
protected void	closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch)	Closes the engine without acquiring the write lock.
protected void	commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog, java.lang.String syncId)	Commits the specified index writer.
void	deactivateThrottling()	Reverses a previous Engine.activateThrottling() call.
Engine.DeleteResult	delete(Engine.Delete delete)	Perform document delete operation on the engine
protected InternalEngine.DeletionStrategy	deletionStrategyForOperation(Engine.Delete delete)
boolean	ensureTranslogSynced(java.util.stream.Stream<Translog.Location> locations)	Ensures that all locations in the given stream have been written to the underlying storage.
int	estimateNumberOfHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo)	Returns the estimated number of history operations whose seq# at least the provided seq# in this engine.
int	fillSeqNoGaps(long primaryTerm)	Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.
Engine.CommitId	flush(boolean force, boolean waitIfOngoing)	Flushes the state of the engine including the transaction log, clearing memory.
void	forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments)	Triggers a forced merge on this engine
protected long	generateSeqNoForOperationOnPrimary(Engine.Operation operation)
Engine.GetResult	get(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory)
java.lang.String	getHistoryUUID()	returns the history uuid for the engine
long	getIndexBufferRAMBytesUsed()	How much heap is used that would be freed by a refresh.
long	getIndexThrottleTimeInMillis()	Returns the number of milliseconds this engine was under index throttling.
protected org.apache.lucene.index.SegmentInfos	getLastCommittedSegmentInfos()
long	getLastSyncedGlobalCheckpoint()	Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e.
long	getMaxSeenAutoIdTimestamp()	Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard via Engine.updateMaxUnsafeAutoIdTimestamp(long).
long	getMaxSeqNoOfUpdatesOrDeletes()	Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number from Engine.advanceMaxSeqNoOfUpdatesOrDeletes(long).
MergeStats	getMergeStats()
long	getMinRetainedSeqNo()	Returns the minimum seqno that is retained in the Lucene index.
long	getPersistedLocalCheckpoint()
long	getProcessedLocalCheckpoint()
protected org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader>	getReferenceManager(Engine.SearcherScope scope)
SafeCommitInfo	getSafeCommitInfo()
SeqNoStats	getSeqNoStats(long globalCheckpoint)
Translog.Location	getTranslogLastWriteLocation()	Returns the last location that the translog of this engine has written into.
TranslogStats	getTranslogStats()
long	getWritingBytes()	Returns how many bytes we are currently moving from indexing buffer to segments on disk
protected boolean	hasBeenProcessedBefore(Engine.Operation op)	Checks if the given operation has been processed in this engine or not.
boolean	hasCompleteOperationHistory(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo)	Checks if this engine has every operations since startingSeqNo(inclusive) in its history (either Lucene or translog)
Engine.IndexResult	index(Engine.Index index)	Perform document index operation on the engine
protected InternalEngine.IndexingStrategy	indexingStrategyForOperation(Engine.Index index)
boolean	isThrottled()	Returns the true iff this engine is currently under index throttling.
boolean	isTranslogSyncNeeded()	Checks if the underlying storage sync is required.
protected void	markSeqNoAsSeen(long seqNo)	Marks the given seq_no as seen and advances the max_seq_no of this engine to at least that value.
protected boolean	maybeFailEngine(java.lang.String source, java.lang.Exception e)	Check whether the engine should be failed
void	maybePruneDeletes()	Tries to prune buffered deletes from the version map.
boolean	maybeRefresh(java.lang.String source)	Synchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.
Translog.Snapshot	newChangesSnapshot(java.lang.String source, MapperService mapperService, long fromSeqNo, long toSeqNo, boolean requiredFullRange)	Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive).
Engine.NoOpResult	noOp(Engine.NoOp noOp)
void	onSettingsChanged(TimeValue translogRetentionAge, ByteSizeValue translogRetentionSize, long softDeletesRetentionOps)
protected InternalEngine.DeletionStrategy	planDeletionAsNonPrimary(Engine.Delete delete)
protected InternalEngine.IndexingStrategy	planIndexingAsNonPrimary(Engine.Index index)
protected java.util.Optional<java.lang.Exception>	preFlightCheckForNoOp(Engine.NoOp noOp)	Executes a pre-flight check for a given NoOp.
Translog.Snapshot	readHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo)	Creates a new history snapshot for reading operations since the provided seqno.
InternalEngine	recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo)	Performs recovery from the transaction log up to recoverUpToSeqNo (inclusive).
void	refresh(java.lang.String source)	Synchronously refreshes the engine for new search operations to reflect the latest changes.
protected void	refreshIfNeeded(java.lang.String source, long requestingSeqNo)	Refresh this engine **internally** iff the requesting seq_no is greater than the last refreshed checkpoint.
int	restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner)	This method replays translog to restore the Lucene index which might be reverted previously.
void	rollTranslogGeneration()	Rolls the translog generation and cleans unneeded.
java.util.List<Segment>	segments(boolean verbose)	The list of segments in the engine.
boolean	shouldPeriodicallyFlush()	Checks if this engine should be flushed periodically.
boolean	shouldRollTranslogGeneration()	Tests whether or not the translog generation should be rolled to a new generation.
void	skipTranslogRecovery()	Do not replay translog operations, but make the engine be ready.
Engine.SyncedFlushResult	syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId)	Attempts to do a special commit where the given syncID is put into the commit data.
void	syncTranslog()
void	trimOperationsFromTranslog(long belowTerm, long aboveSeqNo)	Trims translog for terms below belowTerm and seq# above aboveSeqNo
void	trimUnreferencedTranslogFiles()	checks and removes translog files that no longer need to be retained.
void	updateMaxUnsafeAutoIdTimestamp(long newTimestamp)	Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp.
void	writeIndexingBuffer()	Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.
protected void	writerSegmentStats(SegmentsStats stats)

Methods inherited from class org.elasticsearch.index.engine.Engine

acquireSearcher, acquireSearcher, close, commitStats, completionStats, config, docsStats, docStats, ensureOpen, ensureOpen, failEngine, fillSegmentStats, flush, flushAndClose, forceMerge, getFromSearcher, getLastWriteNanos, guardedRamBytesUsed, refreshNeeded, segmentsStats, verifyEngineBeforeIndexClosing

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

InternalEngine

public InternalEngine(EngineConfig engineConfig)

Method Details

restoreLocalHistoryFromTranslog

public int restoreLocalHistoryFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner) throws java.io.IOException

Description copied from class: Engine

This method replays translog to restore the Lucene index which might be reverted previously. This ensures that all acknowledged writes are restored correctly when this engine is promoted.

Specified by:

restoreLocalHistoryFromTranslog in class Engine

Returns:

the number of translog operations have been recovered

Throws:

java.io.IOException

fillSeqNoGaps

public int fillSeqNoGaps(long primaryTerm) throws java.io.IOException

Description copied from class: Engine

Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.

Specified by:

fillSeqNoGaps in class Engine

Parameters:

primaryTerm - the shards primary term this engine was created for

Returns:

the number of no-ops added

Throws:

java.io.IOException

recoverFromTranslog

public InternalEngine recoverFromTranslog(Engine.TranslogRecoveryRunner translogRecoveryRunner, long recoverUpToSeqNo) throws java.io.IOException

Description copied from class: Engine

Performs recovery from the transaction log up to recoverUpToSeqNo (inclusive). This operation will close the engine if the recovery fails.

Specified by:

recoverFromTranslog in class Engine

Parameters:

translogRecoveryRunner - the translog recovery runner

recoverUpToSeqNo - the upper bound, inclusive, of sequence number to be recovered

Throws:

java.io.IOException

skipTranslogRecovery

public void skipTranslogRecovery()

Description copied from class: Engine

Do not replay translog operations, but make the engine be ready.

Specified by:

skipTranslogRecovery in class Engine

isTranslogSyncNeeded

public boolean isTranslogSyncNeeded()

Description copied from class: Engine

Checks if the underlying storage sync is required.

Specified by:

isTranslogSyncNeeded in class Engine

ensureTranslogSynced

public boolean ensureTranslogSynced(java.util.stream.Stream<Translog.Location> locations) throws java.io.IOException

Description copied from class: Engine

Ensures that all locations in the given stream have been written to the underlying storage.

Specified by:

ensureTranslogSynced in class Engine

Throws:

java.io.IOException

syncTranslog

public void syncTranslog() throws java.io.IOException

Specified by:

syncTranslog in class Engine

Throws:

java.io.IOException

readHistoryOperations

public Translog.Snapshot readHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws java.io.IOException

Creates a new history snapshot for reading operations since the provided seqno. The returned snapshot can be retrieved from either Lucene index or translog files.

Specified by:

readHistoryOperations in class Engine

Throws:

java.io.IOException

estimateNumberOfHistoryOperations

public int estimateNumberOfHistoryOperations(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws java.io.IOException

Returns the estimated number of history operations whose seq# at least the provided seq# in this engine.

Specified by:

estimateNumberOfHistoryOperations in class Engine

Throws:

java.io.IOException

getTranslogStats

public TranslogStats getTranslogStats()

Specified by:

getTranslogStats in class Engine

getTranslogLastWriteLocation

public Translog.Location getTranslogLastWriteLocation()

Description copied from class: Engine

Returns the last location that the translog of this engine has written into.

Specified by:

getTranslogLastWriteLocation in class Engine

getHistoryUUID

public java.lang.String getHistoryUUID()

Description copied from class: Engine

returns the history uuid for the engine

Specified by:

getHistoryUUID in class Engine

getWritingBytes

public long getWritingBytes()

Returns how many bytes we are currently moving from indexing buffer to segments on disk

Specified by:

getWritingBytes in class Engine

get

public Engine.GetResult get(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory) throws EngineException

Specified by:

get in class Engine

Throws:

EngineException

assertPrimaryCanOptimizeAddDocument

protected boolean assertPrimaryCanOptimizeAddDocument(Engine.Index index)

assertPrimaryIncomingSequenceNumber

protected boolean assertPrimaryIncomingSequenceNumber(Engine.Operation.Origin origin, long seqNo)

generateSeqNoForOperationOnPrimary

protected long generateSeqNoForOperationOnPrimary(Engine.Operation operation)

advanceMaxSeqNoOfUpdatesOrDeletesOnPrimary

protected void advanceMaxSeqNoOfUpdatesOrDeletesOnPrimary(long seqNo)

index

public Engine.IndexResult index(Engine.Index index) throws java.io.IOException

Description copied from class: Engine

Perform document index operation on the engine

Specified by:

index in class Engine

Parameters:

index - operation to perform

Returns:

Engine.IndexResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

Throws:

java.io.IOException

planIndexingAsNonPrimary

protected final InternalEngine.IndexingStrategy planIndexingAsNonPrimary(Engine.Index index) throws java.io.IOException

Throws:

java.io.IOException

indexingStrategyForOperation

protected InternalEngine.IndexingStrategy indexingStrategyForOperation(Engine.Index index) throws java.io.IOException

Throws:

java.io.IOException

delete

public Engine.DeleteResult delete(Engine.Delete delete) throws java.io.IOException

Description copied from class: Engine

Perform document delete operation on the engine

Specified by:

delete in class Engine

Parameters:

delete - operation to perform

Returns:

Engine.DeleteResult containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown

Throws:

java.io.IOException

deletionStrategyForOperation

protected InternalEngine.DeletionStrategy deletionStrategyForOperation(Engine.Delete delete) throws java.io.IOException

Throws:

java.io.IOException

planDeletionAsNonPrimary

protected final InternalEngine.DeletionStrategy planDeletionAsNonPrimary(Engine.Delete delete) throws java.io.IOException

Throws:

java.io.IOException

assertNonPrimaryOrigin

protected boolean assertNonPrimaryOrigin(Engine.Operation operation)

maybePruneDeletes

public void maybePruneDeletes()

Description copied from class: Engine

Tries to prune buffered deletes from the version map.

Specified by:

maybePruneDeletes in class Engine

noOp

public Engine.NoOpResult noOp(Engine.NoOp noOp) throws java.io.IOException

Specified by:

noOp in class Engine

Throws:

java.io.IOException

preFlightCheckForNoOp

protected java.util.Optional<java.lang.Exception> preFlightCheckForNoOp(Engine.NoOp noOp) throws java.io.IOException

Executes a pre-flight check for a given NoOp. If this method returns a non-empty result, the engine won't process this NoOp and returns a failure.

Throws:

java.io.IOException

refresh

public void refresh(java.lang.String source) throws EngineException

Description copied from class: Engine

Synchronously refreshes the engine for new search operations to reflect the latest changes.

Specified by:

refresh in class Engine

Throws:

EngineException

maybeRefresh

public boolean maybeRefresh(java.lang.String source) throws EngineException

Description copied from class: Engine

Synchronously refreshes the engine for new search operations to reflect the latest changes unless another thread is already refreshing the engine concurrently.

Specified by:

maybeRefresh in class Engine

Returns:

true if the a refresh happened. Otherwise false

Throws:

EngineException

writeIndexingBuffer

public void writeIndexingBuffer() throws EngineException

Description copied from class: Engine

Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.

Specified by:

writeIndexingBuffer in class Engine

Throws:

EngineException

syncFlush

public Engine.SyncedFlushResult syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId) throws EngineException

Description copied from class: Engine

Attempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.

Specified by:

syncFlush in class Engine

Parameters:

syncId - id of this sync

expectedCommitId - the expected value of

Returns:

true if the sync commit was made, false o.w.

Throws:

EngineException

shouldPeriodicallyFlush

public boolean shouldPeriodicallyFlush()

Description copied from class: Engine

Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.

Specified by:

shouldPeriodicallyFlush in class Engine

flush

public Engine.CommitId flush(boolean force, boolean waitIfOngoing) throws EngineException

Description copied from class: Engine

Flushes the state of the engine including the transaction log, clearing memory.

Specified by:

flush in class Engine

Parameters:

force - if true a lucene commit is executed even if no changes need to be committed.

waitIfOngoing - if true this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.

Returns:

the commit Id for the resulting commit

Throws:

EngineException

rollTranslogGeneration

public void rollTranslogGeneration() throws EngineException

Description copied from class: Engine

Rolls the translog generation and cleans unneeded.

Specified by:

rollTranslogGeneration in class Engine

Throws:

EngineException

trimUnreferencedTranslogFiles

public void trimUnreferencedTranslogFiles() throws EngineException

Description copied from class: Engine

checks and removes translog files that no longer need to be retained. See TranslogDeletionPolicy for details

Specified by:

trimUnreferencedTranslogFiles in class Engine

Throws:

EngineException

shouldRollTranslogGeneration

public boolean shouldRollTranslogGeneration()

Description copied from class: Engine

Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.

Specified by:

shouldRollTranslogGeneration in class Engine

Returns:

true if the current generation should be rolled to a new generation

trimOperationsFromTranslog

public void trimOperationsFromTranslog(long belowTerm, long aboveSeqNo) throws EngineException

Description copied from class: Engine

Trims translog for terms below belowTerm and seq# above aboveSeqNo

Specified by:

trimOperationsFromTranslog in class Engine

Throws:

EngineException

See Also:

Translog.trimOperations(long, long)

forceMerge

public void forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments) throws EngineException, java.io.IOException

Description copied from class: Engine

Triggers a forced merge on this engine

Specified by:

forceMerge in class Engine

Throws:

EngineException

java.io.IOException

acquireLastIndexCommit

public Engine.IndexCommitRef acquireLastIndexCommit(boolean flushFirst) throws EngineException

Description copied from class: Engine

Snapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.

Specified by:

acquireLastIndexCommit in class Engine

Parameters:

flushFirst - indicates whether the engine should flush before returning the snapshot

Throws:

EngineException

acquireSafeIndexCommit

public Engine.IndexCommitRef acquireSafeIndexCommit() throws EngineException

Description copied from class: Engine

Snapshots the most recent safe index commit from the engine.

Specified by:

acquireSafeIndexCommit in class Engine

Throws:

EngineException

getSafeCommitInfo

public SafeCommitInfo getSafeCommitInfo()

Specified by:

getSafeCommitInfo in class Engine

Returns:

a summary of the contents of the current safe commit

maybeFailEngine

protected boolean maybeFailEngine(java.lang.String source, java.lang.Exception e)

Description copied from class: Engine

Check whether the engine should be failed

Overrides:

maybeFailEngine in class Engine

getLastCommittedSegmentInfos

protected org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()

Specified by:

getLastCommittedSegmentInfos in class Engine

writerSegmentStats

protected final void writerSegmentStats(SegmentsStats stats)

Overrides:

writerSegmentStats in class Engine

getIndexBufferRAMBytesUsed

public long getIndexBufferRAMBytesUsed()

Description copied from class: Engine

How much heap is used that would be freed by a refresh. Note that this may throw AlreadyClosedException.

Specified by:

getIndexBufferRAMBytesUsed in class Engine

segments

public java.util.List<Segment> segments(boolean verbose)

Description copied from class: Engine

The list of segments in the engine.

Specified by:

segments in class Engine

closeNoLock

protected final void closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch)

Closes the engine without acquiring the write lock. This should only be called while the write lock is hold or in a disaster condition ie. if the engine is failed.

Specified by:

closeNoLock in class Engine

getReferenceManager

protected final org.apache.lucene.search.ReferenceManager<ElasticsearchDirectoryReader> getReferenceManager(Engine.SearcherScope scope)

Specified by:

getReferenceManager in class Engine

activateThrottling

public void activateThrottling()

Description copied from class: Engine

Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call to Engine.deactivateThrottling().

Specified by:

activateThrottling in class Engine

deactivateThrottling

public void deactivateThrottling()

Description copied from class: Engine

Reverses a previous Engine.activateThrottling() call.

Specified by:

deactivateThrottling in class Engine

isThrottled

public boolean isThrottled()

Description copied from class: Engine

Returns the true iff this engine is currently under index throttling.

Specified by:

isThrottled in class Engine

See Also:

Engine.getIndexThrottleTimeInMillis()

getIndexThrottleTimeInMillis

public long getIndexThrottleTimeInMillis()

Description copied from class: Engine

Returns the number of milliseconds this engine was under index throttling.

Specified by:

getIndexThrottleTimeInMillis in class Engine

commitIndexWriter

protected void commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog, @Nullable java.lang.String syncId) throws java.io.IOException

Commits the specified index writer.

Parameters:

writer - the index writer to commit

translog - the translog

syncId - the sync flush ID (null if not committing a synced flush)

Throws:

java.io.IOException - if an I/O exception occurs committing the specfied writer

onSettingsChanged

public void onSettingsChanged(TimeValue translogRetentionAge, ByteSizeValue translogRetentionSize, long softDeletesRetentionOps)

Overrides:

onSettingsChanged in class Engine

getMergeStats

public MergeStats getMergeStats()

Overrides:

getMergeStats in class Engine

getLastSyncedGlobalCheckpoint

public long getLastSyncedGlobalCheckpoint()

Description copied from class: Engine

Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)

Specified by:

getLastSyncedGlobalCheckpoint in class Engine

getProcessedLocalCheckpoint

public long getProcessedLocalCheckpoint()

getPersistedLocalCheckpoint

public long getPersistedLocalCheckpoint()

Specified by:

getPersistedLocalCheckpoint in class Engine

Returns:

the persisted local checkpoint for this Engine

markSeqNoAsSeen

protected final void markSeqNoAsSeen(long seqNo)

Marks the given seq_no as seen and advances the max_seq_no of this engine to at least that value.

hasBeenProcessedBefore

protected final boolean hasBeenProcessedBefore(Engine.Operation op)

Checks if the given operation has been processed in this engine or not.

Returns:

true if the given operation was processed; otherwise false.

getSeqNoStats

public SeqNoStats getSeqNoStats(long globalCheckpoint)

Specified by:

getSeqNoStats in class Engine

Returns:

a SeqNoStats object, using local state and the supplied global checkpoint

newChangesSnapshot

public Translog.Snapshot newChangesSnapshot(java.lang.String source, MapperService mapperService, long fromSeqNo, long toSeqNo, boolean requiredFullRange) throws java.io.IOException

Description copied from class: Engine

Creates a new history snapshot from Lucene for reading operations whose seqno in the requesting seqno range (both inclusive). This feature requires soft-deletes enabled. If soft-deletes are disabled, this method will throw an IllegalStateException.

Specified by:

newChangesSnapshot in class Engine

Throws:

java.io.IOException

hasCompleteOperationHistory

public boolean hasCompleteOperationHistory(java.lang.String reason, Engine.HistorySource historySource, MapperService mapperService, long startingSeqNo) throws java.io.IOException

Description copied from class: Engine

Checks if this engine has every operations since startingSeqNo(inclusive) in its history (either Lucene or translog)

Specified by:

hasCompleteOperationHistory in class Engine

Throws:

java.io.IOException

getMinRetainedSeqNo

public final long getMinRetainedSeqNo()

Returns the minimum seqno that is retained in the Lucene index. Operations whose seq# are at least this value should exist in the Lucene index.

Specified by:

getMinRetainedSeqNo in class Engine

Returns:

the minimum retained sequence number

acquireHistoryRetentionLock

public java.io.Closeable acquireHistoryRetentionLock(Engine.HistorySource historySource)

Description copied from class: Engine

Acquires a lock on the translog files and Lucene soft-deleted documents to prevent them from being trimmed

Specified by:

acquireHistoryRetentionLock in class Engine

refreshIfNeeded

protected final void refreshIfNeeded(java.lang.String source, long requestingSeqNo)

Refresh this engine **internally** iff the requesting seq_no is greater than the last refreshed checkpoint.

getMaxSeenAutoIdTimestamp

public final long getMaxSeenAutoIdTimestamp()

Description copied from class: Engine

Returns the maximum auto_id_timestamp of all append-only index requests have been processed by this engine or the auto_id_timestamp received from its primary shard via Engine.updateMaxUnsafeAutoIdTimestamp(long). Notes this method returns the auto_id_timestamp of all append-only requests, not max_unsafe_auto_id_timestamp.

Overrides:

getMaxSeenAutoIdTimestamp in class Engine

updateMaxUnsafeAutoIdTimestamp

public final void updateMaxUnsafeAutoIdTimestamp(long newTimestamp)

Description copied from class: Engine

Forces this engine to advance its max_unsafe_auto_id_timestamp marker to at least the given timestamp. The engine will disable optimization for all append-only whose timestamp at most newTimestamp.

Specified by:

updateMaxUnsafeAutoIdTimestamp in class Engine

getMaxSeqNoOfUpdatesOrDeletes

public long getMaxSeqNoOfUpdatesOrDeletes()

Description copied from class: Engine

Returns the maximum sequence number of either update or delete operations have been processed in this engine or the sequence number from Engine.advanceMaxSeqNoOfUpdatesOrDeletes(long). An index request is considered as an update operation if it overwrites the existing documents in Lucene index with the same document id.

A note on the optimization using max_seq_no_of_updates_or_deletes: For each operation O, the key invariants are:

1. I1: There is no operation on docID(O) with seqno that is > MSU(O) and < seqno(O)
2. I2: If MSU(O) < seqno(O) then docID(O) did not exist when O was applied; more precisely, if there is any O' with seqno(O') < seqno(O) and docID(O') = docID(O) then the one with the greatest seqno is a delete.

When a receiving shard (either a replica or a follower) receives an operation O, it must first ensure its own MSU at least MSU(O), and then compares its MSU to its local checkpoint (LCP). If LCP < MSU then there's a gap: there may be some operations that act on docID(O) about which we do not yet know, so we cannot perform an add. Note this also covers the case where a future operation O' with seqNo(O') > seqNo(O) and docId(O') = docID(O) is processed before O. In that case MSU(O') is at least seqno(O') and this means MSU >= seqNo(O') > seqNo(O) > LCP (because O wasn't processed yet).

However, if MSU <= LCP then there is no gap: we have processed every operation <= LCP, and no operation O' with seqno(O') > LCP and seqno(O') < seqno(O) also has docID(O') = docID(O), because such an operation would have seqno(O') > LCP >= MSU >= MSU(O) which contradicts the first invariant. Furthermore in this case we immediately know that docID(O) has been deleted (or never existed) without needing to check Lucene for the following reason. If there's no earlier operation on docID(O) then this is clear, so suppose instead that the preceding operation on docID(O) is O': 1. The first invariant above tells us that seqno(O') <= MSU(O) <= LCP so we have already applied O' to Lucene. 2. Also MSU(O) <= MSU <= LCP < seqno(O) (we discard O if seqno(O) <= LCP) so the second invariant applies, meaning that the O' was a delete.

Therefore, if MSU <= LCP < seqno(O) we know that O can safely be optimized with and added to lucene with addDocument. Moreover, operations that are optimized using the MSU optimization must not be processed twice as this will create duplicates in Lucene. To avoid this we check the local checkpoint tracker to see if an operation was already processed.

Specified by:

getMaxSeqNoOfUpdatesOrDeletes in class Engine

See Also:

Engine.advanceMaxSeqNoOfUpdatesOrDeletes(long)

advanceMaxSeqNoOfUpdatesOrDeletes

public void advanceMaxSeqNoOfUpdatesOrDeletes(long maxSeqNoOfUpdatesOnPrimary)

Description copied from class: Engine

A replica shard receives a new max_seq_no_of_updates from its primary shard, then calls this method to advance this marker to at least the given sequence number.

Specified by:

advanceMaxSeqNoOfUpdatesOrDeletes in class Engine