Package org.elasticsearch.index.engine
Class InternalEngine
- java.lang.Object
-
- org.elasticsearch.index.engine.Engine
-
- org.elasticsearch.index.engine.InternalEngine
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
public class InternalEngine extends Engine
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.elasticsearch.index.engine.Engine
Engine.CommitId, Engine.Delete, Engine.DeleteResult, Engine.EventListener, Engine.Get, Engine.GetResult, Engine.Index, Engine.IndexCommitRef, Engine.IndexResult, Engine.IndexThrottle, Engine.NoOp, Engine.NoOpLock, Engine.NoOpResult, Engine.Operation, Engine.Result, Engine.Searcher, Engine.SearcherScope, Engine.SyncedFlushResult, Engine.Warmer
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID
-
Fields inherited from class org.elasticsearch.index.engine.Engine
allocationId, engineConfig, eventListener, failedEngine, failEngineLock, HISTORY_UUID_KEY, isClosed, lastWriteNanos, logger, readLock, rwl, shardId, store, SYNC_COMMIT_ID, writeLock
-
-
Constructor Summary
Constructors Constructor Description InternalEngine(EngineConfig engineConfig)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description Engine.IndexCommitRef
acquireLastIndexCommit(boolean flushFirst)
Snapshots the most recent index and returns a handle to it.Engine.IndexCommitRef
acquireSafeIndexCommit()
Snapshots the most recent safe index commit from the engine.Engine.Searcher
acquireSearcher(java.lang.String source, Engine.SearcherScope scope)
Returns a new searcher instance.java.io.Closeable
acquireTranslogRetentionLock()
void
activateThrottling()
Request that this engine throttle incoming indexing requests to one thread.protected boolean
assertOriginPrimarySequenceNumber(long seqNo)
protected void
closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch)
Closes the engine without acquiring the write lock.protected void
commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog, java.lang.String syncId)
Commits the specified index writer.void
deactivateThrottling()
Reverses a previousEngine.activateThrottling()
call.Engine.DeleteResult
delete(Engine.Delete delete)
Perform document delete operation on the engineprotected long
doGenerateSeqNoForOperation(Engine.Operation operation)
Generate the sequence number for the specified operation.boolean
ensureTranslogSynced(java.util.stream.Stream<Translog.Location> locations)
Ensures that all locations in the given stream have been written to the underlying storage.int
estimateTranslogOperationsFromMinSeq(long minSeqNo)
Returns the estimated number of translog operations in this engine whose seq# at least the provided seq#.int
fillSeqNoGaps(long primaryTerm)
Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.Engine.CommitId
flush()
Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation.Engine.CommitId
flush(boolean force, boolean waitIfOngoing)
Flushes the state of the engine including the transaction log, clearing memory.void
forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments)
Triggers a forced merge on this engineEngine.GetResult
get(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory)
java.lang.String
getHistoryUUID()
returns the history uuid for the enginelong
getIndexBufferRAMBytesUsed()
How much heap is used that would be freed by a refresh.long
getIndexThrottleTimeInMillis()
Returns the number of milliseconds this engine was under index throttling.protected org.apache.lucene.index.SegmentInfos
getLastCommittedSegmentInfos()
long
getLastSyncedGlobalCheckpoint()
Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e.long
getLocalCheckpoint()
MergeStats
getMergeStats()
SeqNoStats
getSeqNoStats(long globalCheckpoint)
Translog.Location
getTranslogLastWriteLocation()
Returns the last location that the translog of this engine has written into.TranslogStats
getTranslogStats()
long
getWritingBytes()
Returns how many bytes we are currently moving from indexing buffer to segments on diskEngine.IndexResult
index(Engine.Index index)
Perform document index operation on the engineboolean
isRecovering()
Returnstrue
iff this engine is currently recovering from translog.boolean
isThrottled()
Returns thetrue
iff this engine is currently under index throttling.boolean
isTranslogSyncNeeded()
Checks if the underlying storage sync is required.protected boolean
maybeFailEngine(java.lang.String source, java.lang.Exception e)
Check whether the engine should be failedvoid
maybePruneDeletes()
Tries to prune buffered deletes from the version map.Translog.Snapshot
newTranslogSnapshotFromMinSeqNo(long minSeqNo)
Creates a new translog snapshot from this engine for reading translog operations whose seq# at least the provided seq#.Engine.NoOpResult
noOp(Engine.NoOp noOp)
void
onSettingsChanged()
InternalEngine
recoverFromTranslog()
Performs recovery from the transaction log.void
refresh(java.lang.String source)
Synchronously refreshes the engine for new search operations to reflect the latest changes.void
resetLocalCheckpoint(long localCheckpoint)
Reset the local checkpoint in the tracker to the given local checkpointvoid
restoreLocalCheckpointFromTranslog()
Marks operations in the translog as completed.void
rollTranslogGeneration()
Rolls the translog generation and cleans unneeded.java.util.List<Segment>
segments(boolean verbose)
The list of segments in the engine.boolean
shouldPeriodicallyFlush()
Checks if this engine should be flushed periodically.boolean
shouldRollTranslogGeneration()
Tests whether or not the translog generation should be rolled to a new generation.void
skipTranslogRecovery()
Do not replay translog operations, but make the engine be ready.Engine.SyncedFlushResult
syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId)
Attempts to do a special commit where the given syncID is put into the commit data.void
syncTranslog()
void
trimOperationsFromTranslog(long belowTerm, long aboveSeqNo)
Trims translog for terms belowbelowTerm
and seq# aboveaboveSeqNo
void
trimUnreferencedTranslogFiles()
checks and removes translog files that no longer need to be retained.void
waitForOpsToComplete(long seqNo)
Waits for all operations up to the provided sequence number to complete.void
writeIndexingBuffer()
Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.protected void
writerSegmentStats(SegmentsStats stats)
-
Methods inherited from class org.elasticsearch.index.engine.Engine
acquireSearcher, close, commitStats, config, ensureOpen, ensureOpen, failEngine, flushAndClose, forceMerge, getFromSearcher, getLastWriteNanos, getSegmentInfo, guardedRamBytesUsed, isMergedSegment, refreshNeeded, segmentsStats
-
-
-
-
Field Detail
-
MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID
public static final java.lang.String MAX_UNSAFE_AUTO_ID_TIMESTAMP_COMMIT_ID
- See Also:
- Constant Field Values
-
-
Constructor Detail
-
InternalEngine
public InternalEngine(EngineConfig engineConfig)
-
-
Method Detail
-
restoreLocalCheckpointFromTranslog
public void restoreLocalCheckpointFromTranslog() throws java.io.IOException
Description copied from class:Engine
Marks operations in the translog as completed. This is used to restore the state of the local checkpoint tracker on primary promotion.- Specified by:
restoreLocalCheckpointFromTranslog
in classEngine
- Throws:
java.io.IOException
- if an I/O exception occurred reading the translog
-
fillSeqNoGaps
public int fillSeqNoGaps(long primaryTerm) throws java.io.IOException
Description copied from class:Engine
Fills up the local checkpoints history with no-ops until the local checkpoint and the max seen sequence ID are identical.- Specified by:
fillSeqNoGaps
in classEngine
- Parameters:
primaryTerm
- the shards primary term this engine was created for- Returns:
- the number of no-ops added
- Throws:
java.io.IOException
-
recoverFromTranslog
public InternalEngine recoverFromTranslog() throws java.io.IOException
Description copied from class:Engine
Performs recovery from the transaction log. This operation will close the engine if the recovery fails.- Specified by:
recoverFromTranslog
in classEngine
- Throws:
java.io.IOException
-
skipTranslogRecovery
public void skipTranslogRecovery()
Description copied from class:Engine
Do not replay translog operations, but make the engine be ready.- Specified by:
skipTranslogRecovery
in classEngine
-
isTranslogSyncNeeded
public boolean isTranslogSyncNeeded()
Description copied from class:Engine
Checks if the underlying storage sync is required.- Specified by:
isTranslogSyncNeeded
in classEngine
-
ensureTranslogSynced
public boolean ensureTranslogSynced(java.util.stream.Stream<Translog.Location> locations) throws java.io.IOException
Description copied from class:Engine
Ensures that all locations in the given stream have been written to the underlying storage.- Specified by:
ensureTranslogSynced
in classEngine
- Throws:
java.io.IOException
-
syncTranslog
public void syncTranslog() throws java.io.IOException
- Specified by:
syncTranslog
in classEngine
- Throws:
java.io.IOException
-
acquireTranslogRetentionLock
public java.io.Closeable acquireTranslogRetentionLock()
- Specified by:
acquireTranslogRetentionLock
in classEngine
-
newTranslogSnapshotFromMinSeqNo
public Translog.Snapshot newTranslogSnapshotFromMinSeqNo(long minSeqNo) throws java.io.IOException
Description copied from class:Engine
Creates a new translog snapshot from this engine for reading translog operations whose seq# at least the provided seq#. The caller has to close the returned snapshot after finishing the reading.- Specified by:
newTranslogSnapshotFromMinSeqNo
in classEngine
- Throws:
java.io.IOException
-
estimateTranslogOperationsFromMinSeq
public int estimateTranslogOperationsFromMinSeq(long minSeqNo)
Description copied from class:Engine
Returns the estimated number of translog operations in this engine whose seq# at least the provided seq#.- Specified by:
estimateTranslogOperationsFromMinSeq
in classEngine
-
getTranslogStats
public TranslogStats getTranslogStats()
- Specified by:
getTranslogStats
in classEngine
-
getTranslogLastWriteLocation
public Translog.Location getTranslogLastWriteLocation()
Description copied from class:Engine
Returns the last location that the translog of this engine has written into.- Specified by:
getTranslogLastWriteLocation
in classEngine
-
getHistoryUUID
public java.lang.String getHistoryUUID()
Description copied from class:Engine
returns the history uuid for the engine- Specified by:
getHistoryUUID
in classEngine
-
getWritingBytes
public long getWritingBytes()
Returns how many bytes we are currently moving from indexing buffer to segments on disk- Specified by:
getWritingBytes
in classEngine
-
get
public Engine.GetResult get(Engine.Get get, java.util.function.BiFunction<java.lang.String,Engine.SearcherScope,Engine.Searcher> searcherFactory) throws EngineException
- Specified by:
get
in classEngine
- Throws:
EngineException
-
assertOriginPrimarySequenceNumber
protected boolean assertOriginPrimarySequenceNumber(long seqNo)
-
doGenerateSeqNoForOperation
protected long doGenerateSeqNoForOperation(Engine.Operation operation)
Generate the sequence number for the specified operation.- Parameters:
operation
- the operation- Returns:
- the sequence number
-
index
public Engine.IndexResult index(Engine.Index index) throws java.io.IOException
Description copied from class:Engine
Perform document index operation on the engine- Specified by:
index
in classEngine
- Parameters:
index
- operation to perform- Returns:
Engine.IndexResult
containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
java.io.IOException
-
delete
public Engine.DeleteResult delete(Engine.Delete delete) throws java.io.IOException
Description copied from class:Engine
Perform document delete operation on the engine- Specified by:
delete
in classEngine
- Parameters:
delete
- operation to perform- Returns:
Engine.DeleteResult
containing updated translog location, version and document specific failures Note: engine level failures (i.e. persistent engine failures) are thrown- Throws:
java.io.IOException
-
maybePruneDeletes
public void maybePruneDeletes()
Description copied from class:Engine
Tries to prune buffered deletes from the version map.- Specified by:
maybePruneDeletes
in classEngine
-
noOp
public Engine.NoOpResult noOp(Engine.NoOp noOp)
-
refresh
public void refresh(java.lang.String source) throws EngineException
Description copied from class:Engine
Synchronously refreshes the engine for new search operations to reflect the latest changes.- Specified by:
refresh
in classEngine
- Throws:
EngineException
-
writeIndexingBuffer
public void writeIndexingBuffer() throws EngineException
Description copied from class:Engine
Called when our engine is using too much heap and should move buffered indexed/deleted documents to disk.- Specified by:
writeIndexingBuffer
in classEngine
- Throws:
EngineException
-
syncFlush
public Engine.SyncedFlushResult syncFlush(java.lang.String syncId, Engine.CommitId expectedCommitId) throws EngineException
Description copied from class:Engine
Attempts to do a special commit where the given syncID is put into the commit data. The attempt succeeds if there are not pending writes in lucene and the current point is equal to the expected one.- Specified by:
syncFlush
in classEngine
- Parameters:
syncId
- id of this syncexpectedCommitId
- the expected value of- Returns:
- true if the sync commit was made, false o.w.
- Throws:
EngineException
-
shouldPeriodicallyFlush
public boolean shouldPeriodicallyFlush()
Description copied from class:Engine
Checks if this engine should be flushed periodically. This check is mainly based on the uncommitted translog size and the translog flush threshold setting.- Specified by:
shouldPeriodicallyFlush
in classEngine
-
flush
public Engine.CommitId flush() throws EngineException
Description copied from class:Engine
Flushes the state of the engine including the transaction log, clearing memory and persisting documents in the lucene index to disk including a potentially heavy and durable fsync operation. This operation is not going to block if another flush operation is currently running and won't write a lucene commit if nothing needs to be committed.- Specified by:
flush
in classEngine
- Returns:
- the commit Id for the resulting commit
- Throws:
EngineException
-
flush
public Engine.CommitId flush(boolean force, boolean waitIfOngoing) throws EngineException
Description copied from class:Engine
Flushes the state of the engine including the transaction log, clearing memory.- Specified by:
flush
in classEngine
- Parameters:
force
- iftrue
a lucene commit is executed even if no changes need to be committed.waitIfOngoing
- iftrue
this call will block until all currently running flushes have finished. Otherwise this call will return without blocking.- Returns:
- the commit Id for the resulting commit
- Throws:
EngineException
-
rollTranslogGeneration
public void rollTranslogGeneration() throws EngineException
Description copied from class:Engine
Rolls the translog generation and cleans unneeded.- Specified by:
rollTranslogGeneration
in classEngine
- Throws:
EngineException
-
trimUnreferencedTranslogFiles
public void trimUnreferencedTranslogFiles() throws EngineException
Description copied from class:Engine
checks and removes translog files that no longer need to be retained. SeeTranslogDeletionPolicy
for details- Specified by:
trimUnreferencedTranslogFiles
in classEngine
- Throws:
EngineException
-
shouldRollTranslogGeneration
public boolean shouldRollTranslogGeneration()
Description copied from class:Engine
Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.- Specified by:
shouldRollTranslogGeneration
in classEngine
- Returns:
true
if the current generation should be rolled to a new generation
-
trimOperationsFromTranslog
public void trimOperationsFromTranslog(long belowTerm, long aboveSeqNo) throws EngineException
Description copied from class:Engine
Trims translog for terms belowbelowTerm
and seq# aboveaboveSeqNo
- Specified by:
trimOperationsFromTranslog
in classEngine
- Throws:
EngineException
- See Also:
Translog.trimOperations(long, long)
-
forceMerge
public void forceMerge(boolean flush, int maxNumSegments, boolean onlyExpungeDeletes, boolean upgrade, boolean upgradeOnlyAncientSegments) throws EngineException, java.io.IOException
Description copied from class:Engine
Triggers a forced merge on this engine- Specified by:
forceMerge
in classEngine
- Throws:
EngineException
java.io.IOException
-
acquireLastIndexCommit
public Engine.IndexCommitRef acquireLastIndexCommit(boolean flushFirst) throws EngineException
Description copied from class:Engine
Snapshots the most recent index and returns a handle to it. If needed will try and "commit" the lucene index to make sure we have a "fresh" copy of the files to snapshot.- Specified by:
acquireLastIndexCommit
in classEngine
- Parameters:
flushFirst
- indicates whether the engine should flush before returning the snapshot- Throws:
EngineException
-
acquireSafeIndexCommit
public Engine.IndexCommitRef acquireSafeIndexCommit() throws EngineException
Description copied from class:Engine
Snapshots the most recent safe index commit from the engine.- Specified by:
acquireSafeIndexCommit
in classEngine
- Throws:
EngineException
-
maybeFailEngine
protected boolean maybeFailEngine(java.lang.String source, java.lang.Exception e)
Description copied from class:Engine
Check whether the engine should be failed- Overrides:
maybeFailEngine
in classEngine
-
getLastCommittedSegmentInfos
protected org.apache.lucene.index.SegmentInfos getLastCommittedSegmentInfos()
- Specified by:
getLastCommittedSegmentInfos
in classEngine
-
writerSegmentStats
protected final void writerSegmentStats(SegmentsStats stats)
- Overrides:
writerSegmentStats
in classEngine
-
getIndexBufferRAMBytesUsed
public long getIndexBufferRAMBytesUsed()
Description copied from class:Engine
How much heap is used that would be freed by a refresh. Note that this may throwAlreadyClosedException
.- Specified by:
getIndexBufferRAMBytesUsed
in classEngine
-
segments
public java.util.List<Segment> segments(boolean verbose)
Description copied from class:Engine
The list of segments in the engine.
-
closeNoLock
protected final void closeNoLock(java.lang.String reason, java.util.concurrent.CountDownLatch closedLatch)
Closes the engine without acquiring the write lock. This should only be called while the write lock is hold or in a disaster condition ie. if the engine is failed.- Specified by:
closeNoLock
in classEngine
-
acquireSearcher
public Engine.Searcher acquireSearcher(java.lang.String source, Engine.SearcherScope scope)
Description copied from class:Engine
Returns a new searcher instance. The consumer of this API is responsible for releasing the returned searcher in a safe manner, preferably in a try/finally block.- Specified by:
acquireSearcher
in classEngine
- Parameters:
source
- the source API or routing that triggers this searcher acquirescope
- the scope of this searcher ie. if the searcher will be used for get or search purposes- See Also:
Engine.Searcher.close()
-
activateThrottling
public void activateThrottling()
Description copied from class:Engine
Request that this engine throttle incoming indexing requests to one thread. Must be matched by a later call toEngine.deactivateThrottling()
.- Specified by:
activateThrottling
in classEngine
-
deactivateThrottling
public void deactivateThrottling()
Description copied from class:Engine
Reverses a previousEngine.activateThrottling()
call.- Specified by:
deactivateThrottling
in classEngine
-
isThrottled
public boolean isThrottled()
Description copied from class:Engine
Returns thetrue
iff this engine is currently under index throttling.- Specified by:
isThrottled
in classEngine
- See Also:
Engine.getIndexThrottleTimeInMillis()
-
getIndexThrottleTimeInMillis
public long getIndexThrottleTimeInMillis()
Description copied from class:Engine
Returns the number of milliseconds this engine was under index throttling.- Specified by:
getIndexThrottleTimeInMillis
in classEngine
-
commitIndexWriter
protected void commitIndexWriter(org.apache.lucene.index.IndexWriter writer, Translog translog, @Nullable java.lang.String syncId) throws java.io.IOException
Commits the specified index writer.- Parameters:
writer
- the index writer to committranslog
- the translogsyncId
- the sync flush ID (null
if not committing a synced flush)- Throws:
java.io.IOException
- if an I/O exception occurs committing the specfied writer
-
onSettingsChanged
public void onSettingsChanged()
- Overrides:
onSettingsChanged
in classEngine
-
getMergeStats
public MergeStats getMergeStats()
- Overrides:
getMergeStats
in classEngine
-
getLastSyncedGlobalCheckpoint
public long getLastSyncedGlobalCheckpoint()
Description copied from class:Engine
Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)- Specified by:
getLastSyncedGlobalCheckpoint
in classEngine
-
getLocalCheckpoint
public long getLocalCheckpoint()
- Specified by:
getLocalCheckpoint
in classEngine
- Returns:
- the local checkpoint for this Engine
-
waitForOpsToComplete
public void waitForOpsToComplete(long seqNo) throws java.lang.InterruptedException
Description copied from class:Engine
Waits for all operations up to the provided sequence number to complete.- Specified by:
waitForOpsToComplete
in classEngine
- Parameters:
seqNo
- the sequence number that the checkpoint must advance to before this method returns- Throws:
java.lang.InterruptedException
- if the thread was interrupted while blocking on the condition
-
resetLocalCheckpoint
public void resetLocalCheckpoint(long localCheckpoint)
Description copied from class:Engine
Reset the local checkpoint in the tracker to the given local checkpoint- Specified by:
resetLocalCheckpoint
in classEngine
- Parameters:
localCheckpoint
- the new checkpoint to be set
-
getSeqNoStats
public SeqNoStats getSeqNoStats(long globalCheckpoint)
- Specified by:
getSeqNoStats
in classEngine
- Returns:
- a
SeqNoStats
object, using local state and the supplied global checkpoint
-
isRecovering
public boolean isRecovering()
Description copied from class:Engine
Returnstrue
iff this engine is currently recovering from translog.- Overrides:
isRecovering
in classEngine
-
-