Package org.elasticsearch.index.shard

Class IndexShard

    • Field Detail

      • shardRouting

        protected volatile ShardRouting shardRouting

      • pendingPrimaryTerm

        protected volatile long pendingPrimaryTerm

      • operationPrimaryTerm

        protected volatile long operationPrimaryTerm

      • currentEngineReference

        protected final java.util.concurrent.atomic.AtomicReference<Engine> currentEngineReference

    • Method Detail

      • getThreadPool

        public ThreadPool getThreadPool()

      • store

        public Store store()

      • getIndexSort

        public org.apache.lucene.search.Sort getIndexSort()
        Return the sort order of this index, or null if the index has no sort.

      • getPendingPrimaryTerm

        public long getPendingPrimaryTerm()
        USE THIS METHOD WITH CARE! Returns the primary term the index shard is supposed to be on. In case of primary promotion or when a replica learns about a new term due to a new primary, the term that's exposed here will not be the term that the shard internally uses to assign to operations. The shard will auto-correct its internal operation term, but this might take time. See IndexMetaData.primaryTerm(int)

      • getQueryCachingPolicy

        public org.apache.lucene.search.QueryCachingPolicy getQueryCachingPolicy()

      • updateShardState

        public void updateShardState​(ShardRouting newRouting,
                             long newPrimaryTerm,
                             java.util.function.BiConsumer<IndexShard,ActionListener<PrimaryReplicaSyncer.ResyncTask>> primaryReplicaSyncer,
                             long applyingClusterStateVersion,
                             java.util.Set<java.lang.String> inSyncAllocationIds,
                             IndexShardRoutingTable routingTable,
                             java.util.Set<java.lang.String> pre60AllocationIds)
                      throws java.io.IOException
        Description copied from interface: IndicesClusterStateService.Shard
        Updates the shard state based on an incoming cluster state: - Updates and persists the new routing value. - Updates the primary term if this shard is a primary. - Updates the allocation ids that are tracked by the shard if it is a primary. See ReplicationTracker.updateFromMaster(long, Set, IndexShardRoutingTable, Set) for details.
        Specified by:
        updateShardState in interface IndicesClusterStateService.Shard
        Parameters:
        newRouting - the new routing entry
        newPrimaryTerm - the new primary term
        primaryReplicaSyncer - the primary-replica resync action to trigger when a term is increased on a primary
        applyingClusterStateVersion - the cluster state version being applied when updating the allocation IDs from the master
        inSyncAllocationIds - the allocation ids of the currently in-sync shard copies
        routingTable - the shard routing table
        Throws:
        java.io.IOException - if shard state could not be persisted

      • relocated

        public void relocated​(java.util.function.Consumer<ReplicationTracker.PrimaryContext> consumer)
               throws IllegalIndexShardStateException,
                      java.lang.InterruptedException
        Completes the relocation. Operations are blocked and current operations are drained before changing state to relocated. The provided Runnable is executed after all operations are successfully blocked.
        Parameters:
        consumer - a Runnable that is executed after operations are blocked
        Throws:
        IllegalIndexShardStateException - if the shard is not relocating due to concurrent cancellation
        java.lang.InterruptedException - if blocking operations is interrupted

      • applyIndexOperationOnPrimary

        public Engine.IndexResult applyIndexOperationOnPrimary​(long version,
                                                       VersionType versionType,
                                                       SourceToParse sourceToParse,
                                                       long autoGeneratedTimestamp,
                                                       boolean isRetry)
                                                throws java.io.IOException
        Throws:
        java.io.IOException

      • applyIndexOperationOnReplica

        public Engine.IndexResult applyIndexOperationOnReplica​(long seqNo,
                                                       long version,
                                                       VersionType versionType,
                                                       long autoGeneratedTimeStamp,
                                                       boolean isRetry,
                                                       SourceToParse sourceToParse)
                                                throws java.io.IOException
        Throws:
        java.io.IOException

      • markSeqNoAsNoop

        public Engine.NoOpResult markSeqNoAsNoop​(long seqNo,
                                         java.lang.String reason)
                                  throws java.io.IOException
        Throws:
        java.io.IOException

      • getFailedIndexResult

        public Engine.IndexResult getFailedIndexResult​(java.lang.Exception e,
                                               long version)

      • getFailedDeleteResult

        public Engine.DeleteResult getFailedDeleteResult​(java.lang.Exception e,
                                                 long version)

      • applyDeleteOperationOnPrimary

        public Engine.DeleteResult applyDeleteOperationOnPrimary​(long version,
                                                         java.lang.String type,
                                                         java.lang.String id,
                                                         VersionType versionType)
                                                  throws java.io.IOException
        Throws:
        java.io.IOException

      • applyDeleteOperationOnReplica

        public Engine.DeleteResult applyDeleteOperationOnReplica​(long seqNo,
                                                         long version,
                                                         java.lang.String type,
                                                         java.lang.String id,
                                                         VersionType versionType)
                                                  throws java.io.IOException
        Throws:
        java.io.IOException

      • refresh

        public void refresh​(java.lang.String source)
        Writes all indexing changes to disk and opens a new searcher reflecting all changes. This can throw AlreadyClosedException.

      • getWritingBytes

        public long getWritingBytes()
        Returns how many bytes we are currently moving from heap to disk

      • commitStats

        public CommitStats commitStats()
        Returns:
        CommitStats
        Throws:
        org.apache.lucene.store.AlreadyClosedException - if shard is closed

      • seqNoStats

        public SeqNoStats seqNoStats()
        Returns:
        SeqNoStats
        Throws:
        org.apache.lucene.store.AlreadyClosedException - if shard is closed

      • indexingStats

        public IndexingStats indexingStats​(java.lang.String... types)

      • searchStats

        public SearchStats searchStats​(java.lang.String... groups)

      • segmentStats

        public SegmentsStats segmentStats​(boolean includeSegmentFileSizes)

      • fieldDataStats

        public FieldDataStats fieldDataStats​(java.lang.String... fields)

      • completionStats

        public CompletionStats completionStats​(java.lang.String... fields)

      • flush

        public Engine.CommitId flush​(FlushRequest request)
        Executes the given flush request against the engine.
        Parameters:
        request - the flush request
        Returns:
        the commit ID

      • trimTranslog

        public void trimTranslog()
        checks and removes translog files that no longer need to be retained. See TranslogDeletionPolicy for details

      • forceMerge

        public void forceMerge​(ForceMergeRequest forceMerge)
                throws java.io.IOException
        Throws:
        java.io.IOException

      • upgrade

        public org.apache.lucene.util.Version upgrade​(UpgradeRequest upgrade)
                                       throws java.io.IOException
        Upgrades the shard to the current version of Lucene and returns the minimum segment version
        Throws:
        java.io.IOException

      • minimumCompatibleVersion

        public org.apache.lucene.util.Version minimumCompatibleVersion()

      • acquireLastIndexCommit

        public Engine.IndexCommitRef acquireLastIndexCommit​(boolean flushFirst)
                                             throws EngineException
        Creates a new IndexCommit snapshot from the currently running engine. All resources referenced by this commit won't be freed until the commit / snapshot is closed.
        Parameters:
        flushFirst - true if the index should first be flushed to disk / a low level lucene commit should be executed
        Throws:
        EngineException

      • acquireSafeIndexCommit

        public Engine.IndexCommitRef acquireSafeIndexCommit()
                                             throws EngineException
        Snapshots the most recent safe index commit from the currently running engine. All index files referenced by this index commit won't be freed until the commit/snapshot is closed.
        Throws:
        EngineException

      • snapshotStoreMetadata

        public Store.MetadataSnapshot snapshotStoreMetadata()
                                             throws java.io.IOException
        gets a Store.MetadataSnapshot for the current directory. This method is safe to call in all lifecycle of the index shard, without having to worry about the current state of the engine and concurrent flushes.
        Throws:
        org.apache.lucene.index.IndexNotFoundException - if no index is found in the current directory
        org.apache.lucene.index.CorruptIndexException - if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.
        org.apache.lucene.index.IndexFormatTooOldException - if the lucene index is too old to be opened.
        org.apache.lucene.index.IndexFormatTooNewException - if the lucene index is too new to be opened.
        java.io.FileNotFoundException - if one or more files referenced by a commit are not present.
        java.nio.file.NoSuchFileException - if one or more files referenced by a commit are not present.
        java.io.IOException

      • failShard

        public void failShard​(java.lang.String reason,
                      @Nullable
                      java.lang.Exception e)
        Fails the shard and marks the shard store as corrupted if e is caused by index corruption

      • acquireSearcher

        public Engine.Searcher acquireSearcher​(java.lang.String source)

      • close

        public void close​(java.lang.String reason,
                  boolean flushEngine)
           throws java.io.IOException
        Throws:
        java.io.IOException

      • prepareForIndexRecovery

        public void prepareForIndexRecovery()
        called before starting to copy index files over

      • trimOperationOfPreviousPrimaryTerms

        public void trimOperationOfPreviousPrimaryTerms​(long aboveSeqNo)

      • openEngineAndRecoverFromTranslog

        public void openEngineAndRecoverFromTranslog()
                                      throws java.io.IOException
        opens the engine on top of the existing lucene engine and translog. Operations from the translog will be replayed to bring lucene up to date.
        Throws:
        java.io.IOException

      • openEngineAndSkipTranslogRecovery

        public void openEngineAndSkipTranslogRecovery()
                                       throws java.io.IOException
        Opens the engine on top of the existing lucene engine and translog. The translog is kept but its operations won't be replayed.
        Throws:
        java.io.IOException

      • onNewEngine

        protected void onNewEngine​(Engine newEngine)

      • performRecoveryRestart

        public void performRecoveryRestart()
                            throws java.io.IOException
        called if recovery has to be restarted after network error / delay **
        Throws:
        java.io.IOException

      • recoveryStats

        public RecoveryStats recoveryStats()
        returns stats about ongoing recoveries, both source and target

      • finalizeRecovery

        public void finalizeRecovery()
        perform the last stages of recovery once all translog operations are done. note that you should still call postRecovery(String).

      • ignoreRecoveryAttempt

        public boolean ignoreRecoveryAttempt()
        Returns true if this shard can ignore a recovery attempt made to it (since the already doing/done it)

      • isReadAllowed

        public boolean isReadAllowed()
        returns true if the IndexShardState allows reading

      • getIndexBufferRAMBytesUsed

        public long getIndexBufferRAMBytesUsed()
        Returns number of heap bytes used by the indexing buffer for this shard, or 0 if the shard is closed

      • addShardFailureCallback

        public void addShardFailureCallback​(java.util.function.Consumer<IndexShard.ShardFailure> onShardFailure)

      • checkIdle

        public void checkIdle​(long inactiveTimeNS)
        Called by IndexingMemoryController to check whether more than inactiveTimeNS has passed since the last indexing operation, and notify listeners that we are now inactive so e.g. sync'd flush can happen.

      • isActive

        public boolean isActive()

      • recoverFromLocalShards

        public boolean recoverFromLocalShards​(java.util.function.BiConsumer<java.lang.String,MappingMetaData> mappingUpdateConsumer,
                                      java.util.List<IndexShard> localShards)
                               throws java.io.IOException
        Throws:
        java.io.IOException

      • recoverFromStore

        public boolean recoverFromStore()

      • restoreFromRepository

        public boolean restoreFromRepository​(Repository repository)

      • onSettingsChanged

        public void onSettingsChanged()

      • acquireTranslogRetentionLock

        public java.io.Closeable acquireTranslogRetentionLock()
        Acquires a lock on the translog files, preventing them from being trimmed.

      • newTranslogSnapshotFromMinSeqNo

        public Translog.Snapshot newTranslogSnapshotFromMinSeqNo​(long minSeqNo)
                                                  throws java.io.IOException
        Creates a new translog snapshot for reading translog operations whose seq# at least the provided seq#. The caller has to close the returned snapshot after finishing the reading.
        Throws:
        java.io.IOException

      • estimateTranslogOperationsFromMinSeq

        public int estimateTranslogOperationsFromMinSeq​(long minSeqNo)
        Returns the estimated number of operations in translog whose seq# at least the provided seq#.

      • segments

        public java.util.List<Segment> segments​(boolean verbose)

      • flushAndCloseEngine

        public void flushAndCloseEngine()
                         throws java.io.IOException
        Throws:
        java.io.IOException

      • getHistoryUUID

        public java.lang.String getHistoryUUID()

      • activateThrottling

        public void activateThrottling()

      • deactivateThrottling

        public void deactivateThrottling()

      • writeIndexingBuffer

        public void writeIndexingBuffer()
        Called when our shard is using too much heap and should move buffered indexed/deleted documents to disk.

      • updateLocalCheckpointForShard

        public void updateLocalCheckpointForShard​(java.lang.String allocationId,
                                          long checkpoint)
        Notifies the service to update the local checkpoint for the shard with the provided allocation ID. See ReplicationTracker.updateLocalCheckpoint(String, long) for details.
        Parameters:
        allocationId - the allocation ID of the shard to update the local checkpoint for
        checkpoint - the local checkpoint for the shard

      • updateGlobalCheckpointForShard

        public void updateGlobalCheckpointForShard​(java.lang.String allocationId,
                                           long globalCheckpoint)
        Update the local knowledge of the global checkpoint for the specified allocation ID.
        Parameters:
        allocationId - the allocation ID to update the global checkpoint for
        globalCheckpoint - the global checkpoint

      • waitForOpsToComplete

        public void waitForOpsToComplete​(long seqNo)
                          throws java.lang.InterruptedException
        Waits for all operations up to the provided sequence number to complete.
        Parameters:
        seqNo - the sequence number that the checkpoint must advance to before this method returns
        Throws:
        java.lang.InterruptedException - if the thread was interrupted while blocking on the condition

      • initiateTracking

        public void initiateTracking​(java.lang.String allocationId)
        Called when the recovery process for a shard has opened the engine on the target shard. Ensures that the right data structures have been set up locally to track local checkpoint information for the shard and that the shard is added to the replication group.
        Parameters:
        allocationId - the allocation ID of the shard for which recovery was initiated

      • markAllocationIdAsInSync

        public void markAllocationIdAsInSync​(java.lang.String allocationId,
                                     long localCheckpoint)
                              throws java.lang.InterruptedException
        Marks the shard with the provided allocation ID as in-sync with the primary shard. See ReplicationTracker.markAllocationIdAsInSync(String, long) for additional details.
        Parameters:
        allocationId - the allocation ID of the shard to mark as in-sync
        localCheckpoint - the current local checkpoint on the shard
        Throws:
        java.lang.InterruptedException

      • getLocalCheckpoint

        public long getLocalCheckpoint()
        Returns the local checkpoint for the shard.
        Returns:
        the local checkpoint

      • getGlobalCheckpoint

        public long getGlobalCheckpoint()
        Returns the global checkpoint for the shard.
        Returns:
        the global checkpoint

      • getLastSyncedGlobalCheckpoint

        public long getLastSyncedGlobalCheckpoint()
        Returns the latest global checkpoint value that has been persisted in the underlying storage (i.e. translog's checkpoint)

      • getInSyncGlobalCheckpoints

        public com.carrotsearch.hppc.ObjectLongMap<java.lang.String> getInSyncGlobalCheckpoints()
        Get the local knowledge of the global checkpoints for all in-sync allocation IDs.
        Returns:
        a map from allocation ID to the local knowledge of the global checkpoint for that allocation ID

      • maybeSyncGlobalCheckpoint

        public void maybeSyncGlobalCheckpoint​(java.lang.String reason)
        Syncs the global checkpoint to the replicas if the global checkpoint on at least one replica is behind the global checkpoint on the primary.

      • getReplicationGroup

        public ReplicationGroup getReplicationGroup()
        Returns the current replication group for the shard.
        Returns:
        the replication group

      • updateGlobalCheckpointOnReplica

        public void updateGlobalCheckpointOnReplica​(long globalCheckpoint,
                                            java.lang.String reason)
        Updates the global checkpoint on a replica shard after it has been updated by the primary.
        Parameters:
        globalCheckpoint - the global checkpoint
        reason - the reason the global checkpoint was updated

      • activateWithPrimaryContext

        public void activateWithPrimaryContext​(ReplicationTracker.PrimaryContext primaryContext)
        Updates the known allocation IDs and the local checkpoints for the corresponding allocations from a primary relocation source.
        Parameters:
        primaryContext - the sequence number context

      • pendingInSync

        public boolean pendingInSync()
        Check if there are any recoveries pending in-sync.
        Returns:
        true if there is at least one shard pending in-sync, otherwise false

      • noopUpdate

        public void noopUpdate​(java.lang.String type)
        Should be called for each no-op update operation to increment relevant statistics.
        Parameters:
        type - the doc type of the update

      • getEngineOrNull

        protected Engine getEngineOrNull()
        NOTE: returns null if engine is not yet started (e.g. recovery phase 1, copying over index files, is still running), or if engine is closed.

      • isRelocatedPrimary

        public boolean isRelocatedPrimary()
        Returns whether the shard is a relocated primary, i.e. not in charge anymore of replicating changes (see ReplicationTracker).

      • acquirePrimaryOperationPermit

        public void acquirePrimaryOperationPermit​(ActionListener<Releasable> onPermitAcquired,
                                          java.lang.String executorOnDelay,
                                          java.lang.Object debugInfo)
        Acquire a primary operation permit whenever the shard is ready for indexing. If a permit is directly available, the provided ActionListener will be called on the calling thread. During relocation hand-off, permit acquisition can be delayed. The provided ActionListener will then be called using the provided executor.
        Parameters:
        debugInfo - an extra information that can be useful when tracing an unreleased permit. When assertions are enabled the tracing will capture the supplied object's Object.toString() value. Otherwise the object isn't used

      • acquireReplicaOperationPermit

        public void acquireReplicaOperationPermit​(long opPrimaryTerm,
                                          long globalCheckpoint,
                                          ActionListener<Releasable> onPermitAcquired,
                                          java.lang.String executorOnDelay,
                                          java.lang.Object debugInfo)
        Acquire a replica operation permit whenever the shard is ready for indexing (see acquirePrimaryOperationPermit(ActionListener, String, Object)). If the given primary term is lower than then one in shardRouting, the ActionListener.onFailure(Exception) method of the provided listener is invoked with an IllegalStateException. If permit acquisition is delayed, the listener will be invoked on the executor with the specified name.
        Parameters:
        opPrimaryTerm - the operation primary term
        globalCheckpoint - the global checkpoint associated with the request
        onPermitAcquired - the listener for permit acquisition
        executorOnDelay - the name of the executor to invoke the listener on if permit acquisition is delayed
        debugInfo - an extra information that can be useful when tracing an unreleased permit. When assertions are enabled the tracing will capture the supplied object's Object.toString() value. Otherwise the object isn't used

      • getActiveOperationsCount

        public int getActiveOperationsCount()

      • getActiveOperations

        public java.util.List<java.lang.String> getActiveOperations()
        Returns:
        a list of describing each permit that wasn't released yet. The description consist of the debugInfo supplied when the permit was acquired plus a stack traces that was captured when the permit was request.

      • sync

        public final void sync​(Translog.Location location,
                       java.util.function.Consumer<java.lang.Exception> syncListener)
        Syncs the given location with the underlying storage unless already synced. This method might return immediately without actually fsyncing the location until the sync listener is called. Yet, unless there is already another thread fsyncing the transaction log the caller thread will be hijacked to run the fsync for all pending fsync operations. This method allows indexing threads to continue indexing without blocking on fsync calls. We ensure that there is only one thread blocking on the sync an all others can continue indexing. NOTE: if the syncListener throws an exception when it's processed the exception will only be logged. Users should make sure that the listener handles all exception cases internally.

      • sync

        public void sync()
          throws java.io.IOException
        Throws:
        java.io.IOException

      • isSyncNeeded

        public boolean isSyncNeeded()
        Checks if the underlying storage sync is required.

      • getTranslogDurability

        public Translog.Durability getTranslogDurability()
        Returns the current translog durability mode

      • afterWriteOperation

        public void afterWriteOperation()
        Schedules a flush or translog generation roll if needed but will not schedule more than one concurrently. The operation will be executed asynchronously on the flush thread pool.

      • isRefreshNeeded

        public boolean isRefreshNeeded()
        Returns true iff one or more changes to the engine are not visible to via the current searcher *or* there are pending refresh listeners. Otherwise false.
        Throws:
        org.apache.lucene.store.AlreadyClosedException - if the engine or internal indexwriter in the engine is already closed

      • addRefreshListener

        public void addRefreshListener​(Translog.Location location,
                               java.util.function.Consumer<java.lang.Boolean> listener)
        Add a listener for refreshes.
        Parameters:
        location - the location to listen for
        listener - for the refresh. Called with true if registering the listener ran it out of slots and forced a refresh. Called with false otherwise.