Class BlobStoreRepository

java.lang.Object
org.elasticsearch.common.component.AbstractLifecycleComponent
org.elasticsearch.repositories.blobstore.BlobStoreRepository
All Implemented Interfaces:
java.io.Closeable, java.lang.AutoCloseable, LifecycleComponent, Releasable, Repository
Direct Known Subclasses:
FsRepository

public abstract class BlobStoreRepository
extends AbstractLifecycleComponent
implements Repository
BlobStore - based implementation of Snapshot Repository

This repository works with any BlobStore implementation. The blobStore could be (and preferred) lazy initialized in createBlobStore().

For in depth documentation on how exactly implementations of this class interact with the snapshot functionality please refer to the documentation of the package org.elasticsearch.repositories.blobstore.
  • Field Details

  • Constructor Details

    • BlobStoreRepository

      protected BlobStoreRepository​(RepositoryMetadata metadata, boolean compress, org.elasticsearch.common.xcontent.NamedXContentRegistry namedXContentRegistry, ClusterService clusterService)
      Constructs new BlobStoreRepository
      Parameters:
      metadata - The metadata for this repository including name and settings
      clusterService - ClusterService
  • Method Details

    • doStart

      protected void doStart()
      Specified by:
      doStart in class AbstractLifecycleComponent
    • doStop

      protected void doStop()
      Specified by:
      doStop in class AbstractLifecycleComponent
    • doClose

      protected void doClose()
      Specified by:
      doClose in class AbstractLifecycleComponent
    • executeConsistentStateUpdate

      public void executeConsistentStateUpdate​(java.util.function.Function<RepositoryData,​ClusterStateUpdateTask> createUpdateTask, java.lang.String source, java.util.function.Consumer<java.lang.Exception> onFailure)
      Description copied from interface: Repository
      Execute a cluster state update with a consistent view of the current RepositoryData. The ClusterState passed to the task generated through createUpdateTask is guaranteed to point at the same state for this repository as the did the state at the time the RepositoryData was loaded. This allows for operations on the repository that need a consistent view of both the cluster state and the repository contents at one point in time like for example, checking if a snapshot is in the repository before adding the delete operation for it to the cluster state.
      Specified by:
      executeConsistentStateUpdate in interface Repository
      Parameters:
      createUpdateTask - function to supply cluster state update task
      source - the source of the cluster state update task
      onFailure - error handler invoked on failure to get a consistent view of the current RepositoryData
    • updateState

      public void updateState​(ClusterState state)
      Description copied from interface: Repository
      Update the repository with the incoming cluster state. This method is invoked from RepositoriesService.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent) and thus the same semantics as with ClusterStateApplier.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent) apply for the ClusterState that is passed here.
      Specified by:
      updateState in interface Repository
      Parameters:
      state - new cluster state
    • threadPool

      public ThreadPool threadPool()
    • getBlobStore

      protected BlobStore getBlobStore()
    • blobContainer

      protected BlobContainer blobContainer()
      maintains single lazy instance of BlobContainer
    • blobStore

      public BlobStore blobStore()
      Maintains single lazy instance of BlobStore. Public for testing.
    • createBlobStore

      protected abstract BlobStore createBlobStore() throws java.lang.Exception
      Creates new BlobStore to read and write data.
      Throws:
      java.lang.Exception
    • basePath

      public abstract BlobPath basePath()
      Returns base path of the repository
    • isCompress

      protected final boolean isCompress()
      Returns true if metadata and snapshot files should be compressed
      Returns:
      true if compression is needed
    • chunkSize

      protected ByteSizeValue chunkSize()
      Returns data file chunk size.

      This method should return null if no chunking is needed.

      Returns:
      chunk size
    • getMetadata

      public RepositoryMetadata getMetadata()
      Description copied from interface: Repository
      Returns metadata about this repository.
      Specified by:
      getMetadata in interface Repository
    • stats

      public RepositoryStats stats()
      Description copied from interface: Repository
      Returns stats on the repository usage
      Specified by:
      stats in interface Repository
    • initializeSnapshot

      public void initializeSnapshot​(SnapshotId snapshotId, java.util.List<IndexId> indices, Metadata clusterMetadata)
      Description copied from interface: Repository
      Starts snapshotting process
      Specified by:
      initializeSnapshot in interface Repository
      Parameters:
      snapshotId - snapshot id
      indices - list of indices to be snapshotted
      clusterMetadata - cluster metadata
    • deleteSnapshots

      public void deleteSnapshots​(java.util.Collection<SnapshotId> snapshotIds, long repositoryStateId, Version repositoryMetaVersion, ActionListener<java.lang.Void> listener)
      Description copied from interface: Repository
      Deletes snapshots
      Specified by:
      deleteSnapshots in interface Repository
      Parameters:
      snapshotIds - snapshot ids
      repositoryStateId - the unique id identifying the state of the repository when the snapshot deletion began
      repositoryMetaVersion - version of the updated repository metadata to write
      listener - completion listener
    • cleanup

      public void cleanup​(long repositoryStateId, Version repositoryMetaVersion, ActionListener<RepositoryCleanupResult> listener)
      Runs cleanup actions on the repository. Increments the repository state id by one before executing any modifications on the repository. TODO: Add shard level cleanups
      • Deleting stale indices cleanupStaleIndices(java.util.Map<java.lang.String, org.elasticsearch.common.blobstore.BlobContainer>, java.util.Set<java.lang.String>)
      • Deleting unreferenced root level blobs cleanupStaleRootFiles(java.util.List<java.lang.String>)
      Parameters:
      repositoryStateId - Current repository state id
      repositoryMetaVersion - version of the updated repository metadata to write
      listener - Listener to complete when done
    • finalizeSnapshot

      public void finalizeSnapshot​(SnapshotId snapshotId, ShardGenerations shardGenerations, long startTime, java.lang.String failure, int totalShards, java.util.List<SnapshotShardFailure> shardFailures, long repositoryStateId, boolean includeGlobalState, Metadata clusterMetadata, java.util.Map<java.lang.String,​java.lang.Object> userMetadata, Version repositoryMetaVersion, java.util.function.Function<ClusterState,​ClusterState> stateTransformer, ActionListener<org.elasticsearch.common.collect.Tuple<RepositoryData,​SnapshotInfo>> listener)
      Description copied from interface: Repository
      Finalizes snapshotting process

      This method is called on master after all shards are snapshotted.

      Specified by:
      finalizeSnapshot in interface Repository
      Parameters:
      snapshotId - snapshot id
      shardGenerations - updated shard generations
      startTime - start time of the snapshot
      failure - global failure reason or null
      totalShards - total number of shards
      shardFailures - list of shard failures
      repositoryStateId - the unique id identifying the state of the repository when the snapshot began
      includeGlobalState - include cluster global state
      clusterMetadata - cluster metadata
      userMetadata - user metadata
      repositoryMetaVersion - version of the updated repository metadata to write
      stateTransformer - a function that filters the last cluster state update that the snapshot finalization will execute and is used to remove any state tracked for the in-progress snapshot from the cluster state
      listener - listener to be invoked with the new RepositoryData and the snapshot's SnapshotInfo completion of the snapshot
    • getSnapshotInfo

      public SnapshotInfo getSnapshotInfo​(SnapshotId snapshotId)
      Description copied from interface: Repository
      Reads snapshot description from repository.
      Specified by:
      getSnapshotInfo in interface Repository
      Parameters:
      snapshotId - snapshot id
      Returns:
      information about snapshot
    • getSnapshotGlobalMetadata

      public Metadata getSnapshotGlobalMetadata​(SnapshotId snapshotId)
      Description copied from interface: Repository
      Returns global metadata associated with the snapshot.
      Specified by:
      getSnapshotGlobalMetadata in interface Repository
      Parameters:
      snapshotId - the snapshot id to load the global metadata from
      Returns:
      the global metadata about the snapshot
    • getSnapshotIndexMetadata

      public IndexMetadata getSnapshotIndexMetadata​(SnapshotId snapshotId, IndexId index) throws java.io.IOException
      Description copied from interface: Repository
      Returns the index metadata associated with the snapshot.
      Specified by:
      getSnapshotIndexMetadata in interface Repository
      Parameters:
      snapshotId - the snapshot id to load the index metadata from
      index - the IndexId to load the metadata from
      Returns:
      the index metadata about the given index for the given snapshot
      Throws:
      java.io.IOException
    • shardContainer

      public BlobContainer shardContainer​(IndexId indexId, int shardId)
    • getSnapshotThrottleTimeInNanos

      public long getSnapshotThrottleTimeInNanos()
      Description copied from interface: Repository
      Returns snapshot throttle time in nanoseconds
      Specified by:
      getSnapshotThrottleTimeInNanos in interface Repository
    • getRestoreThrottleTimeInNanos

      public long getRestoreThrottleTimeInNanos()
      Description copied from interface: Repository
      Returns restore throttle time in nanoseconds
      Specified by:
      getRestoreThrottleTimeInNanos in interface Repository
    • assertSnapshotOrGenericThread

      protected void assertSnapshotOrGenericThread()
    • startVerification

      public java.lang.String startVerification()
      Description copied from interface: Repository
      Verifies repository on the master node and returns the verification token.

      If the verification token is not null, it's passed to all data nodes for verification. If it's null - no additional verification is required

      Specified by:
      startVerification in interface Repository
      Returns:
      verification token that should be passed to all Index Shard Repositories for additional verification or null
    • endVerification

      public void endVerification​(java.lang.String seed)
      Description copied from interface: Repository
      Called at the end of repository verification process.

      This method should perform all necessary cleanup of the temporary files created in the repository

      Specified by:
      endVerification in interface Repository
      Parameters:
      seed - verification request generated by Repository.startVerification() command
    • getRepositoryData

      public void getRepositoryData​(ActionListener<RepositoryData> listener)
      Description copied from interface: Repository
      Returns a RepositoryData to describe the data in the repository, including the snapshots and the indices across all snapshots found in the repository. Throws a RepositoryException if there was an error in reading the data.
      Specified by:
      getRepositoryData in interface Repository
    • isReadOnly

      public boolean isReadOnly()
      Description copied from interface: Repository
      Returns true if the repository supports only read operations
      Specified by:
      isReadOnly in interface Repository
      Returns:
      true if the repository is read/only
    • writeIndexGen

      protected void writeIndexGen​(RepositoryData repositoryData, long expectedGen, boolean writeShardGens, java.util.function.Function<ClusterState,​ClusterState> stateFilter, ActionListener<RepositoryData> listener)
      Writing a new index generation is a three step process. First, the RepositoryMetadata entry for this repository is set into a pending state by incrementing its pending generation P while its safe generation N remains unchanged. Second, the updated RepositoryData is written to generation P + 1. Lastly, the RepositoryMetadata entry for this repository is updated to the new generation P + 1 and thus pending and safe generation are set to the same value marking the end of the update of the repository data.
      Parameters:
      repositoryData - RepositoryData to write
      expectedGen - expected repository generation at the start of the operation
      writeShardGens - whether to write ShardGenerations to the new RepositoryData blob
      stateFilter - filter for the last cluster state update executed by this method
      listener - completion listener
    • snapshotShard

      public void snapshotShard​(Store store, MapperService mapperService, SnapshotId snapshotId, IndexId indexId, org.apache.lucene.index.IndexCommit snapshotIndexCommit, java.lang.String shardStateIdentifier, IndexShardSnapshotStatus snapshotStatus, Version repositoryMetaVersion, java.util.Map<java.lang.String,​java.lang.Object> userMetadata, ActionListener<java.lang.String> listener)
      Description copied from interface: Repository
      Creates a snapshot of the shard based on the index commit point.

      The index commit point can be obtained by using Engine.acquireLastIndexCommit(boolean) method. Repository implementations shouldn't release the snapshot index commit point. It is done by the method caller.

      As snapshot process progresses, implementation of this method should update IndexShardSnapshotStatus object and check IndexShardSnapshotStatus.isAborted() to see if the snapshot process should be aborted.

      Specified by:
      snapshotShard in interface Repository
      Parameters:
      store - store to be snapshotted
      mapperService - the shards mapper service
      snapshotId - snapshot id
      indexId - id for the index being snapshotted
      snapshotIndexCommit - commit point
      shardStateIdentifier - a unique identifier of the state of the shard that is stored with the shard's snapshot and used to detect if the shard has changed between snapshots. If null is passed as the identifier snapshotting will be done by inspecting the physical files referenced by snapshotIndexCommit
      snapshotStatus - snapshot status
      repositoryMetaVersion - version of the updated repository metadata to write
      userMetadata - user metadata of the snapshot found in SnapshotsInProgress.Entry.userMetadata()
      listener - listener invoked on completion
    • restoreShard

      public void restoreShard​(Store store, SnapshotId snapshotId, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState, ActionListener<java.lang.Void> listener)
      Description copied from interface: Repository
      Restores snapshot of the shard.

      The index can be renamed on restore, hence different shardId and snapshotShardId are supplied.

      Specified by:
      restoreShard in interface Repository
      Parameters:
      store - the store to restore the index into
      snapshotId - snapshot id
      indexId - id of the index in the repository from which the restore is occurring
      snapshotShardId - shard id (in the snapshot)
      recoveryState - recovery state
      listener - listener to invoke once done
    • maybeRateLimitRestores

      public java.io.InputStream maybeRateLimitRestores​(java.io.InputStream stream)
    • maybeRateLimitSnapshots

      public java.io.InputStream maybeRateLimitSnapshots​(java.io.InputStream stream)
    • getShardSnapshotStatus

      public IndexShardSnapshotStatus getShardSnapshotStatus​(SnapshotId snapshotId, IndexId indexId, ShardId shardId)
      Description copied from interface: Repository
      Retrieve shard snapshot status for the stored snapshot
      Specified by:
      getShardSnapshotStatus in interface Repository
      Parameters:
      snapshotId - snapshot id
      indexId - the snapshotted index id for the shard to get status for
      shardId - shard id
      Returns:
      snapshot status
    • verify

      public void verify​(java.lang.String seed, DiscoveryNode localNode)
      Description copied from interface: Repository
      Verifies repository settings on data node.
      Specified by:
      verify in interface Repository
      Parameters:
      seed - value returned by Repository.startVerification()
      localNode - the local node information, for inclusion in verification errors
    • toString

      public java.lang.String toString()
      Overrides:
      toString in class java.lang.Object
    • loadShardSnapshot

      public BlobStoreIndexShardSnapshot loadShardSnapshot​(BlobContainer shardContainer, SnapshotId snapshotId)
      Loads information about shard snapshot