Package org.elasticsearch.repositories.blobstore

Class BlobStoreRepository

java.lang.Object
org.elasticsearch.common.component.AbstractLifecycleComponent
org.elasticsearch.repositories.blobstore.BlobStoreRepository
All Implemented Interfaces:
Closeable, AutoCloseable, LifecycleComponent, org.elasticsearch.core.Releasable, Repository
Direct Known Subclasses:
FsRepository, MeteredBlobStoreRepository
public abstract class BlobStoreRepository extends AbstractLifecycleComponent implements Repository
BlobStore - based implementation of Snapshot Repository

This repository works with any BlobStore implementation. The blobStore could be (and preferred) lazy initialized in createBlobStore().

For in depth documentation on how exactly implementations of this class interact with the snapshot functionality please refer to the documentation of the package org.elasticsearch.repositories.blobstore.

  • Field Details

    • metadata

      protected volatile RepositoryMetadata metadata

    • threadPool

      protected final ThreadPool threadPool

    • SNAPSHOT_PREFIX

      public static final String SNAPSHOT_PREFIX
      See Also:

    • INDEX_FILE_PREFIX

      public static final String INDEX_FILE_PREFIX
      See Also:

    • INDEX_LATEST_BLOB

      public static final String INDEX_LATEST_BLOB
      See Also:

    • METADATA_PREFIX

      public static final String METADATA_PREFIX
      See Also:

    • METADATA_NAME_FORMAT

      public static final String METADATA_NAME_FORMAT
      See Also:

    • SNAPSHOT_NAME_FORMAT

      public static final String SNAPSHOT_NAME_FORMAT
      See Also:

    • UPLOADED_DATA_BLOB_PREFIX

      public static final String UPLOADED_DATA_BLOB_PREFIX
      See Also:

    • URL_REPOSITORY_TYPE

      public static final String URL_REPOSITORY_TYPE
      See Also:

    • READONLY_SETTING_KEY

      public static final String READONLY_SETTING_KEY
      All BlobStoreRepository implementations can be made read-only by setting this key to true in their settings.
      See Also:

    • ALLOW_CONCURRENT_MODIFICATION

      public static final Setting<Boolean> ALLOW_CONCURRENT_MODIFICATION
      When set to true, bestEffortConsistency will be set to true and concurrent modifications of the repository contents will not result in the repository being marked as corrupted. Note: This setting is intended as a backwards compatibility solution for 7.x and will go away in 8.

    • CACHE_REPOSITORY_DATA

      public static final Setting<Boolean> CACHE_REPOSITORY_DATA
      Setting to disable caching of the latest repository data.

    • BUFFER_SIZE_SETTING

      public static final Setting<ByteSizeValue> BUFFER_SIZE_SETTING
      Size hint for the IO buffer size to use when reading from and writing to the repository.

    • SUPPORT_URL_REPO

      public static final Setting<Boolean> SUPPORT_URL_REPO
      Setting to disable writing the index.latest blob which enables the contents of this repository to be used with a url-repository.

    • MAX_SNAPSHOTS_SETTING

      public static final Setting<Integer> MAX_SNAPSHOTS_SETTING
      Setting that defines the maximum number of snapshots to which the repository may grow. Trying to create a snapshot into the repository that would move it above this size will throw an exception.

    • USE_FOR_PEER_RECOVERY_SETTING

      public static final Setting<Boolean> USE_FOR_PEER_RECOVERY_SETTING
      Setting that defines if the repository should be used to recover index files during peer recoveries.

    • supportURLRepo

      protected final boolean supportURLRepo

    • GLOBAL_METADATA_FORMAT

      public static final ChecksumBlobStoreFormat<Metadata> GLOBAL_METADATA_FORMAT

    • INDEX_METADATA_FORMAT

      public static final ChecksumBlobStoreFormat<IndexMetadata> INDEX_METADATA_FORMAT

    • SNAPSHOT_FORMAT

      public static final ChecksumBlobStoreFormat<SnapshotInfo> SNAPSHOT_FORMAT

    • INDEX_SHARD_SNAPSHOT_FORMAT

      public static final ChecksumBlobStoreFormat<BlobStoreIndexShardSnapshot> INDEX_SHARD_SNAPSHOT_FORMAT

    • INDEX_SHARD_SNAPSHOTS_FORMAT

      public static final ChecksumBlobStoreFormat<BlobStoreIndexShardSnapshots> INDEX_SHARD_SNAPSHOTS_FORMAT

    • MAX_SNAPSHOT_BYTES_PER_SEC

      public static final Setting<ByteSizeValue> MAX_SNAPSHOT_BYTES_PER_SEC

    • MAX_RESTORE_BYTES_PER_SEC

      public static final Setting<ByteSizeValue> MAX_RESTORE_BYTES_PER_SEC

    • bigArrays

      protected final BigArrays bigArrays

    • bufferSize

      protected final int bufferSize
      IO buffer size hint for reading and writing to the underlying blob store.

  • Constructor Details

    • BlobStoreRepository

      protected BlobStoreRepository(RepositoryMetadata metadata, boolean compress, org.elasticsearch.xcontent.NamedXContentRegistry namedXContentRegistry, ClusterService clusterService, BigArrays bigArrays, RecoverySettings recoverySettings)
      Constructs new BlobStoreRepository
      Parameters:
      metadata - The metadata for this repository including name and settings
      clusterService - ClusterService

  • Method Details

    • doStart

      protected void doStart()
      Specified by:
      doStart in class AbstractLifecycleComponent

    • doStop

      protected void doStop()
      Specified by:
      doStop in class AbstractLifecycleComponent

    • doClose

      protected void doClose()
      Specified by:
      doClose in class AbstractLifecycleComponent

    • awaitIdle

      public void awaitIdle()
      Description copied from interface: Repository
      Block until all in-flight operations for this repository have completed. Must only be called after this instance has been closed by a call to stop Releasable.close(). Waiting for ongoing operations should be implemented here instead of in LifecycleComponent.stop() or Releasable.close() hooks of this interface as these are expected to be called on the cluster state applier thread (which must not block) if a repository is removed from the cluster. This method is intended to be called on node shutdown instead as a means to ensure no repository operations are leaked.
      Specified by:
      awaitIdle in interface Repository

    • executeConsistentStateUpdate

      public void executeConsistentStateUpdate(Function<RepositoryData,ClusterStateUpdateTask> createUpdateTask, String source, Consumer<Exception> onFailure)
      Description copied from interface: Repository
      Execute a cluster state update with a consistent view of the current RepositoryData. The ClusterState passed to the task generated through createUpdateTask is guaranteed to point at the same state for this repository as the did the state at the time the RepositoryData was loaded. This allows for operations on the repository that need a consistent view of both the cluster state and the repository contents at one point in time like for example, checking if a snapshot is in the repository before adding the delete operation for it to the cluster state.
      Specified by:
      executeConsistentStateUpdate in interface Repository
      Parameters:
      createUpdateTask - function to supply cluster state update task
      source - the source of the cluster state update task
      onFailure - error handler invoked on failure to get a consistent view of the current RepositoryData

    • cloneShardSnapshot

      public void cloneShardSnapshot(SnapshotId source, SnapshotId target, RepositoryShardId shardId, @Nullable ShardGeneration shardGeneration, ActionListener<ShardSnapshotResult> listener)
      Description copied from interface: Repository
      Clones a shard snapshot.
      Specified by:
      cloneShardSnapshot in interface Repository
      Parameters:
      source - source snapshot
      target - target snapshot
      shardId - shard id
      shardGeneration - shard generation in repo
      listener - listener to complete with new shard generation once clone has completed

    • canUpdateInPlace

      public boolean canUpdateInPlace(Settings updatedSettings, Set<String> ignoredSettings)
      Description copied from interface: Repository
      Check if this instances Settings can be changed to the provided updated settings without recreating the repository.
      Specified by:
      canUpdateInPlace in interface Repository
      Parameters:
      updatedSettings - new repository settings
      ignoredSettings - setting names to ignore even if changed
      Returns:
      true if the repository can be updated in place

    • updateState

      public void updateState(ClusterState state)
      Description copied from interface: Repository
      Update the repository with the incoming cluster state. This method is invoked from RepositoriesService.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent) and thus the same semantics as with ClusterStateApplier.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent) apply for the ClusterState that is passed here.
      Specified by:
      updateState in interface Repository
      Parameters:
      state - new cluster state

    • threadPool

      public ThreadPool threadPool()

    • getBlobStore

      protected BlobStore getBlobStore()

    • blobContainer

      protected BlobContainer blobContainer()
      maintains single lazy instance of BlobContainer

    • blobStore

      public BlobStore blobStore()
      Maintains single lazy instance of BlobStore. Public for testing.

    • createBlobStore

      protected abstract BlobStore createBlobStore() throws Exception
      Creates new BlobStore to read and write data.
      Throws:
      Exception

    • basePath

      public abstract BlobPath basePath()
      Returns base path of the repository

    • isCompress

      protected final boolean isCompress()
      Returns true if metadata and snapshot files should be compressed
      Returns:
      true if compression is needed

    • chunkSize

      protected ByteSizeValue chunkSize()
      Returns data file chunk size.

      This method should return null if no chunking is needed.

      Returns:
      chunk size

    • getMetadata

      public RepositoryMetadata getMetadata()
      Description copied from interface: Repository
      Returns metadata about this repository.
      Specified by:
      getMetadata in interface Repository

    • stats

      public RepositoryStats stats()
      Description copied from interface: Repository
      Returns stats on the repository usage
      Specified by:
      stats in interface Repository

    • initializeSnapshot

      public void initializeSnapshot(SnapshotId snapshotId, List<IndexId> indices, Metadata clusterMetadata)
      Description copied from interface: Repository
      Starts snapshotting process
      Specified by:
      initializeSnapshot in interface Repository
      Parameters:
      snapshotId - snapshot id
      indices - list of indices to be snapshotted
      clusterMetadata - cluster metadata

    • deleteSnapshots

      public void deleteSnapshots(Collection<SnapshotId> snapshotIds, long repositoryStateId, Version repositoryMetaVersion, ActionListener<RepositoryData> listener)
      Description copied from interface: Repository
      Deletes snapshots
      Specified by:
      deleteSnapshots in interface Repository
      Parameters:
      snapshotIds - snapshot ids
      repositoryStateId - the unique id identifying the state of the repository when the snapshot deletion began
      repositoryMetaVersion - version of the updated repository metadata to write
      listener - completion listener

    • cleanup

      public void cleanup(long repositoryStateId, Version repositoryMetaVersion, ActionListener<RepositoryCleanupResult> listener)
      Runs cleanup actions on the repository. Increments the repository state id by one before executing any modifications on the repository. TODO: Add shard level cleanups TODO: Add unreferenced index metadata cleanup
      • Deleting stale indices cleanupStaleIndices(java.util.Map<java.lang.String, org.elasticsearch.common.blobstore.BlobContainer>, java.util.Set<java.lang.String>)
      • Deleting unreferenced root level blobs cleanupStaleRootFiles(long, java.util.Collection<org.elasticsearch.snapshots.SnapshotId>, java.util.List<java.lang.String>)
      Parameters:
      repositoryStateId - Current repository state id
      repositoryMetaVersion - version of the updated repository metadata to write
      listener - Listener to complete when done

    • finalizeSnapshot

      public void finalizeSnapshot(FinalizeSnapshotContext finalizeSnapshotContext)
      Description copied from interface: Repository
      Finalizes snapshotting process

      This method is called on master after all shards are snapshotted.

      Specified by:
      finalizeSnapshot in interface Repository
      Parameters:
      finalizeSnapshotContext - finalization context

    • getSnapshotInfo

      public void getSnapshotInfo(GetSnapshotInfoContext context)
      Description copied from interface: Repository
      Reads snapshot descriptions from the repository.
      Specified by:
      getSnapshotInfo in interface Repository
      Parameters:
      context - get-snapshot-info-context

    • getSnapshotGlobalMetadata

      public Metadata getSnapshotGlobalMetadata(SnapshotId snapshotId)
      Description copied from interface: Repository
      Returns global metadata associated with the snapshot.
      Specified by:
      getSnapshotGlobalMetadata in interface Repository
      Parameters:
      snapshotId - the snapshot id to load the global metadata from
      Returns:
      the global metadata about the snapshot

    • getSnapshotIndexMetaData

      public IndexMetadata getSnapshotIndexMetaData(RepositoryData repositoryData, SnapshotId snapshotId, IndexId index) throws IOException
      Description copied from interface: Repository
      Returns the index metadata associated with the snapshot.
      Specified by:
      getSnapshotIndexMetaData in interface Repository
      Parameters:
      repositoryData - current RepositoryData
      snapshotId - the snapshot id to load the index metadata from
      index - the IndexId to load the metadata from
      Returns:
      the index metadata about the given index for the given snapshot
      Throws:
      IOException

    • shardContainer

      public BlobContainer shardContainer(IndexId indexId, int shardId)

    • getSnapshotThrottleTimeInNanos

      public long getSnapshotThrottleTimeInNanos()
      Description copied from interface: Repository
      Returns snapshot throttle time in nanoseconds
      Specified by:
      getSnapshotThrottleTimeInNanos in interface Repository

    • getRestoreThrottleTimeInNanos

      public long getRestoreThrottleTimeInNanos()
      Description copied from interface: Repository
      Returns restore throttle time in nanoseconds
      Specified by:
      getRestoreThrottleTimeInNanos in interface Repository

    • assertSnapshotOrGenericThread

      protected void assertSnapshotOrGenericThread()

    • startVerification

      public String startVerification()
      Description copied from interface: Repository
      Verifies repository on the master node and returns the verification token.

      If the verification token is not null, it's passed to all data nodes for verification. If it's null - no additional verification is required

      Specified by:
      startVerification in interface Repository
      Returns:
      verification token that should be passed to all Index Shard Repositories for additional verification or null

    • endVerification

      public void endVerification(String seed)
      Description copied from interface: Repository
      Called at the end of repository verification process.

      This method should perform all necessary cleanup of the temporary files created in the repository

      Specified by:
      endVerification in interface Repository
      Parameters:
      seed - verification request generated by Repository.startVerification() command

    • getRepositoryData

      public void getRepositoryData(ActionListener<RepositoryData> listener)
      Description copied from interface: Repository
      Returns a RepositoryData to describe the data in the repository, including the snapshots and the indices across all snapshots found in the repository. Throws a RepositoryException if there was an error in reading the data.
      Specified by:
      getRepositoryData in interface Repository
      Parameters:
      listener - listener that may be resolved on different kinds of threads including transport and cluster state applier threads and therefore must fork to a new thread for executing any long running actions

    • isReadOnly

      public boolean isReadOnly()
      Description copied from interface: Repository
      Returns true if the repository supports only read operations
      Specified by:
      isReadOnly in interface Repository
      Returns:
      true if the repository is read/only

    • writeIndexGen

      protected void writeIndexGen(RepositoryData repositoryData, long expectedGen, Version version, Function<ClusterState,ClusterState> stateFilter, ActionListener<RepositoryData> listener)
      Writing a new index generation is a three step process. First, the RepositoryMetadata entry for this repository is set into a pending state by incrementing its pending generation P while its safe generation N remains unchanged. Second, the updated RepositoryData is written to generation P + 1. Lastly, the RepositoryMetadata entry for this repository is updated to the new generation P + 1 and thus pending and safe generation are set to the same value marking the end of the update of the repository data.
      Parameters:
      repositoryData - RepositoryData to write
      expectedGen - expected repository generation at the start of the operation
      version - version of the repository metadata to write
      stateFilter - filter for the last cluster state update executed by this method
      listener - completion listener

    • snapshotShard

      public void snapshotShard(SnapshotShardContext context)
      Description copied from interface: Repository
      Creates a snapshot of the shard referenced by the given SnapshotShardContext.

      As snapshot process progresses, implementation of this method should update IndexShardSnapshotStatus object returned by SnapshotShardContext.status() and check its IndexShardSnapshotStatus.isAborted() to see if the snapshot process should be aborted.

      Specified by:
      snapshotShard in interface Repository
      Parameters:
      context - snapshot shard context that must be completed via SnapshotShardContext.onResponse(org.elasticsearch.repositories.ShardSnapshotResult) or ActionListener.Delegating.onFailure(java.lang.Exception)

    • restoreShard

      public void restoreShard(Store store, SnapshotId snapshotId, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState, ActionListener<Void> listener)
      Description copied from interface: Repository
      Restores snapshot of the shard.

      The index can be renamed on restore, hence different shardId and snapshotShardId are supplied.

      Specified by:
      restoreShard in interface Repository
      Parameters:
      store - the store to restore the index into
      snapshotId - snapshot id
      indexId - id of the index in the repository from which the restore is occurring
      snapshotShardId - shard id (in the snapshot)
      recoveryState - recovery state
      listener - listener to invoke once done

    • maybeRateLimitRestores

      public InputStream maybeRateLimitRestores(InputStream stream)
      Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is reported to the given listener and not otherwise recorded in the value returned by getRestoreThrottleTimeInNanos().

    • maybeRateLimitRestores

      public InputStream maybeRateLimitRestores(InputStream stream, RateLimitingInputStream.Listener throttleListener)
      Wrap the restore rate limiter (controlled by the repository setting `max_restore_bytes_per_sec` and the cluster setting `indices.recovery.max_bytes_per_sec`) around the given stream. Any throttling is recorded in the value returned by getRestoreThrottleTimeInNanos().

    • maybeRateLimitSnapshots

      public InputStream maybeRateLimitSnapshots(InputStream stream)
      Wrap the snapshot rate limiter (controlled by the repository setting `max_snapshot_bytes_per_sec`) around the given stream. Any throttling is recorded in the value returned by getSnapshotThrottleTimeInNanos().

    • maybeRateLimitSnapshots

      public InputStream maybeRateLimitSnapshots(InputStream stream, RateLimitingInputStream.Listener throttleListener)
      Wrap the snapshot rate limiter (controlled by the repository setting `max_snapshot_bytes_per_sec`) around the given stream. Any throttling is reported to the given listener and not otherwise recorded in the value returned by getSnapshotThrottleTimeInNanos().

    • getShardSnapshotStatus

      public IndexShardSnapshotStatus getShardSnapshotStatus(SnapshotId snapshotId, IndexId indexId, ShardId shardId)
      Description copied from interface: Repository
      Retrieve shard snapshot status for the stored snapshot
      Specified by:
      getShardSnapshotStatus in interface Repository
      Parameters:
      snapshotId - snapshot id
      indexId - the snapshotted index id for the shard to get status for
      shardId - shard id
      Returns:
      snapshot status

    • verify

      public void verify(String seed, DiscoveryNode localNode)
      Description copied from interface: Repository
      Verifies repository settings on data node.
      Specified by:
      verify in interface Repository
      Parameters:
      seed - value returned by Repository.startVerification()
      localNode - the local node information, for inclusion in verification errors

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • loadShardSnapshot

      public BlobStoreIndexShardSnapshot loadShardSnapshot(BlobContainer shardContainer, SnapshotId snapshotId)
      Loads information about shard snapshot

    • getBlobStoreIndexShardSnapshots

      public BlobStoreIndexShardSnapshots getBlobStoreIndexShardSnapshots(IndexId indexId, int shardId, @Nullable ShardGeneration shardGen) throws IOException
      Loads all available snapshots in the repository using the given generation for a shard. When shardGen is null it tries to load it using the BwC mode, listing the available index- blobs in the shard container.
      Throws:
      IOException

    • supportURLRepo

      public boolean supportURLRepo()

    • hasAtomicOverwrites

      public boolean hasAtomicOverwrites()
      Returns:
      whether this repository performs overwrites atomically. In practice we only overwrite the `index.latest` blob so this is not very important, but the repository analyzer does test that overwrites happen atomically. It will skip those tests if the repository overrides this method to indicate that it does not support atomic overwrites.

    • getReadBufferSizeInBytes

      public int getReadBufferSizeInBytes()