Class BlobStoreRepository
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,LifecycleComponent
,Releasable
,Repository
- Direct Known Subclasses:
FsRepository
public abstract class BlobStoreRepository extends AbstractLifecycleComponent implements Repository
This repository works with any BlobStore
implementation. The blobStore could be (and preferred) lazy initialized in
createBlobStore()
.
org.elasticsearch.repositories.blobstore
.-
Nested Class Summary
Nested classes/interfaces inherited from interface org.elasticsearch.repositories.Repository
Repository.Factory
-
Field Summary
Fields Modifier and Type Field Description static Setting<java.lang.Boolean>
ALLOW_CONCURRENT_MODIFICATION
When set totrue
,bestEffortConsistency
will be set totrue
and concurrent modifications of the repository contents will not result in the repository being marked as corrupted.static Setting<java.lang.Boolean>
COMPRESS_SETTING
When set to true metadata files are stored in compressed format.static java.lang.String
INDEX_FILE_PREFIX
static java.lang.String
INDEX_LATEST_BLOB
protected RepositoryMetaData
metadata
static java.lang.String
METADATA_NAME_FORMAT
static java.lang.String
METADATA_PREFIX
protected NamedXContentRegistry
namedXContentRegistry
static java.lang.String
SNAPSHOT_CODEC
static java.lang.String
SNAPSHOT_NAME_FORMAT
static java.lang.String
SNAPSHOT_PREFIX
protected ChecksumBlobStoreFormat<SnapshotInfo>
snapshotFormat
protected ThreadPool
threadPool
-
Constructor Summary
Constructors Modifier Constructor Description protected
BlobStoreRepository(RepositoryMetaData metadata, boolean compress, NamedXContentRegistry namedXContentRegistry, ClusterService clusterService)
Constructs new BlobStoreRepository -
Method Summary
Modifier and Type Method Description protected void
assertSnapshotOrGenericThread()
abstract BlobPath
basePath()
Returns base path of the repositoryprotected BlobContainer
blobContainer()
maintains single lazy instance ofBlobContainer
BlobStore
blobStore()
Maintains single lazy instance ofBlobStore
.protected ByteSizeValue
chunkSize()
Returns data file chunk size.void
cleanup(long repositoryStateId, boolean writeShardGens, ActionListener<RepositoryCleanupResult> listener)
Runs cleanup actions on the repository.protected abstract BlobStore
createBlobStore()
Creates new BlobStore to read and write data.void
deleteSnapshot(SnapshotId snapshotId, long repositoryStateId, boolean writeShardGens, ActionListener<java.lang.Void> listener)
Deletes snapshotprotected void
doClose()
protected void
doStart()
protected void
doStop()
void
endVerification(java.lang.String seed)
Called at the end of repository verification process.void
finalizeSnapshot(SnapshotId snapshotId, ShardGenerations shardGenerations, long startTime, java.lang.String failure, int totalShards, java.util.List<SnapshotShardFailure> shardFailures, long repositoryStateId, boolean includeGlobalState, MetaData clusterMetaData, java.util.Map<java.lang.String,java.lang.Object> userMetadata, boolean writeShardGens, ActionListener<SnapshotInfo> listener)
Finalizes snapshotting processprotected BlobStore
getBlobStore()
RepositoryMetaData
getMetadata()
Returns metadata about this repository.void
getRepositoryData(ActionListener<RepositoryData> listener)
Returns aRepositoryData
to describe the data in the repository, including the snapshots and the indices across all snapshots found in the repository.long
getRestoreThrottleTimeInNanos()
Returns restore throttle time in nanosecondsIndexShardSnapshotStatus
getShardSnapshotStatus(SnapshotId snapshotId, IndexId indexId, ShardId shardId)
Retrieve shard snapshot status for the stored snapshotMetaData
getSnapshotGlobalMetaData(SnapshotId snapshotId)
Returns global metadata associated with the snapshot.IndexMetaData
getSnapshotIndexMetaData(SnapshotId snapshotId, IndexId index)
Returns the index metadata associated with the snapshot.SnapshotInfo
getSnapshotInfo(SnapshotId snapshotId)
Reads snapshot description from repository.long
getSnapshotThrottleTimeInNanos()
Returns snapshot throttle time in nanosecondsvoid
initializeSnapshot(SnapshotId snapshotId, java.util.List<IndexId> indices, MetaData clusterMetaData)
Starts snapshotting processprotected boolean
isCompress()
Returns true if metadata and snapshot files should be compressedboolean
isReadOnly()
Returns true if the repository supports only read operationsvoid
restoreShard(Store store, SnapshotId snapshotId, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState, ActionListener<java.lang.Void> listener)
Restores snapshot of the shard.void
snapshotShard(Store store, MapperService mapperService, SnapshotId snapshotId, IndexId indexId, org.apache.lucene.index.IndexCommit snapshotIndexCommit, IndexShardSnapshotStatus snapshotStatus, boolean writeShardGens, ActionListener<java.lang.String> listener)
Creates a snapshot of the shard based on the index commit point.java.lang.String
startVerification()
Verifies repository on the master node and returns the verification token.ThreadPool
threadPool()
java.lang.String
toString()
void
updateState(ClusterState state)
Update the repository with the incoming cluster state.void
verify(java.lang.String seed, DiscoveryNode localNode)
Verifies repository settings on data node.protected void
writeIndexGen(RepositoryData repositoryData, long expectedGen, boolean writeShardGens, ActionListener<java.lang.Void> listener)
Writing a new index generation is a three step process.Methods inherited from class org.elasticsearch.common.component.AbstractLifecycleComponent
addLifecycleListener, close, lifecycleState, removeLifecycleListener, start, stop
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
Methods inherited from interface org.elasticsearch.common.component.LifecycleComponent
addLifecycleListener, lifecycleState, removeLifecycleListener, start, stop
-
Field Details
-
metadata
-
namedXContentRegistry
-
threadPool
-
SNAPSHOT_PREFIX
public static final java.lang.String SNAPSHOT_PREFIX- See Also:
- Constant Field Values
-
SNAPSHOT_CODEC
public static final java.lang.String SNAPSHOT_CODEC- See Also:
- Constant Field Values
-
INDEX_FILE_PREFIX
public static final java.lang.String INDEX_FILE_PREFIX- See Also:
- Constant Field Values
-
INDEX_LATEST_BLOB
public static final java.lang.String INDEX_LATEST_BLOB- See Also:
- Constant Field Values
-
METADATA_PREFIX
public static final java.lang.String METADATA_PREFIX- See Also:
- Constant Field Values
-
METADATA_NAME_FORMAT
public static final java.lang.String METADATA_NAME_FORMAT- See Also:
- Constant Field Values
-
SNAPSHOT_NAME_FORMAT
public static final java.lang.String SNAPSHOT_NAME_FORMAT- See Also:
- Constant Field Values
-
COMPRESS_SETTING
When set to true metadata files are stored in compressed format. This setting doesn’t affect index files that are already compressed by default. Changing the setting does not invalidate existing files since reads do not observe the setting, instead they examine the file to see if it is compressed or not. -
ALLOW_CONCURRENT_MODIFICATION
When set totrue
,bestEffortConsistency
will be set totrue
and concurrent modifications of the repository contents will not result in the repository being marked as corrupted. Note: This setting is intended as a backwards compatibility solution for 7.x and will go away in 8. -
snapshotFormat
-
-
Constructor Details
-
BlobStoreRepository
protected BlobStoreRepository(RepositoryMetaData metadata, boolean compress, NamedXContentRegistry namedXContentRegistry, ClusterService clusterService)Constructs new BlobStoreRepository- Parameters:
metadata
- The metadata for this repository including name and settingsclusterService
- ClusterService
-
-
Method Details
-
doStart
protected void doStart()- Specified by:
doStart
in classAbstractLifecycleComponent
-
doStop
protected void doStop()- Specified by:
doStop
in classAbstractLifecycleComponent
-
doClose
protected void doClose()- Specified by:
doClose
in classAbstractLifecycleComponent
-
updateState
Description copied from interface:Repository
Update the repository with the incoming cluster state. This method is invoked fromRepositoriesService.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent)
and thus the same semantics as withClusterStateApplier.applyClusterState(org.elasticsearch.cluster.ClusterChangedEvent)
apply for theClusterState
that is passed here.- Specified by:
updateState
in interfaceRepository
- Parameters:
state
- new cluster state
-
threadPool
-
getBlobStore
-
blobContainer
maintains single lazy instance ofBlobContainer
-
blobStore
Maintains single lazy instance ofBlobStore
. Public for testing. -
createBlobStore
Creates new BlobStore to read and write data.- Throws:
java.lang.Exception
-
basePath
Returns base path of the repository -
isCompress
protected final boolean isCompress()Returns true if metadata and snapshot files should be compressed- Returns:
- true if compression is needed
-
chunkSize
Returns data file chunk size.This method should return null if no chunking is needed.
- Returns:
- chunk size
-
getMetadata
Description copied from interface:Repository
Returns metadata about this repository.- Specified by:
getMetadata
in interfaceRepository
-
initializeSnapshot
public void initializeSnapshot(SnapshotId snapshotId, java.util.List<IndexId> indices, MetaData clusterMetaData)Description copied from interface:Repository
Starts snapshotting process- Specified by:
initializeSnapshot
in interfaceRepository
- Parameters:
snapshotId
- snapshot idindices
- list of indices to be snapshottedclusterMetaData
- cluster metadata
-
deleteSnapshot
public void deleteSnapshot(SnapshotId snapshotId, long repositoryStateId, boolean writeShardGens, ActionListener<java.lang.Void> listener)Description copied from interface:Repository
Deletes snapshot- Specified by:
deleteSnapshot
in interfaceRepository
- Parameters:
snapshotId
- snapshot idrepositoryStateId
- the unique id identifying the state of the repository when the snapshot deletion beganwriteShardGens
- if shard generations should be written to the repositorylistener
- completion listener
-
cleanup
public void cleanup(long repositoryStateId, boolean writeShardGens, ActionListener<RepositoryCleanupResult> listener)Runs cleanup actions on the repository. Increments the repository state id by one before executing any modifications on the repository. TODO: Add shard level cleanups- Deleting stale indices
cleanupStaleIndices(java.util.Map<java.lang.String, org.elasticsearch.common.blobstore.BlobContainer>, java.util.Set<java.lang.String>)
- Deleting unreferenced root level blobs
cleanupStaleRootFiles(java.util.List<java.lang.String>)
- Parameters:
repositoryStateId
- Current repository state idwriteShardGens
- If shard generations should be written to the repositorylistener
- Listener to complete when done
- Deleting stale indices
-
finalizeSnapshot
public void finalizeSnapshot(SnapshotId snapshotId, ShardGenerations shardGenerations, long startTime, java.lang.String failure, int totalShards, java.util.List<SnapshotShardFailure> shardFailures, long repositoryStateId, boolean includeGlobalState, MetaData clusterMetaData, java.util.Map<java.lang.String,java.lang.Object> userMetadata, boolean writeShardGens, ActionListener<SnapshotInfo> listener)Description copied from interface:Repository
Finalizes snapshotting processThis method is called on master after all shards are snapshotted.
- Specified by:
finalizeSnapshot
in interfaceRepository
- Parameters:
snapshotId
- snapshot idshardGenerations
- updated shard generationsstartTime
- start time of the snapshotfailure
- global failure reason or nulltotalShards
- total number of shardsshardFailures
- list of shard failuresrepositoryStateId
- the unique id identifying the state of the repository when the snapshot beganincludeGlobalState
- include cluster global stateclusterMetaData
- cluster metadatauserMetadata
- user metadatawriteShardGens
- if shard generations should be written to the repositorylistener
- listener to be called on completion of the snapshot
-
getSnapshotInfo
Description copied from interface:Repository
Reads snapshot description from repository.- Specified by:
getSnapshotInfo
in interfaceRepository
- Parameters:
snapshotId
- snapshot id- Returns:
- information about snapshot
-
getSnapshotGlobalMetaData
Description copied from interface:Repository
Returns global metadata associated with the snapshot.- Specified by:
getSnapshotGlobalMetaData
in interfaceRepository
- Parameters:
snapshotId
- the snapshot id to load the global metadata from- Returns:
- the global metadata about the snapshot
-
getSnapshotIndexMetaData
public IndexMetaData getSnapshotIndexMetaData(SnapshotId snapshotId, IndexId index) throws java.io.IOExceptionDescription copied from interface:Repository
Returns the index metadata associated with the snapshot.- Specified by:
getSnapshotIndexMetaData
in interfaceRepository
- Parameters:
snapshotId
- the snapshot id to load the index metadata fromindex
- theIndexId
to load the metadata from- Returns:
- the index metadata about the given index for the given snapshot
- Throws:
java.io.IOException
-
getSnapshotThrottleTimeInNanos
public long getSnapshotThrottleTimeInNanos()Description copied from interface:Repository
Returns snapshot throttle time in nanoseconds- Specified by:
getSnapshotThrottleTimeInNanos
in interfaceRepository
-
getRestoreThrottleTimeInNanos
public long getRestoreThrottleTimeInNanos()Description copied from interface:Repository
Returns restore throttle time in nanoseconds- Specified by:
getRestoreThrottleTimeInNanos
in interfaceRepository
-
assertSnapshotOrGenericThread
protected void assertSnapshotOrGenericThread() -
startVerification
public java.lang.String startVerification()Description copied from interface:Repository
Verifies repository on the master node and returns the verification token.If the verification token is not null, it's passed to all data nodes for verification. If it's null - no additional verification is required
- Specified by:
startVerification
in interfaceRepository
- Returns:
- verification token that should be passed to all Index Shard Repositories for additional verification or null
-
endVerification
public void endVerification(java.lang.String seed)Description copied from interface:Repository
Called at the end of repository verification process.This method should perform all necessary cleanup of the temporary files created in the repository
- Specified by:
endVerification
in interfaceRepository
- Parameters:
seed
- verification request generated byRepository.startVerification()
command
-
getRepositoryData
Description copied from interface:Repository
Returns aRepositoryData
to describe the data in the repository, including the snapshots and the indices across all snapshots found in the repository. Throws aRepositoryException
if there was an error in reading the data.- Specified by:
getRepositoryData
in interfaceRepository
-
isReadOnly
public boolean isReadOnly()Description copied from interface:Repository
Returns true if the repository supports only read operations- Specified by:
isReadOnly
in interfaceRepository
- Returns:
- true if the repository is read/only
-
writeIndexGen
protected void writeIndexGen(RepositoryData repositoryData, long expectedGen, boolean writeShardGens, ActionListener<java.lang.Void> listener)Writing a new index generation is a three step process. First, theRepositoryMetaData
entry for this repository is set into a pending state by incrementing its pending generationP
while its safe generationN
remains unchanged. Second, the updatedRepositoryData
is written to generationP + 1
. Lastly, theRepositoryMetaData
entry for this repository is updated to the new generationP + 1
and thus pending and safe generation are set to the same value marking the end of the update of the repository data.- Parameters:
repositoryData
- RepositoryData to writeexpectedGen
- expected repository generation at the start of the operationwriteShardGens
- whether to writeShardGenerations
to the newRepositoryData
bloblistener
- completion listener
-
snapshotShard
public void snapshotShard(Store store, MapperService mapperService, SnapshotId snapshotId, IndexId indexId, org.apache.lucene.index.IndexCommit snapshotIndexCommit, IndexShardSnapshotStatus snapshotStatus, boolean writeShardGens, ActionListener<java.lang.String> listener)Description copied from interface:Repository
Creates a snapshot of the shard based on the index commit point.The index commit point can be obtained by using
Engine.acquireLastIndexCommit(boolean)
method. Repository implementations shouldn't release the snapshot index commit point. It is done by the method caller.As snapshot process progresses, implementation of this method should update
IndexShardSnapshotStatus
object and checkIndexShardSnapshotStatus.isAborted()
to see if the snapshot process should be aborted.- Specified by:
snapshotShard
in interfaceRepository
- Parameters:
store
- store to be snapshottedmapperService
- the shards mapper servicesnapshotId
- snapshot idindexId
- id for the index being snapshottedsnapshotIndexCommit
- commit pointsnapshotStatus
- snapshot statuslistener
- listener invoked on completion
-
restoreShard
public void restoreShard(Store store, SnapshotId snapshotId, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState, ActionListener<java.lang.Void> listener)Description copied from interface:Repository
Restores snapshot of the shard.The index can be renamed on restore, hence different
shardId
andsnapshotShardId
are supplied.- Specified by:
restoreShard
in interfaceRepository
- Parameters:
store
- the store to restore the index intosnapshotId
- snapshot idindexId
- id of the index in the repository from which the restore is occurringsnapshotShardId
- shard id (in the snapshot)recoveryState
- recovery statelistener
- listener to invoke once done
-
getShardSnapshotStatus
public IndexShardSnapshotStatus getShardSnapshotStatus(SnapshotId snapshotId, IndexId indexId, ShardId shardId)Description copied from interface:Repository
Retrieve shard snapshot status for the stored snapshot- Specified by:
getShardSnapshotStatus
in interfaceRepository
- Parameters:
snapshotId
- snapshot idindexId
- the snapshotted index id for the shard to get status forshardId
- shard id- Returns:
- snapshot status
-
verify
Description copied from interface:Repository
Verifies repository settings on data node.- Specified by:
verify
in interfaceRepository
- Parameters:
seed
- value returned byRepository.startVerification()
localNode
- the local node information, for inclusion in verification errors
-
toString
public java.lang.String toString()- Overrides:
toString
in classjava.lang.Object
-