Class BlobStoreRepository
- java.lang.Object
-
- org.elasticsearch.common.component.AbstractLifecycleComponent
-
- org.elasticsearch.repositories.blobstore.BlobStoreRepository
-
- All Implemented Interfaces:
java.io.Closeable
,java.lang.AutoCloseable
,LifecycleComponent
,Releasable
,Repository
- Direct Known Subclasses:
FsRepository
public abstract class BlobStoreRepository extends AbstractLifecycleComponent implements Repository
BlobStore - based implementation of Snapshot RepositoryThis repository works with any
BlobStore
implementation. The blobStore could be (and preferred) lazy initialized increateBlobStore()
.BlobStoreRepository maintains the following structure in the blob store
STORE_ROOT |- index-N - JSON serialized {@link RepositoryData} containing a list of all snapshot ids and the indices belonging to | each snapshot, N is the generation of the file |- index.latest - contains the numeric value of the latest generation of the index file (i.e. N from above) |- incompatible-snapshots - list of all snapshot ids that are no longer compatible with the current version of the cluster |- snap-20131010.dat - SMILE serialized {@link SnapshotInfo} for snapshot "20131010" |- meta-20131010.dat - SMILE serialized {@link MetaData} for snapshot "20131010" (includes only global metadata) |- snap-20131011.dat - SMILE serialized {@link SnapshotInfo} for snapshot "20131011" |- meta-20131011.dat - SMILE serialized {@link MetaData} for snapshot "20131011" ..... |- indices/ - data for all indices |- Ac1342-B_x/ - data for index "foo" which was assigned the unique id of Ac1342-B_x in the repository | |- meta-20131010.dat - JSON Serialized {@link IndexMetaData} for index "foo" | |- 0/ - data for shard "0" of index "foo" | | |- __1 \ (files with numeric names were created by older ES versions) | | |- __2 | | | |- __VPO5oDMVT5y4Akv8T_AO_A |- files from different segments see snap-* for their mappings to real segment files | | |- __1gbJy18wS_2kv1qI7FgKuQ | | | |- __R8JvZAHlSMyMXyZc2SS8Zg / | | ..... | | |- snap-20131010.dat - SMILE serialized {@link BlobStoreIndexShardSnapshot} for snapshot "20131010" | | |- snap-20131011.dat - SMILE serialized {@link BlobStoreIndexShardSnapshot} for snapshot "20131011" | | |- index-123 - SMILE serialized {@link BlobStoreIndexShardSnapshots} for the shard | | | |- 1/ - data for shard "1" of index "foo" | | |- __1 | | ..... | | | |-2/ | ...... | |- 1xB0D8_B3y/ - data for index "bar" which was assigned the unique id of 1xB0D8_B3y in the repository ......
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from interface org.elasticsearch.repositories.Repository
Repository.Factory
-
-
Field Summary
Fields Modifier and Type Field Description static java.lang.String
INDEX_FILE_PREFIX
static java.lang.String
INDEX_LATEST_BLOB
protected java.util.concurrent.atomic.AtomicLong
latestKnownRepoGen
protected RepositoryMetaData
metadata
static java.lang.String
METADATA_NAME_FORMAT
static java.lang.String
METADATA_PREFIX
protected NamedXContentRegistry
namedXContentRegistry
static java.lang.String
SNAPSHOT_CODEC
static java.lang.String
SNAPSHOT_NAME_FORMAT
static java.lang.String
SNAPSHOT_PREFIX
protected ChecksumBlobStoreFormat<SnapshotInfo>
snapshotFormat
protected ThreadPool
threadPool
-
Fields inherited from class org.elasticsearch.common.component.AbstractLifecycleComponent
lifecycle
-
-
Constructor Summary
Constructors Modifier Constructor Description protected
BlobStoreRepository(RepositoryMetaData metadata, boolean compress, NamedXContentRegistry namedXContentRegistry, ThreadPool threadPool)
Constructs new BlobStoreRepository
-
Method Summary
Modifier and Type Method Description protected void
assertSnapshotOrGenericThread()
abstract BlobPath
basePath()
Returns base path of the repositoryprotected BlobContainer
blobContainer()
maintains single lazy instance ofBlobContainer
BlobStore
blobStore()
Maintains single lazy instance ofBlobStore
.protected ByteSizeValue
chunkSize()
Returns data file chunk size.void
cleanup(long repositoryStateId, ActionListener<RepositoryCleanupResult> listener)
Runs cleanup actions on the repository.protected abstract BlobStore
createBlobStore()
Creates new BlobStore to read and write data.void
deleteSnapshot(SnapshotId snapshotId, long repositoryStateId, ActionListener<java.lang.Void> listener)
Deletes snapshotprotected void
doClose()
protected void
doStart()
protected void
doStop()
void
endVerification(java.lang.String seed)
Called at the end of repository verification process.void
finalizeSnapshot(SnapshotId snapshotId, java.util.List<IndexId> indices, long startTime, java.lang.String failure, int totalShards, java.util.List<SnapshotShardFailure> shardFailures, long repositoryStateId, boolean includeGlobalState, MetaData clusterMetaData, java.util.Map<java.lang.String,java.lang.Object> userMetadata, ActionListener<SnapshotInfo> listener)
Finalizes snapshotting processprotected BlobStore
getBlobStore()
RepositoryMetaData
getMetadata()
Returns metadata about this repository.RepositoryData
getRepositoryData()
Returns aRepositoryData
to describe the data in the repository, including the snapshots and the indices across all snapshots found in the repository.long
getRestoreThrottleTimeInNanos()
Returns restore throttle time in nanosecondsIndexShardSnapshotStatus
getShardSnapshotStatus(SnapshotId snapshotId, Version version, IndexId indexId, ShardId shardId)
Retrieve shard snapshot status for the stored snapshotMetaData
getSnapshotGlobalMetaData(SnapshotId snapshotId)
Returns global metadata associated with the snapshot.IndexMetaData
getSnapshotIndexMetaData(SnapshotId snapshotId, IndexId index)
Returns the index metadata associated with the snapshot.SnapshotInfo
getSnapshotInfo(SnapshotId snapshotId)
Reads snapshot description from repository.long
getSnapshotThrottleTimeInNanos()
Returns snapshot throttle time in nanosecondsvoid
initializeSnapshot(SnapshotId snapshotId, java.util.List<IndexId> indices, MetaData clusterMetaData)
Starts snapshotting processprotected boolean
isCompress()
Returns true if metadata and snapshot files should be compressedboolean
isReadOnly()
Returns true if the repository supports only read operationsvoid
restoreShard(Store store, SnapshotId snapshotId, Version version, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState)
Restores snapshot of the shard.void
snapshotShard(Store store, MapperService mapperService, SnapshotId snapshotId, IndexId indexId, org.apache.lucene.index.IndexCommit snapshotIndexCommit, IndexShardSnapshotStatus snapshotStatus, ActionListener<java.lang.Void> listener)
Creates a snapshot of the shard based on the index commit point.java.lang.String
startVerification()
Verifies repository on the master node and returns the verification token.ThreadPool
threadPool()
java.lang.String
toString()
void
verify(java.lang.String seed, DiscoveryNode localNode)
Verifies repository settings on data node.protected void
writeIndexGen(RepositoryData repositoryData, long expectedGen)
-
Methods inherited from class org.elasticsearch.common.component.AbstractLifecycleComponent
addLifecycleListener, close, lifecycleState, removeLifecycleListener, start, stop
-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.elasticsearch.common.component.LifecycleComponent
addLifecycleListener, lifecycleState, removeLifecycleListener, start, stop
-
Methods inherited from interface org.elasticsearch.common.lease.Releasable
close
-
Methods inherited from interface org.elasticsearch.repositories.Repository
restoreShard
-
-
-
-
Field Detail
-
metadata
protected final RepositoryMetaData metadata
-
namedXContentRegistry
protected final NamedXContentRegistry namedXContentRegistry
-
threadPool
protected final ThreadPool threadPool
-
SNAPSHOT_PREFIX
public static final java.lang.String SNAPSHOT_PREFIX
- See Also:
- Constant Field Values
-
SNAPSHOT_CODEC
public static final java.lang.String SNAPSHOT_CODEC
- See Also:
- Constant Field Values
-
INDEX_FILE_PREFIX
public static final java.lang.String INDEX_FILE_PREFIX
- See Also:
- Constant Field Values
-
INDEX_LATEST_BLOB
public static final java.lang.String INDEX_LATEST_BLOB
- See Also:
- Constant Field Values
-
METADATA_PREFIX
public static final java.lang.String METADATA_PREFIX
- See Also:
- Constant Field Values
-
METADATA_NAME_FORMAT
public static final java.lang.String METADATA_NAME_FORMAT
- See Also:
- Constant Field Values
-
SNAPSHOT_NAME_FORMAT
public static final java.lang.String SNAPSHOT_NAME_FORMAT
- See Also:
- Constant Field Values
-
snapshotFormat
protected ChecksumBlobStoreFormat<SnapshotInfo> snapshotFormat
-
latestKnownRepoGen
protected final java.util.concurrent.atomic.AtomicLong latestKnownRepoGen
-
-
Constructor Detail
-
BlobStoreRepository
protected BlobStoreRepository(RepositoryMetaData metadata, boolean compress, NamedXContentRegistry namedXContentRegistry, ThreadPool threadPool)
Constructs new BlobStoreRepository- Parameters:
metadata
- The metadata for this repository including name and settingsthreadPool
- Threadpool to run long running repository manipulations on asynchronously
-
-
Method Detail
-
doStart
protected void doStart()
- Specified by:
doStart
in classAbstractLifecycleComponent
-
doStop
protected void doStop()
- Specified by:
doStop
in classAbstractLifecycleComponent
-
doClose
protected void doClose()
- Specified by:
doClose
in classAbstractLifecycleComponent
-
threadPool
public ThreadPool threadPool()
-
getBlobStore
protected BlobStore getBlobStore()
-
blobContainer
protected BlobContainer blobContainer()
maintains single lazy instance ofBlobContainer
-
blobStore
public BlobStore blobStore()
Maintains single lazy instance ofBlobStore
. Public for testing.
-
createBlobStore
protected abstract BlobStore createBlobStore() throws java.lang.Exception
Creates new BlobStore to read and write data.- Throws:
java.lang.Exception
-
basePath
public abstract BlobPath basePath()
Returns base path of the repository
-
isCompress
protected final boolean isCompress()
Returns true if metadata and snapshot files should be compressed- Returns:
- true if compression is needed
-
chunkSize
protected ByteSizeValue chunkSize()
Returns data file chunk size.This method should return null if no chunking is needed.
- Returns:
- chunk size
-
getMetadata
public RepositoryMetaData getMetadata()
Description copied from interface:Repository
Returns metadata about this repository.- Specified by:
getMetadata
in interfaceRepository
-
initializeSnapshot
public void initializeSnapshot(SnapshotId snapshotId, java.util.List<IndexId> indices, MetaData clusterMetaData)
Description copied from interface:Repository
Starts snapshotting process- Specified by:
initializeSnapshot
in interfaceRepository
- Parameters:
snapshotId
- snapshot idindices
- list of indices to be snapshottedclusterMetaData
- cluster metadata
-
deleteSnapshot
public void deleteSnapshot(SnapshotId snapshotId, long repositoryStateId, ActionListener<java.lang.Void> listener)
Description copied from interface:Repository
Deletes snapshot- Specified by:
deleteSnapshot
in interfaceRepository
- Parameters:
snapshotId
- snapshot idrepositoryStateId
- the unique id identifying the state of the repository when the snapshot deletion beganlistener
- completion listener
-
cleanup
public void cleanup(long repositoryStateId, ActionListener<RepositoryCleanupResult> listener)
Runs cleanup actions on the repository. Increments the repository state id by one before executing any modifications on the repository. TODO: Add shard level cleanups- Deleting stale indices
cleanupStaleIndices(java.util.Map<java.lang.String, org.elasticsearch.common.blobstore.BlobContainer>, java.util.Set<java.lang.String>)
- Deleting unreferenced root level blobs
cleanupStaleRootFiles(java.util.List<java.lang.String>)
- Parameters:
repositoryStateId
- Current repository state idlistener
- Lister to complete when done
- Deleting stale indices
-
finalizeSnapshot
public void finalizeSnapshot(SnapshotId snapshotId, java.util.List<IndexId> indices, long startTime, java.lang.String failure, int totalShards, java.util.List<SnapshotShardFailure> shardFailures, long repositoryStateId, boolean includeGlobalState, MetaData clusterMetaData, java.util.Map<java.lang.String,java.lang.Object> userMetadata, ActionListener<SnapshotInfo> listener)
Description copied from interface:Repository
Finalizes snapshotting processThis method is called on master after all shards are snapshotted.
- Specified by:
finalizeSnapshot
in interfaceRepository
- Parameters:
snapshotId
- snapshot idindices
- list of indices in the snapshotstartTime
- start time of the snapshotfailure
- global failure reason or nulltotalShards
- total number of shardsshardFailures
- list of shard failuresrepositoryStateId
- the unique id identifying the state of the repository when the snapshot beganincludeGlobalState
- include cluster global statelistener
- listener to be called on completion of the snapshot
-
getSnapshotInfo
public SnapshotInfo getSnapshotInfo(SnapshotId snapshotId)
Description copied from interface:Repository
Reads snapshot description from repository.- Specified by:
getSnapshotInfo
in interfaceRepository
- Parameters:
snapshotId
- snapshot id- Returns:
- information about snapshot
-
getSnapshotGlobalMetaData
public MetaData getSnapshotGlobalMetaData(SnapshotId snapshotId)
Description copied from interface:Repository
Returns global metadata associated with the snapshot.- Specified by:
getSnapshotGlobalMetaData
in interfaceRepository
- Parameters:
snapshotId
- the snapshot id to load the global metadata from- Returns:
- the global metadata about the snapshot
-
getSnapshotIndexMetaData
public IndexMetaData getSnapshotIndexMetaData(SnapshotId snapshotId, IndexId index) throws java.io.IOException
Description copied from interface:Repository
Returns the index metadata associated with the snapshot.- Specified by:
getSnapshotIndexMetaData
in interfaceRepository
- Parameters:
snapshotId
- the snapshot id to load the index metadata fromindex
- theIndexId
to load the metadata from- Returns:
- the index metadata about the given index for the given snapshot
- Throws:
java.io.IOException
-
getSnapshotThrottleTimeInNanos
public long getSnapshotThrottleTimeInNanos()
Description copied from interface:Repository
Returns snapshot throttle time in nanoseconds- Specified by:
getSnapshotThrottleTimeInNanos
in interfaceRepository
-
getRestoreThrottleTimeInNanos
public long getRestoreThrottleTimeInNanos()
Description copied from interface:Repository
Returns restore throttle time in nanoseconds- Specified by:
getRestoreThrottleTimeInNanos
in interfaceRepository
-
assertSnapshotOrGenericThread
protected void assertSnapshotOrGenericThread()
-
startVerification
public java.lang.String startVerification()
Description copied from interface:Repository
Verifies repository on the master node and returns the verification token.If the verification token is not null, it's passed to all data nodes for verification. If it's null - no additional verification is required
- Specified by:
startVerification
in interfaceRepository
- Returns:
- verification token that should be passed to all Index Shard Repositories for additional verification or null
-
endVerification
public void endVerification(java.lang.String seed)
Description copied from interface:Repository
Called at the end of repository verification process.This method should perform all necessary cleanup of the temporary files created in the repository
- Specified by:
endVerification
in interfaceRepository
- Parameters:
seed
- verification request generated byRepository.startVerification()
command
-
getRepositoryData
public RepositoryData getRepositoryData()
Description copied from interface:Repository
Returns aRepositoryData
to describe the data in the repository, including the snapshots and the indices across all snapshots found in the repository. Throws aRepositoryException
if there was an error in reading the data.- Specified by:
getRepositoryData
in interfaceRepository
-
isReadOnly
public boolean isReadOnly()
Description copied from interface:Repository
Returns true if the repository supports only read operations- Specified by:
isReadOnly
in interfaceRepository
- Returns:
- true if the repository is read/only
-
writeIndexGen
protected void writeIndexGen(RepositoryData repositoryData, long expectedGen) throws java.io.IOException
- Throws:
java.io.IOException
-
snapshotShard
public void snapshotShard(Store store, MapperService mapperService, SnapshotId snapshotId, IndexId indexId, org.apache.lucene.index.IndexCommit snapshotIndexCommit, IndexShardSnapshotStatus snapshotStatus, ActionListener<java.lang.Void> listener)
Description copied from interface:Repository
Creates a snapshot of the shard based on the index commit point.The index commit point can be obtained by using
Engine.acquireLastIndexCommit(boolean)
method. Repository implementations shouldn't release the snapshot index commit point. It is done by the method caller.As snapshot process progresses, implementation of this method should update
IndexShardSnapshotStatus
object and checkIndexShardSnapshotStatus.isAborted()
to see if the snapshot process should be aborted.- Specified by:
snapshotShard
in interfaceRepository
- Parameters:
store
- store to be snapshottedmapperService
- the shards mapper servicesnapshotId
- snapshot idindexId
- id for the index being snapshottedsnapshotIndexCommit
- commit pointsnapshotStatus
- snapshot statuslistener
- listener invoked on completion
-
restoreShard
public void restoreShard(Store store, SnapshotId snapshotId, Version version, IndexId indexId, ShardId snapshotShardId, RecoveryState recoveryState)
Description copied from interface:Repository
Restores snapshot of the shard.The index can be renamed on restore, hence different
shardId
andsnapshotShardId
are supplied.- Specified by:
restoreShard
in interfaceRepository
- Parameters:
store
- the store to restore the index intosnapshotId
- snapshot idversion
- version of elasticsearch that created this snapshotindexId
- id of the index in the repository from which the restore is occurringsnapshotShardId
- shard id (in the snapshot)recoveryState
- recovery state
-
getShardSnapshotStatus
public IndexShardSnapshotStatus getShardSnapshotStatus(SnapshotId snapshotId, Version version, IndexId indexId, ShardId shardId)
Description copied from interface:Repository
Retrieve shard snapshot status for the stored snapshot- Specified by:
getShardSnapshotStatus
in interfaceRepository
- Parameters:
snapshotId
- snapshot idversion
- version of elasticsearch that created this snapshotindexId
- the snapshotted index id for the shard to get status forshardId
- shard id- Returns:
- snapshot status
-
verify
public void verify(java.lang.String seed, DiscoveryNode localNode)
Description copied from interface:Repository
Verifies repository settings on data node.- Specified by:
verify
in interfaceRepository
- Parameters:
seed
- value returned byRepository.startVerification()
localNode
- the local node information, for inclusion in verification errors
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
-