Package org.elasticsearch.snapshots
This package exposes the Elasticsearch Snapshot functionality.
Preliminaries
There are two communication channels between all nodes and master in the snapshot functionality:
- The master updates the cluster state by adding, removing or altering the contents of its custom entry
SnapshotsInProgress
. All nodes consume the state of theSnapshotsInProgress
and will start or abort relevant shard snapshot tasks accordingly. - Nodes that are executing shard snapshot tasks report either success or failure of their snapshot task by submitting a
UpdateIndexShardSnapshotStatusRequest
to the master node that will update the snapshot's entry in the cluster state accordingly.
Snapshot Creation
Snapshots are created by the following sequence of events:
- First the
SnapshotsService
determines the primary shards' assignments for all indices that are being snapshotted and creates aSnapshotsInProgress.Entry
with stateSTARTED
and adds the map ofShardId
toSnapshotsInProgress.ShardSnapshotStatus
that tracks the assignment of which node is to snapshot which shard. All shard snapshots are executed on the shard's primary node. Thus all shards for which the primary node was found to have a healthy copy of the shard are marked as being in stateINIT
in this map. If the primary for a shard is unassigned, it is marked asMISSING
in this map. In case the primary is initializing at this point, it is marked as in stateWAITING
. In case a shard's primary is relocated at any point after itsSnapshotsInProgress.Entry
was created and thus been assigned to a specific cluster node, that shard's snapshot will fail and move to stateFAILED
. - The new
SnapshotsInProgress.Entry
is then observed bySnapshotShardsService.clusterChanged(org.elasticsearch.cluster.ClusterChangedEvent)
on all nodes and since the entry is in stateSTARTED
theSnapshotShardsService
will check if any local primary shards are to be snapshotted (signaled by the shard's snapshot state beingINIT
). For those local primary shards found in stateINIT
) the snapshot process of writing the shard's data files to the snapshot'sRepository
is executed. Once the snapshot execution finishes for a shard anUpdateIndexShardSnapshotStatusRequest
is sent to the master node signaling either statusSUCCESS
orFAILED
. The master node will then update a shard's state in the snapshotsSnapshotsInProgress.Entry
whenever it receives such aUpdateIndexShardSnapshotStatusRequest
. - If as a result of the received status update requests, all shards in the cluster state are in a completed state, i.e are marked as
either
SUCCESS
,FAILED
orMISSING
, theSnapshotShardsService
will update the state of theEntry
itself and mark it asSUCCESS
. At the same timeSnapshotsService.endSnapshot(org.elasticsearch.cluster.SnapshotsInProgress.Entry, org.elasticsearch.cluster.metadata.Metadata, org.elasticsearch.repositories.RepositoryData)
is executed, writing the metadata necessary to finalize the snapshot in the repository to the repository. - After writing the final metadata to the repository, a cluster state update to remove the snapshot from the cluster state is
submitted and the removal of the snapshot's
SnapshotsInProgress.Entry
from the cluster state completes the snapshot process.
Deleting a Snapshot
Deleting a snapshot can take the form of either simply deleting it from the repository or (if it has not completed yet) aborting it and subsequently deleting it from the repository.
Aborting a Snapshot
- Aborting a snapshot starts by updating the state of the snapshot's
SnapshotsInProgress.Entry
toABORTED
. - The snapshot's state change to
ABORTED
in cluster state is then picked up by theSnapshotShardsService
on all nodes. Those nodes that have shard snapshot actions for the snapshot assigned to them, will abort them and notify master about the shards snapshot status accordingly. If the shard snapshot action completed or was in stateFINALIZE
when the abort was registered by theSnapshotShardsService
, then the shard's state will be reported to master asSUCCESS
. Otherwise, it will be reported asFAILED
. - Once all the shards are reported to master as either
SUCCESS
orFAILED
theSnapshotsService
on the master will finish the snapshot process as all shard's states are now completed and hence the snapshot can be completed as explained in point 4 of the snapshot creation section above.
Deleting a Snapshot from a Repository
- Assuming there are no entries in the cluster state's
SnapshotsInProgress
, deleting a snapshot starts by theSnapshotsService
creating an entry for deleting the snapshot in the cluster state'sSnapshotDeletionsInProgress
. - Once the cluster state contains the deletion entry in
SnapshotDeletionsInProgress
theSnapshotsService
will invokeRepository.deleteSnapshots(java.util.Collection<org.elasticsearch.snapshots.SnapshotId>, long, org.elasticsearch.Version, org.elasticsearch.action.ActionListener<org.elasticsearch.repositories.RepositoryData>)
for the given snapshot, which will remove files associated with the snapshot from the repository as well as update its meta-data to reflect the deletion of the snapshot. - After the deletion of the snapshot's data from the repository finishes, the
SnapshotsService
will submit a cluster state update to remove the deletion's entry inSnapshotDeletionsInProgress
which concludes the process of deleting a snapshot.
Concurrent Snapshot Operations
Snapshot create and delete operations may be started concurrently. Operations targeting different repositories run independently of each other. Multiple operations targeting the same repository are executed according to the following rules:Concurrent Snapshot Creation
If multiple snapshot creation jobs are started at the same time, the data-node operations of multiple snapshots may run in parallel across different shards. If multiple snapshots want to snapshot a certain shard, then the shard snapshots for that shard will be executed one by one. This is enforced by the master node setting the shard's snapshot state toSnapshotsInProgress.ShardSnapshotStatus.UNASSIGNED_QUEUED
for all but one snapshot. The order of
operations on a single shard is given by the order in which the snapshots were started.
As soon as all shards for a given snapshot have finished, it will be finalized as explained above. Finalization will happen one snapshot
at a time, working in the order in which snapshots had their shards completed.
Concurrent Snapshot Deletes
A snapshot delete will be executed as soon as there are no more shard snapshots or snapshot finalizations executing running for a given repository. Before a delete is executed on the repository it will be set to stateSnapshotDeletionsInProgress.State.STARTED
. If it cannot be executed when it is received it will be
set to state SnapshotDeletionsInProgress.State.WAITING
initially.
If a delete is received for a given repository while there is already an ongoing delete for the same repository, there are two possible
scenarios:
1. If the delete is in state META_DATA
(i.e. already running on the repository) then the new delete will be added in state
WAITING
and will be executed after the current delete. The only exception here would be the case where the new delete covers
the exact same snapshots as the already running delete. In this case no new delete operation is added and second delete request will
simply wait for the existing delete to return.
2. If the existing delete is in state WAITING
then the existing
SnapshotDeletionsInProgress.Entry
in the cluster state will be updated to cover both the snapshots
in the existing delete as well as additional snapshots that may be found in the second delete request.
In either of the above scenarios, in-progress snapshots will be aborted in the same cluster state update that adds a delete to the
cluster state, if a delete applies to them.
If a snapshot request is received while there already is a delete in the cluster state for the same repository, that snapshot will not
start doing any shard snapshots until the delete has been executed.-
Class Summary Class Description RestoreInfo Information about successfully completed restore operation.RestoreService Service responsible for restoring snapshotsRestoreService.RestoreCompletionResponse RestoreService.RestoreInProgressUpdater Snapshot Basic information about a snapshot - a SnapshotId and the repository that the snapshot belongs to.SnapshotId SnapshotId - snapshot name + snapshot UUIDSnapshotInfo Information about a snapshotSnapshotInfo.SnapshotInfoBuilder SnapshotShardFailure Stores information about failures that occurred during shard snapshotting processSnapshotShardsService This service runs on data nodes and controls currently running shard snapshots on these nodes.SnapshotsService Service responsible for creating snapshots.SnapshotUtils Snapshot utilitiesUpdateIndexShardSnapshotStatusRequest Internal request that is used to send changes in snapshot status to master -
Enum Summary Enum Description SnapshotState Represents the state that a snapshot can be in -
Exception Summary Exception Description AbortedSnapshotException ConcurrentSnapshotExecutionException Thrown when a user tries to multiple conflicting snapshot/restore operations at the same time.InvalidSnapshotNameException Thrown on the attempt to create a snapshot with invalid nameSnapshotCreationException Thrown when snapshot creation fails completelySnapshotException Generic snapshot exceptionSnapshotInProgressException Thrown on the attempt to execute an action that requires that no snapshot is in progress.SnapshotMissingException Thrown if requested snapshot doesn't existSnapshotRestoreException Snapshot restore exception