java.io.Closeable
, java.lang.AutoCloseable
, RefCounted
, IndexShardComponent
public class Store extends AbstractIndexShardComponent implements java.io.Closeable, RefCounted
Note: If you use a store it's reference count should be increased before using it by calling #incRef and a corresponding #decRef must be called in a try/finally block to release the store again ie.:
store.incRef(); try { // use the store... } finally { store.decRef(); }
Modifier and Type | Class | Description |
---|---|---|
static class |
Store.MetadataSnapshot |
Represents a snapshot of the current directory build from the latest Lucene commit.
|
static interface |
Store.OnClose |
A listener that is executed once the store is closed and all references to it are released
|
static class |
Store.RecoveryDiff |
A class representing the diff between a recovery source and recovery target
|
Modifier and Type | Field | Description |
---|---|---|
static Setting<TimeValue> |
INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING |
deprecationLogger, indexSettings, logger, shardId
Constructor | Description |
---|---|
Store(ShardId shardId,
IndexSettings indexSettings,
DirectoryService directoryService,
ShardLock shardLock) |
|
Store(ShardId shardId,
IndexSettings indexSettings,
DirectoryService directoryService,
ShardLock shardLock,
Store.OnClose onClose) |
Modifier and Type | Method | Description |
---|---|---|
void |
associateIndexWithNewTranslog(java.lang.String translogUUID) |
Force bakes the given translog generation as recovery information in the lucene index.
|
void |
bootstrapNewHistory() |
Marks an existing lucene index with a new history uuid.
|
static boolean |
canOpenIndex(org.apache.logging.log4j.Logger logger,
java.nio.file.Path indexLocation,
ShardId shardId,
NodeEnvironment.ShardLocker shardLocker) |
Returns
true iff the given location contains an index an the index
can be successfully opened. |
org.apache.lucene.index.CheckIndex.Status |
checkIndex(java.io.PrintStream out) |
Checks and returns the status of the existing index in this store.
|
static void |
checkIntegrity(StoreFileMetaData md,
org.apache.lucene.store.Directory directory) |
|
boolean |
checkIntegrityNoException(StoreFileMetaData md) |
|
static boolean |
checkIntegrityNoException(StoreFileMetaData md,
org.apache.lucene.store.Directory directory) |
|
void |
cleanupAndVerify(java.lang.String reason,
Store.MetadataSnapshot sourceMetaData) |
This method deletes every file in this store that is not contained in the given source meta data or is a
legacy checksum file.
|
void |
close() |
|
void |
createEmpty() |
creates an empty lucene index and a corresponding empty translog.
|
org.apache.lucene.store.IndexOutput |
createVerifyingOutput(java.lang.String fileName,
StoreFileMetaData metadata,
org.apache.lucene.store.IOContext context) |
The returned IndexOutput validates the files checksum.
|
void |
decRef() |
Decreases the refCount of this Store instance.
|
void |
deleteQuiet(java.lang.String... files) |
|
static java.lang.String |
digestToString(long digest) |
Produces a string representation of the given digest value.
|
org.apache.lucene.store.Directory |
directory() |
|
boolean |
ensureIndexHas6xCommitTags() |
A 5.x index does not have either historyUUDID or sequence number markers as these markers are introduced in 6.0+.
|
void |
exorciseIndex(org.apache.lucene.index.CheckIndex.Status status) |
Repairs the index using the previous returned status from
checkIndex(PrintStream) . |
void |
failIfCorrupted() |
|
Store.MetadataSnapshot |
getMetadata(org.apache.lucene.index.IndexCommit commit) |
Returns a new MetadataSnapshot for the given commit.
|
Store.MetadataSnapshot |
getMetadata(org.apache.lucene.index.IndexCommit commit,
boolean lockDirectory) |
Returns a new MetadataSnapshot for the given commit.
|
void |
incRef() |
Increments the refCount of this Store instance.
|
static boolean |
isAutogenerated(java.lang.String name) |
Returns true if the file is auto-generated by the store and shouldn't be deleted during cleanup.
|
boolean |
isMarkedCorrupted() |
|
static SequenceNumbers.CommitInfo |
loadSeqNoInfo(org.apache.lucene.index.IndexCommit commit) |
Loads the maximum sequence number and local checkpoint from the given Lucene commit point or the latest if not provided.
|
void |
markStoreCorrupted(java.io.IOException exception) |
Marks this store as corrupted.
|
org.apache.lucene.store.IndexInput |
openVerifyingInput(java.lang.String filename,
org.apache.lucene.store.IOContext context,
StoreFileMetaData metadata) |
|
org.apache.lucene.index.SegmentInfos |
readLastCommittedSegmentsInfo() |
Returns the last committed segments info for this store
|
static Store.MetadataSnapshot |
readMetadataSnapshot(java.nio.file.Path indexLocation,
ShardId shardId,
NodeEnvironment.ShardLocker shardLocker,
org.apache.logging.log4j.Logger logger) |
Reads a MetadataSnapshot from the given index locations or returns an empty snapshot if it can't be read.
|
int |
refCount() |
Returns the current reference count.
|
void |
removeCorruptionMarker() |
Deletes all corruption markers from this store.
|
void |
renameTempFilesSafe(java.util.Map<java.lang.String,java.lang.String> tempFileMap) |
Renames all the given files from the key of the map to the
value of the map.
|
StoreStats |
stats() |
|
void |
trimUnsafeCommits(long lastSyncedGlobalCheckpoint,
long minRetainedTranslogGen,
Version indexVersionCreated) |
Keeping existing unsafe commits when opening an engine can be problematic because these commits are not safe
at the recovering time but they can suddenly become safe in the future.
|
boolean |
tryIncRef() |
Tries to increment the refCount of this Store instance.
|
static void |
tryOpenIndex(java.nio.file.Path indexLocation,
ShardId shardId,
NodeEnvironment.ShardLocker shardLocker,
org.apache.logging.log4j.Logger logger) |
Tries to open an index for the given location.
|
static void |
verify(org.apache.lucene.store.IndexInput input) |
|
static void |
verify(org.apache.lucene.store.IndexOutput output) |
indexSettings, nodeName, shardId
public Store(ShardId shardId, IndexSettings indexSettings, DirectoryService directoryService, ShardLock shardLock) throws java.io.IOException
java.io.IOException
public Store(ShardId shardId, IndexSettings indexSettings, DirectoryService directoryService, ShardLock shardLock, Store.OnClose onClose) throws java.io.IOException
java.io.IOException
public org.apache.lucene.store.Directory directory()
public org.apache.lucene.index.SegmentInfos readLastCommittedSegmentsInfo() throws java.io.IOException
java.io.IOException
- if the index is corrupted or the segments file is not presentpublic static SequenceNumbers.CommitInfo loadSeqNoInfo(org.apache.lucene.index.IndexCommit commit) throws java.io.IOException
commit
- the commit point to load seqno stats, or the last commit in the store if the parameter is nullSequenceNumbers.CommitInfo
containing information about the last commitjava.io.IOException
- if an I/O exception occurred reading the latest Lucene commit point from diskpublic Store.MetadataSnapshot getMetadata(org.apache.lucene.index.IndexCommit commit) throws java.io.IOException
null
the latest commit point is used.
Note that this method requires the caller verify it has the right to access the store and
no concurrent file changes are happening. If in doubt, you probably want to use one of the following:
readMetadataSnapshot(Path, ShardId, NodeEnvironment.ShardLocker, Logger)
to read a meta data while locking
IndexShard.snapshotStoreMetadata()
to safely read from an existing shard
IndexShard.acquireLastIndexCommit(boolean)
to get an IndexCommit
which is safe to use but has to be freedcommit
- the index commit to read the snapshot from or null
if the latest snapshot should be read from the
directoryorg.apache.lucene.index.CorruptIndexException
- if the lucene index is corrupted. This can be caused by a checksum mismatch or an
unexpected exception when opening the index reading the segments file.org.apache.lucene.index.IndexFormatTooOldException
- if the lucene index is too old to be opened.org.apache.lucene.index.IndexFormatTooNewException
- if the lucene index is too new to be opened.java.io.FileNotFoundException
- if one or more files referenced by a commit are not present.java.nio.file.NoSuchFileException
- if one or more files referenced by a commit are not present.org.apache.lucene.index.IndexNotFoundException
- if the commit point can't be found in this storejava.io.IOException
public Store.MetadataSnapshot getMetadata(org.apache.lucene.index.IndexCommit commit, boolean lockDirectory) throws java.io.IOException
null
the latest commit point is used.
Note that this method requires the caller verify it has the right to access the store and
no concurrent file changes are happening. If in doubt, you probably want to use one of the following:
readMetadataSnapshot(Path, ShardId, NodeEnvironment.ShardLocker, Logger)
to read a meta data while locking
IndexShard.snapshotStoreMetadata()
to safely read from an existing shard
IndexShard.acquireLastIndexCommit(boolean)
to get an IndexCommit
which is safe to use but has to be freedcommit
- the index commit to read the snapshot from or null
if the latest snapshot should be read from the
directorylockDirectory
- if true
the index writer lock will be obtained before reading the snapshot. This should
only be used if there is no started shard using this store.org.apache.lucene.index.CorruptIndexException
- if the lucene index is corrupted. This can be caused by a checksum mismatch or an
unexpected exception when opening the index reading the segments file.org.apache.lucene.index.IndexFormatTooOldException
- if the lucene index is too old to be opened.org.apache.lucene.index.IndexFormatTooNewException
- if the lucene index is too new to be opened.java.io.FileNotFoundException
- if one or more files referenced by a commit are not present.java.nio.file.NoSuchFileException
- if one or more files referenced by a commit are not present.org.apache.lucene.index.IndexNotFoundException
- if the commit point can't be found in this storejava.io.IOException
public void renameTempFilesSafe(java.util.Map<java.lang.String,java.lang.String> tempFileMap) throws java.io.IOException
java.io.IOException
public org.apache.lucene.index.CheckIndex.Status checkIndex(java.io.PrintStream out) throws java.io.IOException
out
- where infoStream messages should go. See CheckIndex.setInfoStream(PrintStream)
java.io.IOException
public void exorciseIndex(org.apache.lucene.index.CheckIndex.Status status) throws java.io.IOException
checkIndex(PrintStream)
.java.io.IOException
public StoreStats stats() throws java.io.IOException
java.io.IOException
public final void incRef()
decRef()
, in a finally clause; otherwise the store may never be closed. Note that
close()
simply calls decRef(), which means that the Store will not really be closed until decRef()
has been called for all outstanding references.
Note: Close can safely be called multiple times.
incRef
in interface RefCounted
org.apache.lucene.store.AlreadyClosedException
- iff the reference counter can not be incremented.decRef()
,
tryIncRef()
public final boolean tryIncRef()
decRef()
, in a finally clause; otherwise the store may never be closed. Note that
close()
simply calls decRef(), which means that the Store will not really be closed until decRef()
has been called for all outstanding references.
Note: Close can safely be called multiple times.
tryIncRef
in interface RefCounted
decRef()
,
incRef()
public final void decRef()
decRef
in interface RefCounted
incRef()
public void close()
close
in interface java.lang.AutoCloseable
close
in interface java.io.Closeable
public static Store.MetadataSnapshot readMetadataSnapshot(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger) throws java.io.IOException
java.io.IOException
- if the index we try to read is corruptedpublic static boolean canOpenIndex(org.apache.logging.log4j.Logger logger, java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker) throws java.io.IOException
true
iff the given location contains an index an the index
can be successfully opened. This includes reading the segment infos and possible
corruption markers.java.io.IOException
public static void tryOpenIndex(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger) throws java.io.IOException, ShardLockObtainFailedException
java.io.IOException
ShardLockObtainFailedException
public org.apache.lucene.store.IndexOutput createVerifyingOutput(java.lang.String fileName, StoreFileMetaData metadata, org.apache.lucene.store.IOContext context) throws java.io.IOException
Note: Checksums are calculated by default since version 4.8.0. This method only adds the verification against the checksum in the given metadata and does not add any significant overhead.
java.io.IOException
public static void verify(org.apache.lucene.store.IndexOutput output) throws java.io.IOException
java.io.IOException
public org.apache.lucene.store.IndexInput openVerifyingInput(java.lang.String filename, org.apache.lucene.store.IOContext context, StoreFileMetaData metadata) throws java.io.IOException
java.io.IOException
public static void verify(org.apache.lucene.store.IndexInput input) throws java.io.IOException
java.io.IOException
public boolean checkIntegrityNoException(StoreFileMetaData md)
public static boolean checkIntegrityNoException(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
public static void checkIntegrity(StoreFileMetaData md, org.apache.lucene.store.Directory directory) throws java.io.IOException
java.io.IOException
public boolean isMarkedCorrupted() throws java.io.IOException
java.io.IOException
public void removeCorruptionMarker() throws java.io.IOException
java.io.IOException
public void failIfCorrupted() throws java.io.IOException
java.io.IOException
public void cleanupAndVerify(java.lang.String reason, Store.MetadataSnapshot sourceMetaData) throws java.io.IOException
reason
- the reason for this cleanup operation logged for each deleted filesourceMetaData
- the metadata used for cleanup. all files in this metadata should be kept around.java.io.IOException
- if an IOException occursjava.lang.IllegalStateException
- if the latest snapshot in this store differs from the given one after the cleanup.public int refCount()
public static boolean isAutogenerated(java.lang.String name)
public static java.lang.String digestToString(long digest)
public void deleteQuiet(java.lang.String... files)
public void markStoreCorrupted(java.io.IOException exception) throws java.io.IOException
isMarkedCorrupted()
will return true
.java.io.IOException
public void createEmpty() throws java.io.IOException
java.io.IOException
public void bootstrapNewHistory() throws java.io.IOException
java.io.IOException
public void associateIndexWithNewTranslog(java.lang.String translogUUID) throws java.io.IOException
java.io.IOException
public boolean ensureIndexHas6xCommitTags() throws java.io.IOException
true
if a new commit is flushed, otherwise return falsejava.io.IOException
public void trimUnsafeCommits(long lastSyncedGlobalCheckpoint, long minRetainedTranslogGen, Version indexVersionCreated) throws java.io.IOException
1. Replica can use unsafe commit in peer-recovery. This happens when a replica with a safe commit c1(max_seqno=1) and an unsafe commit c2(max_seqno=2) recovers from a primary with c1(max_seqno=1). If a new document(seqno=2) is added without flushing, the global checkpoint is advanced to 2; and the replica recovers again, it will use the unsafe commit c2(max_seqno=2 at most gcp=2) as the starting commit for sequenced-based recovery even the commit c2 contains a stale operation and the document(with seqno=2) will not be replicated to the replica.
2. Min translog gen for recovery can go backwards in peer-recovery. This happens when are replica with a safe commit c1(local_checkpoint=1, recovery_translog_gen=1) and an unsafe commit c2(local_checkpoint=2, recovery_translog_gen=2). The replica recovers from a primary, and keeps c2 as the last commit, then sets last_translog_gen to 2. Flushing a new commit on the replica will cause exception as the new last commit c3 will have recovery_translog_gen=1. The recovery translog generation of a commit is calculated based on the current local checkpoint. The local checkpoint of c3 is 1 while the local checkpoint of c2 is 2.
3. Commit without translog can be used in recovery. An old index, which was created before multiple-commits is introduced (v6.2), may not have a safe commit. If that index has a snapshotted commit without translog and an unsafe commit, the policy can consider the snapshotted commit as a safe commit for recovery even the commit does not have translog.
java.io.IOException