org.elasticsearch.index.store

Class Store

java.lang.Object

org.elasticsearch.index.shard.AbstractIndexShardComponent

org.elasticsearch.index.store.Store

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, RefCounted, IndexShardComponent

public class Store
extends AbstractIndexShardComponent
implements java.io.Closeable, RefCounted

A Store provides plain access to files written by an elasticsearch index shard. Each shard has a dedicated store that is uses to access Lucene's Directory which represents the lowest level of file abstraction in Lucene used to read and write Lucene indices. This class also provides access to metadata information like checksums for committed files. A committed file is a file that belongs to a segment written by a Lucene commit. Files that have not been committed ie. created during a merge or a shard refresh / NRT reopen are not considered in the MetadataSnapshot.

Note: If you use a store it's reference count should be increased before using it by calling #incRef and a corresponding #decRef must be called in a try/finally block to release the store again ie.:

      store.incRef();
      try {
        // use the store...

      } finally {
          store.decRef();
      }
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Store.MetadataSnapshot Represents a snapshot of the current directory build from the latest Lucene commit.
static interface	Store.OnClose A listener that is executed once the store is closed and all references to it are released
static class	Store.RecoveryDiff A class representing the diff between a recovery source and recovery target

Field Summary

Fields

Modifier and Type	Field and Description
static Setting<TimeValue>	INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING

Fields inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent

deprecationLogger, indexSettings, logger, shardId

Constructor Summary

Constructors

Constructor and Description
Store(ShardId shardId, IndexSettings indexSettings, DirectoryService directoryService, ShardLock shardLock)
Store(ShardId shardId, IndexSettings indexSettings, DirectoryService directoryService, ShardLock shardLock, Store.OnClose onClose)

Method Summary

Modifier and Type	Method and Description
static boolean	canOpenIndex(org.apache.logging.log4j.Logger logger, java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker) Returns true iff the given location contains an index an the index can be successfully opened.
static void	checkIntegrity(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
boolean	checkIntegrityNoException(StoreFileMetaData md)
static boolean	checkIntegrityNoException(StoreFileMetaData md, org.apache.lucene.store.Directory directory)
void	cleanupAndVerify(java.lang.String reason, Store.MetadataSnapshot sourceMetaData) This method deletes every file in this store that is not contained in the given source meta data or is a legacy checksum file.
void	close()
org.apache.lucene.store.IndexOutput	createVerifyingOutput(java.lang.String fileName, StoreFileMetaData metadata, org.apache.lucene.store.IOContext context) The returned IndexOutput validates the files checksum.
void	decRef() Decreases the refCount of this Store instance.If the refCount drops to 0, then this store is closed.
void	deleteQuiet(java.lang.String... files)
static java.lang.String	digestToString(long digest) Produces a string representation of the given digest value.
org.apache.lucene.store.Directory	directory()
void	failIfCorrupted()
Store.MetadataSnapshot	getMetadata(org.apache.lucene.index.IndexCommit commit) Returns a new MetadataSnapshot for the given commit.
Store.MetadataSnapshot	getMetadata(org.apache.lucene.index.IndexCommit commit, boolean lockDirectory) Returns a new MetadataSnapshot for the given commit.
void	incRef() Increments the refCount of this Store instance.
static boolean	isAutogenerated(java.lang.String name) Returns true if the file is auto-generated by the store and shouldn't be deleted during cleanup.
boolean	isMarkedCorrupted()
void	markStoreCorrupted(java.io.IOException exception) Marks this store as corrupted.
org.apache.lucene.store.IndexInput	openVerifyingInput(java.lang.String filename, org.apache.lucene.store.IOContext context, StoreFileMetaData metadata)
org.apache.lucene.index.SegmentInfos	readLastCommittedSegmentsInfo() Returns the last committed segments info for this store
static Store.MetadataSnapshot	readMetadataSnapshot(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger) Reads a MetadataSnapshot from the given index locations or returns an empty snapshot if it can't be read.
int	refCount() Returns the current reference count.
void	removeCorruptionMarker() Deletes all corruption markers from this store.
void	renameTempFilesSafe(java.util.Map<java.lang.String,java.lang.String> tempFileMap) Renames all the given files from the key of the map to the value of the map.
StoreStats	stats()
boolean	tryIncRef() Tries to increment the refCount of this Store instance.
static void	tryOpenIndex(java.nio.file.Path indexLocation, ShardId shardId, NodeEnvironment.ShardLocker shardLocker, org.apache.logging.log4j.Logger logger) Tries to open an index for the given location.
static void	verify(org.apache.lucene.store.IndexInput input)
static void	verify(org.apache.lucene.store.IndexOutput output)

Methods inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent

indexSettings, nodeName, shardId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING

public static final Setting<TimeValue> INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING

Constructor Detail

Store

public Store(ShardId shardId,
             IndexSettings indexSettings,
             DirectoryService directoryService,
             ShardLock shardLock)
      throws java.io.IOException

Throws:

java.io.IOException

Store

public Store(ShardId shardId,
             IndexSettings indexSettings,
             DirectoryService directoryService,
             ShardLock shardLock,
             Store.OnClose onClose)
      throws java.io.IOException

Throws:

java.io.IOException

Method Detail

directory

public org.apache.lucene.store.Directory directory()

readLastCommittedSegmentsInfo

public org.apache.lucene.index.SegmentInfos readLastCommittedSegmentsInfo()
                                                                   throws java.io.IOException

Returns the last committed segments info for this store

Throws:

java.io.IOException - if the index is corrupted or the segments file is not present

getMetadata

public Store.MetadataSnapshot getMetadata(org.apache.lucene.index.IndexCommit commit)
                                   throws java.io.IOException

Returns a new MetadataSnapshot for the given commit. If the given commit is null the latest commit point is used. Note that this method requires the caller verify it has the right to access the store and no concurrent file changes are happening. If in doubt, you probably want to use one of the following: readMetadataSnapshot(Path, ShardId, NodeEnvironment.ShardLocker, Logger) to read a meta data while locking IndexShard.snapshotStoreMetadata() to safely read from an existing shard IndexShard.acquireIndexCommit(boolean) to get an IndexCommit which is safe to use but has to be freed

Parameters:

commit - the index commit to read the snapshot from or null if the latest snapshot should be read from the directory

Throws:

org.apache.lucene.index.CorruptIndexException - if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.

org.apache.lucene.index.IndexFormatTooOldException - if the lucene index is too old to be opened.

org.apache.lucene.index.IndexFormatTooNewException - if the lucene index is too new to be opened.

java.io.FileNotFoundException - if one or more files referenced by a commit are not present.

java.nio.file.NoSuchFileException - if one or more files referenced by a commit are not present.

org.apache.lucene.index.IndexNotFoundException - if the commit point can't be found in this store

java.io.IOException

getMetadata

public Store.MetadataSnapshot getMetadata(org.apache.lucene.index.IndexCommit commit,
                                          boolean lockDirectory)
                                   throws java.io.IOException

Returns a new MetadataSnapshot for the given commit. If the given commit is null the latest commit point is used. Note that this method requires the caller verify it has the right to access the store and no concurrent file changes are happening. If in doubt, you probably want to use one of the following: readMetadataSnapshot(Path, ShardId, NodeEnvironment.ShardLocker, Logger) to read a meta data while locking IndexShard.snapshotStoreMetadata() to safely read from an existing shard IndexShard.acquireIndexCommit(boolean) to get an IndexCommit which is safe to use but has to be freed

Parameters:

commit - the index commit to read the snapshot from or null if the latest snapshot should be read from the directory

lockDirectory - if true the index writer lock will be obtained before reading the snapshot. This should only be used if there is no started shard using this store.

Throws:

org.apache.lucene.index.CorruptIndexException - if the lucene index is corrupted. This can be caused by a checksum mismatch or an unexpected exception when opening the index reading the segments file.

org.apache.lucene.index.IndexFormatTooOldException - if the lucene index is too old to be opened.

org.apache.lucene.index.IndexFormatTooNewException - if the lucene index is too new to be opened.

java.io.FileNotFoundException - if one or more files referenced by a commit are not present.

java.nio.file.NoSuchFileException - if one or more files referenced by a commit are not present.

org.apache.lucene.index.IndexNotFoundException - if the commit point can't be found in this store

java.io.IOException

renameTempFilesSafe

public void renameTempFilesSafe(java.util.Map<java.lang.String,java.lang.String> tempFileMap)
                         throws java.io.IOException

Renames all the given files from the key of the map to the value of the map. All successfully renamed files are removed from the map in-place.

Throws:

java.io.IOException

stats

public StoreStats stats()
                 throws java.io.IOException

Throws:

java.io.IOException

incRef

public final void incRef()

Increments the refCount of this Store instance. RefCounts are used to determine when a Store can be closed safely, i.e. as soon as there are no more references. Be sure to always call a corresponding decRef(), in a finally clause; otherwise the store may never be closed. Note that close() simply calls decRef(), which means that the Store will not really be closed until decRef() has been called for all outstanding references.

Note: Close can safely be called multiple times.

Specified by:

incRef in interface RefCounted

Throws:

org.apache.lucene.store.AlreadyClosedException - iff the reference counter can not be incremented.

See Also:

decRef(), tryIncRef()

tryIncRef

public final boolean tryIncRef()

Tries to increment the refCount of this Store instance. This method will return true iff the refCount was incremented successfully otherwise false. RefCounts are used to determine when a Store can be closed safely, i.e. as soon as there are no more references. Be sure to always call a corresponding decRef(), in a finally clause; otherwise the store may never be closed. Note that close() simply calls decRef(), which means that the Store will not really be closed until decRef() has been called for all outstanding references.

Note: Close can safely be called multiple times.

Specified by:

tryIncRef in interface RefCounted

See Also:

decRef(), incRef()

decRef

public final void decRef()

Decreases the refCount of this Store instance.If the refCount drops to 0, then this store is closed.

Specified by:

decRef in interface RefCounted

See Also:

incRef()

close

public void close()

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

readMetadataSnapshot

public static Store.MetadataSnapshot readMetadataSnapshot(java.nio.file.Path indexLocation,
                                                          ShardId shardId,
                                                          NodeEnvironment.ShardLocker shardLocker,
                                                          org.apache.logging.log4j.Logger logger)
                                                   throws java.io.IOException

Reads a MetadataSnapshot from the given index locations or returns an empty snapshot if it can't be read.

Throws:

java.io.IOException - if the index we try to read is corrupted

canOpenIndex

public static boolean canOpenIndex(org.apache.logging.log4j.Logger logger,
                                   java.nio.file.Path indexLocation,
                                   ShardId shardId,
                                   NodeEnvironment.ShardLocker shardLocker)
                            throws java.io.IOException

Returns true iff the given location contains an index an the index can be successfully opened. This includes reading the segment infos and possible corruption markers.

Throws:

java.io.IOException

tryOpenIndex

public static void tryOpenIndex(java.nio.file.Path indexLocation,
                                ShardId shardId,
                                NodeEnvironment.ShardLocker shardLocker,
                                org.apache.logging.log4j.Logger logger)
                         throws java.io.IOException,
                                ShardLockObtainFailedException

Tries to open an index for the given location. This includes reading the segment infos and possible corruption markers. If the index can not be opened, an exception is thrown

Throws:

java.io.IOException

ShardLockObtainFailedException

createVerifyingOutput

public org.apache.lucene.store.IndexOutput createVerifyingOutput(java.lang.String fileName,
                                                                 StoreFileMetaData metadata,
                                                                 org.apache.lucene.store.IOContext context)
                                                          throws java.io.IOException

The returned IndexOutput validates the files checksum.

Note: Checksums are calculated by default since version 4.8.0. This method only adds the verification against the checksum in the given metadata and does not add any significant overhead.

Throws:

java.io.IOException

verify

public static void verify(org.apache.lucene.store.IndexOutput output)
                   throws java.io.IOException

Throws:

java.io.IOException

openVerifyingInput

public org.apache.lucene.store.IndexInput openVerifyingInput(java.lang.String filename,
                                                             org.apache.lucene.store.IOContext context,
                                                             StoreFileMetaData metadata)
                                                      throws java.io.IOException

Throws:

java.io.IOException

verify

public static void verify(org.apache.lucene.store.IndexInput input)
                   throws java.io.IOException

Throws:

java.io.IOException

checkIntegrityNoException

public boolean checkIntegrityNoException(StoreFileMetaData md)

checkIntegrityNoException

public static boolean checkIntegrityNoException(StoreFileMetaData md,
                                                org.apache.lucene.store.Directory directory)

checkIntegrity

public static void checkIntegrity(StoreFileMetaData md,
                                  org.apache.lucene.store.Directory directory)
                           throws java.io.IOException

Throws:

java.io.IOException

isMarkedCorrupted

public boolean isMarkedCorrupted()
                          throws java.io.IOException

Throws:

java.io.IOException

removeCorruptionMarker

public void removeCorruptionMarker()
                            throws java.io.IOException

Deletes all corruption markers from this store.

Throws:

java.io.IOException

failIfCorrupted

public void failIfCorrupted()
                     throws java.io.IOException

Throws:

java.io.IOException

cleanupAndVerify

public void cleanupAndVerify(java.lang.String reason,
                             Store.MetadataSnapshot sourceMetaData)
                      throws java.io.IOException

This method deletes every file in this store that is not contained in the given source meta data or is a legacy checksum file. After the delete it pulls the latest metadata snapshot from the store and compares it to the given snapshot. If the snapshots are inconsistent an illegal state exception is thrown.

Parameters:

reason - the reason for this cleanup operation logged for each deleted file

sourceMetaData - the metadata used for cleanup. all files in this metadata should be kept around.

Throws:

java.io.IOException - if an IOException occurs

java.lang.IllegalStateException - if the latest snapshot in this store differs from the given one after the cleanup.

refCount

public int refCount()

Returns the current reference count.

isAutogenerated

public static boolean isAutogenerated(java.lang.String name)

Returns true if the file is auto-generated by the store and shouldn't be deleted during cleanup. This includes write lock and checksum files

digestToString

public static java.lang.String digestToString(long digest)

Produces a string representation of the given digest value.

deleteQuiet

public void deleteQuiet(java.lang.String... files)

markStoreCorrupted

public void markStoreCorrupted(java.io.IOException exception)
                        throws java.io.IOException

Marks this store as corrupted. This method writes a corrupted_${uuid} file containing the given exception message. If a store contains a corrupted_${uuid} file isMarkedCorrupted() will return true.

Throws:

java.io.IOException