Package org.elasticsearch.index.translog

Class Translog

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, IndexShardComponent
    public class Translog
extends AbstractIndexShardComponent
implements IndexShardComponent, java.io.Closeable
    A Translog is a per index shard component that records all non-committed index operations in a durable manner. In Elasticsearch there is one Translog instance per InternalEngine. The engine records the current translog generation getGeneration() in it's commit metadata using TRANSLOG_GENERATION_KEY to reference the generation that contains all operations that have not yet successfully been committed to the engines lucene index. Additionally, since Elasticsearch 2.0 the engine also records a TRANSLOG_UUID_KEY with each commit to ensure a strong association between the lucene index an the transaction log file. This UUID is used to prevent accidental recovery from a transaction log that belongs to a different engine.

    Each Translog has only one translog file open for writes at any time referenced by a translog generation ID. This ID is written to a translog.ckp file that is designed to fit in a single disk block such that a write of the file is atomic. The checkpoint file is written on each fsync operation of the translog and records the number of operations written, the current translog's file generation, its fsynced offset in bytes, and other important statistics.

    When the current translog file reaches a certain size (IndexSettings.INDEX_TRANSLOG_GENERATION_THRESHOLD_SIZE_SETTING, or when a clear separation between old and new operations (upon change in primary term), the current file is reopened for read only and a new write only file is created. Any non-current, read only translog file always has a translog-${gen}.ckp associated with it which is an fsynced copy of its last translog.ckp such that in disaster recovery last fsynced offsets, number of operation etc. are still preserved.

    • Constructor Detail

      • Translog

        public Translog​(TranslogConfig config,
                java.lang.String translogUUID,
                TranslogDeletionPolicy deletionPolicy,
                java.util.function.LongSupplier globalCheckpointSupplier,
                java.util.function.LongSupplier primaryTermSupplier)
         throws java.io.IOException
        Creates a new Translog instance. This method will create a new transaction log unless the given Translog.TranslogGeneration is null. If the generation is null this method is destructive and will delete all files in the translog path given. If the generation is not null, this method tries to open the given translog generation. The generation is treated as the last generation referenced from already committed data. This means all operations that have not yet been committed should be in the translog file referenced by this generation. The translog creation will fail if this generation can't be opened.
        Parameters:
        config - the configuration of this translog
        translogUUID - the translog uuid to open, null for a new translog
        deletionPolicy - an instance of TranslogDeletionPolicy that controls when a translog file can be safely deleted
        globalCheckpointSupplier - a supplier for the global checkpoint
        primaryTermSupplier - a supplier for the latest value of primary term of the owning index shard. The latest term value is examined and stored in the header whenever a new generation is rolled. It's guaranteed from outside that a new generation is rolled when the term is increased. This guarantee allows to us to validate and reject operation whose term is higher than the primary term stored in the translog header.
        Throws:
        java.io.IOException

    • Method Detail

      • parseIdFromFileName

        public static long parseIdFromFileName​(java.nio.file.Path translogFile)
        Extracts the translog generation from a file name.
        Throws:
        java.lang.IllegalArgumentException - if the path doesn't match the expected pattern.

      • isOpen

        public boolean isOpen()
        Returns true if this Translog is still open.

      • close

        public void close()
           throws java.io.IOException
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
        Throws:
        java.io.IOException

      • location

        public java.nio.file.Path location()
        Returns all translog locations as absolute paths. These paths don't contain actual translog files they are directories holding the transaction logs.

      • currentFileGeneration

        public long currentFileGeneration()
        Returns the generation of the current transaction log.

      • getMinFileGeneration

        public long getMinFileGeneration()
        Returns the minimum file generation referenced by the translog

      • totalOperations

        public int totalOperations()
        Returns the number of operations in the translog files

      • sizeInBytes

        public long sizeInBytes()
        Returns the size in bytes of the v files

      • totalOperationsByMinGen

        public int totalOperationsByMinGen​(long minGeneration)
        Returns the number of operations in the translog files at least the given generation

      • estimateTotalOperationsFromMinSeq

        public int estimateTotalOperationsFromMinSeq​(long minSeqNo)
        Returns the number of operations in the transaction files that contain operations with seq# above the given number.

      • sizeInBytesByMinGen

        public long sizeInBytesByMinGen​(long minGeneration)
        Returns the size in bytes of the translog files at least the given generation

      • add

        public Translog.Location add​(Translog.Operation operation)
                      throws java.io.IOException
        Adds an operation to the transaction log.
        Parameters:
        operation - the operation to add
        Returns:
        the location of the operation in the translog
        Throws:
        java.io.IOException - if adding the operation to the translog resulted in an I/O exception

      • shouldRollGeneration

        public boolean shouldRollGeneration()
        Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.
        Returns:
        true if the current generation should be rolled to a new generation

      • getLastSyncedGlobalCheckpoint

        public long getLastSyncedGlobalCheckpoint()
        The last synced checkpoint for this translog.
        Returns:
        the last synced checkpoint

      • newSnapshot

        public Translog.Snapshot newSnapshot()
                              throws java.io.IOException
        Snapshots the current transaction log allowing to safely iterate over the snapshot. Snapshots are fixed in time and will not be updated with future operations.
        Throws:
        java.io.IOException

      • readOperation

        public Translog.Operation readOperation​(Translog.Location location)
                                 throws java.io.IOException
        Reads and returns the operation from the given location if the generation it references is still available. Otherwise this method will return null.
        Throws:
        java.io.IOException

      • newSnapshotFromMinSeqNo

        public Translog.Snapshot newSnapshotFromMinSeqNo​(long minSeqNo)
                                          throws java.io.IOException
        Throws:
        java.io.IOException

      • acquireRetentionLock

        public java.io.Closeable acquireRetentionLock()
        Acquires a lock on the translog files, preventing them from being trimmed

      • sync

        public void sync()
          throws java.io.IOException
        Sync's the translog.
        Throws:
        java.io.IOException

      • syncNeeded

        public boolean syncNeeded()
        Returns true if an fsync is required to ensure durability of the translogs operations or it's metadata.

      • getFilename

        public static java.lang.String getFilename​(long generation)
        package private for testing

      • trimOperations

        public void trimOperations​(long belowTerm,
                           long aboveSeqNo)
                    throws java.io.IOException
        Trims translog for terms of files below belowTerm and seq# above aboveSeqNo. Effectively it moves max visible seq# Checkpoint.trimmedAboveSeqNo therefore TranslogSnapshot skips those operations.
        Throws:
        java.io.IOException

      • ensureSynced

        public boolean ensureSynced​(Translog.Location location)
                     throws java.io.IOException
        Ensures that the given location has be synced / written to the underlying storage.
        Returns:
        Returns true iff this call caused an actual sync operation otherwise false
        Throws:
        java.io.IOException

      • ensureSynced

        public boolean ensureSynced​(java.util.stream.Stream<Translog.Location> locations)
                     throws java.io.IOException
        Ensures that all locations in the given stream have been synced / written to the underlying storage. This method allows for internal optimization to minimize the amount of fsync operations if multiple locations must be synced.
        Returns:
        Returns true iff this call caused an actual sync operation otherwise false
        Throws:
        java.io.IOException

      • closeOnTragicEvent

        protected void closeOnTragicEvent​(java.lang.Exception ex)
        Closes the translog if the current translog writer experienced a tragic exception. Note that in case this thread closes the translog it must not already be holding a read lock on the translog as it will acquire a write lock in the course of closing the translog
        Parameters:
        ex - if an exception occurs closing the translog, it will be suppressed into the provided exception

      • writeOperations

        public static void writeOperations​(StreamOutput outStream,
                                   java.util.List<Translog.Operation> toWrite)
                            throws java.io.IOException
        Writes all operations in the given iterable to the given output stream including the size of the array use readOperations(StreamInput, String) to read it back.
        Throws:
        java.io.IOException

      • getMinGenerationForSeqNo

        public Translog.TranslogGeneration getMinGenerationForSeqNo​(long seqNo)
        Gets the minimum generation that could contain any sequence number after the specified sequence number, or the current generation if there is no generation that could any such sequence number.
        Parameters:
        seqNo - the sequence number
        Returns:
        the minimum generation for the sequence number

      • rollGeneration

        public void rollGeneration()
                    throws java.io.IOException
        Roll the current translog generation into a new generation. This does not commit the translog.
        Throws:
        java.io.IOException - if an I/O exception occurred during any file operations

      • trimUnreferencedReaders

        public void trimUnreferencedReaders()
                             throws java.io.IOException
        Trims unreferenced translog generations by asking TranslogDeletionPolicy for the minimum required generation
        Throws:
        java.io.IOException

      • getGeneration

        public Translog.TranslogGeneration getGeneration()
        Returns the current generation of this translog. This corresponds to the latest uncommitted translog generation

      • isCurrent

        public boolean isCurrent​(Translog.TranslogGeneration generation)
        Returns true iff the given generation is the current generation of this translog

      • getTragicException

        public java.lang.Exception getTragicException()
        If this Translog was closed as a side-effect of a tragic exception, e.g. disk full while flushing a new segment, this returns the root cause exception. Otherwise (no tragic exception has occurred) it returns null.

      • readGlobalCheckpoint

        public static long readGlobalCheckpoint​(java.nio.file.Path location,
                                        java.lang.String expectedTranslogUUID)
                                 throws java.io.IOException
        Reads the sequence numbers global checkpoint from the translog checkpoint. This ensures that the translogUUID from this translog matches with the provided translogUUID.
        Parameters:
        location - the location of the translog
        Returns:
        the global checkpoint
        Throws:
        java.io.IOException - if an I/O exception occurred reading the checkpoint
        TranslogCorruptedException - if the translog is corrupted or mismatched with the given uuid

      • readMinTranslogGeneration

        public static long readMinTranslogGeneration​(java.nio.file.Path location,
                                             java.lang.String expectedTranslogUUID)
                                      throws java.io.IOException
        Returns the minimum translog generation retained by the translog at the given location. This ensures that the translogUUID from this translog matches with the provided translogUUID.
        Parameters:
        location - the location of the translog
        Returns:
        the minimum translog generation
        Throws:
        java.io.IOException - if an I/O exception occurred reading the checkpoint
        TranslogCorruptedException - if the translog is corrupted or mismatched with the given uuid

      • getTranslogUUID

        public java.lang.String getTranslogUUID()
        Returns the translog uuid used to associate a lucene index with a translog.

      • getMaxSeqNo

        public long getMaxSeqNo()
        Returns the max seq_no of translog operations found in this translog. Since this value is calculated based on the current existing readers, this value is not necessary to be the max seq_no of all operations have been stored in this translog.

      • createEmptyTranslog

        public static java.lang.String createEmptyTranslog​(java.nio.file.Path location,
                                                   long initialGlobalCheckpoint,
                                                   ShardId shardId,
                                                   long primaryTerm)
                                            throws java.io.IOException
        Throws:
        java.io.IOException