Class Translog

All Implemented Interfaces:, java.lang.AutoCloseable, IndexShardComponent

public class Translog
extends AbstractIndexShardComponent
implements IndexShardComponent,
A Translog is a per index shard component that records all non-committed index operations in a durable manner. In Elasticsearch there is one Translog instance per InternalEngine. The engine records the current translog generation getGeneration() in it's commit metadata using TRANSLOG_GENERATION_KEY to reference the generation that contains all operations that have not yet successfully been committed to the engines lucene index. Additionally, since Elasticsearch 2.0 the engine also records a TRANSLOG_UUID_KEY with each commit to ensure a strong association between the lucene index an the transaction log file. This UUID is used to prevent accidental recovery from a transaction log that belongs to a different engine.

Each Translog has only one translog file open for writes at any time referenced by a translog generation ID. This ID is written to a translog.ckp file that is designed to fit in a single disk block such that a write of the file is atomic. The checkpoint file is written on each fsync operation of the translog and records the number of operations written, the current translog's file generation, its fsynced offset in bytes, and other important statistics.

When the current translog file reaches a certain size (IndexSettings.INDEX_TRANSLOG_GENERATION_THRESHOLD_SIZE_SETTING, or when a clear separation between old and new operations (upon change in primary term), the current file is reopened for read only and a new write only file is created. Any non-current, read only translog file always has a translog-${gen}.ckp associated with it which is an fsynced copy of its last translog.ckp such that in disaster recovery last fsynced offsets, number of operation etc. are still preserved.

  • Field Details

  • Constructor Details

    • Translog

      public Translog​(TranslogConfig config, java.lang.String translogUUID, TranslogDeletionPolicy deletionPolicy, java.util.function.LongSupplier globalCheckpointSupplier, java.util.function.LongSupplier primaryTermSupplier, java.util.function.LongConsumer persistedSequenceNumberConsumer) throws
      Creates a new Translog instance. This method will create a new transaction log unless the given Translog.TranslogGeneration is null. If the generation is null this method is destructive and will delete all files in the translog path given. If the generation is not null, this method tries to open the given translog generation. The generation is treated as the last generation referenced from already committed data. This means all operations that have not yet been committed should be in the translog file referenced by this generation. The translog creation will fail if this generation can't be opened.
      config - the configuration of this translog
      translogUUID - the translog uuid to open, null for a new translog
      deletionPolicy - an instance of TranslogDeletionPolicy that controls when a translog file can be safely deleted
      globalCheckpointSupplier - a supplier for the global checkpoint
      primaryTermSupplier - a supplier for the latest value of primary term of the owning index shard. The latest term value is examined and stored in the header whenever a new generation is rolled. It's guaranteed from outside that a new generation is rolled when the term is increased. This guarantee allows to us to validate and reject operation whose term is higher than the primary term stored in the translog header.
      persistedSequenceNumberConsumer - a callback that's called whenever an operation with a given sequence number is successfully persisted.
  • Method Details

    • parseIdFromFileName

      public static long parseIdFromFileName​(java.nio.file.Path translogFile)
      Extracts the translog generation from a file name.
      java.lang.IllegalArgumentException - if the path doesn't match the expected pattern.
    • isOpen

      public boolean isOpen()
      Returns true if this Translog is still open.
    • close

      public void close() throws
      Specified by:
      close in interface java.lang.AutoCloseable
      Specified by:
      close in interface
    • location

      public java.nio.file.Path location()
      Returns all translog locations as absolute paths. These paths don't contain actual translog files they are directories holding the transaction logs.
    • currentFileGeneration

      public long currentFileGeneration()
      Returns the generation of the current transaction log.
    • getMinFileGeneration

      public long getMinFileGeneration()
      Returns the minimum file generation referenced by the translog
    • totalOperations

      public int totalOperations()
      Returns the number of operations in the translog files
    • sizeInBytes

      public long sizeInBytes()
      Returns the size in bytes of the v files
    • totalOperationsByMinGen

      public int totalOperationsByMinGen​(long minGeneration)
      Returns the number of operations in the translog files at least the given generation
    • estimateTotalOperationsFromMinSeq

      public int estimateTotalOperationsFromMinSeq​(long minSeqNo)
      Returns the number of operations in the transaction files that contain operations with seq# above the given number.
    • sizeInBytesByMinGen

      public long sizeInBytesByMinGen​(long minGeneration)
      Returns the size in bytes of the translog files at least the given generation
    • add

      public Translog.Location add​(Translog.Operation operation) throws
      Adds an operation to the transaction log.
      operation - the operation to add
      the location of the operation in the translog
      Throws: - if adding the operation to the translog resulted in an I/O exception
    • shouldRollGeneration

      public boolean shouldRollGeneration()
      Tests whether or not the translog generation should be rolled to a new generation. This test is based on the size of the current generation compared to the configured generation threshold size.
      true if the current generation should be rolled to a new generation
    • getLastWriteLocation

      public Translog.Location getLastWriteLocation()
      The a Translog.Location that will sort after the Translog.Location returned by the last write but before any locations which can be returned by the next write.
    • getLastSyncedGlobalCheckpoint

      public long getLastSyncedGlobalCheckpoint()
      The last synced checkpoint for this translog.
      the last synced checkpoint
    • newSnapshot

      public Translog.Snapshot newSnapshot() throws
      Snapshots the current transaction log allowing to safely iterate over the snapshot. Snapshots are fixed in time and will not be updated with future operations.
    • newSnapshotFromGen

      public Translog.Snapshot newSnapshotFromGen​(Translog.TranslogGeneration fromGeneration, long upToSeqNo) throws
    • readOperation

      public Translog.Operation readOperation​(Translog.Location location) throws
      Reads and returns the operation from the given location if the generation it references is still available. Otherwise this method will return null.
    • newSnapshotFromMinSeqNo

      public Translog.Snapshot newSnapshotFromMinSeqNo​(long minSeqNo) throws
    • acquireRetentionLock

      public acquireRetentionLock()
      Acquires a lock on the translog files, preventing them from being trimmed
    • sync

      public void sync() throws
      Sync's the translog.
    • syncNeeded

      public boolean syncNeeded()
      Returns true if an fsync is required to ensure durability of the translogs operations or it's metadata.
    • getFilename

      public static java.lang.String getFilename​(long generation)
      package private for testing
    • trimOperations

      public void trimOperations​(long belowTerm, long aboveSeqNo) throws
      Trims translog for terms of files below belowTerm and seq# above aboveSeqNo. Effectively it moves max visible seq# Checkpoint.trimmedAboveSeqNo therefore TranslogSnapshot skips those operations.
    • ensureSynced

      public boolean ensureSynced​(Translog.Location location) throws
      Ensures that the given location has be synced / written to the underlying storage.
      Returns true iff this call caused an actual sync operation otherwise false
    • ensureSynced

      public boolean ensureSynced​(<Translog.Location> locations) throws
      Ensures that all locations in the given stream have been synced / written to the underlying storage. This method allows for internal optimization to minimize the amount of fsync operations if multiple locations must be synced.
      Returns true iff this call caused an actual sync operation otherwise false
    • closeOnTragicEvent

      protected void closeOnTragicEvent​(java.lang.Exception ex)
      Closes the translog if the current translog writer experienced a tragic exception. Note that in case this thread closes the translog it must not already be holding a read lock on the translog as it will acquire a write lock in the course of closing the translog
      ex - if an exception occurs closing the translog, it will be suppressed into the provided exception
    • stats

      public TranslogStats stats()
      return stats
    • getConfig

      public TranslogConfig getConfig()
    • getDeletionPolicy

      public TranslogDeletionPolicy getDeletionPolicy()
    • readOperations

      public static java.util.List<Translog.Operation> readOperations​(StreamInput input, java.lang.String source) throws
      Reads a list of operations written with writeOperations(StreamOutput, List)
    • writeOperations

      public static void writeOperations​(StreamOutput outStream, java.util.List<Translog.Operation> toWrite) throws
      Writes all operations in the given iterable to the given output stream including the size of the array use readOperations(StreamInput, String) to read it back.
    • writeOperationNoSize

      public static void writeOperationNoSize​(BufferedChecksumStreamOutput out, Translog.Operation op) throws
    • getMinGenerationForSeqNo

      public Translog.TranslogGeneration getMinGenerationForSeqNo​(long seqNo)
      Gets the minimum generation that could contain any sequence number after the specified sequence number, or the current generation if there is no generation that could any such sequence number.
      seqNo - the sequence number
      the minimum generation for the sequence number
    • rollGeneration

      public void rollGeneration() throws
      Roll the current translog generation into a new generation. This does not commit the translog.
      Throws: - if an I/O exception occurred during any file operations
    • trimUnreferencedReaders

      public void trimUnreferencedReaders() throws
      Trims unreferenced translog generations by asking TranslogDeletionPolicy for the minimum required generation
    • getGeneration

      public Translog.TranslogGeneration getGeneration()
      Returns the current generation of this translog. This corresponds to the latest uncommitted translog generation
    • getTragicException

      public java.lang.Exception getTragicException()
      If this Translog was closed as a side-effect of a tragic exception, e.g. disk full while flushing a new segment, this returns the root cause exception. Otherwise (no tragic exception has occurred) it returns null.
    • readGlobalCheckpoint

      public static long readGlobalCheckpoint​(java.nio.file.Path location, java.lang.String expectedTranslogUUID) throws
      Reads the sequence numbers global checkpoint from the translog checkpoint. This ensures that the translogUUID from this translog matches with the provided translogUUID.
      location - the location of the translog
      the global checkpoint
      Throws: - if an I/O exception occurred reading the checkpoint
      TranslogCorruptedException - if the translog is corrupted or mismatched with the given uuid
    • readMinTranslogGeneration

      public static long readMinTranslogGeneration​(java.nio.file.Path location, java.lang.String expectedTranslogUUID) throws
      Returns the minimum translog generation retained by the translog at the given location. This ensures that the translogUUID from this translog matches with the provided translogUUID.
      location - the location of the translog
      the minimum translog generation
      Throws: - if an I/O exception occurred reading the checkpoint
      TranslogCorruptedException - if the translog is corrupted or mismatched with the given uuid
    • getTranslogUUID

      public java.lang.String getTranslogUUID()
      Returns the translog uuid used to associate a lucene index with a translog.
    • getMaxSeqNo

      public long getMaxSeqNo()
      Returns the max seq_no of translog operations found in this translog. Since this value is calculated based on the current existing readers, this value is not necessary to be the max seq_no of all operations have been stored in this translog.
    • createEmptyTranslog

      public static java.lang.String createEmptyTranslog​(java.nio.file.Path location, long initialGlobalCheckpoint, ShardId shardId, long primaryTerm) throws