org.elasticsearch.index.translog

Class Translog

java.lang.Object

org.elasticsearch.index.shard.AbstractIndexShardComponent

org.elasticsearch.index.translog.Translog

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable, org.apache.lucene.index.TwoPhaseCommit, IndexShardComponent

public class Translog
extends AbstractIndexShardComponent
implements IndexShardComponent, java.io.Closeable, org.apache.lucene.index.TwoPhaseCommit

A Translog is a per index shard component that records all non-committed index operations in a durable manner. In Elasticsearch there is one Translog instance per InternalEngine. The engine records the current translog generation getGeneration() in it's commit metadata using TRANSLOG_GENERATION_KEY to reference the generation that contains all operations that have not yet successfully been committed to the engines lucene index. Additionally, since Elasticsearch 2.0 the engine also records a TRANSLOG_UUID_KEY with each commit to ensure a strong association between the lucene index an the transaction log file. This UUID is used to prevent accidential recovery from a transaction log that belongs to a different engine.

Each Translog has only one translog file open at any time referenced by a translog generation ID. This ID is written to a translog.ckp file that is designed to fit in a single disk block such that a write of the file is atomic. The checkpoint file is written on each fsync operation of the translog and records the number of operations written, the current tranlogs file generation and it's fsynced offset in bytes.

When a translog is opened the checkpoint is use to retrieve the latest translog file generation and subsequently to open the last written file to recovery operations. The Translog.TranslogGeneration, given when the translog is opened / constructed is compared against the latest generation and all consecutive translog files singe the given generation and the last generation in the checkpoint will be recovered and preserved until the next generation is committed using commit(). In the common case the translog file generation in the checkpoint and the generation passed to the translog on creation are the same. The only situation when they can be different is when an actual translog commit fails in between prepareCommit() and commit(). In such a case the currently being committed translog file will not be deleted since it's commit was not successful. Yet, a new/current translog file is already opened at that point such that there is more than one translog file present. Such an uncommitted translog file always has a translog-${gen}.ckp associated with it which is an fsynced copy of the it's last translog.ckp such that in disaster recovery last fsynced offsets, number of operation etc. are still preserved.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Translog.Delete
static class	Translog.Durability
static class	Translog.Index
static class	Translog.Location
static interface	Translog.Operation A generic interface representing an operation performed on the transaction log.
static interface	Translog.Snapshot A snapshot of the transaction log, allows to iterate over all the transaction log operations.
static class	Translog.Source
static class	Translog.TranslogGeneration References a transaction log generation
class	Translog.View a view into the translog, capturing all translog file at the moment of creation and updated with any future translog.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CHECKPOINT_FILE_NAME
static java.lang.String	CHECKPOINT_SUFFIX
protected ReleasableLock	readLock
static java.lang.String	TRANSLOG_FILE_PREFIX
static java.lang.String	TRANSLOG_FILE_SUFFIX
static java.lang.String	TRANSLOG_GENERATION_KEY
static java.lang.String	TRANSLOG_UUID_KEY
protected ReleasableLock	writeLock

Fields inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent

deprecationLogger, indexSettings, logger, shardId

Constructor Summary

Constructors

Constructor and Description
Translog(TranslogConfig config, Translog.TranslogGeneration translogGeneration) Creates a new Translog instance.

Method Summary

Modifier and Type	Method and Description
Translog.Location	add(Translog.Operation operation) Adds a delete / index operations to the transaction log.
void	close()
long	commit()
long	currentFileGeneration() Returns the generation of the current transaction log.
boolean	ensureSynced(java.util.stream.Stream<Translog.Location> locations) Ensures that all locations in the given stream have been synced / written to the underlying storage.
boolean	ensureSynced(Translog.Location location) Ensures that the given location has be synced / written to the underlying storage.
TranslogConfig	getConfig()
static java.lang.String	getFilename(long generation) package private for testing
Translog.TranslogGeneration	getGeneration() Returns the current generation of this translog.
Translog.Location	getLastWriteLocation() The a Translog.Location that will sort after the Translog.Location returned by the last write but before any locations which can be returned by the next write.
java.lang.Exception	getTragicException() If this Translog was closed as a side-effect of a tragic exception, e.g.
java.lang.String	getTranslogUUID() Returns the translog uuid used to associate a lucene index with a translog.
boolean	isCurrent(Translog.TranslogGeneration generation) Returns true iff the given generation is the current gbeneration of this translog
boolean	isOpen() Returns true if this Translog is still open.
java.nio.file.Path	location() Returns all translog locations as absolute paths.
Translog.Snapshot	newSnapshot() Snapshots the current transaction log allowing to safely iterate over the snapshot.
Translog.View	newView() Returns a view into the current translog that is guaranteed to retain all current operations while receiving future ones as well
static long	parseIdFromFileName(java.nio.file.Path translogFile) Extracts the translog generation from a file name.
long	prepareCommit()
static java.util.List<Translog.Operation>	readOperations(StreamInput input) Reads a list of operations written with writeOperations(StreamOutput, List)
void	rollback()
long	sizeInBytes() Returns the size in bytes of the translog files that aren't committed to lucene.
TranslogStats	stats() return stats
void	sync() Sync's the translog.
boolean	syncNeeded()
int	totalOperations() Returns the number of operations in the transaction files that aren't committed to lucene..
static void	writeOperationNoSize(BufferedChecksumStreamOutput out, Translog.Operation op)
static void	writeOperations(StreamOutput outStream, java.util.List<Translog.Operation> toWrite) Writes all operations in the given iterable to the given output stream including the size of the array use readOperations(StreamInput) to read it back.

Methods inherited from class org.elasticsearch.index.shard.AbstractIndexShardComponent

indexSettings, nodeName, shardId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.elasticsearch.index.shard.IndexShardComponent

indexSettings, shardId

Field Detail

TRANSLOG_GENERATION_KEY

public static final java.lang.String TRANSLOG_GENERATION_KEY

See Also:

Constant Field Values

TRANSLOG_UUID_KEY

public static final java.lang.String TRANSLOG_UUID_KEY

See Also:

Constant Field Values

TRANSLOG_FILE_PREFIX

public static final java.lang.String TRANSLOG_FILE_PREFIX

See Also:

Constant Field Values

TRANSLOG_FILE_SUFFIX

public static final java.lang.String TRANSLOG_FILE_SUFFIX

See Also:

Constant Field Values

CHECKPOINT_SUFFIX

public static final java.lang.String CHECKPOINT_SUFFIX

See Also:

Constant Field Values

CHECKPOINT_FILE_NAME

public static final java.lang.String CHECKPOINT_FILE_NAME

See Also:

Constant Field Values

readLock

protected final ReleasableLock readLock

writeLock

protected final ReleasableLock writeLock

Constructor Detail

Translog

public Translog(TranslogConfig config,
                Translog.TranslogGeneration translogGeneration)
         throws java.io.IOException

Creates a new Translog instance. This method will create a new transaction log unless the given TranslogConfig has a non-null Translog.TranslogGeneration. If the generation is null this method us destructive and will delete all files in the translog path given.

Parameters:

config - the configuration of this translog

translogGeneration - the translog generation to open. If this is null a new translog is created. If non-null the translog tries to open the given translog generation. The generation is treated as the last generation referenced form already committed data. This means all operations that have not yet been committed should be in the translog file referenced by this generation. The translog creation will fail if this generation can't be opened.

Throws:

java.io.IOException

See Also:

TranslogConfig.getTranslogPath()

Method Detail

parseIdFromFileName

public static long parseIdFromFileName(java.nio.file.Path translogFile)

Extracts the translog generation from a file name.

Throws:

java.lang.IllegalArgumentException - if the path doesn't match the expected pattern.

isOpen

public boolean isOpen()

Returns true if this Translog is still open.

close

public void close()
           throws java.io.IOException

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

location

public java.nio.file.Path location()

Returns all translog locations as absolute paths. These paths don't contain actual translog files they are directories holding the transaction logs.

currentFileGeneration

public long currentFileGeneration()

Returns the generation of the current transaction log.

totalOperations

public int totalOperations()

Returns the number of operations in the transaction files that aren't committed to lucene..

sizeInBytes

public long sizeInBytes()

Returns the size in bytes of the translog files that aren't committed to lucene.

add

public Translog.Location add(Translog.Operation operation)
                      throws java.io.IOException

Adds a delete / index operations to the transaction log.

Throws:

java.io.IOException

See Also:

Translog.Operation, Translog.Index, Translog.Delete

getLastWriteLocation

public Translog.Location getLastWriteLocation()

The a Translog.Location that will sort after the Translog.Location returned by the last write but before any locations which can be returned by the next write.

newSnapshot

public Translog.Snapshot newSnapshot()

Snapshots the current transaction log allowing to safely iterate over the snapshot. Snapshots are fixed in time and will not be updated with future operations.

newView

public Translog.View newView()

Returns a view into the current translog that is guaranteed to retain all current operations while receiving future ones as well

sync

public void sync()
          throws java.io.IOException

Sync's the translog.

Throws:

java.io.IOException

syncNeeded

public boolean syncNeeded()

getFilename

public static java.lang.String getFilename(long generation)

package private for testing

ensureSynced

public boolean ensureSynced(Translog.Location location)
                     throws java.io.IOException

Ensures that the given location has be synced / written to the underlying storage.

Returns:

Returns true iff this call caused an actual sync operation otherwise false

Throws:

java.io.IOException

ensureSynced

public boolean ensureSynced(java.util.stream.Stream<Translog.Location> locations)
                     throws java.io.IOException

Ensures that all locations in the given stream have been synced / written to the underlying storage. This method allows for internal optimization to minimize the amout of fsync operations if multiple locations must be synced.

Returns:

Returns true iff this call caused an actual sync operation otherwise false

Throws:

java.io.IOException

stats

public TranslogStats stats()

return stats

getConfig

public TranslogConfig getConfig()

readOperations

public static java.util.List<Translog.Operation> readOperations(StreamInput input)
                                                         throws java.io.IOException

Reads a list of operations written with writeOperations(StreamOutput, List)

Throws:

java.io.IOException

writeOperations

public static void writeOperations(StreamOutput outStream,
                                   java.util.List<Translog.Operation> toWrite)
                            throws java.io.IOException

Writes all operations in the given iterable to the given output stream including the size of the array use readOperations(StreamInput) to read it back.

Throws:

java.io.IOException

writeOperationNoSize

public static void writeOperationNoSize(BufferedChecksumStreamOutput out,
                                        Translog.Operation op)
                                 throws java.io.IOException

Throws:

java.io.IOException

prepareCommit

public long prepareCommit()
                   throws java.io.IOException

Specified by:

prepareCommit in interface org.apache.lucene.index.TwoPhaseCommit

Throws:

java.io.IOException

commit

public long commit()
            throws java.io.IOException

Specified by:

commit in interface org.apache.lucene.index.TwoPhaseCommit

Throws:

java.io.IOException

rollback

public void rollback()
              throws java.io.IOException

Specified by:

rollback in interface org.apache.lucene.index.TwoPhaseCommit

Throws:

java.io.IOException

getGeneration

public Translog.TranslogGeneration getGeneration()

Returns the current generation of this translog. This corresponds to the latest uncommitted translog generation

isCurrent

public boolean isCurrent(Translog.TranslogGeneration generation)

Returns true iff the given generation is the current gbeneration of this translog

getTragicException

public java.lang.Exception getTragicException()

If this Translog was closed as a side-effect of a tragic exception, e.g. disk full while flushing a new segment, this returns the root cause exception. Otherwise (no tragic exception has occurred) it returns null.

getTranslogUUID

public java.lang.String getTranslogUUID()

Returns the translog uuid used to associate a lucene index with a translog.