Package org.elasticsearch.indices.recovery

Class MultiFileTransfer<Request extends MultiFileTransfer.ChunkRequest>

java.lang.Object

org.elasticsearch.indices.recovery.MultiFileTransfer<Request>

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public abstract class MultiFileTransfer<Request extends MultiFileTransfer.ChunkRequest>
extends java.lang.Object
implements java.io.Closeable

File chunks are sent/requested sequentially by at most one thread at any time. However, the sender/requestor won't wait for the response before processing the next file chunk request to reduce the recovery time especially on secure/compressed or high latency communication.

The sender/requestor can send up to maxConcurrentFileChunks file chunk requests without waiting for responses. Since the recovery target can receive file chunks out of order, it has to buffer those file chunks in memory and only flush to disk when there's no gap. To ensure the recover target never buffers more than maxConcurrentFileChunks file chunks, we allow the sender/requestor to send only up to maxConcurrentFileChunks file chunk requests from the last flushed (and acknowledged) file chunk. We leverage the local checkpoint tracker for this purpose. We generate a new sequence number and assign it to each file chunk request before sending; then mark that sequence number as processed when we receive a response for the corresponding file chunk request. With the local checkpoint tracker, we know the last acknowledged-flushed file-chunk is a file chunk whose requestSeqId equals to the local checkpoint because the recover target can flush all file chunks up to the local checkpoint.

When the number of un-replied file chunk requests reaches the limit (i.e. the gap between the max_seq_no and the local checkpoint is greater than maxConcurrentFileChunks), the sending/requesting thread will abort its execution. That process will be resumed by one of the networking threads which receive/handle the responses of the current pending file chunk requests. This process will continue until all chunk requests are sent/responded.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	MultiFileTransfer.ChunkRequest

Constructor Summary

Constructors

Modifier	Constructor	Description
protected	MultiFileTransfer(org.apache.logging.log4j.Logger logger, ThreadContext threadContext, ActionListener<java.lang.Void> listener, int maxConcurrentFileChunks, java.util.List<StoreFileMetaData> files)

Method Summary

Modifier and Type	Method	Description
protected abstract void	executeChunkRequest(Request request, ActionListener<java.lang.Void> listener)
protected abstract void	handleError(StoreFileMetaData md, java.lang.Exception e)
protected abstract Request	nextChunkRequest(StoreFileMetaData md)
protected abstract void	onNewFile(StoreFileMetaData md)	This method is called when starting sending/requesting a new file.
void	start()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable

close

Constructor Details

MultiFileTransfer

protected MultiFileTransfer(org.apache.logging.log4j.Logger logger, ThreadContext threadContext, ActionListener<java.lang.Void> listener, int maxConcurrentFileChunks, java.util.List<StoreFileMetaData> files)

Method Details

start

public final void start()

onNewFile

protected abstract void onNewFile(StoreFileMetaData md) throws java.io.IOException

This method is called when starting sending/requesting a new file. Subclasses should override this method to reset the file offset or close the previous file and open a new file if needed.

Throws:

java.io.IOException

nextChunkRequest

protected abstract Request nextChunkRequest(StoreFileMetaData md) throws java.io.IOException

Throws:

java.io.IOException

executeChunkRequest

protected abstract void executeChunkRequest(Request request, ActionListener<java.lang.Void> listener)

handleError

protected abstract void handleError(StoreFileMetaData md, java.lang.Exception e) throws java.lang.Exception

Throws:

java.lang.Exception