Package dev.vortex.spark.io

Class HadoopReadable

java.lang.Object
dev.vortex.io.PooledReadable
dev.vortex.spark.io.HadoopReadable
All Implemented Interfaces:
dev.vortex.io.NativeReadable, Closeable, AutoCloseable
public final class HadoopReadable extends dev.vortex.io.PooledReadable
A NativeReadable that serves Vortex's reads from a Hadoop FileSystem.

  • Nested Class Summary

    Nested classes/interfaces inherited from class dev.vortex.io.PooledReadable

    dev.vortex.io.PooledReadable.PositionalStream, dev.vortex.io.PooledReadable.ScratchBytes

  • Field Summary

    Fields inherited from interface dev.vortex.io.NativeReadable

    DEFAULT_COALESCE_DISTANCE, DEFAULT_COALESCE_MAX_SIZE

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    coalesceDistance()
     
    int
    coalesceMaxSize()
     
    static HadoopReadable
    open(org.apache.hadoop.conf.Configuration conf, String path)
    Opens a readable over path, stating the file for its length.
    static HadoopReadable
    open(org.apache.hadoop.conf.Configuration conf, String path, long length)
    Opens a readable over path.
    static HadoopReadable
    open(org.apache.hadoop.conf.Configuration conf, String path, long length, int coalesceDistance, int coalesceMaxSize)
    Opens a readable with the given coalescing gap and maximum merged size, both in bytes.
    protected dev.vortex.io.PooledReadable.PositionalStream
    openStream()
     

    Methods inherited from class dev.vortex.io.PooledReadable

    close, length, name, readFully

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • open

      public static HadoopReadable open(org.apache.hadoop.conf.Configuration conf, String path)
      Opens a readable over path, stating the file for its length.

    • open

      public static HadoopReadable open(org.apache.hadoop.conf.Configuration conf, String path, long length)
      Opens a readable over path. A non-negative length is taken as authoritative — the driver already stats every file when it plans the scan, so executors need not stat them again.

    • open

      public static HadoopReadable open(org.apache.hadoop.conf.Configuration conf, String path, long length, int coalesceDistance, int coalesceMaxSize)
      Opens a readable with the given coalescing gap and maximum merged size, both in bytes. Native code rejects a negative gap or non-positive size when the source is registered.

    • coalesceDistance

      public int coalesceDistance()

    • coalesceMaxSize

      public int coalesceMaxSize()

    • openStream

      protected dev.vortex.io.PooledReadable.PositionalStream openStream() throws IOException
      Specified by:
      openStream in class dev.vortex.io.PooledReadable
      Throws:
      IOException