NAME

Hadoop::IO::SequenceFile::HDFSWriter

VERSION

version 0.003

DESCRIPTION

This is a simple wrapper around WebHDFS providing a more file-like interface.

NAME

Hadoop::IO::SequenceFile::HDFSWriter - buffering HDFS file writer.

METHODS

new

    $class->new($webhdfs, $path, %options)

Construct new HDFSFile writer. Arguments:

$webhdfs

An instance of Net::Hadoop::WebHDFS.

$path

HDFS path to the destination file. If it doesn't exist it will be created. Can not point at a directory.

%opts

Other options:

truncate

If true, file will be truncated to zero length. If not set or false, do nothing here, and write calls just append to the existing file.

chunk_size

Buffer this many bytes before actually writing to HDFS. Default is 8 Mb.

on_write

This coderef will be called without arguments before each request to WebHDFS service.

write

    $writer->write($data)

Append $data to the file. Buffers up to chunk_size (see "new") bytes before flushing out to HDFS.

Call "flush" after last write to ensure all data reaches the destination file.

flush

    $writer->flush()

Flush buffered data out to HDFS. Normally you only need to call this once at the end of all writes.

rename

    $writer->rename($new_name)

Rename destination file to a new name. If $new_name contains '/' character, path is treated as absolute and used as is, otherwise only last component of the path is updated, ie file is renamed but stays in the same directory.

AUTHORS

• Philippe Bruhat

• Sabbir Ahmed

• Somesh Malviya

• Vikentiy Fesunov

COPYRIGHT AND LICENSE

This software is copyright (c) 2023 by Booking.com.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.