NAME

Doit::Lwp - commands for dealing with WWW access

SYNOPSIS

    use Doit;
    my $doit = Doit->init;
    $doit->add_component('lwp');

    $doit->lwp_mirror('http://example.com', 'mirrored-file');

DESCRIPTION

Doit::Lwp is a Doit component providing commands for dealing with WWW access. It has to be added to a script using Doit's add_component:

    $doit->add_component('lwp');

DOIT COMMANDS

The following commands are added to the Doit runner object:

lwp_mirror

    $doit->lwp_mirror($url, $filename);
    $doit->lwp_mirror($url, $filename, refresh => "always"|"never"|"unconditionally", ua => $ua);
    $doit->lwp_mirror($url, $filename, refresh => ["digest", $digest_type, $digest], ua => $ua);

Make sure that the resource at $url is fetched and stored on the local system as $filename. LWP::UserAgent's mirror is used to fetch the resource, so if the remote resource did not change and the remote server deals correctly with If-Modified-Since requests, then nothing happens.

The refresh option may be set to the following options:

always (default)

A normal If-Modified-Since check with the remote server is done, skipping the download if the remote server supports this operation and nothing changed since the last download.

never

The resource will fetched only once, and never refreshed.

unconditionally

Download the resource unconditionally, even if there's no change (no If-Modified-Since check will be done).

["digest", $digest_type, $digest]

If the local file already exists, then calculate the digest using $digest_type (e.g. MD5 (default), SHA-1 etc., see Digest for supported algorithms). If this matches $digest, then no download is done. Probably useful to minimize remote calls, or if even a "304 Not Modified" response is slower than the local digest calculation, or if the remote server does not support If-Modified-Since requests.

An example call could look like this:

    my $downloaded = $doit->lwp_mirror(
        "http://httpbingo.org/base64/aHR0cGJpbmdvLm9yZw==",
        $local_file,
        refresh => ['digest', '7f7652b0379b30f50d4daafb05d82c79', 'MD5'],
    );

By default, LWP::UserAgent is used for doing the mirror. It is possible to use HTTP::Tiny by passing an object of this class to the ua option.

Please note that LWP::UserAgent's mirror is probably only atomic with recent LWP (>= 6.61) and running on Unix-like systems. See also https://github.com/libwww-perl/libwww-perl/issues/217 and https://github.com/libwww-perl/libwww-perl/pull/401.

The command returns 1 if the resource was downloaded, otherwise 0.

AUTHOR

Slaven Rezic <srezic@cpan.org>

COPYRIGHT

Copyright (c) 2017,2018,2023 Slaven Rezic. All rights reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

Doit, LWP::UserAgent, HTTP::Tiny.