NAME

   Catmandu - a data toolkit

SYNOPSIS

       # From the command line

       # Convert data from one format to another
       $ catmandu convert JSON to CSV  < data.json
       $ catmandu convert CSV  to YAML < data.csv
       $ catmandu convert MARC to YAML < data.mrc

       # Fix data, add, delete, change fields
       $ catmandu convert JSON --fix 'move_field(title,my_title)' < data.json
       $ catmandu convert JSON --fix all_my_fixes.txt < data.json
       # Use a moustache preprocessor on the fix script
       $ catmandu convert JSON --fix all_my_fixes.txt --var opt1=foo --var opt2=bar < data.json

       # Import data into a database
       # Requires: Catmandu::MongoDB and Catmandu::ElasticSearch
       $ catmandu import YAML to MongoDB --database_name bibliography < data.yml
       $ catmandu import CSV to ElasticSearch --index_name mystuff < data.csv

       # Export data from a database
       # Requires: Catmandu::MongoDB and Catmandu::ElasticSearch
       $ catmandu export MongoDB --database_name bibliography to YAML > data.yml
       $ catmandu export ElasticSearch --index_name mystuff to CSV > data.csv

       # Copy data from one store to another
       $ catmandu copy MongoDB --database_name mydb to ElasticSearch --index_name mydb

       # Show the contents of catmandu.yml
       $ catmandu config

       # Count items in a store
       $ catmandu count test1

       # Delete items from store
       $ catmandu delete test1 --query 'title:"My Rabbit"'

       # run a fix script
       $ catmandu run myfixes.fix

       # or, create an executable fix script
       $ cat myfixes.fix
       #!/usr/local/bin/catmandu run
       do importer(OAI,url:"http://biblio.ugent.be/oai")
           retain(_id)
       end
       $ chmod 755 myfixes.fix
       $ ./myfixes.fix

       # From Perl
       use Catmandu;

       # If you have Catmandu::OAI and Catmandu::MongoDB installed
       my $importer = Catmandu->importer('OAI',url => 'https://biblio.ugent.be/oai')
       my $store    = Catmandu->store('MongoDB',database_name => 'test');

       # Import all the OAI records into MongoDB
       $store->add_many($importer);

       # Export all the MongoDB records to YAML and apply some fixes
       # myfixes.txt:
       #   upcase(title.*)
       #   remove_field(_metadata)
       #   join_field(creator,'; ')
       #   join_field(subject,'-- ')
       my $fixer    = Catmandu->fixer('myfixes.txt');
       my $exporter = Catmandu->exporter('YAML');

       $exporter->add_many( $fixer->fix($store) );
       $exporter->commit;

DESCRIPTION

   Catmandu provides a command line client and a Perl API to ease the
   export (E) transformation (T) and loading (L) of data into databases or
   data file, ETL in short.

   Most of the daily work processing structured data can be done on the
   command line executing the catmandu command. With our catmandu command
   ETL processing is available in a Perl context. Catmandu is different
   from other ETL tools by its focus on command line processing with much
   support for dataformats available in (academic) libraries: MARC, MODS,
   OAI and SRU. But, also generic formats such as JSON, YAML, CVS, Excel,
   XML, RDF, Atom are supported.

   Read :

     * Catmandu::Introduction for a primer on the command line
     capabilities of Catmandu.

     * Catmandu::Importer for the basics of importing

     * Catmandu::Fix for the basics of transformations

     * Catmandu::Exporter for the basics of exporting

     * Catmandu::Store for the basics of storing information

     * Or, visit our website at http://librecat.org/ and our blog
     https://librecatproject.wordpress.com/ for many tutorials

   The documentation below describes the methods available when including
   Catmandu as part of a Perl script. For an overview of the command line
   tool itself read the documentation on catmandu.

USE

   To include Catmandu in a Perl script it should be loaded with a use
   command:

       use Catmandu;

   By default no methods are imported into the Perl context. To import all
   or some Catmandu methods, provide them as a list to the use command:

       use Catmandu -all;
       use Catmandu qw(config store exporter);

   Catmandu can load configuration options for exports, importers, fixers
   via configuration files (see the CONFIG section below). When adding the
   --load option (optionally with a path) to the use command, these
   configuration files will be loaded at the start of your script.

       use Catmandu -load;
       use Catmandu --load => ['/my/config/directory'];

       # or use all the options
       use Catmandu -all -load => [qw(/config/path' '/another/config/path)];

CLASS METHODS

log

   Return the current Log::Any logger.

       use Catmandu;
       use Log::Any::Adapter;
       use Log::Log4perl;

       Log::Any::Adapter->set('Log4perl'); # requires Log::Any::Adapter::Log4perl
       Log::Log4perl::init('./log4perl.conf');

       my $logger = Catmandu->log;
       $logger->info("Starting main program");

   with log4perl.conf like:

       # Send a copy of all logging messages to STDERR
       log4perl.rootLogger=DEBUG,STDERR

       # Logging specific for your main program
       log4perl.category.myprog=INFO,STDERR

       # Logging specific for on part of Catmandu
       log4perl.category.Catmandu::Fix=DEBUG,STDERR

       # Where to send the STDERR output
       log4perl.appender.STDERR=Log::Log4perl::Appender::Screen
       log4perl.appender.STDERR.stderr=1
       log4perl.appender.STDERR.utf8=1

       log4perl.appender.STDERR.layout=PatternLayout
       log4perl.appender.STDERR.layout.ConversionPattern=%d [%P] - %p %l time=%r : %m%n

default_load_path(['/default/path'])

   Returns the default location where Catmandu looks for configuration and
   lib when called with no argument. Sets the default location if a path
   is given. The default load path is the script directory or it's parent
   if the script directory is bin.

load

   Load all the configuration options in the catmandu.yml configuration
   file. See CONFIG below for extended examples of configuration options.

load('/path', '/another/path')

   Load all the configuration options stored at alternative paths.

   A load path ':up' will search upwards from your program for
   configuration.

   See CONFIG below for extended examples of configuration options.

roots

   Returns an ARRAYREF of paths where configuration was found. Note that
   this list is empty before load.

root

   Returns the first path where configuration was found. Note that this is
   undef before load.

config

   Returns the current configuration as a HASHREF.

config($config)

   Set a new configuration and reload the environment.

default_store

   Return the name of the default store.

store([NAME])

   Return an instance of Catmandu::Store. The NAME is a name of a
   Catmandu::Store or the name of a store configured in a catmandu.yml
   configuration file. When no NAME is given, the 'default' store in the
   configuration file will be used.

   E.g. if the configuration file 'catmandu.yml' contains:

    store:
     default:
      package: ElasticSearch
      options:
        index_name: blog
     test:
      package: Mock

   then in your program:

       # This will use ElasticSearch
       my $store = Catmandu->store('ElasticSearch', index_name => 'blog');

       # or because we have a 'default' set in the configuration file

       my $store = Catmandu->store('default');

       # or because 'default' will be used when no name was provided

       my $store = Catmandu->store;

       # This will use Mock
       my $store = Catmandu->store('test');

   Configuration settings can be overwritten by the store command:

     my $store2 = Catmandu->store('default', index_name => 'test2');

default_fixer

   Return the name of the default fixer.

fixer(NAME)

fixer(FIX,FIX)

fixer([FIX])

   Return an instance of Catmandu::Fix. NAME can be the name of a fixer
   section in a catmandu.yml file. Or, one or more Catmandu::Fix-es can be
   provided inline.

   E.g. if the configuration file 'catmandu.yml' contains:

    fixer:
     default:
       - do_this()
       - do_that()

   then in your program al these lines below will create the same fixer:

       my $fixer = Catmandu->fixer('do_this()', 'do_that()');
       my $fixer = Catmandu->fixer(['do_this()', 'do_that()']);
       my $fixer = Catmandu->fixer('default');
       my $fixer = Catmandu->fixer(); # The default name is 'default'

   FIX-es can be also written to a Fix script. E.g. if myfixes.txt
   contains:

    do_this()
    do_that()

   then the above code will even be equivalent to:

      my $fixer = Catmandu->fixer('myfixes.txt');

default_importer

   Return the name of the default importer.

default_importer_package

   Return the name of the default importer package if no package name is
   given in the config or as a param.

importer(NAME)

   Return an instance of Catmandu::Importer. The NAME is a name of a
   Catmandu::Importer or the name of a importer configured in a
   catmandu.yml configuration file. When no NAME is given, the 'default'
   importer in the configuration file will be used.

   E.g. if the configuration file 'catmandu.yml' contains:

     importer:
       default:
         package: OAI
         options:
           url: http://www.instute.org/oai/

   then in your program all these lines will be equivalent:

     my $importer = Catmandu->importer('OAI', url => 'http://www.instute.org/oai/');
     my $importer = Catmandu->importer('default');
     my $importer = Catmandu->importer(); # The default name is 'default'

   Configuration settings can be overwritten by the importer command:

     my $importer2 = Catmandu->importer('default', url => 'http://other.institute.org');

default_exporter

   Return the name of the default exporter.

default_exporter_package

   Return the name of the default exporter package if no package name is
   given in the config or as a param.

exporter([NAME])

   Return an instance of Catmandu::Exporter with name NAME (or the default
   when no name is given). The NAME is set in the configuration file (see
   'importer').

export($data,[NAME])

   Export data using a default or named exporter or exporter instance.

       Catmandu->export({ foo=>'bar'});

       my $importer = Catmandu::Importer::Mock->new;
       Catmandu->export($importer, 'YAML', file => '/my/file');
       Catmandu->export($importer, 'my_exporter');
       Catmandu->export($importer, 'my_exporter', exporter_option => '...' , ...);
       Catmantu->export($importer, Catmandu::Exporter::YAML->new);

export_to_string

   Export data using a default or named exporter to a string.

       my $importer = Catmandu::Importer::Mock->new;
       my $yaml = Catmandu->export_to_string($importer, 'YAML');
       # is the same as
       my $yaml = "";
       Catmandu->export($importer, 'YAML', file => \$yaml);

import_from_string

   Import data from a string using a default or named importer. Return
   value should be an array of hashes.

       my $json = qq([{"name":"Nicolas"}]);
       {
           my $record = Catmandu->import_from_string( $json, "JSON" );
       }
       # is the same as
       {
           my $record = Catmandu->importer('JSON', file => \$json)->to_array()
       }

define_importer

   Configure a new named importer.

       Catmandu->define_importer(books => CSV => (fields => 'title,author,publisher'));
       Catmandu->importer(books => (file => 'mybooks.csv'))->each(sub {
           my $book = shift;
           say $book->{title};
       });

       # this is equivalent to

       Catmandu->config->{importer}{books} = {
           package => 'CSV',
           options => {
               fields => 'title,author,publisher',
           },
       }

define_exporter

   Configure a new named exporter.

       Catmandu->define_exporter('books', 'CSV', fix => 'capitalize(title)');
       my $csv = Catmandu->export_to_string({title => 'nexus'}, 'books');

       # this is equivalent to

       Catmandu->config->{exporter}{books} = {
           package => 'CSV',
           options => {
               fix => 'capitalize(title)',
           },
       }

define_store

   Configure a new named store.

       Catmandu->define_store(mydb => MongoDB => (database_name => 'mydb'));
       Catmandu->store->bag('books')->get(1234);

       # this is equivalent to

       Catmandu->config->{store}{mydb} = {
           package => 'MongoDB',
           options => {
               database_name => 'mydb',
           },
       }

define_fixer

   Configure a new named fixer.

       Catmandu->define_fixer('cleanup', [
           'trim(title)',
           'capitalize(title)',
           'remove_field(junk)',
           # ...
       ]);
       Catmandu->fixer('cleanup')->fix($record);

EXPORTS

   config

     Same as Catmandu->config.

   store

     Same as Catmandu->store.

   importer

     Same as Catmandu->importer.

   exporter

     Same as Catmandu->exporter.

   export

     Same as Catmandu->export.

   export_to_string

     Same as Catmandu->export_to_string.

   import_from_string

     Same as Catmandu->import_from_string.

   fixer

     Same as Catmandu->fixer.

   log

     Same as Catmandu->log.

   -all/:all

     Import everything.

   -load/:load

         use Catmandu -load;
         use Catmandu -load => [];
         # is the same as
         Catmandu->load;

         use Catmandu -load => ['/config/path'];
         # is the same as
         Catmandu->load('/config/path');

CONFIG

   Catmandu configuration options can be stored in files in the root
   directory of your programming project. The file can be YAML, JSON or
   Perl and is called catmandu.yml, catmandu.json or catmandu.pl. In this
   file you can set the default Catmandu stores and exporters to be used.
   Here is an example of a catmandu.yml file:

       store:
         default:
           package: ElasticSearch
           options:
             index_name: myrepository

       exporter:
         default:
           package: YAML

Split config

   For large configs it's more convenient to split the config into several
   files. You can do so by having multiple config files starting with
   catmandu*.

       catmandu.general.yml
       catmandu.db.yml
       ...

   Split config files are processed and merged by Config::Onion.

Deeply nested config structures

   Config files can indicate a path under which their keys will be nested.
   This makes your configuration more readable by keeping indentation to a
   minimum.

   A config file containing

       _prefix:
           foo:
               bar:
       baz: 1

   will be loaded as

       foo:
         bar:
           baz: 1

   See Config::Onion for more information on how this works.

SEE ALSO

   documentation

     http://librecat.org/Catmandu/

   command line client

     catmandu

   core modules

     Catmandu::Importer Catmandu::Exporter, Catmandu::Store,
     Catmandu::Fix, Catmandu::Iterable

   extended features

     Catmandu::Validator

AUTHOR

   Nicolas Steenlant, <nicolas.steenlant at ugent.be>

CONTRIBUTORS

   Magnus Enger, magnus at enger.priv.no

   Nicolas Franck, nicolas.franck at ugent.be

   Patrick Hochstenbach, patrick.hochstenbach at ugent.be

   Vitali Peil, vitali.peil at uni-bielefeld.de

   Christian Pietsch, christian.pietsch at uni-bielefeld.de

   Dave Sherohman, dave.sherohman at ub.lu.se

   Jakob Voss, nichtich at cpan.org

   Snorri Briem, snorri.briem at ub.lu.se

   Johann Rolschewski, jorol at cpan.org

   Pieter De Praetere, pieter.de.praetere at helptux.be

   Doug Bell

   Upsana, me at upasana.me

   Stefan Weil

   Tom Hukins

QUESTIONS, ISSUES & BUG REPORTS

   For any questions on the use of our modules please join our mailing
   list at:

       [email protected]

   or send in your bug reports or feature requests to our issue tracker
   at:

       https://github.com/LibreCat/Catmandu/issues

LICENSE AND COPYRIGHT

   This program is free software; you can redistribute it and/or modify it
   under the terms of either: the GNU General Public License as published
   by the Free Software Foundation; or the Artistic License.

   See http://dev.perl.org/licenses/ for more information.

You are viewing proxied material from ftp.icm.edu.pl. The copyright of proxied material belongs to its original authors. Any comments or complaints in relation to proxied material should be directed to the original authors of the content concerned. Please see the disclaimer for more details.