Skip to content

camillebaronnet/php-etl-framework

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

20 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ETL Framwork

Create simple imports with the Extract, Transform, Load pattern.

Latest Stable Version Minimum PHP Version GitHub issues GitHub license

Installation

composer require camillebaronnet/php-etl

Usage

This example extract some Github's repositories, apply some transformations

<?php

use Camillebaronnet\ETL\Etl;
use Camillebaronnet\ETL\Extractor\Http;
use Camillebaronnet\ETL\Loader\DebugLoader;
use Camillebaronnet\ETL\Transformer\DateTime;
use Camillebaronnet\ETL\Transformer\Decode;
use Camillebaronnet\ETL\Transformer\Flatten;
use Camillebaronnet\ETL\Transformer\Map;
use Camillebaronnet\ETL\Transformer\Sleep;

$etl = (new Etl)
    ->extract(Http::class, ['url' => 'https://api.github.com/users/camillebaronnet/repos'])
    ->add(Decode::class)
    ->add(Sleep::class, ['seconds' => .2])
    ->add(Flatten::class, ['glue' => '_'])
    ->add(Map::class, [
        'fields' => [
            'id',
            'name',
            'full_name' => 'fullName',
            'owner_login' => 'ownerLogin',
            'owner_url' => 'ownerUrl',
            'url',
            'ssh_url' => 'sshUrl',
            'created_at' => 'createdAt'
        ]
    ])
    ->add(DateTime::class, [
        'fields' => ['createdAt'],
        'from' => 'Y-m-d\TH:i:s\Z',
        'to' => 'd/m/Y',
    ])
;

$etl->process(DebugLoader::class);

The process explained

  • EXTRACT : Extract can output one or more items

  • TRANFORM : A transform step takes the result of the previous step (extractor or transformer) apply an operation and optionally split the input into several subsets of items (example with Decode).

  • LOADER : A loader can by placed at the end of the pipeline or between transformers. Several Loader can be setting up.

Collection

Extractors

Name Description
HTTP Simple wrapper for the libCurl

Transformers

Name Description
Decode Decode JSON, YAML, XML, CSV and more using Symfony's DecoderInterface
Map Rename, keep and remove some fields
Flatten Flattens a multi-dimensional collection into a single dimension
Trim Strip whitespace from the beginning and end of a string
Sleep Delay execution
DateTime Parse/Convert dates

Loaders

Name Description
Debug Display items in output

Extendable

You can easily create your own custom Extractors, Transformers, Loader or Strategy by implementing the corresponding interface.

Submit yours. Send a pull-request