PHP-ETL - Getting Started

🦢 Sylius the e-commerce framework based on symfony

The sylius bundle

The sylius bundle uses the Symfony bundle (maybe pretty obvious). It therefore allows the usage of the library in symfony. It adds the required “commands” as well as “services” to make the etl easy to use.

Install

Start by installing the symfony bundle

1. Start by installing the necessary dependencies

    composer require oliverde8/php-etl-bundle
   

2. in /config/ create a directory etl

3. Enable bundle:

   \Oliverde8\PhpEtlBundle\Oliverde8PhpEtlBundle::class => ['all' => true],
   

4. Optional You can enable queue’s if you have an interface allowing users to execute etl processes (Easy Admin for example).

   framework:
     messenger:
    routing:
        "Oliverde8\PhpEtlBundle\Message\EtlExecutionMessage": async
   

5. Optional: Enable creation of individual files for each log by editing the monolog.yaml

   etl:
     type: service
     id: Oliverde8\PhpEtlBundle\Services\ChainExecutionLogger
     level: debug
     channels: ["!event"] 
   

Now let’s install the sylius bundle

1. Install the additional dependency

   composer require oliverde8/php-etl-sylius-admin-bundle
   

2. Create EtlExecution table via migrations

3. Import configs

   # config/packages/etl.yaml
   imports:
     - { resource: "@Oliverde8PhpEtlSyliusAdminBundle/Resources/config/config.yaml" }
   

4. Import routes

   # config/routes/etl.yaml
   oliverde8_etl:
     resource: '@Oliverde8PhpEtlSyliusAdminBundle/Resources/config/routing.yaml'
   

5. Optional: Configure EtlExecution Message:

   # config/packages/messenger.yaml
   framework:
    messenger:
        # Uncomment this (and the failed transport below) to send failed messages to this transport for later handling.
        failure_transport: failed
   
        transports:
            failed: 'doctrine://default?queue_name=failed'
            generic_with_retry:
                dsn: 'doctrine://default?queue_name=generic_with_retry'
                retry_strategy:
                    max_retries: 3
                    multiplier: 4
                    delay: 3600000 #1H first retry, 4H second retry, 16H third retry (see multiplier) 
            etl_async:
                dsn: 'doctrine://default?queue_name=etl_async'
                retry_strategy:
                    max_retries: 0
   
        routing:
            'Oliverde8\PhpEtlBundle\Message\EtlExecutionMessage': etl_async
   

Usage

Creating an ETL chain

Each chain is declared in a single file. The name of the chain is the name of the file created in /config/etl/.

Example:

chain:
  "Dummy Step":
    operation: rule-engine-transformer
    options:
      add: true
      columns:
        test:
          rules:
            - get : {field: [0, 'uid']}

Executing a chain

./bin/console etl:execute demo '[["test1"],["test2"]]' '{"opt1": "val1"}'

The first argument is the input, depending on your chain it can be empty. The second are parameters that will be available in the context of each link in the chain.

Get a definition

./bin/console etl:get-definition demo

Get definition graph

./bin/console etl:definition:graph demo

This will return a mermaid graph. Adding a -u will return the url to the mermaid graph image.