PHP-ETL - Getting Started

🎵 Symfony Framework

The Symfony Bundle

The Php etl bundle allows the usage of the library in symfony. It adds the required “commands” as well as “services” to make the etl easy to use.

Install

1. Start by installing the necessary dependencies

    composer require oliverde8/php-etl-bundle
   

2. in /config/ create a directory etl

3. Enable bundle:

   \Oliverde8\PhpEtlBundle\Oliverde8PhpEtlBundle::class => ['all' => true],
   

4. Optional You can enable queue’s if you have an interface allowing users to execute etl processes (Easy Admin for example).

   framework:
     messenger:
    routing:
        "Oliverde8\PhpEtlBundle\Message\EtlExecutionMessage": async
   

5. Optional: Enable creation of individual files for each log by editing the monolog.yaml

   etl:
     type: service
     id: Oliverde8\PhpEtlBundle\Services\ChainExecutionLogger
     level: debug
     channels: ["!event"] 
   

Usage

Creating an ETL chain

Each chain is declared in a single file. The name of the chain is the name of the file created in /config/etl/.

Example:

chain:
  "Dummy Step":
    operation: rule-engine-transformer
    options:
      add: true
      columns:
        test:
          rules:
            - get : {field: [0, 'uid']}

Executing a chain

./bin/console etl:execute demo '[["test1"],["test2"]]' '{"opt1": "val1"}'

The first argument is the input, depending on your chain it can be empty. The second are parameters that will be available in the context of each link in the chain.

Get a definition

./bin/console etl:get-definition demo

Get definition graph

./bin/console etl:definition:graph demo

This will return a mermaid graph. Adding a -u will return the url to the mermaid graph image.

Adding an Easyadmin interface

If you a use easyadmin with your symfony project you can have an admin interface allowing you to monitor & execute etl processes (see enable queue’s for allowing creation of tasks)

1. Install the necessary dependencies

    composer require oliverde8/php-etl-easyadmin-bundle
   

2. Enable the bundle

   \Oliverde8\PhpEtlBundle\Oliverde8PhpEtlEasyAdminBundle::class => ['all' => true],
   

3. Add to easy admin

   yield MenuItem::linktoRoute("Job Dashboard", 'fas fa-chart-bar', "etl_execution_dashboard");
   yield MenuItem::linkToCrud('Etl Executions', 'fas fa-list', EtlExecution::class);
   

4. Enable routes

   etl_bundle:
     resource: '@Oliverde8PhpEtlEasyAdminBundle/Controller'
     type: annotation
     prefix: /admin
   

See the github repository for additional information.

Changing the location of the contextual file system dir

Every PHP-ETL execution is tied to a dedicated directory, which serves as a central location for:

• Input and output files
• Logs and debug artifacts

This design allows each execution to be self-contained, making logs and file traces easy to access and audit—without requiring individual operations to manage paths or storage manually.

By default, PHP-ETL stores these execution directories on the local filesystem in var/etl of the symfony project. But what if:

• You want to store files on remote storage, like Amazon S3 or Google Cloud Storage?
• You need to move or centralize execution data across environments?

PHP-ETL uses Flysystem as its file abstraction layer—wrapped in its own internal abstraction. This allows you to fully control where and how files are stored, using any Flysystem-compatible adapter (S3, SFTP, etc.).

To customize where the execution directory is stored, you can override the default FileSystemFactory.

1. Create a custom implementation of the FileSystemFactoryInterface.

2. Register your service in Symfony, replacing the default implementation:

services:
  Oliverde8\PhpEtlBundle\Services\FileSystemFactoryInterface: '@App\Services\FileSystemFactory'