PHP-ETL - Understand the ETL

Execution Context - Why to have an execution context & what it does

In most of our examples our chain had access to the whole file system. This means having multiple chains running together, or having a list of files each execution has generated is impossible.

Both the 🎵 Symfony Bundle(and therefore the 🦢 Sylius integration) and the Magento2 Module will use contextual chains. This means the “main” operations have only access to a particular directory created for the execution of the chain.

This directory might be locally available on the server or it might be a remote file system. This can be usefull if php-etl is used on a multi server setup for example to share files between the servers.

Additional operations such as the ExternalFileFinderOperation and ExternalFileProcessor will be use to process files that are either on a remote directory (sftp, bucket s3…) or files that are on the local file system. Because operations such as the CsvLoader will not have access to those files unless they are copied into the contextual directory of the current execution.

So both or ExternalFile & our context can be a remote, they could be the same remote, or 2 different remotes.

Let start by a simple example.

Write the result of an API to a CSV File.

For this we will first create a new ContextFactory using PerExecutionContextFactory. This context factory will create unique contexts for each execution. This means a unique directory to run the etl in; and a unique logger.

This is only needed if you are running the etl in 🐘 standalone. With any integration this should be automatically handled for you.

<?php
use Oliverde8\Component\PhpEtl\ChainWorkDirManager;
use Oliverde8\Component\PhpEtl\Factory\NullLoggerFactory;
use Oliverde8\Component\PhpEtl\Factory\LocalFileSystemFactory;
use Oliverde8\Component\PhpEtl\PerExecutionContextFactory;

$workdir = __DIR__ . "/var/";
$dirManager = new ChainWorkDirManager($workdir);
$loggerFactory = new NullLoggerFactory();
$fileFactory = new LocalFileSystemFactory($dirManager);

return new PerExecutionContextFactory(
        $dirManager,
        $fileFactory,
        $loggerFactory
    );

Next, we configure the ETL chain using the new type-safe configuration approach. We create a ChainConfig object and add operation configurations to it. Each operation is configured using typed configuration classes with named parameters for better IDE support and validation.

<?php
use Oliverde8\Component\PhpEtl\ChainConfig;
use Oliverde8\Component\PhpEtl\OperationConfig\Transformer\SimpleHttpConfig;
use Oliverde8\Component\PhpEtl\OperationConfig\Transformer\SplitItemConfig;
use Oliverde8\Component\PhpEtl\OperationConfig\Loader\CsvFileWriterConfig;

$chainConfig = new ChainConfig();
$chainConfig->addLink(new SimpleHttpConfig(
        method: 'GET',
        url: 'https://63b687951907f863aaf90ab1.mockapi.io/test',
        responseIsJson: true
    ))
    ->addLink(new SplitItemConfig(
        keys: ['content'],
        singleElement: true
    ))
    ->addLink(new CsvFileWriterConfig('output.csv'));

Finally, we create the chain processor from our configuration and execute it with the execution context. The execution is identified with objects of type ExecutionInterface set in the options array:

use Oliverde8\Component\PhpEtl\ChainBuilderV2;
use Oliverde8\Component\PhpEtl\Item\DataItem;
use Oliverde8\Component\PhpEtl\Model\PockExecution;

// Assuming $chainBuilder is a configured ChainBuilderV2 instance
$chainProcessor = $chainBuilder->createChain($chainConfig);

$options = [
    'etl' => [
        'execution' => new PockExecution(new DateTime())
    ]
];

$chainProcessor->process(
    new ArrayIterator([new DataItem([])]),
    $options
);

Executing this will create a directory in var/ with the output result. Everytime you execute the chain a new directory wil be created.

Isolating context in sub-chains

The same ExecutionContext instance flows by reference through every operation in a run, including into the sub-chains wrapped by Split, Merge, Repeat, FailSafe, and If. This means a parameter set with $context->setParameter() inside a branch, a repeat iteration, or a retry attempt is visible everywhere else in that run — sibling branches, the main chain after the operation finishes, later items, all of it.

That’s usually what you want, but not always: a retried API call shouldn’t be able to permanently clobber a parameter the rest of the pipeline depends on just because one attempt failed halfway through. For this, ChainSplitConfig, ChainMergeConfig, ChainRepeatConfig, FailSafeConfig, and IfConfig all accept an isolateContext option. When true, the sub-chain runs against its own clone of the context instead of the shared one, so any parameters it sets are discarded once the operation finishes.

Only context parameters are cloned — the underlying file system and logger stay shared, since those are meant to remain common resources rather than be duplicated per branch or per attempt.

See the isolation section on each operation’s page for the specifics: Split, Merge, Repeat, Safe, If.