Pipeline

Variant with System.Threading.Channels for concurrent processing and back-pressure.

#When to use it

When steps are slow and it's beneficial to parallelize them with natural back-pressure.

using System.Threading.Channels;

var inputs = Channel.CreateBounded<string>(100);
var processed = Channel.CreateBounded<byte[]>(100);

// Stage 1
_ = Task.Run(async () =>
{
    await foreach (var s in inputs.Reader.ReadAllAsync())
        await processed.Writer.WriteAsync(SHA256.HashData(Encoding.UTF8.GetBytes(s.Trim())));
    processed.Writer.Complete();
});

// Producer
await inputs.Writer.WriteAsync("hola");
inputs.Writer.Complete();

// Consumer
await foreach (var hash in processed.Reader.ReadAllAsync())
    Console.WriteLine(Convert.ToHexString(hash));

#pipeline #channels #async