svrnty/dotnet-cqrs
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity
feature/what-am-i-doing
dotnet-cqrs/docs/event-streaming/projections/projection-options.md

10 KiB
Raw Permalink Blame History

Projection Options

Configure projection behavior with auto-start, batching, and checkpoint intervals.

Overview

Projection options control how projections run and process events:

  • Auto-Start - Start projections automatically on application startup
  • Batch Size - Number of events to process per batch
  • Checkpoint Interval - How often to save checkpoints
  • Error Handling - Retry strategies and dead letter queues

Quick Start

using Svrnty.CQRS.Events;

var builder = WebApplication.CreateBuilder(args);

// Register projections with options
builder.Services.AddDynamicProjections(options =>
{
    options.AutoStart = true;
    options.BatchSize = 100;
    options.CheckpointInterval = TimeSpan.FromSeconds(5);
    options.CatchUpBatchSize = 1000;
    options.MaxDegreeOfParallelism = 1;
});

var app = builder.Build();
app.Run();

Projection Options

public class ProjectionOptions
{
    public bool AutoStart { get; set; }                      // Auto-start on startup
    public int BatchSize { get; set; }                       // Events per batch
    public TimeSpan CheckpointInterval { get; set; }         // Checkpoint frequency
    public int CatchUpBatchSize { get; set; }                // Batch size for catch-up
    public int MaxDegreeOfParallelism { get; set; }          // Parallel projections
    public TimeSpan PollingInterval { get; set; }            // Poll for new events
    public TimeSpan ErrorRetryDelay { get; set; }            // Retry delay
    public int MaxRetryAttempts { get; set; }                // Max retries
    public bool EnableDeadLetterQueue { get; set; }          // DLQ for failures
}

Auto-Start

Automatically start projections on application startup:

// ✅ Auto-start enabled (default)
builder.Services.AddDynamicProjections(options =>
{
    options.AutoStart = true;
});

// ❌ Manual start required
builder.Services.AddDynamicProjections(options =>
{
    options.AutoStart = false;
});

// Manual start
var projectionService = app.Services.GetRequiredService<IProjectionService>();
await projectionService.StartProjectionAsync("order-summary");

Batch Size

Control how many events are processed per batch:

// Small batches - lower latency, more checkpoints
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 10;
});

// Medium batches - balanced (default)
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 100;
});

// Large batches - higher throughput, fewer checkpoints
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 1000;
});

Batch Size Impact

// Small batch: 10 events
// - Checkpoint every 10 events
// - Lower latency (5-10ms)
// - More database writes

// Medium batch: 100 events
// - Checkpoint every 100 events
// - Moderate latency (50-100ms)
// - Balanced performance

// Large batch: 1000 events
// - Checkpoint every 1000 events
// - Higher latency (500ms-1s)
// - Fewer database writes, better throughput

Checkpoint Interval

Control how often checkpoints are saved:

// Frequent checkpoints - every 1 second
builder.Services.AddDynamicProjections(options =>
{
    options.CheckpointInterval = TimeSpan.FromSeconds(1);
});

// Moderate checkpoints - every 5 seconds (default)
builder.Services.AddDynamicProjections(options =>
{
    options.CheckpointInterval = TimeSpan.FromSeconds(5);
});

// Infrequent checkpoints - every 30 seconds
builder.Services.AddDynamicProjections(options =>
{
    options.CheckpointInterval = TimeSpan.FromSeconds(30);
});

Checkpoint Strategies

// Time-based checkpointing
builder.Services.AddDynamicProjections(options =>
{
    options.CheckpointInterval = TimeSpan.FromSeconds(5);  // Every 5 seconds
});

// Batch-based checkpointing
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 100;  // Checkpoint every 100 events
});

// Combined checkpointing (whichever comes first)
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 100;
    options.CheckpointInterval = TimeSpan.FromSeconds(5);
});

Catch-Up Mode

Optimize for rebuilding projections from scratch:

// Standard mode
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 100;  // Standard batch size
});

// Catch-up mode - larger batches for faster rebuild
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 100;           // Real-time batch size
    options.CatchUpBatchSize = 5000;   // Catch-up batch size
});

// Projection detects catch-up automatically
public async Task RunAsync(CancellationToken ct)
{
    var checkpoint = await _checkpointStore.GetCheckpointAsync(ProjectionName);
    var streamHead = await _eventStore.GetStreamHeadAsync("orders");
    var lag = streamHead - checkpoint;

    // Use larger batch if lagging
    var batchSize = lag > 10000 ? _options.CatchUpBatchSize : _options.BatchSize;

    await foreach (var @event in _eventStore.ReadStreamAsync(
        "orders",
        fromOffset: checkpoint + 1,
        batchSize: batchSize,
        cancellationToken: ct))
    {
        await HandleEventAsync(@event, ct);
    }
}

Parallel Projections

Run multiple projections in parallel:

// Sequential projections (default)
builder.Services.AddDynamicProjections(options =>
{
    options.MaxDegreeOfParallelism = 1;
});

// Parallel projections
builder.Services.AddDynamicProjections(options =>
{
    options.MaxDegreeOfParallelism = 4;  // Run 4 projections concurrently
});

// Caution: Only use for independent projections
// Don't parallelize projections that update the same data

Polling Interval

Control how often to check for new events:

// Aggressive polling - every 100ms
builder.Services.AddDynamicProjections(options =>
{
    options.PollingInterval = TimeSpan.FromMilliseconds(100);
});

// Moderate polling - every 1 second (default)
builder.Services.AddDynamicProjections(options =>
{
    options.PollingInterval = TimeSpan.FromSeconds(1);
});

// Relaxed polling - every 5 seconds
builder.Services.AddDynamicProjections(options =>
{
    options.PollingInterval = TimeSpan.FromSeconds(5);
});

Error Handling

Configure retry and dead letter queue:

// Retry with backoff
builder.Services.AddDynamicProjections(options =>
{
    options.MaxRetryAttempts = 5;
    options.ErrorRetryDelay = TimeSpan.FromSeconds(10);
});

// Dead letter queue for permanent failures
builder.Services.AddDynamicProjections(options =>
{
    options.MaxRetryAttempts = 3;
    options.EnableDeadLetterQueue = true;
});

// No retry - fail fast
builder.Services.AddDynamicProjections(options =>
{
    options.MaxRetryAttempts = 0;
});

Per-Projection Configuration

// Global defaults
builder.Services.AddDynamicProjections(options =>
{
    options.BatchSize = 100;
    options.CheckpointInterval = TimeSpan.FromSeconds(5);
});

// Per-projection override
public class OrderSummaryProjection : IDynamicProjection
{
    public string ProjectionName => "order-summary";

    public ProjectionOptions Options => new()
    {
        BatchSize = 1000,  // Override global setting
        CheckpointInterval = TimeSpan.FromSeconds(10)
    };

    public async Task RunAsync(CancellationToken ct)
    {
        // Use per-projection options
    }
}

Configuration Examples

Real-Time Projection

// Optimize for low latency
builder.Services.AddDynamicProjections(options =>
{
    options.AutoStart = true;
    options.BatchSize = 10;                          // Small batches
    options.CheckpointInterval = TimeSpan.FromSeconds(1);  // Frequent checkpoints
    options.PollingInterval = TimeSpan.FromMilliseconds(100);  // Aggressive polling
});

Batch Projection

// Optimize for throughput
builder.Services.AddDynamicProjections(options =>
{
    options.AutoStart = true;
    options.BatchSize = 5000;                        // Large batches
    options.CheckpointInterval = TimeSpan.FromSeconds(30);  // Infrequent checkpoints
    options.PollingInterval = TimeSpan.FromSeconds(5);  // Relaxed polling
    options.CatchUpBatchSize = 10000;                // Very large catch-up batches
});

Resilient Projection

// Optimize for reliability
builder.Services.AddDynamicProjections(options =>
{
    options.AutoStart = true;
    options.BatchSize = 100;
    options.CheckpointInterval = TimeSpan.FromSeconds(5);
    options.MaxRetryAttempts = 10;                   // Many retries
    options.ErrorRetryDelay = TimeSpan.FromSeconds(30);
    options.EnableDeadLetterQueue = true;            // DLQ for failures
});

Monitoring Options

// Log projection configuration
var options = app.Services.GetRequiredService<IOptions<ProjectionOptions>>().Value;

_logger.LogInformation(
    "Projection options: AutoStart={AutoStart}, BatchSize={BatchSize}, CheckpointInterval={Interval}",
    options.AutoStart,
    options.BatchSize,
    options.CheckpointInterval);

// Monitor projection performance
var metrics = new
{
    ProjectionName = "order-summary",
    BatchSize = options.BatchSize,
    CheckpointInterval = options.CheckpointInterval,
    EventsPerSecond = await GetProjectionThroughputAsync("order-summary"),
    Lag = await GetProjectionLagAsync("order-summary")
};

if (metrics.Lag > 10000)
{
    _logger.LogWarning(
        "Projection {Name} lagging: {Lag} events behind",
        metrics.ProjectionName,
        metrics.Lag);

    // Consider increasing batch size
}

Best Practices

✅ DO

  • Use auto-start for production projections
  • Start with default settings and tune based on metrics
  • Use larger batches for catch-up mode
  • Enable DLQ for critical projections
  • Monitor projection lag
  • Use appropriate checkpoint intervals
  • Test with production-like volumes
  • Configure retries for transient failures

❌ DON'T

  • Don't use very small batches (< 10) in production
  • Don't checkpoint too frequently (< 1 second)
  • Don't poll too aggressively (< 100ms)
  • Don't run dependent projections in parallel
  • Don't ignore projection errors
  • Don't use same options for all projections
  • Don't forget to monitor lag
  • Don't disable auto-start in production

See Also