dotnet-cqrs/RABBITMQ-GUIDE.md

# RabbitMQ Cross-Service Event Streaming Guide

**Phase 4 Feature**: Cross-service event streaming via RabbitMQ

## Overview

The Svrnty.CQRS.Events.RabbitMQ package provides automatic cross-service event streaming using RabbitMQ as the message broker. Events published by one service can be consumed by other services with zero RabbitMQ knowledge required from developers.

## Features

- ✅ **Automatic Topology Management** - Exchanges, queues, and bindings created automatically
- ✅ **Connection Resilience** - Automatic reconnection and recovery
- ✅ **Publisher Confirms** - Reliable message delivery with acknowledgments
- ✅ **Consumer Acknowledgments** - Manual or automatic ack/nack support
- ✅ **Dead Letter Queue** - Failed messages automatically routed to DLQ
- ✅ **Message Persistence** - Messages survive broker restarts
- ✅ **Zero Developer Friction** - Just configure streams, framework handles RabbitMQ

## Quick Start

### 1. Install Package

```bash
dotnet add package Svrnty.CQRS.Events.RabbitMQ
```

### 2. Configure RabbitMQ Provider

```csharp
using Svrnty.CQRS.Events.RabbitMQ;

var builder = WebApplication.CreateBuilder(args);

// Register RabbitMQ event delivery
builder.Services.AddRabbitMQEventDelivery(options =>
{
    options.ConnectionString = "amqp://guest:guest@localhost:5672/";
    options.ExchangePrefix = "myapp";  // Optional: prefix for all exchanges
    options.DefaultExchangeType = "topic";
    options.EnablePublisherConfirms = true;
    options.AutoDeclareTopology = true;  // Auto-create exchanges/queues
});

var app = builder.Build();
app.Run();
```

### 3. Publish Events Externally

Events published from workflows are automatically sent to RabbitMQ when configured:

```csharp
// Service A: Publishing Service
public class UserCreatedEvent : ICorrelatedEvent
{
    public string EventId { get; set; } = Guid.NewGuid().ToString();
    public string? CorrelationId { get; set; }
    public int UserId { get; set; }
    public string Email { get; set; } = string.Empty;
    public DateTimeOffset CreatedAt { get; set; }
}

public class CreateUserCommandHandler : ICommandHandlerWithWorkflow<CreateUserCommand, int, UserWorkflow>
{
    public async Task<int> HandleAsync(
        CreateUserCommand command,
        UserWorkflow workflow,
        CancellationToken ct)
    {
        // Create user in database
        var userId = await _repository.CreateUserAsync(command.Email);

        // Emit event - this will be published to RabbitMQ
        workflow.Emit(new UserCreatedEvent
        {
            UserId = userId,
            Email = command.Email,
            CreatedAt = DateTimeOffset.UtcNow
        });

        return userId;
    }
}
```

### 4. Subscribe to External Events

```csharp
// Service B: Consuming Service
using Svrnty.CQRS.Events.Abstractions;

public class UserEventConsumer : BackgroundService
{
    private readonly IExternalEventDeliveryProvider _rabbitMq;
    private readonly ILogger<UserEventConsumer> _logger;

    public UserEventConsumer(
        IExternalEventDeliveryProvider rabbitMq,
        ILogger<UserEventConsumer> logger)
    {
        _rabbitMq = rabbitMq;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await _rabbitMq.SubscribeExternalAsync(
            streamName: "user-events",
            subscriptionId: "email-service",
            consumerId: "worker-1",
            eventHandler: HandleEventAsync,
            cancellationToken: stoppingToken);
    }

    private async Task HandleEventAsync(
        ICorrelatedEvent @event,
        IDictionary<string, string> metadata,
        CancellationToken ct)
    {
        switch (@event)
        {
            case UserCreatedEvent userCreated:
                _logger.LogInformation("Sending welcome email to {Email}", userCreated.Email);
                await SendWelcomeEmailAsync(userCreated.Email, ct);
                break;
        }
    }

    private async Task SendWelcomeEmailAsync(string email, CancellationToken ct)
    {
        // Send email logic
        await Task.Delay(100, ct); // Simulate email sending
    }
}
```

## Configuration Reference

### Connection Settings

```csharp
options.ConnectionString = "amqp://username:password@hostname:port/virtualhost";
// Examples:
// - Local: "amqp://guest:guest@localhost:5672/"
// - Remote: "amqp://user:pass@rabbitmq.example.com:5672/production"
// - SSL: "amqps://user:pass@rabbitmq.example.com:5671/"

options.HeartbeatInterval = TimeSpan.FromSeconds(60);
options.AutoRecovery = true;
options.RecoveryInterval = TimeSpan.FromSeconds(10);
```

### Exchange Configuration

```csharp
options.ExchangePrefix = "myapp";  // Prefix for all exchanges
options.DefaultExchangeType = "topic";  // topic, fanout, direct, headers
options.DurableExchanges = true;  // Survive broker restart
options.AutoDeclareTopology = true;  // Auto-create exchanges
```

### Queue Configuration

```csharp
options.DurableQueues = true;  // Survive broker restart
options.PrefetchCount = 10;  // Number of unacked messages per consumer
options.MessageTTL = TimeSpan.FromDays(7);  // Message expiration (optional)
options.MaxQueueLength = 10000;  // Max queue size (optional)
```

### Routing Configuration

```csharp
options.DefaultRoutingKeyStrategy = "EventType";  // EventType, StreamName, Wildcard
// EventType: Routes by event class name (UserCreatedEvent)
// StreamName: Routes by stream name (user-events)
// Wildcard: Routes to all consumers (#)
```

### Reliability Configuration

```csharp
options.PersistentMessages = true;  // Messages survive broker restart
options.EnablePublisherConfirms = true;  // Wait for broker acknowledgment
options.PublisherConfirmTimeout = TimeSpan.FromSeconds(5);

options.MaxPublishRetries = 3;
options.PublishRetryDelay = TimeSpan.FromSeconds(1);

options.MaxConnectionRetries = 5;
options.ConnectionRetryDelay = TimeSpan.FromSeconds(5);
```

### Dead Letter Queue

```csharp
options.DeadLetterExchange = "dlx.events";  // Dead letter exchange name
// Failed messages are automatically routed to this exchange
```

## Subscription Modes

### Broadcast Mode
Each consumer gets a copy of every event.

```csharp
await rabbitMq.SubscribeExternalAsync(
    streamName: "user-events",
    subscriptionId: "analytics",
    consumerId: "analytics-worker-1",  // Each worker gets own queue
    eventHandler: HandleEventAsync,
    cancellationToken: stoppingToken);
```

**RabbitMQ Topology:**
- Queue: `myapp.analytics.analytics-worker-1` (auto-delete)
- Binding: All events routed to this queue

### Consumer Group Mode
Events load-balanced across multiple consumers.

```csharp
// Consumer 1
await rabbitMq.SubscribeExternalAsync(
    streamName: "user-events",
    subscriptionId: "email-service",
    consumerId: "worker-1",
    eventHandler: HandleEventAsync,
    cancellationToken: stoppingToken);

// Consumer 2
await rabbitMq.SubscribeExternalAsync(
    streamName: "user-events",
    subscriptionId: "email-service",
    consumerId: "worker-2",
    eventHandler: HandleEventAsync,
    cancellationToken: stoppingToken);
```

**RabbitMQ Topology:**
- Queue: `myapp.email-service` (shared by all workers)
- Binding: Events distributed round-robin

## Message Format

Events are serialized to JSON with metadata in message headers:

**Headers:**
- `event-type`: Event class name (e.g., "UserCreatedEvent")
- `event-id`: Unique event identifier
- `correlation-id`: Workflow correlation ID
- `timestamp`: Event occurrence time (ISO 8601)
- `assembly-qualified-name`: Full type name for deserialization

**Body:**
```json
{
  "eventId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "correlationId": "workflow-12345",
  "userId": 42,
  "email": "user@example.com",
  "createdAt": "2025-12-10T10:30:00Z"
}
```

## Topology Naming Conventions

### Exchange Names
Format: `{ExchangePrefix}.{StreamName}`

Examples:
- Stream: `user-events`, Prefix: `myapp` → Exchange: `myapp.user-events`
- Stream: `orders`, Prefix: `` → Exchange: `orders`

### Queue Names

**Broadcast Mode:**
Format: `{ExchangePrefix}.{SubscriptionId}.{ConsumerId}`

Example: `myapp.analytics.worker-1`

**Consumer Group / Exclusive Mode:**
Format: `{ExchangePrefix}.{SubscriptionId}`

Example: `myapp.email-service`

### Routing Keys

Determined by `DefaultRoutingKeyStrategy`:
- **EventType**: `UserCreatedEvent`, `OrderPlacedEvent`
- **StreamName**: `user-events`, `order-events`
- **Wildcard**: `#` (matches all)

## Error Handling

### Automatic Retry

```csharp
private async Task HandleEventAsync(
    ICorrelatedEvent @event,
    IDictionary<string, string> metadata,
    CancellationToken ct)
{
    try
    {
        // Process event
        await ProcessEventAsync(@event);
        // Auto-ACK on success (if using default behavior)
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, "Failed to process event {EventId}", @event.EventId);
        // Auto-NACK with requeue on exception
        throw;
    }
}
```

### Dead Letter Queue

Events that fail after max retries are sent to the dead letter exchange:

```csharp
options.DeadLetterExchange = "dlx.events";
```

Monitor the DLQ for failed messages:
```bash
# List messages in DLQ
rabbitmqadmin get queue=dlx.events count=10
```

## Production Best Practices

### 1. Use Connection Pooling

RabbitMQ provider automatically manages connections. Don't create multiple instances.

```csharp
// Good: Single instance registered in DI
services.AddRabbitMQEventDelivery(connectionString);

// Bad: Don't create multiple instances manually
```

### 2. Configure Prefetch

Balance throughput vs memory usage:

```csharp
options.PrefetchCount = 10;  // Low: Better for heavy processing
options.PrefetchCount = 100; // High: Better for lightweight processing
```

### 3. Enable Publisher Confirms

For critical events, always enable confirms:

```csharp
options.EnablePublisherConfirms = true;
options.PublisherConfirmTimeout = TimeSpan.FromSeconds(5);
```

### 4. Set Message TTL

Prevent queue buildup with old messages:

```csharp
options.MessageTTL = TimeSpan.FromDays(7);
```

### 5. Monitor Queue Lengths

```csharp
options.MaxQueueLength = 100000;  // Prevent unbounded growth
```

### 6. Use Durable Queues and Exchanges

For production:

```csharp
options.DurableExchanges = true;
options.DurableQueues = true;
options.PersistentMessages = true;
```

### 7. Configure Dead Letter Exchange

Always configure DLQ for production:

```csharp
options.DeadLetterExchange = "dlx.events";
```

## Monitoring

### Health Checks

```csharp
var provider = serviceProvider.GetRequiredService<IExternalEventDeliveryProvider>();

if (provider.IsHealthy())
{
    Console.WriteLine($"RabbitMQ is healthy. Active consumers: {provider.GetActiveConsumerCount()}");
}
else
{
    Console.WriteLine("RabbitMQ connection is down!");
}
```

### Metrics to Monitor

1. **Connection Status**: `IsHealthy()`
2. **Active Consumers**: `GetActiveConsumerCount()`
3. **Queue Length**: Monitor via RabbitMQ Management UI
4. **Message Rate**: Publish/Consume rates
5. **Error Rate**: Failed messages / DLQ depth

### RabbitMQ Management UI

Access at `http://localhost:15672` (default credentials: guest/guest)

Monitor:
- Exchanges and their message rates
- Queues and their depths
- Connections and channels
- Consumer status

## Troubleshooting

### Connection Failures

**Symptom:** `Failed to connect to RabbitMQ`

**Solutions:**
1. Check connection string format
2. Verify RabbitMQ is running: `docker ps` or `rabbitmqctl status`
3. Check network connectivity: `telnet localhost 5672`
4. Review firewall rules

### Messages Not Delivered

**Symptom:** Publisher succeeds but consumer doesn't receive messages

**Solutions:**
1. Check exchange exists: `rabbitmqadmin list exchanges`
2. Check queue exists and is bound: `rabbitmqadmin list bindings`
3. Verify routing keys match
4. Check consumer is connected: `rabbitmqadmin list consumers`

### Type Resolution Errors

**Symptom:** `Could not resolve event type`

**Solutions:**
1. Ensure event classes have same namespace in both services
2. Check `assembly-qualified-name` header matches actual type
3. Verify event assemblies are loaded

### High Memory Usage

**Symptom:** Consumer process uses excessive memory

**Solutions:**
1. Lower prefetch count: `options.PrefetchCount = 10;`
2. Add message TTL: `options.MessageTTL = TimeSpan.FromHours(24);`
3. Implement backpressure in event handlers

## Docker Setup

### docker-compose.yml

```yaml
version: '3.8'

services:
  rabbitmq:
    image: rabbitmq:3-management-alpine
    container_name: rabbitmq
    ports:
      - "5672:5672"    # AMQP
      - "15672:15672"  # Management UI
    environment:
      RABBITMQ_DEFAULT_USER: guest
      RABBITMQ_DEFAULT_PASS: guest
    volumes:
      - rabbitmq_data:/var/lib/rabbitmq
    healthcheck:
      test: rabbitmq-diagnostics -q ping
      interval: 10s
      timeout: 5s
      retries: 5

volumes:
  rabbitmq_data:
```

### Start RabbitMQ

```bash
docker-compose up -d rabbitmq
```

### Stop RabbitMQ

```bash
docker-compose down
```

## Example: Cross-Service Communication

See `CROSS-SERVICE-EXAMPLE.md` for a complete example with two microservices communicating via RabbitMQ.

## Advanced Topics

### Custom Routing Keys

```csharp
// Publisher sets custom routing key
var metadata = new Dictionary<string, string>
{
    { "routing-key", "user.created.premium" }
};

await rabbitMq.PublishExternalAsync(
    streamName: "user-events",
    @event: userCreatedEvent,
    metadata: metadata);
```

### Message Priority

RabbitMQ supports message priority (requires queue declaration with priority support):

```csharp
// Set priority in metadata
var metadata = new Dictionary<string, string>
{
    { "priority", "5" }  // 0-9, higher = more important
};
```

### Manual Topology Management

If you prefer to manage topology externally:

```csharp
options.AutoDeclareTopology = false;
```

Then create exchanges and queues manually via RabbitMQ Management UI or CLI.

## Migration Guide

### From Direct RabbitMQ Usage

**Before:**
```csharp
var factory = new ConnectionFactory { Uri = new Uri("amqp://localhost") };
using var connection = await factory.CreateConnectionAsync();
using var channel = await connection.CreateChannelAsync();

await channel.ExchangeDeclareAsync("user-events", "topic", durable: true);
await channel.QueueDeclareAsync("email-service", durable: true);
await channel.QueueBindAsync("email-service", "user-events", "#");

// Complex publish logic...
```

**After:**
```csharp
// Just configuration
services.AddRabbitMQEventDelivery("amqp://localhost");

// Events automatically published
workflow.Emit(new UserCreatedEvent { ... });
```

## Summary

The RabbitMQ integration provides enterprise-grade cross-service event streaming with minimal configuration. The framework handles all RabbitMQ complexity, allowing developers to focus on business logic.

**Key Benefits:**
- Zero RabbitMQ knowledge required
- Production-ready out of the box
- Automatic topology management
- Built-in resilience and reliability
- Comprehensive monitoring and logging

For questions or issues, see the main repository: https://git.openharbor.io/svrnty/dotnet-cqrs