dotnet-cqrs/docs/grpc-integration/grpc-troubleshooting.md

# gRPC Troubleshooting

Common gRPC issues and solutions.

## Connection Issues

### Cannot Connect to Server

**Symptoms:**
```
Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="failed to connect to all addresses")
```

**Common Causes:**

1. **Wrong address/port:**
   ```csharp
   // ❌ Wrong
   var channel = GrpcChannel.ForAddress("http://localhost:5000");

   // ✅ Correct - Check server is listening on this port
   var channel = GrpcChannel.ForAddress("https://localhost:5001");
   ```

2. **HTTP vs HTTPS mismatch:**
   ```csharp
   // Server configured for HTTPS
   app.Urls.Add("https://localhost:5001");

   // Client must use HTTPS
   var channel = GrpcChannel.ForAddress("https://localhost:5001");
   ```

3. **Server not running:**
   ```bash
   # Check if server is running
   netstat -an | grep 5001

   # Or on Windows
   netstat -an | findstr 5001
   ```

### SSL/TLS Certificate Errors

**Symptoms:**
```
The SSL connection could not be established
The remote certificate is invalid according to the validation procedure
```

**Solutions:**

**Development (disable SSL validation):**
```csharp
var handler = new HttpClientHandler
{
    ServerCertificateCustomValidationCallback =
        HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
};

var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
{
    HttpHandler = handler
});
```

**Production (trust certificate):**
```bash
# macOS - Trust certificate
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain cert.crt

# Linux - Add to trusted certificates
sudo cp cert.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# Windows - Import to Trusted Root
certutil -addstore -f "ROOT" cert.crt
```

**Use valid certificate in production:**
```csharp
builder.WebHost.ConfigureKestrel(options =>
{
    options.Listen(IPAddress.Any, 5001, listenOptions =>
    {
        listenOptions.UseHttps("certificate.pfx", "password");
    });
});
```

### HTTP/2 Not Supported

**Symptoms:**
```
The request was aborted. The HTTP/2 connection closed.
```

**Solution - Enable HTTP/2:**
```csharp
// Client
var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
{
    HttpVersion = new Version(2, 0)
});

// Server (Kestrel)
builder.WebHost.ConfigureKestrel(options =>
{
    options.ConfigureEndpointDefaults(listenOptions =>
    {
        listenOptions.Protocols = HttpProtocols.Http2;
    });
});
```

## Code Generation Issues

### Proto Files Not Generating C# Code

**Symptoms:**
- No generated files in `obj/` directory
- Build succeeds but classes not available

**Solutions:**

1. **Verify .csproj configuration:**
   ```xml
   <ItemGroup>
     <Protobuf Include="Protos\cqrs_services.proto" GrpcServices="Server" />
   </ItemGroup>

   <ItemGroup>
     <PackageReference Include="Grpc.AspNetCore" Version="2.68.0" />
     <PackageReference Include="Grpc.Tools" Version="2.76.0">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
     </PackageReference>
   </ItemGroup>
   ```

2. **Clean and rebuild:**
   ```bash
   dotnet clean
   dotnet build
   ```

3. **Check build output:**
   ```bash
   dotnet build -v detailed
   ```
   Look for lines containing "protoc" and "grpc_csharp_plugin"

4. **Verify Grpc.Tools is restored:**
   ```bash
   dotnet restore
   ```

### Source Generator Not Running

**Symptoms:**
- `CommandServiceImpl` and `QueryServiceImpl` not found
- Generator package installed but no output

**Solutions:**

1. **Verify package reference:**
   ```xml
   <PackageReference Include="Svrnty.CQRS.Grpc.Generators" Version="1.0.0" />
   ```

2. **Check generated files location:**
   ```bash
   # View generated files
   ls obj/Debug/net10.0/generated/Svrnty.CQRS.Grpc.Generators/

   # Or on Windows
   dir obj\Debug\net10.0\generated\Svrnty.CQRS.Grpc.Generators\
   ```

3. **Restart IDE:**
   - Close and reopen Visual Studio or Rider
   - Sometimes IDEs cache analyzer/generator state

4. **Force regeneration:**
   ```bash
   dotnet clean
   rm -rf obj bin
   dotnet build
   ```

### Type Mismatch Errors

**Symptoms:**
```
Cannot convert from 'CreateUserCommand' to 'CreateUserCommand'
```

**Cause:** Two types with same name - one from proto, one from C#

**Solution:**

Use the proto-generated type OR ensure your C# type matches proto exactly:

**Option 1 - Use proto-generated type:**
```csharp
// Use the generated type from proto
var command = new CreateUserCommand
{
    Name = "John",
    Email = "john@example.com"
};
```

**Option 2 - Match proto exactly:**
```protobuf
message CreateUserCommand {
  string name = 1;
  string email = 2;
  int32 age = 3;
}
```

```csharp
// C# type must match proto fields exactly
public record CreateUserCommand
{
    public string Name { get; init; } = string.Empty;
    public string Email { get; init; } = string.Empty;
    public int Age { get; init; }
}
```

## Validation Issues

### Validation Errors Not Returned Correctly

**Symptoms:**
- Validation fails but client receives generic error
- Rich Error Model not working

**Solutions:**

1. **Ensure Rich Error Model packages:**
   ```xml
   <PackageReference Include="Grpc.StatusProto" Version="2.71.0" />
   <PackageReference Include="Google.Rpc" Version="2.0.0" />
   ```

2. **Import google/rpc/status.proto:**
   ```protobuf
   import "google/rpc/status.proto";
   import "google/rpc/error_details.proto";
   ```

3. **Client-side error handling:**
   ```csharp
   using Grpc.Core;
   using Google.Rpc;

   try
   {
       var response = await client.CreateUserAsync(command);
   }
   catch (RpcException ex) when (ex.StatusCode == StatusCode.InvalidArgument)
   {
       // Get detailed status
       var status = ex.GetRpcStatus();

       if (status != null)
       {
           var badRequest = status.GetDetail<BadRequest>();

           foreach (var violation in badRequest.FieldViolations)
           {
               Console.WriteLine($"{violation.Field}: {violation.Description}");
           }
       }
   }
   ```

### FluentValidation Not Triggered

**Symptoms:**
- Validators registered but not executing
- Invalid data accepted

**Solutions:**

1. **Verify validator registration:**
   ```csharp
   builder.Services.AddTransient<IValidator<CreateUserCommand>, CreateUserCommandValidator>();
   ```

2. **Check handler registration:**
   ```csharp
   builder.Services.AddCommand<CreateUserCommand, int, CreateUserCommandHandler>();
   ```

3. **Ensure validation logic is correct:**
   ```csharp
   public class CreateUserCommandValidator : AbstractValidator<CreateUserCommand>
   {
       public CreateUserCommandValidator()
       {
           RuleFor(x => x.Name)
               .NotEmpty()
               .WithMessage("Name is required");

           RuleFor(x => x.Email)
               .NotEmpty()
               .EmailAddress()
               .WithMessage("Valid email is required");
       }
   }
   ```

## Performance Issues

### Slow Response Times

**Causes and Solutions:**

1. **Not reusing channels:**
   ```csharp
   // ❌ Bad - Creates new connection per call
   public async Task<int> CreateUser(string name, string email)
   {
       var channel = GrpcChannel.ForAddress("https://localhost:5001");
       var client = new CommandService.CommandServiceClient(channel);
       var response = await client.CreateUserAsync(new CreateUserCommand { Name = name, Email = email });
       await channel.ShutdownAsync();
       return response.UserId;
   }

   // ✅ Good - Reuse channel
   private readonly GrpcChannel _channel = GrpcChannel.ForAddress("https://localhost:5001");

   public async Task<int> CreateUser(string name, string email)
   {
       var client = new CommandService.CommandServiceClient(_channel);
       var response = await client.CreateUserAsync(new CreateUserCommand { Name = name, Email = email });
       return response.UserId;
   }
   ```

2. **Enable connection pooling:**
   ```csharp
   builder.Services.AddGrpcClient<CommandService.CommandServiceClient>(options =>
   {
       options.Address = new Uri("https://localhost:5001");
   })
   .ConfigurePrimaryHttpMessageHandler(() =>
   {
       return new SocketsHttpHandler
       {
           PooledConnectionIdleTimeout = Timeout.InfiniteTimeSpan,
           KeepAlivePingDelay = TimeSpan.FromSeconds(60),
           KeepAlivePingTimeout = TimeSpan.FromSeconds(30),
           EnableMultipleHttp2Connections = true
       };
   });
   ```

3. **Use streaming for large datasets:**
   ```csharp
   // Instead of single large response
   rpc GetAllUsers (GetAllUsersQuery) returns (UserListResponse);

   // Use server streaming
   rpc StreamUsers (StreamUsersQuery) returns (stream UserDto);
   ```

### High Memory Usage

**Solutions:**

1. **Limit message size:**
   ```csharp
   builder.Services.AddGrpc(options =>
   {
       options.MaxReceiveMessageSize = 4 * 1024 * 1024;  // 4 MB
       options.MaxSendMessageSize = 4 * 1024 * 1024;     // 4 MB
   });
   ```

2. **Use pagination:**
   ```protobuf
   message ListUsersQuery {
     int32 page = 1;
     int32 page_size = 2;
   }

   message UserListResponse {
     repeated UserDto users = 1;
     int32 total_count = 2;
   }
   ```

3. **Stream large responses:**
   ```csharp
   public override async Task StreamUsers(
       StreamUsersQuery request,
       IServerStreamWriter<UserDto> responseStream,
       ServerCallContext context)
   {
       var users = await _repository.GetUsersAsync();

       foreach (var user in users)
       {
           await responseStream.WriteAsync(user);
       }
   }
   ```

## Deadline Exceeded Errors

**Symptoms:**
```
Grpc.Core.RpcException: Status(StatusCode="DeadlineExceeded", Detail="Deadline Exceeded")
```

**Solutions:**

1. **Increase client deadline:**
   ```csharp
   var deadline = DateTime.UtcNow.AddSeconds(30);

   var response = await client.CreateUserAsync(
       new CreateUserCommand { ... },
       deadline: deadline);
   ```

2. **Set default deadline:**
   ```csharp
   var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
   {
       MaxReceiveMessageSize = null,
       MaxSendMessageSize = null,
       ServiceConfig = new ServiceConfig
       {
           MethodConfigs =
           {
               new MethodConfig
               {
                   Names = { MethodName.Default },
                   Timeout = TimeSpan.FromSeconds(30)
               }
           }
       }
   });
   ```

3. **Server-side - respect cancellation:**
   ```csharp
   public override async Task<CreateUserResponse> CreateUser(
       CreateUserCommand request,
       ServerCallContext context)
   {
       // Check cancellation token
       context.CancellationToken.ThrowIfCancellationRequested();

       // Pass to handler
       var userId = await _handler.HandleAsync(request, context.CancellationToken);

       return new CreateUserResponse { UserId = userId };
   }
   ```

## Metadata/Headers Issues

### Headers Not Received

**Problem:**
```csharp
// Client sends header
var metadata = new Metadata { { "Authorization", "Bearer token" } };
await client.CreateUserAsync(request, headers: metadata);

// Server doesn't see it
var authHeader = context.RequestHeaders.GetValue("Authorization");  // null
```

**Solution:**

Check header name matches exactly (case-sensitive in some implementations):

```csharp
// Client
var metadata = new Metadata
{
    { "authorization", "Bearer token" }  // lowercase
};

// Server
var authHeader = context.RequestHeaders.GetValue("authorization");
```

### CORS Issues with Metadata

**Problem:**
gRPC-Web calls fail with CORS errors when sending custom headers.

**Solution:**

```csharp
builder.Services.AddCors(options =>
{
    options.AddPolicy("AllowGrpcWeb", policy =>
    {
        policy.WithOrigins("http://localhost:3000")
            .AllowAnyHeader()
            .AllowAnyMethod()
            .WithExposedHeaders(
                "Grpc-Status",
                "Grpc-Message",
                "Grpc-Encoding",
                "Grpc-Accept-Encoding",
                "Authorization",        // Your custom headers
                "X-Request-ID");
    });
});

app.UseGrpcWeb();
app.UseCors("AllowGrpcWeb");
```

## Reflection Issues

### Reflection Not Working

**Symptoms:**
```bash
grpcurl -plaintext localhost:5001 list
# Error: server does not support the reflection API
```

**Solutions:**

1. **Install package:**
   ```bash
   dotnet add package Grpc.AspNetCore.Server.Reflection
   ```

2. **Register and map service:**
   ```csharp
   builder.Services.AddGrpcReflection();

   app.MapGrpcReflectionService();
   ```

3. **Verify service is mapped:**
   ```bash
   grpcurl -plaintext localhost:5001 list

   # Should show:
   # grpc.reflection.v1alpha.ServerReflection
   # myapp.CommandService
   # myapp.QueryService
   ```

## Common Client Errors

### Connection Refused (Browser)

**Problem:**
gRPC doesn't work directly in browsers - need gRPC-Web.

**Solution:**

1. **Server - Enable gRPC-Web:**
   ```csharp
   builder.Services.AddGrpc();

   app.UseGrpcWeb();
   app.MapGrpcService<CommandServiceImpl>().EnableGrpcWeb();
   ```

2. **Client - Use grpc-web:**
   ```bash
   npm install grpc-web
   npm install google-protobuf
   ```

3. **Generate grpc-web code:**
   ```bash
   protoc -I=. cqrs_services.proto \
     --js_out=import_style=commonjs:. \
     --grpc-web_out=import_style=typescript,mode=grpcwebtext:.
   ```

### Unimplemented Method

**Symptoms:**
```
Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Method is unimplemented")
```

**Causes:**

1. **Service not mapped:**
   ```csharp
   // Missing
   app.MapGrpcService<CommandServiceImpl>();
   ```

2. **Method not implemented:**
   ```csharp
   // Handler not registered
   builder.Services.AddCommand<CreateUserCommand, int, CreateUserCommandHandler>();
   ```

3. **Proto method name mismatch:**
   ```protobuf
   // Proto
   rpc CreateUser (CreateUserCommand) returns (CreateUserResponse);

   // Client must match exactly
   await client.CreateUserAsync(...);  // ✅
   await client.CreateUSERAsync(...);  // ❌ Wrong case
   ```

## Debugging Tips

### Enable Detailed Logging

```csharp
builder.Logging.AddFilter("Grpc", LogLevel.Debug);
builder.Logging.AddFilter("Microsoft.AspNetCore.Server.Kestrel", LogLevel.Debug);
```

### Inspect Network Traffic

**Using Wireshark:**
1. Install Wireshark
2. Capture on loopback interface
3. Filter: `tcp.port == 5001`
4. Follow TCP stream to see gRPC frames

**Using grpcurl:**
```bash
# Verbose output
grpcurl -v -plaintext -d '{"name": "John"}' localhost:5001 myapp.CommandService/CreateUser

# Test connectivity
grpcurl -plaintext localhost:5001 list

# Describe service
grpcurl -plaintext localhost:5001 describe myapp.CommandService
```

### Test with BloomRPC

1. Download BloomRPC
2. Import .proto file
3. Set server address
4. Test requests with GUI

## Production Checklist

Before deploying gRPC to production:

- [ ] Use HTTPS with valid certificates
- [ ] Configure authentication (JWT, mTLS, etc.)
- [ ] Set appropriate deadlines
- [ ] Implement health checks
- [ ] Configure logging
- [ ] Set message size limits
- [ ] Enable connection pooling
- [ ] Disable reflection (or require auth)
- [ ] Configure CORS for gRPC-Web
- [ ] Monitor error rates
- [ ] Set up load balancing
- [ ] Test failover scenarios

## See Also

- [gRPC Integration Overview](README.md)
- [Getting Started](getting-started-grpc.md)
- [gRPC Clients](grpc-clients.md)
- [Service Implementation](service-implementation.md)
- [gRPC .NET Troubleshooting](https://learn.microsoft.com/en-us/aspnet/core/grpc/troubleshoot)
- [gRPC Status Codes](https://grpc.io/docs/guides/status-codes/)