dotnet-cqrs/docs/grpc-integration/grpc-troubleshooting.md

gRPC Troubleshooting

Common gRPC issues and solutions.

Connection Issues

Cannot Connect to Server

Symptoms:

Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="failed to connect to all addresses")

Common Causes:

1. Wrong address/port:

   // ❌ Wrong
   var channel = GrpcChannel.ForAddress("http://localhost:5000");
   
   // ✅ Correct - Check server is listening on this port
   var channel = GrpcChannel.ForAddress("https://localhost:5001");
   

2. HTTP vs HTTPS mismatch:

   // Server configured for HTTPS
   app.Urls.Add("https://localhost:5001");
   
   // Client must use HTTPS
   var channel = GrpcChannel.ForAddress("https://localhost:5001");
   

3. Server not running:

   # Check if server is running
   netstat -an | grep 5001
   
   # Or on Windows
   netstat -an | findstr 5001
   

SSL/TLS Certificate Errors

Symptoms:

The SSL connection could not be established
The remote certificate is invalid according to the validation procedure

Solutions:

Development (disable SSL validation):

var handler = new HttpClientHandler
{
    ServerCertificateCustomValidationCallback =
        HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
};

var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
{
    HttpHandler = handler
});

Production (trust certificate):

# macOS - Trust certificate
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain cert.crt

# Linux - Add to trusted certificates
sudo cp cert.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# Windows - Import to Trusted Root
certutil -addstore -f "ROOT" cert.crt

Use valid certificate in production:

builder.WebHost.ConfigureKestrel(options =>
{
    options.Listen(IPAddress.Any, 5001, listenOptions =>
    {
        listenOptions.UseHttps("certificate.pfx", "password");
    });
});

HTTP/2 Not Supported

Symptoms:

The request was aborted. The HTTP/2 connection closed.

Solution - Enable HTTP/2:

// Client
var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
{
    HttpVersion = new Version(2, 0)
});

// Server (Kestrel)
builder.WebHost.ConfigureKestrel(options =>
{
    options.ConfigureEndpointDefaults(listenOptions =>
    {
        listenOptions.Protocols = HttpProtocols.Http2;
    });
});

Code Generation Issues

Proto Files Not Generating C# Code

Symptoms:

• No generated files in obj/ directory
• Build succeeds but classes not available

Solutions:

1. Verify .csproj configuration:

   <ItemGroup>
     <Protobuf Include="Protos\cqrs_services.proto" GrpcServices="Server" />
   </ItemGroup>
   
   <ItemGroup>
     <PackageReference Include="Grpc.AspNetCore" Version="2.68.0" />
     <PackageReference Include="Grpc.Tools" Version="2.76.0">
       <PrivateAssets>all</PrivateAssets>
       <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
     </PackageReference>
   </ItemGroup>
   

2. Clean and rebuild:

   dotnet clean
   dotnet build
   

3. Check build output:

   dotnet build -v detailed
   

   Look for lines containing "protoc" and "grpc_csharp_plugin"

4. Verify Grpc.Tools is restored:

   dotnet restore
   

Source Generator Not Running

Symptoms:

• CommandServiceImpl and QueryServiceImpl not found
• Generator package installed but no output

Solutions:

1. Verify package reference:

   <PackageReference Include="Svrnty.CQRS.Grpc.Generators" Version="1.0.0" />
   

2. Check generated files location:

   # View generated files
   ls obj/Debug/net10.0/generated/Svrnty.CQRS.Grpc.Generators/
   
   # Or on Windows
   dir obj\Debug\net10.0\generated\Svrnty.CQRS.Grpc.Generators\
   

3. Restart IDE:

   • Close and reopen Visual Studio or Rider
   • Sometimes IDEs cache analyzer/generator state

4. Force regeneration:

   dotnet clean
   rm -rf obj bin
   dotnet build
   

Type Mismatch Errors

Symptoms:

Cannot convert from 'CreateUserCommand' to 'CreateUserCommand'

Cause: Two types with same name - one from proto, one from C#

Solution:

Use the proto-generated type OR ensure your C# type matches proto exactly:

Option 1 - Use proto-generated type:

// Use the generated type from proto
var command = new CreateUserCommand
{
    Name = "John",
    Email = "john@example.com"
};

Option 2 - Match proto exactly:

message CreateUserCommand {
  string name = 1;
  string email = 2;
  int32 age = 3;
}

// C# type must match proto fields exactly
public record CreateUserCommand
{
    public string Name { get; init; } = string.Empty;
    public string Email { get; init; } = string.Empty;
    public int Age { get; init; }
}

Validation Issues

Validation Errors Not Returned Correctly

Symptoms:

• Validation fails but client receives generic error
• Rich Error Model not working

Solutions:

1. Ensure Rich Error Model packages:

   <PackageReference Include="Grpc.StatusProto" Version="2.71.0" />
   <PackageReference Include="Google.Rpc" Version="2.0.0" />
   

2. Import google/rpc/status.proto:

   import "google/rpc/status.proto";
   import "google/rpc/error_details.proto";
   

3. Client-side error handling:

   using Grpc.Core;
   using Google.Rpc;
   
   try
   {
       var response = await client.CreateUserAsync(command);
   }
   catch (RpcException ex) when (ex.StatusCode == StatusCode.InvalidArgument)
   {
       // Get detailed status
       var status = ex.GetRpcStatus();
   
       if (status != null)
       {
           var badRequest = status.GetDetail<BadRequest>();
   
           foreach (var violation in badRequest.FieldViolations)
           {
               Console.WriteLine($"{violation.Field}: {violation.Description}");
           }
       }
   }
   

FluentValidation Not Triggered

Symptoms:

• Validators registered but not executing
• Invalid data accepted

Solutions:

1. Verify validator registration:

   builder.Services.AddTransient<IValidator<CreateUserCommand>, CreateUserCommandValidator>();
   

2. Check handler registration:

   builder.Services.AddCommand<CreateUserCommand, int, CreateUserCommandHandler>();
   

3. Ensure validation logic is correct:

   public class CreateUserCommandValidator : AbstractValidator<CreateUserCommand>
   {
       public CreateUserCommandValidator()
       {
           RuleFor(x => x.Name)
               .NotEmpty()
               .WithMessage("Name is required");
   
           RuleFor(x => x.Email)
               .NotEmpty()
               .EmailAddress()
               .WithMessage("Valid email is required");
       }
   }
   

Performance Issues

Slow Response Times

Causes and Solutions:

1. Not reusing channels:

   // ❌ Bad - Creates new connection per call
   public async Task<int> CreateUser(string name, string email)
   {
       var channel = GrpcChannel.ForAddress("https://localhost:5001");
       var client = new CommandService.CommandServiceClient(channel);
       var response = await client.CreateUserAsync(new CreateUserCommand { Name = name, Email = email });
       await channel.ShutdownAsync();
       return response.UserId;
   }
   
   // ✅ Good - Reuse channel
   private readonly GrpcChannel _channel = GrpcChannel.ForAddress("https://localhost:5001");
   
   public async Task<int> CreateUser(string name, string email)
   {
       var client = new CommandService.CommandServiceClient(_channel);
       var response = await client.CreateUserAsync(new CreateUserCommand { Name = name, Email = email });
       return response.UserId;
   }
   

2. Enable connection pooling:

   builder.Services.AddGrpcClient<CommandService.CommandServiceClient>(options =>
   {
       options.Address = new Uri("https://localhost:5001");
   })
   .ConfigurePrimaryHttpMessageHandler(() =>
   {
       return new SocketsHttpHandler
       {
           PooledConnectionIdleTimeout = Timeout.InfiniteTimeSpan,
           KeepAlivePingDelay = TimeSpan.FromSeconds(60),
           KeepAlivePingTimeout = TimeSpan.FromSeconds(30),
           EnableMultipleHttp2Connections = true
       };
   });
   

3. Use streaming for large datasets:

   // Instead of single large response
   rpc GetAllUsers (GetAllUsersQuery) returns (UserListResponse);
   
   // Use server streaming
   rpc StreamUsers (StreamUsersQuery) returns (stream UserDto);
   

High Memory Usage

Solutions:

1. Limit message size:

   builder.Services.AddGrpc(options =>
   {
       options.MaxReceiveMessageSize = 4 * 1024 * 1024;  // 4 MB
       options.MaxSendMessageSize = 4 * 1024 * 1024;     // 4 MB
   });
   

2. Use pagination:

   message ListUsersQuery {
     int32 page = 1;
     int32 page_size = 2;
   }
   
   message UserListResponse {
     repeated UserDto users = 1;
     int32 total_count = 2;
   }
   

3. Stream large responses:

   public override async Task StreamUsers(
       StreamUsersQuery request,
       IServerStreamWriter<UserDto> responseStream,
       ServerCallContext context)
   {
       var users = await _repository.GetUsersAsync();
   
       foreach (var user in users)
       {
           await responseStream.WriteAsync(user);
       }
   }
   

Deadline Exceeded Errors

Symptoms:

Grpc.Core.RpcException: Status(StatusCode="DeadlineExceeded", Detail="Deadline Exceeded")

Solutions:

1. Increase client deadline:

   var deadline = DateTime.UtcNow.AddSeconds(30);
   
   var response = await client.CreateUserAsync(
       new CreateUserCommand { ... },
       deadline: deadline);
   

2. Set default deadline:

   var channel = GrpcChannel.ForAddress("https://localhost:5001", new GrpcChannelOptions
   {
       MaxReceiveMessageSize = null,
       MaxSendMessageSize = null,
       ServiceConfig = new ServiceConfig
       {
           MethodConfigs =
           {
               new MethodConfig
               {
                   Names = { MethodName.Default },
                   Timeout = TimeSpan.FromSeconds(30)
               }
           }
       }
   });
   

3. Server-side - respect cancellation:

   public override async Task<CreateUserResponse> CreateUser(
       CreateUserCommand request,
       ServerCallContext context)
   {
       // Check cancellation token
       context.CancellationToken.ThrowIfCancellationRequested();
   
       // Pass to handler
       var userId = await _handler.HandleAsync(request, context.CancellationToken);
   
       return new CreateUserResponse { UserId = userId };
   }
   

Metadata/Headers Issues

Headers Not Received

Problem:

// Client sends header
var metadata = new Metadata { { "Authorization", "Bearer token" } };
await client.CreateUserAsync(request, headers: metadata);

// Server doesn't see it
var authHeader = context.RequestHeaders.GetValue("Authorization");  // null

Solution:

Check header name matches exactly (case-sensitive in some implementations):

// Client
var metadata = new Metadata
{
    { "authorization", "Bearer token" }  // lowercase
};

// Server
var authHeader = context.RequestHeaders.GetValue("authorization");

CORS Issues with Metadata

Problem: gRPC-Web calls fail with CORS errors when sending custom headers.

Solution:

builder.Services.AddCors(options =>
{
    options.AddPolicy("AllowGrpcWeb", policy =>
    {
        policy.WithOrigins("http://localhost:3000")
            .AllowAnyHeader()
            .AllowAnyMethod()
            .WithExposedHeaders(
                "Grpc-Status",
                "Grpc-Message",
                "Grpc-Encoding",
                "Grpc-Accept-Encoding",
                "Authorization",        // Your custom headers
                "X-Request-ID");
    });
});

app.UseGrpcWeb();
app.UseCors("AllowGrpcWeb");

Reflection Issues

Reflection Not Working

Symptoms:

grpcurl -plaintext localhost:5001 list
# Error: server does not support the reflection API

Solutions:

1. Install package:

   dotnet add package Grpc.AspNetCore.Server.Reflection
   

2. Register and map service:

   builder.Services.AddGrpcReflection();
   
   app.MapGrpcReflectionService();
   

3. Verify service is mapped:

   grpcurl -plaintext localhost:5001 list
   
   # Should show:
   # grpc.reflection.v1alpha.ServerReflection
   # myapp.CommandService
   # myapp.QueryService
   

Common Client Errors

Connection Refused (Browser)

Problem: gRPC doesn't work directly in browsers - need gRPC-Web.

Solution:

1. Server - Enable gRPC-Web:

   builder.Services.AddGrpc();
   
   app.UseGrpcWeb();
   app.MapGrpcService<CommandServiceImpl>().EnableGrpcWeb();
   

2. Client - Use grpc-web:

   npm install grpc-web
   npm install google-protobuf
   

3. Generate grpc-web code:

   protoc -I=. cqrs_services.proto \
     --js_out=import_style=commonjs:. \
     --grpc-web_out=import_style=typescript,mode=grpcwebtext:.
   

Unimplemented Method

Symptoms:

Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Method is unimplemented")

Causes:

1. Service not mapped:

   // Missing
   app.MapGrpcService<CommandServiceImpl>();
   

2. Method not implemented:

   // Handler not registered
   builder.Services.AddCommand<CreateUserCommand, int, CreateUserCommandHandler>();
   

3. Proto method name mismatch:

   // Proto
   rpc CreateUser (CreateUserCommand) returns (CreateUserResponse);
   
   // Client must match exactly
   await client.CreateUserAsync(...);  // ✅
   await client.CreateUSERAsync(...);  // ❌ Wrong case
   

Debugging Tips

Enable Detailed Logging

builder.Logging.AddFilter("Grpc", LogLevel.Debug);
builder.Logging.AddFilter("Microsoft.AspNetCore.Server.Kestrel", LogLevel.Debug);

Inspect Network Traffic

Using Wireshark:

1. Install Wireshark
2. Capture on loopback interface
3. Filter: tcp.port == 5001
4. Follow TCP stream to see gRPC frames

Using grpcurl:

# Verbose output
grpcurl -v -plaintext -d '{"name": "John"}' localhost:5001 myapp.CommandService/CreateUser

# Test connectivity
grpcurl -plaintext localhost:5001 list

# Describe service
grpcurl -plaintext localhost:5001 describe myapp.CommandService

Test with BloomRPC

1. Download BloomRPC
2. Import .proto file
3. Set server address
4. Test requests with GUI

Production Checklist

Before deploying gRPC to production:

• Use HTTPS with valid certificates
• Configure authentication (JWT, mTLS, etc.)
• Set appropriate deadlines
• Implement health checks
• Configure logging
• Set message size limits
• Enable connection pooling
• Disable reflection (or require auth)
• Configure CORS for gRPC-Web
• Monitor error rates
• Set up load balancing
• Test failover scenarios

See Also

• gRPC Integration Overview
• Getting Started
• gRPC Clients
• Service Implementation
• gRPC .NET Troubleshooting
• gRPC Status Codes