Fix: gRPC Error Not Working — Status Codes, Connection Failed, or Deadline Exceeded

The Problem

A gRPC client throws UNAVAILABLE when connecting to a server:

Error: 14 UNAVAILABLE: Connection refused
Error: 14 UNAVAILABLE: failed to connect to all addresses

Or a call fails with DEADLINE_EXCEEDED even though the server is running:

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded

Or the server returns UNIMPLEMENTED:

Error: 12 UNIMPLEMENTED: Method not found

Or TLS setup fails in production:

Error: 14 UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection failure

Why This Happens

gRPC uses HTTP/2 and has its own status code system separate from HTTP. Common failure causes:

• Wrong port or address — gRPC servers default to port 50051 (convention, not a standard). If the client targets the wrong address, it gets UNAVAILABLE.
• TLS mismatch — using a TLS client against a plaintext server (or vice versa) causes connection failures. InsecureChannel vs SslChannel must match the server.
• No deadline set — without a deadline, gRPC calls can hang indefinitely if the server is slow or stuck. This also triggers the default DEADLINE_EXCEEDED if the framework sets one.
• Proto mismatch — if client and server use different versions of the .proto file, the server doesn’t recognize the method name and returns UNIMPLEMENTED.
• Load balancer not gRPC-aware — standard HTTP/1.1 load balancers don’t understand gRPC’s HTTP/2 multiplexing. All requests go to one backend, and the connection drops after the LB’s HTTP timeout.

The deeper reason gRPC errors confuse newcomers is that the framework spans three independent layers — HTTP/2 transport, gRPC framing, and your application status — and each layer can emit UNAVAILABLE or INTERNAL from a different failure mode. A TCP RST from a misconfigured firewall, an HTTP/2 GOAWAY from a server shutting down, and a status.Error(codes.Unavailable, ...) returned by your handler all surface as code 14 UNAVAILABLE to the client. The client cannot tell them apart without server-side logs. Always pair gRPC status codes with structured logs that capture peer, method, and HTTP/2 stream state at both ends.

The second source of confusion is that gRPC propagates errors via metadata (the grpc-status and grpc-message HTTP/2 trailers) rather than the HTTP response body. Any proxy or load balancer in the path must understand HTTP/2 trailers; otherwise the trailers are stripped and the client sees INTERNAL: Trailers missing even though the server returned a clean NOT_FOUND. This is the most common cause of intermittent gRPC failures behind misconfigured ingress.

How Other Tools Handle This

gRPC’s error model differs sharply from REST, GraphQL, JSON-RPC, and SOAP — knowing the differences saves hours of debugging.

gRPC status codes vs HTTP status codes. gRPC defines 17 canonical status codes (numbered 0 through 16) carried in HTTP/2 trailers, intentionally decoupled from HTTP’s 1xx-5xx range. The HTTP response is always 200 OK if the gRPC stream succeeds at the transport layer — even when the application returns NOT_FOUND. This is the opposite of REST, where 404 Not Found is the HTTP status itself. The benefit is that gRPC errors survive cleanly across HTTP/2 proxies that would otherwise rewrite status codes; the cost is that L7 firewalls, CDNs, and HTTP logs cannot distinguish a successful call from a PERMISSION_DENIED without parsing the trailers.

gRPC errors vs GraphQL errors. GraphQL responds with HTTP 200 and embeds errors in the errors array of the JSON body, with no canonical taxonomy — you invent your own error codes per schema. gRPC’s codes.NotFound is universally meaningful; GraphQL’s extensions.code is project-specific. gRPC also has one error per call, while GraphQL can return partial data with multiple errors attached to specific field paths. Migrating from GraphQL to gRPC usually means choosing a canonical mapping (your INPUT_VALIDATION_FAILED becomes INVALID_ARGUMENT) and accepting that gRPC cannot represent “this field failed but the rest of the response is valid.”

gRPC vs JSON-RPC vs SOAP faults. JSON-RPC 2.0 returns { "error": { "code": -32601, "message": "Method not found" } } with a small set of reserved codes (-32700 to -32603) for protocol-level errors and an open range for application errors. SOAP uses XML <soap:Fault> elements with faultcode, faultstring, and a <detail> element for typed exceptions. gRPC sits between the two: a small canonical set (like JSON-RPC) plus rich typed details via google.rpc.Status and google.rpc.ErrorInfo (like SOAP’s <detail>). The gRPC Status message can carry retry hints (RetryInfo), quota failure details (QuotaFailure), and request validation problems (BadRequest.FieldViolation) without the verbosity of SOAP.

Error metadata: trailers vs headers vs body. gRPC uses HTTP/2 trailers for grpc-status, grpc-message, and grpc-status-details-bin. Anything you want to attach to an error (an internal request ID, retry-after seconds) goes in the trailing metadata or the binary Status.details. Compared to REST, where you choose between Retry-After headers, body fields, or both, gRPC’s structure is more disciplined but requires every proxy in the path to forward trailers correctly. Envoy and modern gRPC-aware nginx do; older HAProxy versions and AWS Classic Load Balancer do not.

Deadline semantics. gRPC deadlines are absolute timestamps propagated through metadata across service hops. A client setting a 5-second deadline causes every downstream service to inherit that 5-second budget; a downstream service spending 4 seconds leaves only 1 second for further calls. This is fundamentally different from HTTP’s Timeout header (rarely respected) or REST’s per-hop timeouts. The trade-off is that a single misconfigured deadline at the edge cascades into DEADLINE_EXCEEDED across the entire call graph, which can look like a backend failure when it’s really an aggressive client. Tools like OpenTelemetry’s gRPC instrumentation expose the inherited deadline at every hop and make this debuggable.

In Production: Incident Lens

Production gRPC outages almost never start with the gRPC code itself — they start with the path between client and server. The most common incident is UNAVAILABLE: name resolution failed after a Kubernetes service is renamed or its ClusterIP is recycled. The client’s resolver cached the old IP, and every call returns UNAVAILABLE until the connection’s keepalive eventually triggers re-resolution. Setting grpc.WithDefaultServiceConfig with "healthCheckConfig": {"serviceName": ""} plus a short MAX_CONNECTION_AGE on the server forces periodic re-resolution and recovers automatically.

The second recurring incident is silent DEADLINE_EXCEEDED cascades during regional failover. Your edge service has a 2-second deadline. A downstream service in the failed-over region is cold-starting and takes 3 seconds for the first request. The edge cancels at 2 seconds, the downstream completes at 3 seconds with no client to send the response to, and your dashboard shows 0% error rate on the downstream service and 100% DEADLINE_EXCEEDED on the edge. Propagating context cancellation correctly (ctx.Done() in Go, awaiting cancellation in Node) and emitting a grpc.method + grpc.status_code histogram split by deadline budget makes the cause visible at first glance.

The third incident pattern is connection storms after a deployment. Pods rolling restart, every client’s keepalive ping fails simultaneously, every client opens a new connection at the same moment, and the new pods are overwhelmed before they finish health-check warmup. Configure MAX_CONNECTION_AGE_GRACE to stagger pod-side connection terminations, and configure clients with WithBackoffConfig to jitter reconnects across a 1-10 second window instead of reconnecting immediately.

Fix 1: Verify Connection and Credentials

Check the most basic issues first — address and TLS:

// Node.js — @grpc/grpc-js
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');

const packageDefinition = protoLoader.loadSync('service.proto');
const proto = grpc.loadPackageDefinition(packageDefinition);

// WRONG — using secure credentials against a plaintext server
const client = new proto.MyService(
  'localhost:50051',
  grpc.credentials.createSsl()  // Server not using TLS → UNAVAILABLE
);

// CORRECT — plaintext for development
const client = new proto.MyService(
  'localhost:50051',
  grpc.credentials.createInsecure()
);

// CORRECT — TLS for production
const client = new proto.MyService(
  'api.example.com:443',
  grpc.credentials.createSsl()  // Uses system root CAs
);

// CORRECT — TLS with custom certificate
const rootCert = fs.readFileSync('ca.crt');
const client = new proto.MyService(
  'api.example.com:443',
  grpc.credentials.createSsl(rootCert)
);

Go client:

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/credentials/insecure"
)

// Plaintext (development)
conn, err := grpc.NewClient("localhost:50051",
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

// TLS (production)
creds, err := credentials.NewClientTLSFromFile("ca.crt", "")
conn, err := grpc.NewClient("api.example.com:443",
    grpc.WithTransportCredentials(creds),
)

Fix 2: Set Deadlines on Every Call

Always set a deadline (timeout) on gRPC calls:

// Node.js — set deadline as a Date object
const deadline = new Date();
deadline.setSeconds(deadline.getSeconds() + 10);  // 10 second timeout

client.GetUser({ id: 'user-123' }, { deadline }, (err, response) => {
  if (err) {
    if (err.code === grpc.status.DEADLINE_EXCEEDED) {
      console.error('Request timed out');
    } else {
      console.error('gRPC error:', err.code, err.message);
    }
    return;
  }
  console.log(response);
});

// With async/await using util.promisify
const { promisify } = require('util');
const getUser = promisify(client.GetUser.bind(client));

try {
  const deadline = new Date(Date.now() + 10000);  // 10 seconds from now
  const user = await getUser({ id: 'user-123' }, { deadline });
} catch (err) {
  if (err.code === grpc.status.DEADLINE_EXCEEDED) {
    // Handle timeout
  }
}

Go — use context with timeout:

import (
    "context"
    "time"
)

// Set timeout via context
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

response, err := client.GetUser(ctx, &pb.GetUserRequest{Id: "user-123"})
if err != nil {
    st, ok := status.FromError(err)
    if ok && st.Code() == codes.DeadlineExceeded {
        log.Println("Request timed out")
    }
    return nil, err
}

Python:

import grpc

channel = grpc.insecure_channel('localhost:50051')
stub = pb2_grpc.MyServiceStub(channel)

try:
    # timeout in seconds
    response = stub.GetUser(pb2.GetUserRequest(id='user-123'), timeout=10)
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
        print("Request timed out")
    elif e.code() == grpc.StatusCode.UNAVAILABLE:
        print("Server unavailable:", e.details())
    else:
        print(f"gRPC error {e.code()}: {e.details()}")

Fix 3: Return Proper Status Codes from the Server

Servers should return meaningful gRPC status codes instead of generic errors:

// Go server — return proper status codes
import (
    "google.golang.org/grpc/codes"
    "google.golang.org/grpc/status"
)

func (s *server) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
    user, err := s.db.GetUser(ctx, req.Id)

    if err == sql.ErrNoRows {
        // Resource not found
        return nil, status.Errorf(codes.NotFound, "user %s not found", req.Id)
    }
    if err != nil {
        // Internal server error
        return nil, status.Errorf(codes.Internal, "failed to get user: %v", err)
    }

    if req.Id == "" {
        // Invalid argument
        return nil, status.Error(codes.InvalidArgument, "user ID is required")
    }

    // Check auth from context
    if !isAuthorized(ctx) {
        return nil, status.Error(codes.PermissionDenied, "not authorized")
    }

    return userToProto(user), nil
}

gRPC status code reference:

Fix 4: Fix UNIMPLEMENTED Errors

UNIMPLEMENTED means the server doesn’t recognize the method. Common causes:

# 1. Check proto files are in sync
# Client and server must use the same .proto file version
# Regenerate stubs from the latest .proto:

# Node.js
npx grpc_tools_node_protoc \
  --js_out=import_style=commonjs,binary:. \
  --grpc_out=grpc_js:. \
  service.proto

# Go
protoc --go_out=. --go-grpc_out=. service.proto

# Python
python -m grpc_tools.protoc \
  -I. \
  --python_out=. \
  --grpc_python_out=. \
  service.proto

// Go server — ensure all methods are implemented
// If proto defines GetUser and ListUsers, you MUST implement both
// or embed the Unimplemented stub

type server struct {
    pb.UnimplementedMyServiceServer  // ← This provides default "UNIMPLEMENTED" responses
}

// Only implement what you need — unimplemented methods return UNIMPLEMENTED automatically
func (s *server) GetUser(ctx context.Context, req *pb.GetUserRequest) (*pb.User, error) {
    // ... implementation
}
// ListUsers will return UNIMPLEMENTED (from the embedded struct)

// Node.js server — implement ALL methods from the proto
const server = new grpc.Server();

server.addService(proto.MyService.service, {
  // Each method in the proto must have a matching key here
  getUser: (call, callback) => { ... },
  listUsers: (call, callback) => { ... },  // Don't miss any
  createUser: (call, callback) => { ... },
});

Fix 5: Use Interceptors for Consistent Error Handling

Interceptors (middleware) centralize error logging and retry logic:

// Node.js client interceptor — retry on UNAVAILABLE
const retryInterceptor = (options, nextCall) => {
  let savedMetadata;
  let savedSendMessage;
  let savedCallback;

  const requester = {
    start(metadata, listener, next) {
      savedMetadata = metadata;
      const newListener = {
        onReceiveMessage(message, next) { next(message); },
        onReceiveStatus(status, next) {
          if (
            status.code === grpc.status.UNAVAILABLE &&
            (options.retries || 0) < 3
          ) {
            // Retry
            const newCall = nextCall({ ...options, retries: (options.retries || 0) + 1 });
            newCall.start(savedMetadata, listener);
            savedSendMessage && newCall.sendMessage(savedSendMessage);
            newCall.halfClose();
          } else {
            next(status);
          }
        },
        onReceiveMetadata(metadata, next) { next(metadata); },
      };
      next(metadata, newListener);
    },
    sendMessage(message, next) {
      savedSendMessage = message;
      next(message);
    },
    halfClose(next) { next(); },
    cancel(message, next) { next(); },
  };

  return new grpc.InterceptingCall(nextCall(options), requester);
};

const client = new proto.MyService(
  'localhost:50051',
  grpc.credentials.createInsecure(),
  { interceptors: [retryInterceptor] }
);

Go server interceptor — log all errors:

import "google.golang.org/grpc"

func loggingInterceptor(
    ctx context.Context,
    req interface{},
    info *grpc.UnaryServerInfo,
    handler grpc.UnaryHandler,
) (interface{}, error) {
    resp, err := handler(ctx, req)
    if err != nil {
        st, _ := status.FromError(err)
        log.Printf("gRPC error: method=%s code=%s message=%s",
            info.FullMethod, st.Code(), st.Message())
    }
    return resp, err
}

server := grpc.NewServer(
    grpc.UnaryInterceptor(loggingInterceptor),
)

Fix 6: Configure gRPC for Load Balancers

Standard HTTP/1.1 load balancers (AWS ALB, nginx without gRPC module) break gRPC:

# nginx — enable gRPC proxying
server {
    listen 443 ssl http2;  # http2 is required for gRPC

    location / {
        grpc_pass grpc://backend:50051;

        # For TLS backend
        # grpc_pass grpcs://backend:50051;
    }
}

upstream backend {
    server backend-1:50051;
    server backend-2:50051;
    server backend-3:50051;
}

AWS ALB — use gRPC as the target group protocol:

# Terraform — ALB target group for gRPC
resource "aws_lb_target_group" "grpc" {
  name        = "grpc-targets"
  port        = 50051
  protocol    = "HTTP"
  target_type = "ip"

  protocol_version = "GRPC"  # ← This enables gRPC health checks and routing

  health_check {
    protocol = "HTTP"
    path     = "/grpc.health.v1.Health/Check"
    matcher  = "0"  # gRPC OK status code
  }
}

Client-side load balancing (no proxy needed):

// Go — use DNS resolver for client-side LB
conn, err := grpc.NewClient(
    "dns:///api.example.com:50051",  // DNS resolves to multiple IPs
    grpc.WithTransportCredentials(creds),
    grpc.WithDefaultServiceConfig(`{"loadBalancingPolicy":"round_robin"}`),
)

Still Not Working?

gRPC reflection for debugging — enable server reflection to inspect the server without having the .proto file. Use grpcurl to test:

# Install grpcurl
brew install grpcurl  # macOS

# List services
grpcurl -plaintext localhost:50051 list

# Describe a service
grpcurl -plaintext localhost:50051 describe MyPackage.MyService

# Call a method
grpcurl -plaintext -d '{"id": "user-123"}' localhost:50051 MyPackage.MyService/GetUser

RST_STREAM errors — the server reset the stream. This often means the server crashed or the connection was terminated by an upstream proxy. Check server logs and any intermediate load balancers for their HTTP timeout settings. gRPC long-running streams often need the LB timeout raised to hours.

Keep-alive settings — in Kubernetes, pods behind a service may have TCP connections closed by the network layer. Configure gRPC keep-alive:

conn, err := grpc.NewClient("...",
    grpc.WithKeepaliveParams(keepalive.ClientParameters{
        Time:                10 * time.Second,
        Timeout:             time.Second,
        PermitWithoutStream: true,
    }),
)

For related API issues, see Fix: GraphQL 400 Bad Request, Fix: axios Network Error, Fix: GraphQL Error Handling Not Working, and Fix: aiohttp Not Working.