Google Cloud Endpoints

After a lot of fighting and tricky debugging, I managed to get Google Cloud Endpoints working for some rust gRPC services. In order for others to benefit from it, I wanted to document some parts of the process here.

Overview of Google Cloud Endpoints:

Example Service

Lets start with an already implemented hello world example gRPC service; for this example, the service listens on port 50095.

Protocol Buffers schema - example.proto:

syntax = "proto3";

package example.helloworld;

// The greeting service definition.
service Greeter {
    // Sends a greeting
    rpc SayHello (HelloRequest) returns (HelloReply) {}

// The request message containing the user's name.
message HelloRequest {
    string name = 1;

// The response message containing the greetings
message HelloReply {
    string message = 1;

Google Cloud Endpoints will reflect on your service’s API using a protobuf descriptor file, which you can generate using the protobuf compiler protoc:

protoc --proto_path=. --include_imports --include_source_info --descriptor_set_out=example.pb example.proto

The descriptor set name doesn’t actually matter; I’ll use example.pb in this example.

Endpoint Configuration

The endpoints are configured with a custom yaml file (app_config.yaml). This file specifies the description of the service (i.e. Example gRPC API), the service name where the endpoint is deployed (i.e., the authentication/security rules (ignored in this example), and the protobuf service path.

For the service path (i.e. example.helloworld.Greeter), this needs to exactly match your protobuf file. Take the package name (i.e. example.helloworld) and append the service name (i.e. Greeter).

# The configuration schema is defined by service.proto file
type: google.api.Service
config_version: 3

# Name of the service configuration

# API title to appear in the user interface (Google Cloud Console)
title: Example gRPC API
- name: example.helloworld.Greeter

# API usage restrictions
  # Allow unregistered calls for all methods.
  - selector: "*"
    allow_unregistered_calls: true

With a valid app_config.yaml and example.pb generated, the endpoint can be deployed to GCP:

gcloud endpoints services deploy example.pb api_config.yaml

Successful configuration will look like:

% Operation finished successfully. The following command can describe the Operation details:
% gcloud endpoints operations describe operations/

% Operation finished successfully. The following command can describe the Operation details:
% gcloud endpoints operations describe operations/

% Enabling service on project ea-seed-gcp...

% Operation finished successfully. The following command can describe the Operation details:
% gcloud services operations describe operations/tmo-acf.118fae92-8cc8-48d4-b0d3-fee58a1afac8

% Service Configuration [2019-01-17r0] uploaded for service []

% To manage your API, go to:

GCP Service Account and Secret

The ESP container needs access to the GCP meta data registry, so a service account needs to exist with access to the following roles:

  • Service Controller
  • Cloud Trace Agent

This will produce a .json credentials file that looks something like this:

    "type": "service_account",
    "project_id": "ea-seed-gcp",
    "private_key_id": "REDACTED",
    "private_key": "-----BEGIN PRIVATE KEY-----\nREDACTED\n-----END PRIVATE KEY-----\n",
    "client_email": "",
    "client_id": "108507644160716230735",
    "auth_uri": "",
    "token_uri": "",
    "auth_provider_x509_cert_url": "",
    "client_x509_cert_url": ""

Using this file, you need to create a new secret in Kubernetes:

kubectl create secret generic esp-service-account-creds \

Kubernetes Deployment

The following manifest deploys our example gRPC service (example-service) on that listens on port 50095.

  • gRPC clients can just access the load balancer (via the external IP) on port 80. All traffic (gRPC and JSON/HTTP2) will redirect to the ESP proxy on port 9000
  • The ESP proxy is a modified nginx server, and will properly route gRPC traffic directly to the backend (i.e. example-service on port 50095), or transcode HTTP/1.1 and JSON/REST into HTTP2 and gRPC before routing into the backend.

Features implemented below:

  • Keel auto-deploy
  • Jaeger agent side-car
  • Load balancer across N-replicas
  • Public accessible external IP
  • Google Cloud Endpoints proxy (ESP)
  • Mount service account secret into volume path for ESP

Kubernetes manifest - example-service.yaml:

apiVersion: apps/v1
kind: Deployment
  name: example-service
    name: "example-service" force poll
    # "@every 10m"
      app: example-service
  replicas: 4
        app: example-service
        - name: esp-service-account-creds
            secretName: esp-service-account-creds
      - name: example-service
        - containerPort: 50095
      - name: esp
        args: [
        - containerPort: 9000
          - mountPath: /etc/nginx/creds
            name: esp-service-account-creds
            readOnly: true
      - name: jaeger-agent
        image: jaegertracing/jaeger-agent
        - containerPort: 5775
          protocol: UDP
        - containerPort: 6831
          protocol: UDP
        - containerPort: 6832
          protocol: UDP
        - containerPort: 5778
          protocol: TCP
        - "/go/bin/agent-linux"
        - ""
apiVersion: v1
kind: Service
  name: example-service
    app: example-service
  type: LoadBalancer
  # Port that accepts gRPC and JSON/HTTP2 requests over HTTP.
  - port: 80
    targetPort: 9000
    protocol: TCP
    name: http2
    app: example-service

The manifest can be deployed normally:

$ kubectl apply -f example-service.yaml

deployment.apps "example-service" created
service "example-service" created

If everything deploys correctly, you can confirm the load balancer is running, and it has an external IP (this could take a couple minutes to be created):

$ kubectl get svc example-service

NAME              TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)        AGE
example-service   LoadBalancer   80:30234/TCP   59s

Though, even if the load balancer is up, you also need to confirm that the pods being served by the load balancer are also running successfully:

If you get errors like the above, check the log files to diagnose the reason.

In this example, I clicked on one of the example-service* pods that lists CrashLoopBackOff:

The last container to worry about is the ESP itself. The issue here is our example-service container isn’t deploying correctly. ImagePullBackOff usually signifies that GCP cannot fetch the Docker image from the registry (i.e. It looks like I forgot to deploy the example-service image to, oops!

After uploading the example-service image, things look much better:

Client Connection

With the Kubernetes deployment (load balancer, ESP proxy, and gRPC service) running correctly, you should be able to connect to the load balancer (i.e. on port 80, in this example) with a gRPC client using the same example.proto schema.

extern crate futures;
extern crate futures_cpupool;
extern crate env_logger;
extern crate http;
extern crate log;
extern crate grpc;
extern crate file;

use std::io::{BufReader, Read};
use std::fs::File;
use std::env;

use helloworld_grpc::*;
use helloworld::*;

fn main() {
    let _ = ::env_logger::init();

    let client = GreeterClient::new_plain("", 80, Default::default()).unwrap();

    let mut req = HelloRequest::new();
    let res = client.say_hello(grpc::RequestOptions::new(), req);
    println!("{:?}", res.wait());

Running the client against the load balancer should correctly route through all the layers of madness and give us a sensible and expected result!

$ cargo run --bin client
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/client`
Ok((Metadata { entries: [MetadataEntry { key: MetadataKey { name: "server" }, value: b"nginx" }, MetadataEntry { key: MetadataKey { name: "date" }, value: b"Thurs, 1 Jan 2019 13:55:24 GMT" }, MetadataEntry { key: MetadataKey { name: "content-type" }, value: b"application/grpc" }] }, message: "Zomg, it works!", Metadata { entries: [] }))

Endpoint Developer Portal

All endpoints get automatically added to a developer portal, along with generated documentation based on the protobuf (which can be given proper descriptions using a variety of annotations). - (i.e.

Endpoint StackDriver Logs, Tracing, Monitoring

Using Google Cloud Endpoints, we automatically get extensive and rich logs, tracing, monitoring, profiling, etc… with many inter-connected features in the Google ecosystem (especially StackDriver).

Future Improvements

  • Look into connecting VMs to Google Cloud Endpoints (i.e. NOMAD Windows and macOS servers)
  • HTTP/1.1 JSON/REST transcoding
  • Investigate alternate load balancing schemes for gRPC
    • Custom L7 load balancing might be ideal in many cases (but would be lots of work outside of just using a dumb round robin).
    • gRPC Load Balancing

© 2018. All rights reserved.