Custom Load Balancing Policies
Explains how custom load balancing policies can help optimize load balancing under unique circumstances.
Custom Load Balancing Policies
One of the key features of gRPC is load balancing, which allows requests from clients to be distributed across multiple servers. This helps prevent any one server from becoming overloaded and allows the system to scale up by adding more servers.
A gRPC load balancing policy is given a list of server IP addresses by the name resolver. The policy is responsible for maintaining connections (subchannels) to the servers and picking a connection to use when an RPC is sent.
Implementing Your Own Policy
By default the
pick_first policy will be used. This policy actually does no
load balancing but just tries each address it gets from the name resolver and
uses the first one it can connect to. By updating the gRPC service config you
can also switch to using
round_robin that connects to every address it gets
and rotates through the connected backends for each RPC. There are also some
other load balancing policies available, but the exact set varies by language.
If the built-in policies do not meet your needs you can also implement you own
This involves implementing a load balancer interface in the language you are using. At a high level, you will have to:
- Register your implementation in the load balancer registry so that it can be referred to from the service config
- Parse the JSON configuration object of your implementation. This allows your load balancer to be configured in the service config with any arbitrary JSON you choose to support
- Manage what backends to maintain a connection with
- Implement a
pickerthat will choose which backend to connect to when an RPC is made. Note that this needs to be a fast operation as it is on the RPC call path
- To enable your load balancer, configure it in your service config
The exact steps vary by language, see the language support section for some concrete examples in your language.
flowchart TD NR(Name Resolver) -->|Provides addresses &\nLB config| LB(Load Balancer) LB --> |Provides a picker| C(Channel) C -->|Requests\na subchannel| P(Picker) LB --> |Manages subchannels\nto backends| SC(Subchannel 1..n) LB -. Creates .-> P P --> |Picks one| SC
What if your load balancing policy needs to know what is going on with the backend servers in real-time? For this you can rely on backend metrics. You can have metrics provided to you either in-band, in the backend RPC responses, or out-of-band as separate RPCs from the backends. Standard metrics like CPU and memory utilization are provided but you can also implement your own, custom metrics.
For more information on this, please see the custom backend metrics guide (TBD)
If you have a service mesh setup where a central control plane is coordinating the configuration of your microservices, you cannot configure your custom load balancer directly via the service config. But support is provided to do this with the xDS protocol that your control plane uses to communicate with your gRPC clients. Please refer to your control plane documentation to determine how custom load balancing configuration is supported.
For more details, please see gRPC proposal A52.
|Go||Example and xDS support upcoming|
|C++||Not yet supported|