Protobuf and Python

Proto files (.proto) can be compiled to Python code (_pb2.py) using protoc from protobuf repo. E.g.

protoc --proto_path=proto --python_out=. foo.proto

For a simple proto file like

syntax = "proto3";

message Foo {
  string input = 1;
}

The generated Python code looks like

Note, I reformatted the generated code with black for better readability.

The generated code is relatively lightweight because the main mechanism for building the corresponding Python class (class Foo) is still implemented in the python directory of the protobuf repo, using a Python metaclass, instead of in the generated code. The implementation of the key metaclass GeneratedProtocolMessageType can be found in python/google/protobuf/internal/python_message.py.

Note, the protobuf repo implements all of the proto compiler (protoc), the protobuf Python library as well as runtime libraries for about nine other languages.

The python part of protobuf repo is published on PyPi also by the name protobuf.

gRPC and Python

Proto files (.proto) with service definitions can be compiled to Python modules (_pb2.py and _pb2_grpc.py) using grpcio-tools from PyPi.

E.g.

python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. my_service.proto

Note, the implementation grpcio-tools is in the [tools/distrib/python directory] of the grpc repo.

For a simple proto file

syntax = "proto3";

message Foo {
    string input = 1;
}

message Bar {
    uint32 output = 1;
}

service MyService{
    rpc GetBar(Foo) returns (Bar);
}

The generated _pb2.py file is like

Note it contains two _descriptor.Descriptor instances (_FOO and _BAR) and one _descriptor.ServiceDescriptor instance (_MYSERVICE), and two Python classes (Foo and Bar) created via metaclass GeneratedProtocolMessageType, similar to what’s described in the Protobuf and Python section.

The generated _pb2_grpc.py file is like

Note it implements three classes:

  • MyServiceStub, to be used by a gRPC client for sending request to the gRPC server.
  • MyServiceServicer, to be subclassed in the gRPC server code for implementing the real logic of GetBar
  • MyService: an experimental API.

and a function add_MyServiceServicer_to_server, which is used to add the subclass of MyServiceServicer to an grpc._server._Server instance (returned by grpc.server function).

The generated code imports grpc module, which is implemented in the src/python/grpcio directory of the grpc repo.

References