Multi-language and other challenges
While developing backup solutions for environments like Oracle Standalone, Oracle RAC, and SAP HANA in the product, we had to deal with some unique difficulties. Most of our infrastructure is written in Golang. We faced programming language issues in situations where the application that we are protecting is written in C/C++ and expected the backup API specification to be implemented as a C/C++ library. We also faced challenges in clustered environments like Oracle RAC and SAP HANA where the application streams data in parallel from multiple nodes but parts of our product required us to create server sessions only from a single node.
In this blog, we explain how gRPC helped us simplify the overall architecture and deal with multi-language and multi-node issues.
Why we did not use CGO
Druva’s Oracle DTC (Direct To Cloud) solution helps customers protect their Oracle standalone as well as clustered environments (RAC) and allows them to stream data directly to the cloud without provisioning any local backup storage. Our solution leverages the Oracle SBT API published by Oracle. Backup Vendors need to implement the SBT API in the form of a library. The SBT API is written in C. Hence, the library needs to be implemented in C/C++ because the Oracle Database Server Processes are implemented in C/C++ and load the SBT library to stream data to and from backup storage. As most of Druva’s infrastructure is written in Golang, the only option was to implement the API in Golang and expose C APIs using CGO. CGO complicated things because marshaling and unmarshalling SBT API input/output data structures became extremely complicated, in some cases impossible to implement.
Using Golang RPC
We decided to explore RPC instead of CGO. RPC is a form of Client-Server Communication method that uses a form of the function call using IDL (Interface Definition Language) as a form of contract on functions. Using RPC, the idea was to implement a pass-through SBT API C layer in the form of a shared library and this shared library would talk to a Golang RPC server which implements actual APIs to stream data to and from cloud backup storage. For RPC we could have written our own implementation on top of raw sockets but that would have been a significant effort. We were looking for a ready-made open-source infrastructure that we can quickly build upon. gRPC came to the rescue.
What is gRPC
gRPC (g Remote Procedure Calls) allows you to communicate with other applications using function calls rather than HTTP calls. This abstracts the network from you, letting you call methods as if they were local code. gRPC uses something known as Protocol Buffers to serialize data between clients and servers. You use a .proto file for defining the services used in your applications. With the help of a gRPC plugin for protoc you can generate code that will give you the methods needed to call a given service, all with native typing in your language of choice.