NewRelic + Rust = Ackorelic
This post highlights the work we have done to implement the integration of Rust with New Relic.
New Relic is one of the most popular Real User Monitoring(RUM) tool with capabilities ranging from page load timing, session tracking, requests per minute stats, database query timing, external calls timing, etc. We are already using New Relic for instrumenting our frontend and backend, written in Go & Python.
With our Backend APIs written in Rust, it became inevitable to instrument the transactions and check the response times at peak and also the success/failure rates. New relic provides an agent popular languages like Java and Python but nothing of that sort exists for Rust. So, we decided to build an agent for Rust.
New relic provides a C SDK for all languages for which they don’t have official SDKs. Only pre-requisite is that it requires a daemon to be running to send instrumented data to New Relic. We decided to leverage this SDK. Rust has FFI to connect to Newrelic C SDK.
New Relic requires us to start a transaction manually and then using the same transaction handle, create different types of segments. Segments can be of three types- Custom, Datastore and External. They are pretty much self-explanatory. Methods can be instrumented using Custom segments while Datastore calls like Postgres, Redis can be instrumented using Datastore calls and calls to External services (3rd Party APIs) can be instrumented using External segments.
Problems & Solutions
In this section, we try to list the problems and roadblocks we faced during the execution and the solutions we employed to overcome these problems.
1) The Thread Local Way
The problem using the traditional approach is that these transactions and segments need to be started and stopped explicitly by writing code. Also, transaction handle needs to pass through each and every method so essentially we would have modified each method in Rust. But by using this approach we would have ended up writing more code than the business logic itself.
We did a couple of things to solve this.
- We made use of something called Thread Local. For those who don’t know what Thread Local means, it acts as a static or a global memory local to a thread. In short, in a multi-threaded environment each thread will have a dedicated memory which won’t be shared across threads. This Wikipedia link is quite comprehensive:
- Now our API serves many concurrent threads. So, we placed each transaction handle into the thread local memory for each thread and this provided us with the flexibility to globally use the handle through our code. This means no more passing the transaction handle to each and every method.
P.S. – This is just an internal implementation of the library and the user of this library is agnostic of this.
- Rust uses an ORM called Diesel to query Postgres database. Essentially we would have to modify the Diesel framework to ping database queries to New Relic along with the querying database. Which brings us to the second problem.
2) Hidden Diesel Methods
We had to dig down thoroughly into the Diesel framework code to understand how that works and how it can be leveraged to ping New Relic and also serve the queries. Luckily it had provided a public interface called PgConnection to query Postgres. We decided to implement that interface and wrap all query methods with the logic to ping New Relic along with serving the query. The methods where actually magic happened were not documented. We had also taken help from popular forum for Rust Diesel.
3) Custom SQL parser
To realize the full potential of Database queries in New Relic it requires us to inform it about all the details of the query like the database name, table name, the type of query (select, insert or update) etc. For this purpose, we had to write our own custom SQL parser to parse the query and extract relevant fields.
4) Fabric and Supervisor
New relic requires us to start the daemon before starting the application. So, we had to add a daemon script(written in C) to pull the latest daemon from git and build a library out of that and start it before starting our Rust API server. This has to be added to the supervisor to make it production-ready.
This section tries to list future improvements that can be incorporated into this library.
- New Relic has a very strong plugin for Java which embeds itself into the JVM(Java runs itself inside a Virtual memory called the Java Virtual Memory) and produces loads of useful insights like Heap memory usage, garbage memory freeing time etc. This library needs to be elevated to that level by creating traits for the same. There is a trait for calculating heap size. This could be a good starting point.
- Although this is a plug and use library, to make this library work without writing even a single piece of code, appropriate annotations need to be introduced. For example, @NewRelicTransaction should initiate a transaction to New Relic.
- Currently, the database section only shows the operation and table name. The actual parameterized query should be visible in Newrelic.
- New Relic has a capability of Distributed tracing which keeps track of the request in cross applications by embedding some New Relic specific headers to the request. In our case, where we have UI calling Go API which in turn calls Python and Rust to fetch data, this would provide us with meaningful insights about performance. This library can be extended to cover this functionality.
This is the location of New Relic Rust public crate which shows ~2.5K downloads as of Dec 2019.
We encourage the community to try, contribute or fork to our codebase & enhance this crate.