writing instrumentation

Warning: The repository is still in the process of migrating to the structure described here.

Any time we want to add OpenTelemetry support for a new Java library, e.g., so usage
of that library has tracing, we must write new instrumentation for that library. Let’s
go over some terms first.

任何时候我们想为一个新的Java库添加OpenTelemetry支持，例如，使该库的使用有追踪功能，我们必须为该库编写新的仪器。让我们先看一下一些术语。

Library instrumentation: This is logic that creates spans and enriches them with data
using library-specific monitoring APIs. For example, when instrumenting an RPC library,
the instrumentation will use some library-specific functionality to listen to events such
as the start and end of a request and will execute code to start and end spans in these
listeners. Many of these libraries will provide interception type APIs such as the gRPC
ClientInterceptor or servlet’s Filter. Others will provide a Java interface whose methods
correspond to a request, and instrumentation can define an implementation which delegates
to the standard, wrapping methods with the logic to manage spans. Users will add code to their
apps that initialize the classes provided by library instrumentation, and the library instrumentation
can be found inside the user’s app itself.

这是创建Span的逻辑，并使用库特定的监控API来丰富它们的数据。例如，当对一个RPC库进行仪器化时，仪器化将使用一些库特有的功能来监听事件，如请求的开始和结束，并在这些监听器中执行代码来开始和结束span。许多这些库将提供拦截类型的API，如gRPC ClientInterceptor或servlet的Filter。还有一些库将提供一个Java接口，其方法与请求相对应，仪器化可以定义一个实现，委托给标准，用管理跨度的逻辑来包装方法。用户会在他们的应用程序中添加代码，初始化库工具化所提供的类，库工具化可以在用户的应用程序本身中找到。

Some libraries will have no way of intercepting requests because they only expose static APIs
and no interception hooks. For these libraries it is not possible to create library
instrumentation.

有些库没有办法拦截请求，因为它们只暴露了静态API，没有拦截钩。对于这些库来说，不可能创建库的工具化。

Java agent instrumentation: This is logic that is similar to library instrumentation, but instead
of a user initializing classes themselves, a Java agent automatically initializes them during
class loading by manipulating byte code. This allows a user to develop their apps without thinking
about instrumentation and get it “for free”. Often, the agent instrumentation will generate
bytecode that is more or less identical to what a user would have written themselves in their app.

这是一种类似于库工具化的逻辑，但不是用户自己初始化类，而是Java代理通过操纵字节码在类加载过程中自动初始化它们。这使得用户在开发他们的应用程序时不需要考虑工具化的问题，并且可以 “免费 “获得工具化。通常情况下，代理工具会产生或多或少与用户在其应用程序中自己编写的字节码相同。

In addition to automatically initializing library instrumentation, agent instrumentation can be used
for libraries where library instrumentation is not possible, such as URLConnection, because it can
intercept even the JDK’s classes. Such libraries will not have library instrumentation but will have
agent instrumentation.

除了自动初始化库工具，代理工具还可以用于不可能有库工具的库，比如URLConnection，因为它甚至可以拦截JDK的类。这样的库不会有库工具化，但会有代理工具化。

Folder Structure

Please also refer to some of our existing instrumentation for examples of our structure, for example,
aws-sdk-2.2.

When writing new instrumentation, create a new subfolder of instrumentation to correspond to the
instrumented library and the oldest version being targeted. Ideally an old version of the library is
targeted in a way that the instrumentation applies to a large range of versions, but this may be
restricted by the interception APIs provided by the library.

Within the subfolder, create three folders library (skip if library instrumentation is not possible),
javaagent, and testing.

For example, if we are targeting an RPC framework yarpc at version 1.0 we would have a tree like

instrumentation ->
    ...
    yarpc-1.0 ->
        javaagent
            yarpc-1.0-javaagent.gradle
        library
            yarpc-1.0-library.gradle
        testing
            yarpc-1.0-testing.gradle

and in the top level settings.gradle

include 'instrumentation:yarpc-1.0:javaagent'
include 'instrumentation:yarpc-1.0:library'
include 'instrumentation:yarpc-1.0:testing'

Writing library instrumentation

Begin by writing the instrumentation for the library in library. This generally involves defining a
Tracer and using the typed tracers in our instrumentation-common library to create and annotate
spans as part of the implementation of an interceptor for the library. The module should generally
only depend on the OpenTelemetry API, instrumentation-common, and the instrumented library itself.
instrumentation-library.gradle needs to be applied to
configure build tooling for the library.

Writing instrumentation tests

Once the instrumentation is completed, we add tests to the testing module. Tests will
generally apply to both library and agent instrumentation, with the only difference being how a client
or server is initialized. In a library test, there will be code calling into the instrumentation API,
while in an agent test, it will generally just use the underlying library’s API as is. Create tests in an
abstract class with an abstract method that returns an instrumented object like a client. The class
should itself extend from InstrumentationSpecification to be recognized by Spock and include helper
methods for assertions.

After writing a test or two, go back to the library package, make sure it has a test dependency on the
testing submodule and add a test that inherits from the abstract test class. You should implement
the method to initialize the client using the library’s mechanism to register interceptors, perhaps
a method like registerInterceptor or wrapping the result of a library factory when delegating. The
test should implement the LibraryTestTrait trait for common setup logic. If the tests pass,
library instrumentation is working OK.

Writing Java agent instrumentation

Now that we have working instrumentation, we can implement agent instrumentation so users of the agent
do not have to modify their apps to use it. Make sure the javaagent submodule has a dependency on the
library submodule and a test dependency on the testing submodule. Agent instrumentation defines
classes to match against to generate bytecode for. You will often match against the class you used
in the test for library instrumentation, for example the builder of a client. And then you could
match against the method that creates the builder, for example its constructor. Agent instrumentation
can inject byte code to be run after the constructor returns, which would invoke e.g.,
registerInterceptor and initialize the instrumentation. Often, the code inside the byte code
decorator will be identical to the one in the test you wrote above - the agent does the work for
initializing the instrumentation library, so a user doesn’t have to.
You can find a detailed explanation of how to implement a javaagent instrumentation
here.

With that written, let’s add tests for the agent instrumentation. We basically want to ensure that
the instrumentation works without the user knowing about the instrumentation. Add a test that extends
the base class you wrote earlier, but in this, create a client using none of the APIs in our project,
only the ones offered by the library. Implement the AgentTestTrait trait for common setup logic,
and try running. All the tests should pass for agent instrumentation too.

Note that all the tests inside the javaagent module will be run using the shaded -javaagent
in order to perform the same bytecode instrumentation as when the agent is run against a normal app.
This means that the javaagent instrumentation will be inside the javaagent (inside of the
AgentClassLoader) and will not be directly accessible to your test code. See the next section in
case you need to write unit tests that directly access the javaagent instrumentation.

Writing Java agent unit tests

As mentioned above, tests in the javaagent module cannot access the javaagent instrumentation
classes directly.

Ideally javaagent instrumentation is just a thin wrapper over library instrumentation, and so there
is no need to write unit tests that directly access the javaagent instrumentation classes.

If you still want to write a unit test against javaagent instrumentation, add another module
named javaagent-unit-tests. Continuing with the example above:

instrumentation ->
    ...
    yarpc-1.0 ->
        javaagent
            yarpc-1.0-javaagent.gradle
        javaagent-unit-tests
            yarpc-1.0-javaagent-unit-tests.gradle
        ...

Various instrumentation gotchas

Instrumenting code that is not available as a maven dependency

If instrumented server or library jar isn’t available from a maven repository you can create a
module with stub classes that define only the methods that you need for writing the integration.
Methods in stub class can just throw new UnsupportedOperationException() these classes are only
used to compile the advice classes and won’t be packaged into agent. During runtime real classes
from instrumented server or library will be used.

Create a module called compile-stub and add compile-stub.gradle with following content

apply from: "$rootDir/gradle/java.gradle"

In javaagent module add compile only dependency with

compileOnly project(':instrumentation:xxx:compile-stub')