Perl SIG: current level of interest?

[tedsuo]

Hello OTel community. I've had one or two conversations with organizations interested in using a Perl implementation of OpenTelemetry. This issue is to gauge interest in maintaining a Perl implementation.

Since newcomers may be reading this issue, here's brief overview of the components that an OpenTelemetry client is composed of:

• API: the public interfaces used by developers to instrument their code. At runtime, the API can be bound to an implementation. By default, the API provides a no-op implementation. The SDK and a mock/test implementation are the two other common implementations.
• SDK: a official implementation, which follows the SDK spec. The SDK is a framework which consists of plugin interfaces for Exporters, SpanProcessors, etc.
• SDK Plugins: official implementations of common plugins, such as Zipkin, Prometheus, and OTLP exporters.
• Instrumentation Libraries: packages which use the API to provide instrumentation for HTTP clients, databases, web frameworks, and other common libraries.
• Semantic Conventions: Constants and helpers which ensure that instrumentation libraries produce consistent data across different languages.

All of the nitty-gritty details can be found in the OpenTelemetry Specification.

At minimum, this group would need to do the following:

• Define the OpenTelemetry API and Semantic Conventions in Perl, following the spec.
• Define an installation mechanism for the OpenTelemetry SDK, plugins, and instrumentation libraries
  • In some languages we try to automate installation, but that is not a requirement.
• Help the community maintain instrumentation for common Perl libraries.

The group would also need to pick one of two approaches to maintaining an SDK implementation:

• Maintain an SDK and the required SDK plugins in Perl.
• Maintain an FFI bridge to another SDK implementation, such as C++. I believe that SWIG and Perl XS are the available FFI options.

The FFI approach may be attractive, as maintaining a production ready SDK represents a lot of work. But I'm not a Perl expert and I don't know what kinds of disadvantages a FFI implementation may come with.

If anyone has interest in contributing to this effort, please introduce yourself! Questions are welcome.

[avillela-tc]

Folks in my org would love to contribute. We have lots of Perl code that could use some OTel love!

[rabbiveesh]

I'm more than happy to help maintain a perl SDK, for what it's worth.

Regarding FFI, the best option is FFI::Platypus. I am familiar with using it, it's quite pleasant

[jjatria]

Ah, I'm glad I found this! I was about to post a similar issue myself. 🎉

For the past couple of weeks I've been playing around with a Perl implementation of the OpenTelemetry API. It's now at a point where it can start to be used for testing (there is a placeholder integration for HTTP::Tiny, for example).

I've also been in recent talks with @tm604 (the maintainer of the OpenTracing distribution on CPAN) and @leonerd (the maintainer of Metrics::Any, among ... other things), who have said they'd be interested in this as well.

As far as the implementation of the SDK, I agree FFI::Platypus is excellent, but binding with C++ has been tricky in my experience at least. Coming out of the implementation of the API, I would be up for attempting a Perl implementation of the SDK as well.

Incidentally, I suspect @plicease (the maintainer of FFI::Platypus, but perhaps more relevantly of NewFangle) would also be interested in this project.

[leonerd]

Hello there - Yes I'm around :)

[plicease]

Yes, we will probably need to switch from NewFangle to something that uses OpenTelemetry for our Perl app.

[tm604]

So if anyone's interested in working on this, would suggest moving the discussion to a realtime system?

I'm on irc.perl.org/#io-async as tom_m or irc.libera.chat/#perl as tm604, or can use Slack/Discord/Gitter or anything else as long as it can run in the browser.

The OpenTelemetry specification is now quite detailed and well-defined, and a pure Perl implementation is quite feasible: the main challenge in the past was "how to support spans and traces in async code" (as in https://metacpan.org/pod/Future::AsyncAwait), but https://metacpan.org/pod/Syntax::Keyword::Dynamically does most of the heavy lifting there.

For metrics and logs, my current approach is to let these two modules handle the client side:

• https://metacpan.org/pod/Metrics::Any
• https://metacpan.org/pod/Log::Any

and just provide ::Adapter::OpenTelemetry to receive the metrics+logs and combine them with traces.

Using FFI for an alternative implementation may be useful for comparison, but not something I'd be likely to have any time to spend on.

I have access to https://datadoghq.com, https://pyroscope.io and https://honeycomb.io for cross-platform testing + agent integration, plus Jægertracing of course, and for trace propagation can do some testing with AWS X-Ray/ADOT and GCP.

[arhuman]

Hello,

Any update on this (very interesting) project?

[jjatria]

Apologies for not writing earlier. The API implementation I mentioned above is still in roughly the same place (= partially implemented but in working order, although lacking an SDK backend it can plug into).

I have since started porting the Ruby SDK implementation, which I have not yet pushed to Github, and that attempt was making good progress until work responsibilities shifted me away from it. I did hit a stumbling block when it came to running a background process to send batches of logs to, but that's probably more out of my own lack of experience and time rather than a fundamental blocker.

If others are willing to help with some of the dev work, I'd be happy to put what I have in a public facing repository. Or if there are other attempts at implementing the SDK (eg. with FFI) then we can try to hook it into the API implementation and see what is missing.

All in all, I'm very keen on seeing this make progress, so any help would be welcome! :D

---

Edit: The partial SDK implementation now lives at https://github.com/jjatria/perl-opentelemetry-sdk

[jjatria]

I've continued to work on the version of the SDK in Perl (without FFI) and it is now in a state where I can manually create spans as desired, automatically create spans with integration distributions that can be provided by third parties (there are currently experimental ones for HTTP::Tiny and DBI), export those spans synchronously to the console or asynchronously to an OTLP collector, and attach those spans to traces coming from other services.

Although far from complete, this is now in a state where I think it can be taken as a valid MVP, and I think it would benefit from more eyes. 🎉

For completeness' sake, here are links to the relevant repositories I've been working with:

• The API: https://github.com/jjatria/perl-opentelemetry
• The SDK: https://github.com/jjatria/perl-opentelemetry-sdk
• The OTLP exporter: https://github.com/jjatria/perl-opentelemetry-exporter-otlp

I've been trying to add tests and documentation as I go, and this is relatively complete for the API, not so much for the rest. This area will definitely need more work.

I've been testing this locally using a fork of the OpenTelemetry demo, where I've ported the email service to Perl (with apologies to the Ruby folks) and hooked things up to the above repositories. You can see this in motion by cloning the perl branch of https://github.com/jjatria/opentelemetry-demo and bringing the stack up with docker compose up --build (it will take some 20 minutes the first time it runs).

Special thanks to @tm604 for the pointers regarding Metrics::Any and Syntax::Keyword::Dynamically, and to @leonerd for writing those distributions.

I'm keen to hear your thoughts.

[jjatria]

Things have continued to move forward. Since my last post, the API and the SDK distributions have landed on CPAN:

• https://metacpan.org/pod/OpenTelemetry
• https://metacpan.org/pod/OpenTelemetry::SDK

I still have not shipped the OTLP exporter, since that one still needs documenting, and there are parts of it that haven't really been implemented yet (eg. retries, etc).

One big change in the OTLP exporter is that it now supports both JSON and Protobuf exporting, so it should be installable in systems without the protobuf headers or a compiler.

I'm still interested in hearing feedback about the code and the documentation. Similarly, I wonder what the next steps are in terms of putting together a proper SIG to move this along.

[trask]

what the next steps are in terms of putting together a proper SIG to move this along.

hey @jjatria! check out https://github.com/open-telemetry/community/blob/main/project-management.md and feel free to ask questions about this process