Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Create Logs SDK LoggerProvider #1517

Merged
merged 19 commits into from
Jan 22, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions logs_sdk/Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,8 @@
source 'https://rubygems.org'

gemspec

gem 'opentelemetry-api', path: '../api'
gem 'opentelemetry-logs-api', path: '../logs_api'
gem 'opentelemetry-sdk', path: '../sdk'
gem 'opentelemetry-test-helpers', path: '../test_helpers'
3 changes: 2 additions & 1 deletion logs_sdk/lib/opentelemetry-logs-sdk.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@
#
# SPDX-License-Identifier: Apache-2.0

require 'opentelemetry'
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
require 'opentelemetry/sdk'
require 'opentelemetry/sdk/logs'
require 'opentelemetry/sdk/logs/version'
4 changes: 4 additions & 0 deletions logs_sdk/lib/opentelemetry/sdk/logs.rb
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,10 @@
# SPDX-License-Identifier: Apache-2.0

require_relative 'logs/version'
require_relative 'logs/logger'
require_relative 'logs/logger_provider'
require_relative 'logs/log_record_processor'
require_relative 'logs/export'

module OpenTelemetry
module SDK
Expand Down
24 changes: 24 additions & 0 deletions logs_sdk/lib/opentelemetry/sdk/logs/export.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# frozen_string_literal: true

# Copyright The OpenTelemetry Authors
#
# SPDX-License-Identifier: Apache-2.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This matches the pattern in OpenTelemetry::SDK::Trace::Export

The ExportError class is not present because it was not needed for the LoggerProvider. It will be added once it's used in subsequent PRs.

module OpenTelemetry
module SDK
module Logs
# The export module contains result codes for LoggerProvider#force_flush
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should also be result codes for exporters - their use shouldn't just relate to these two methods.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great point. Updated to be more general in: d63a4e9

# and LoggerProvider#shutdown
module Export
# The operation finished successfully.
SUCCESS = 0

# The operation finished with an error.
FAILURE = 1

# The operation timed out.
TIMEOUT = 2
end
end
end
end
47 changes: 47 additions & 0 deletions logs_sdk/lib/opentelemetry/sdk/logs/log_record_processor.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# frozen_string_literal: true

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is a boilerplate/placeholder so the LogRecordProcessor methods can be called. They will be implemented in a later PR.

# Copyright The OpenTelemetry Authors
#
# SPDX-License-Identifier: Apache-2.0

module OpenTelemetry
module SDK
module Logs
# LogRecordProcessor describes a duck type and provides synchronous no-op hooks for when a
# {LogRecord} is started or when a {LogRecord} is ended. It is not required to subclass this
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
# class to provide an implementation of LogRecordProcessor, provided the interface is
# satisfied.
class LogRecordProcessor
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
# Called when a {LogRecord} is emitted. Subsequent calls are not
# permitted after shutdown is called.
# @param [LogRecord] log_record The emitted {LogRecord}
# @param [Context] context The resolved Context
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the "resolved Context" here? Who does the resolution? (Note that I haven't read this part of the spec or all of this PR, or the logs API, so maybe this is answered clearly somewhere else - just asking as a naive reviewer.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "resolved Context" here is coming from the Logs SDK spec description for the method arguments.

https://opentelemetry.io/docs/specs/otel/logs/sdk/#onemit

The description has a bit more detail to include that might clear things up. I've updated the description to include the rest of the text in: 5528277

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure who does the resolution. I'm also not quite sure how we'll use this argument.

I'll consult some of the implementations in other languages and come back with more info.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fbogsany - I can't find any examples where the context argument is used in other languages.

  • Python doesn't include it anywhere.
  • PHP and Java have it as an argument but do not use it in the onEmit functions.
  • Javascript doesn't include it as an argument in their Simple or Batch processors, but the duck type includes it as an argument.

So I don't have a great answer about who does the resolution. I didn't end up needing this argument in the Simple or Batch processors because I used the SpanContext to pull trace_id, span_id, and trace_flags. Those are the only context-related attributes in the Logs Data Model.

Do you have a sense of what approach you'd prefer Ruby take?

def on_emit(log_record, context); end

# Export all log records to the configured `Exporter` that have not yet
# been exported.
#
# This method should only be called in cases where it is absolutely
# necessary, such as when using some FaaS providers that may suspend
# the process after an invocation, but before the `Processor` exports
# the completed spans.
#
# @param [optional Numeric] timeout An optional timeout in seconds.
# @return [Integer] Export::SUCCESS if no error occurred, Export::FAILURE if
# a non-specific failure occurred, Export::TIMEOUT if a timeout occurred.
def force_flush(timeout: nil)
Export::SUCCESS
end

# Called when {LoggerProvider#shutdown} is called.
#
# @param [optional Numeric] timeout An optional timeout in seconds.
# @return [Integer] Export::SUCCESS if no error occurred, Export::FAILURE if
# a non-specific failure occurred, Export::TIMEOUT if a timeout occurred.
def shutdown(timeout: nil)
Export::SUCCESS
end
end
end
end
end
33 changes: 33 additions & 0 deletions logs_sdk/lib/opentelemetry/sdk/logs/logger.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# frozen_string_literal: true

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Logger SDK will be fully written in a separate PR. The code here represents what's necessary to fulfill the LoggerProvider SDK spec.

# Copyright The OpenTelemetry Authors
#
# SPDX-License-Identifier: Apache-2.0

module OpenTelemetry
module SDK
module Logs
# The SDK implementation of OpenTelemetry::Logs::Logger
class Logger < OpenTelemetry::Logs::Logger
attr_reader :instrumentation_scope, :logger_provider
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm curious why we expose these attributes. In the Tracer and Meter, they're private, so the inconsistency is a little weird.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This attr_reader call has been removed. I'm not sure why I initially broke from Tracer and Meter.


# @api private
#
# Returns a new {OpenTelemetry::SDK::Logs::Logger} instance. This should
# not be called directly. New loggers should be created using
# {LoggerProvider#logger}.
#
# @param [String] name Instrumentation package name
# @param [String] version Instrumentation package version
# @param [LoggerProvider] logger_provider The {LoggerProvider} that
# initialized the logger
#
# @return [OpenTelemetry::SDK::Logs::Logger]
def initialize(name, version, logger_provider)
@instrumentation_scope = InstrumentationScope.new(name, version)
@logger_provider = logger_provider
end
Comment on lines +12 to +27
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works toward the SDK Logger Provider => Logger Creation spec fulfillment

https://opentelemetry.io/docs/specs/otel/logs/sdk/#logger-creation

I didn't put any restrictions on initializing a new Logger in the code, but they are present in the documentation. I'm open to suggestions on how to limit Logger creation to only the LoggerProvider#logger method.

Name validation logic is handled in LoggerProvider#logger.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The @api private comment is our standard way of marking "internal" methods. There is no practical way to enforce it.

end
end
end
end
127 changes: 127 additions & 0 deletions logs_sdk/lib/opentelemetry/sdk/logs/logger_provider.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# frozen_string_literal: true

# Copyright The OpenTelemetry Authors
#
# SPDX-License-Identifier: Apache-2.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The specs for SDK LoggerProvider and SDK TracerProvider are extremely similar.

Given this, the code is very similar to the already implemented TracerProvider in the SDK gem.

module OpenTelemetry
module SDK
module Logs
# The SDK implementation of OpenTelemetry::Logs::LoggerProvider.
class LoggerProvider < OpenTelemetry::Logs::LoggerProvider
attr_reader :resource, :log_record_processors
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log_record_processors need safe concurrent access. By exposing this as a reader, external callers may gain access to the array in an unsafe way.

This attribute should not be exposed as a public value

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is also not exposed in the spec.

Suggested change
attr_reader :resource, :log_record_processors
attr_reader :resource


UNEXPECTED_ERROR_MESSAGE = 'unexpected error in ' \
'OpenTelemetry::SDK::Logs::LoggerProvider#%s'
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
# Returns a new LoggerProvider instance.
#
# @param [optional Resource] resource The resource to associate with
# new LogRecords created by {Logger}s created by this LoggerProvider.
# @param [optional Array] log_record_processors The
# {LogRecordProcessor}s to associate with this LoggerProvider.
#
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might notice that limits are not included in the parameters to initialize a LoggerProvider. This is because the limits are implemented in issue #1516.

# @return [OpenTelemetry::SDK::Logs::LoggerProvider]
def initialize(
resource: OpenTelemetry::SDK::Resources::Resource.create,
log_record_processors: []
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Providing the log_record_processors at creation time is MAY in the spec. We don't do that in the MeterProvider (for metric_readers) or the TracerProvider (for span_processors). Instead, we rely on the add_... methods to add them individually. I would start with not exposing it during creation. We can always add it later if that's a common pattern (and then consider adding to TracerProvider and MeterProvider as well).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy to remove the arg from the initialize method. I'll double-check about its commonality in other languages. I've also removed the public reader. See: 37befe6

)
@log_record_processors = log_record_processors
@mutex = Mutex.new
@resource = resource
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A LoggerProvider MUST provide a way to allow a Resource to be specified. If a Resource is specified, it SHOULD be associated with all the LogRecords produced by any Logger from the LoggerProvider.
source

The Resource can be passed as an arg. If none is supplied, a new Resource will be created on initialization.

The Log Record Definition includes Resource as an attribute. The LogRecord class and the act of emitting log records from a logger are not in the scope of this PR.

The statement "it SHOULD be associated with all the LogRecords produced by any Logger from the LoggerProvider" will be addressed in a subsequent PR. The groundwork laid here enables a logger to access the logger provider's resource: logger.instance_variable_get(:@logger_provider).instance_variable_get(:@resource).

@stopped = false
end

# Creates an {OpenTelemetry::SDK::Logs::Logger} instance.
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
#
# @param [optional String] name Instrumentation package name
# @param [optional String] version Instrumentation package version
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
#
# @return [OpenTelemetry::SDK::Logs::Logger]
def logger(name = nil, version = nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This signature is inconsistent between TracerProvider and MeterProvider. In the latter, we used:

def meter(name, version: nil)

so name is required and version is a keyword arg. IIRC we're going to have to add one or two more args at some point 🔜 , so keywords for optional args are vastly preferred.

We should go back and fix this for TracerProvider. I think that can be done in a backwards-compatible fashion.

For MeterProvider, we opted to require name. If a user explicitly passes nil or "", we can handle it, but those are both considered invalid - I'd rather have an explicitly invalid required arg than an invalid default arg.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds great. I prefer the keyword args, I just defaulted to what was in TracerProvider since that was already stable.

Updated in: ca7c15f

name ||= ''
version ||= ''

OpenTelemetry.logger.warn('LoggerProvider#logger called without providing a logger name.') if name.empty?

OpenTelemetry::SDK::Logs::Logger.new(name, version, self)
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
end

# Adds a new log record processor to this LoggerProvider's
# log_record_processors.
#
# @param [LogRecordProcessor] log_record_processor The
# {LogRecordProcessor} to add to this LoggerProvider.
def add_log_record_processor(log_record_processor)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LoggerProvider MAY provide methods to update the configuration. If configuration is updated (e.g., adding a LogRecordProcessor), the updated configuration MUST also apply to all already returned Loggers (i.e. it MUST NOT matter whether a Logger was obtained from the LoggerProvider before or after the configuration change). Note: Implementation-wise, this could mean that Logger instances have a reference to their LoggerProvider and access configuration only via this reference.

source

Every logger instance has a reference to the logger provider that created it. (see L50) This gives loggers access to the log record processors.

@mutex.synchronize do
@log_record_processors = log_record_processors.dup.push(log_record_processor)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is copied from TracerProvider#add_span_processor. @fallwith and I were chatting about this LOC. He was asking about the use of .dup.push here.

Does dup.push provide atomicity in a way that the push method alone couldn't deliver? Or is there a different reason dup.push was used?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm uncertain, but @fbogsany probably knows. I think it has to do with mutating the processors array. Looking at TracerProvider a span gets a reference to processors on start. If a span processor is added after a span is started, and the processors array is mutated, a span will see a processor that wasn't there when it was started. I'm not sure if this would happen in the real world, or if this is what we're protecting against, but that is a reason why we might prefer dup.push over push.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's exactly the reason - each SDK Span instance has a reference to the list of span processors, and that list should not change between the span calling on_start and on_finish. IDK if that applies in your case - I need to read the rest of the PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's worth considering whether it should be valid to add log record processors after @stopped. In the TracerProvider, we warn and don't allow that.

end
end

# Attempts to stop all the activity for this LoggerProvider. Calls
# {LogRecordProcessor#shutdown} for all registered {LogRecordProcessor}s.
#
# This operation may block until all log records are processed. Must
# be called before turning off the main application to ensure all data
# are processed and exported.
#
# After this is called all newly created {LogRecord}s will be no-op.
#
# @param [optional Numeric] timeout An optional timeout in seconds.
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
# @return [Integer] Export::SUCCESS if no error occurred, Export::FAILURE if
# a non-specific failure occurred, Export::TIMEOUT if a timeout occurred.
def shutdown(timeout: nil)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See: https://opentelemetry.io/docs/specs/otel/logs/sdk/#shutdown

  • Only called once by setting @stopped to true L92
  • Provides a way to let the caller know whether it succeeded, failed, or timed out by returning OpenTelemetry::SDK::Logs::EXPORT constants (L81, L87, L93)
  • Implements a timeout through calls to OpenTelemetry::Common::Utilities.timeout_timestamp and OpenTelemetry::Common::Utilities.maybe_timeout (L84, L86)
  • Invokes shutdown on all the processors (L85 - 89)

@mutex.synchronize do
if @stopped
OpenTelemetry.logger.warn('LoggerProvider#shutdown called multiple times.')
return OpenTelemetry::SDK::Logs::Export::FAILURE
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
end

start_time = OpenTelemetry::Common::Utilities.timeout_timestamp
results = log_record_processors.map do |processor|
remaining_timeout = OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start_time)
break [OpenTelemetry::SDK::Logs::Export::TIMEOUT] if remaining_timeout&.zero?
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved

processor.shutdown(timeout: remaining_timeout)
end

@stopped = true
results.max || OpenTelemetry::SDK::Logs::Export::SUCCESS
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
end
rescue StandardError => e
OpenTelemetry.handle_error(exception: e, message: UNEXPECTED_ERROR_MESSAGE % __method__)
Export::FAILURE
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the TracerProvider and MeterProvider we expect the processors/readers to not raise from #shutdown or #force_flush and instead only return one of the 3 supported result codes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. Fixed in: a619d36

end

# Immediately export all {LogRecord}s that have not yet been exported
# for all the registered {LogRecordProcessor}s.
#
# This method should only be called in cases where it is absolutely
# necessary, such as when using some FaaS providers that may suspend
# the process after an invocation, but before the {LogRecordProcessor}
# exports the completed {LogRecord}s.
#
# @param [optional Numeric] timeout An optional timeout in seconds.
kaylareopelle marked this conversation as resolved.
Show resolved Hide resolved
# @return [Integer] Export::SUCCESS if no error occurred, Export::FAILURE if
# a non-specific failure occurred, Export::TIMEOUT if a timeout occurred.
def force_flush(timeout: nil)
@mutex.synchronize do
return Export::SUCCESS if @stopped

start_time = OpenTelemetry::Common::Utilities.timeout_timestamp
results = log_record_processors.map do |processor|
remaining_timeout = OpenTelemetry::Common::Utilities.maybe_timeout(timeout, start_time)
return Export::TIMEOUT if remaining_timeout&.zero?

processor.force_flush(timeout: remaining_timeout)
end

results.max || Export::SUCCESS
end
rescue StandardError => e
OpenTelemetry.handle_error(exception: e, message: UNEXPECTED_ERROR_MESSAGE % __method__)
Export::FAILURE
end
end
end
end
end
15 changes: 9 additions & 6 deletions logs_sdk/opentelemetry-logs-sdk.gemspec
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,18 @@ Gem::Specification.new do |spec|
spec.require_paths = ['lib']
spec.required_ruby_version = '>= 3.0'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updating dependencies to more recent versions. This came about when I realized the SimpleCov version that was installed didn't include a feature I wanted to use.

spec.add_dependency 'opentelemetry-logs-api', '~> 0.1.0'
spec.add_dependency 'opentelemetry-api', '~> 1.2'
spec.add_dependency 'opentelemetry-logs-api', '~> 0.1'
spec.add_dependency 'opentelemetry-sdk', '~> 1.3'

spec.add_development_dependency 'bundler', '>= 1.17'
spec.add_development_dependency 'minitest', '~> 5.0'
spec.add_development_dependency 'rake', '~> 12.0'
spec.add_development_dependency 'rubocop', '~> 1.51.0'
spec.add_development_dependency 'simplecov', '~> 0.17'
spec.add_development_dependency 'minitest', '~> 5.19'
spec.add_development_dependency 'opentelemetry-test-helpers', '~> 0.4'
spec.add_development_dependency 'rake', '~> 13.0'
spec.add_development_dependency 'rubocop', '~> 1.56'
spec.add_development_dependency 'simplecov', '~> 0.22'
spec.add_development_dependency 'yard', '~> 0.9'
spec.add_development_dependency 'yard-doctest', '~> 0.1.6'
spec.add_development_dependency 'yard-doctest', '~> 0.1.17'

if spec.respond_to?(:metadata)
spec.metadata['changelog_uri'] = "https://open-telemetry.github.io/opentelemetry-ruby/opentelemetry-logs-sdk/v#{OpenTelemetry::SDK::Logs::VERSION}/file.CHANGELOG.html"
Expand Down
22 changes: 3 additions & 19 deletions logs_sdk/test/.rubocop.yml
Original file line number Diff line number Diff line change
@@ -1,24 +1,8 @@
# inherit_from: .rubocop_todo.yml
inherit_from: ../.rubocop.yml

AllCops:
TargetRubyVersion: '3.0'

Lint/UnusedMethodArgument:
Enabled: false
Metrics/AbcSize:
Metrics/BlockLength:
Enabled: false
Metrics/LineLength:
Enabled: false
Metrics/MethodLength:
Max: 50
Metrics/PerceivedComplexity:
Max: 30
Metrics/CyclomaticComplexity:
Max: 20
Metrics/ParameterLists:
Enabled: false
Naming/FileName:
Exclude:
- 'lib/opentelemetry-logs-sdk.rb'
Style/ModuleFunction:
Metrics/AbcSize:
Enabled: false
Loading
Loading