Added Otlp UDP Exporter for Lambda Support #129

AsakerMohd · 2024-10-22T02:39:53Z

Description of changes:
Added a new OTLP UDP exporter to export Activities in the form of OTLP spans over UDP. Most of the code is commented to explain how that was achieved but here is a summary:

Copied over protobuf files from https://github.com/open-telemetry/opentelemetry-proto/ to build the assemblies locally.
Used reflection to access private OTEL methods used to build OTLP spans such as ToOtlpSpan function and ToOtlpResource function.
To Review, you don't need to look at any file under opentelemetry/proto as those files were copied directly from the proto repo and it builds properly after updating the project file. The main files to focus on are Plugin.cs and the new OtlpUdpExporter.cs file that has the bulk of the logic. I've added comments throughout OtlpUdpExporter.cs to explain what each function does at a high level.
Changes in Plugin.cs are based on similar changes that are made in NodeJS, mainly checking for the function name to decide on adding the udp exporter and disabling metrics for lambda environment. One thing to point out, we can't remove existing otlp exporters in code and that could only be done by setting OTLP_TRACES_EXPORTER to 'none'. This will be done in the lambda layer.
Will create a follow up task to add unit tests.
Tested by building the OTLP span and then printing it out in JSON format as shown below before exporting. Simulated by setting AWS_LAMBDA_FUNCTION_NAME and running the X-Ray Daemon locally to test E2E to X-Ray. You can also see that we maintained the resource attributes which are important info holding like sdk version:

{
  "resourceSpans": [
    {
      "resource": {
        "attributes": [
          {
            "key": "telemetry.distro.name",
            "value": {
              "stringValue": "aws-otel-dotnet-instrumentation"
            }
          },
          {
            "key": "telemetry.distro.version",
            "value": {
              "stringValue": "1.3.2.dev0-aws"
            }
          },
          {
            "key": "host.name",
            "value": {
              "stringValue": "e0d3c5d8a30e"
            }
          },
          {
            "key": "process.owner",
            "value": {
              "stringValue": "root"
            }
          },
          {
            "key": "process.pid",
            "value": {
              "intValue": "1"
            }
          },
          {
            "key": "process.runtime.description",
            "value": {
              "stringValue": ".NET 6.0.35"
            }
          },
          {
            "key": "process.runtime.name",
            "value": {
              "stringValue": ".NET"
            }
          },
          {
            "key": "process.runtime.version",
            "value": {
              "stringValue": "6.0.35"
            }
          },
          {
            "key": "container.id",
            "value": {
              "stringValue": "e0d3c5d8a30e18106c2a461e31288e0bc361ee8284d17c6f8ed107e38b2b43e0"
            }
          },
          {
            "key": "telemetry.sdk.name",
            "value": {
              "stringValue": "opentelemetry"
            }
          },
          {
            "key": "telemetry.sdk.language",
            "value": {
              "stringValue": "dotnet"
            }
          },
          {
            "key": "telemetry.sdk.version",
            "value": {
              "stringValue": "1.9.0"
            }
          },
          {
            "key": "service.name",
            "value": {
              "stringValue": "aws-otel-integ-test"
            }
          }
        ]
      },
      "scopeSpans": [
        {
          "spans": [
            {
              "traceId": "Z9I7Hvm82FAkZRQ3cnHdpQ==",
              "spanId": "trSjM9DjIm4=",
              "parentSpanId": "AXFO8jW3C3g=",
              "name": "GET",
              "kind": "SPAN_KIND_CLIENT",
              "startTimeUnixNano": "1729621150065744900",
              "endTimeUnixNano": "1729621150471858000",
              "attributes": [
                {
                  "key": "aws.local.operation",
                  "value": {
                    "stringValue": "GET /outgoing-http-call"
                  }
                },
                {
                  "key": "http.request.method",
                  "value": {
                    "stringValue": "GET"
                  }
                },
                {
                  "key": "server.address",
                  "value": {
                    "stringValue": "aws.amazon.com"
                  }
                },
                {
                  "key": "server.port",
                  "value": {
                    "intValue": "443"
                  }
                },
                {
                  "key": "url.full",
                  "value": {
                    "stringValue": "https://aws.amazon.com/"
                  }
                },
                {
                  "key": "network.protocol.version",
                  "value": {
                    "stringValue": "1.1"
                  }
                },
                {
                  "key": "http.response.status_code",
                  "value": {
                    "intValue": "200"
                  }
                },
                {
                  "key": "aws.local.service",
                  "value": {
                    "stringValue": "aws-otel-integ-test"
                  }
                },
                {
                  "key": "aws.remote.service",
                  "value": {
                    "stringValue": "aws.amazon.com:443"
                  }
                },
                {
                  "key": "aws.remote.operation",
                  "value": {
                    "stringValue": "GET /"
                  }
                },
                {
                  "key": "aws.span.kind",
                  "value": {
                    "stringValue": "CLIENT"
                  }
                }
              ],
              "flags": 257
            },
            {
              "traceId": "Z9I7Hvm82FAkZRQ3cnHdpQ==",
              "spanId": "AXFO8jW3C3g=",
              "name": "GET outgoing-http-call",
              "kind": "SPAN_KIND_SERVER",
              "startTimeUnixNano": "1729621149714236200",
              "endTimeUnixNano": "1729621150624477300",
              "attributes": [
                {
                  "key": "aws.local.operation",
                  "value": {
                    "stringValue": "GET outgoing-http-call"
                  }
                },
                {
                  "key": "server.address",
                  "value": {
                    "stringValue": "localhost"
                  }
                },
                {
                  "key": "server.port",
                  "value": {
                    "intValue": "8080"
                  }
                },
                {
                  "key": "http.request.method",
                  "value": {
                    "stringValue": "GET"
                  }
                },
                {
                  "key": "url.scheme",
                  "value": {
                    "stringValue": "http"
                  }
                },
                {
                  "key": "url.path",
                  "value": {
                    "stringValue": "/outgoing-http-call"
                  }
                },
                {
                  "key": "network.protocol.version",
                  "value": {
                    "stringValue": "1.1"
                  }
                },
                {
                  "key": "user_agent.original",
                  "value": {
                    "stringValue": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0"
                  }
                },
                {
                  "key": "http.route",
                  "value": {
                    "stringValue": "outgoing-http-call"
                  }
                },
                {
                  "key": "http.response.status_code",
                  "value": {
                    "intValue": "200"
                  }
                },
                {
                  "key": "aws.local.service",
                  "value": {
                    "stringValue": "aws-otel-integ-test"
                  }
                },
                {
                  "key": "aws.span.kind",
                  "value": {
                    "stringValue": "LOCAL_ROOT"
                  }
                }
              ],
              "flags": 257
            },
            {
              "traceId": "TwDhnrhg3OuvbXBODBihUQ==",
              "spanId": "5Wg7o0N8vM0=",
              "name": "GET",
              "kind": "SPAN_KIND_SERVER",
              "startTimeUnixNano": "1729621150710909400",
              "endTimeUnixNano": "1729621150712945200",
              "attributes": [
                {
                  "key": "aws.local.operation",
                  "value": {
                    "stringValue": "GET /favicon.ico"
                  }
                },
                {
                  "key": "server.address",
                  "value": {
                    "stringValue": "localhost"
                  }
                },
                {
                  "key": "server.port",
                  "value": {
                    "intValue": "8080"
                  }
                },
                {
                  "key": "http.request.method",
                  "value": {
                    "stringValue": "GET"
                  }
                },
                {
                  "key": "url.scheme",
                  "value": {
                    "stringValue": "http"
                  }
                },
                {
                  "key": "url.path",
                  "value": {
                    "stringValue": "/favicon.ico"
                  }
                },
                {
                  "key": "network.protocol.version",
                  "value": {
                    "stringValue": "1.1"
                  }
                },
                {
                  "key": "user_agent.original",
                  "value": {
                    "stringValue": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0"
                  }
                },
                {
                  "key": "http.response.status_code",
                  "value": {
                    "intValue": "0"
                  }
                },
                {
                  "key": "aws.local.service",
                  "value": {
                    "stringValue": "aws-otel-integ-test"
                  }
                },
                {
                  "key": "aws.span.kind",
                  "value": {
                    "stringValue": "LOCAL_ROOT"
                  }
                }
              ],
              "flags": 257
            }
          ]
        }
      ]
    }
  ]
}

Below is the base64 string from the udp exporter:

{"format":"json","version":1}
T1SCv0PCo4ECjoKFXRlbGVtZXRyeS5kaXN0cm8ubmFtZRIhCh9hd3Mtb3RlbC1kb3RuZXQtaW5zdHJ1bWVudGF0aW9uCiwKGHRlbGVtZXRyeS5kaXN0cm8udmVyc2lvbhIQCg4xLjMuMi5kZXYwLWF3cwobCglob3N0Lm5hbWUSDgoMMzhmNmRhOTE1ZTdlChcKDXByb2Nlc3Mub3duZXISBgoEcm9vdAoRCgtwcm9jZXNzLnBpZBICGAEKLAobcHJvY2Vzcy5ydW50aW1lLmRlc2NyaXB0aW9uEg0KCy5ORVQgNi4wLjM1Ch4KFHByb2Nlc3MucnVudGltZS5uYW1lEgYKBC5ORVQKIwoXcHJvY2Vzcy5ydW50aW1lLnZlcnNpb24SCAoGNi4wLjM1ClIKDGNvbnRhaW5lci5pZBJCCkAzOGY2ZGE5MTVlN2U3NGM2NjgxNTg5ZjMwMDMyZjc3N2E1MWIxYjA4NTMyZjUzZGQwOTIxZDc4MDc2Mjc3NzdhCiUKEnRlbGVtZXRyeS5zZGsubmFtZRIPCg1vcGVudGVsZW1ldHJ5CiIKFnRlbGVtZXRyeS5zZGsubGFuZ3VhZ2USCAoGZG90bmV0CiAKFXRlbGVtZXRyeS5zZGsudmVyc2lvbhIHCgUxLjkuMAolCgxzZXJ2aWNlLm5hbWUSFQoTYXdzLW90ZWwtaW50ZWctdGVzdBLpCxLNAwoQd0PT9A+8pnmOG65zlHdPEBIIik2Y+P6J8roiCF20dGn+D2icKgNHRVQwAzk8MRy8OKUAGEF4aDLTOKUAGEowChNhd3MubG9jYWwub3BlcmF0aW9uEhkKF0dFVCAvb3V0Z29pbmctaHR0cC1jYWxsShwKE2h0dHAucmVxdWVzdC5tZXRob2QSBQoDR0VUSiIKDnNlcnZlci5hZGRyZXNzEhAKDmF3cy5hbWF6b24uY29tShIKC3NlcnZlci5wb3J0EgMYuwNKJQoIdXJsLmZ1bGwSGQoXaHR0cHM6Ly9hd3MuYW1hem9uLmNvbS9KIQoYbmV0d29yay5wcm90b2NvbC52ZXJzaW9uEgUKAzEuMUogChlodHRwLnJlc3BvbnNlLnN0YXR1c19jb2RlEgMYyAFKKgoRYXdzLmxvY2FsLnNlcnZpY2USFQoTYXdzLW90ZWwtaW50ZWctdGVzdEoqChJhd3MucmVtb3RlLnNlcnZpY2USFAoSYXdzLmFtYXpvbi5jb206NDQzSh8KFGF3cy5yZW1vdGUub3BlcmF0aW9uEgcKBUdFVCAvShkKDWF3cy5zcGFuLmtpbmQSCAoGQ0xJRU5UhQEBAQAAEqwEChB3Q9P0D7ymeY4brnOUd08QEghdtHRp/g9onCoWR0VUIG91dGdvaW5nLWh0dHAtY2FsbDACOawZT6k4pQAYQWjvxts4pQAYSi8KE2F3cy5sb2NhbC5vcGVyYXRpb24SGAoWR0VUIG91dGdvaW5nLWh0dHAtY2FsbEodCg5zZXJ2ZXIuYWRkcmVzcxILCglsb2NhbGhvc3RKEgoLc2VydmVyLnBvcnQSAxiQP0ocChNodHRwLnJlcXVlc3QubWV0aG9kEgUKA0dFVEoUCgp1cmwuc2NoZW1lEgYKBGh0dHBKIQoIdXJsLnBhdGgSFQoTL291dGdvaW5nLWh0dHAtY2FsbEohChhuZXR3b3JrLnByb3RvY29sLnZlcnNpb24SBQoDMS4xSm0KE3VzZXJfYWdlbnQub3JpZ2luYWwSVgpUTW96aWxsYS81LjAgKE1hY2ludG9zaDsgSW50ZWwgTWFjIE9TIFggMTAuMTU7IHJ2OjEzMS4wKSBHZWNrby8yMDEwMDEwMSBGaXJlZm94LzEzMS4wSiIKCmh0dHAucm91dGUSFAoSb3V0Z29pbmctaHR0cC1jYWxsSiAKGWh0dHAucmVzcG9uc2Uuc3RhdHVzX2NvZGUSAxjIAUoqChFhd3MubG9jYWwuc2VydmljZRIVChNhd3Mtb3RlbC1pbnRlZy10ZXN0Sh0KDWF3cy5zcGFuLmtpbmQSDAoKTE9DQUxfUk9PVIUBAQEAABLnAwoQXpxo3t3+a9LuFTj5JTf+9RIIF+3bV/DBzWcqA0dFVDACOcwIz+I4pQAYQeSg5+I4pQAYSikKE2F3cy5sb2NhbC5vcGVyYXRpb24SEgoQR0VUIC9mYXZpY29uLmljb0odCg5zZXJ2ZXIuYWRkcmVzcxILCglsb2NhbGhvc3RKEgoLc2VydmVyLnBvcnQSAxiQP0ocChNodHRwLnJlcXVlc3QubWV0aG9kEgUKA0dFVEoUCgp1cmwuc2NoZW1lEgYKBGh0dHBKGgoIdXJsLnBhdGgSDgoML2Zhdmljb24uaWNvSiEKGG5ldHdvcmsucHJvdG9jb2wudmVyc2lvbhIFCgMxLjFKbQoTdXNlcl9hZ2VudC5vcmlnaW5hbBJWClRNb3ppbGxhLzUuMCAoTWFjaW50b3NoOyBJbnRlbCBNYWMgT1MgWCAxMC4xNTsgcnY6MTMxLjApIEdlY2tvLzIwMTAwMTAxIEZpcmVmb3gvMTMxLjBKHwoZaHR0cC5yZXNwb25zZS5zdGF0dXNfY29kZRICGABKKgoRYXdzLmxvY2FsLnNlcnZpY2USFQoTYXdzLW90ZWwtaW50ZWctdGVzdEodCg1hd3Muc3Bhbi5raW5kEgwKCkxPQ0FMX1JPT1SFAQEBAAA=

Below is the raw trace data from the xray console:

{
    "Id": "1-67d23b1e-f9bcd850246514377271dda5",
    "Duration": 0.91,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "01714ef235b70b78",
            "Document": {
                "id": "01714ef235b70b78",
                "name": "aws-otel-integ-test",
                "start_time": 1729621149.7142363,
                "trace_id": "1-67d23b1e-f9bcd850246514377271dda5",
                "end_time": 1729621150.6244771,
                "http": {
                    "request": {
                        "method": "GET",
                        "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0"
                    },
                    "response": {
                        "status": 200
                    }
                },
                "aws": {
                    "span.kind": "LOCAL_ROOT"
                },
                "annotations": {
                    "aws.local.service": "aws-otel-integ-test",
                    "span.name": "GET outgoing-http-call",
                    "aws.local.operation": "GET outgoing-http-call",
                    "span.kind": "SERVER"
                },
                "metadata": {
                    "user_agent.original": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:131.0) Gecko/20100101 Firefox/131.0",
                    "telemetry.distro.version": "1.3.2.dev0-aws",
                    "service.name": "aws-otel-integ-test",
                    "network.protocol.version": "1.1",
                    "telemetry.distro.name": "aws-otel-dotnet-instrumentation",
                    "process.runtime.version": "6.0.35",
                    "url.scheme": "http",
                    "process.pid": 1,
                    "container.id": "e0d3c5d8a30e18106c2a461e31288e0bc361ee8284d17c6f8ed107e38b2b43e0",
                    "telemetry.sdk.name": "opentelemetry",
                    "server.address": "localhost",
                    "process.owner": "root",
                    "aws.span.kind": "LOCAL_ROOT",
                    "telemetry.sdk.language": "dotnet",
                    "url.path": "/outgoing-http-call",
                    "process.runtime.name": ".NET",
                    "http.request.method": "GET",
                    "http.route": "outgoing-http-call",
                    "server.port": 8080,
                    "host.name": "e0d3c5d8a30e",
                    "telemetry.sdk.version": "1.9.0",
                    "process.runtime.description": ".NET 6.0.35",
                    "http.response.status_code": 200
                },
                "subsegments": [
                    {
                        "id": "b6b4a333d0e3226e",
                        "name": "aws.amazon.com:443",
                        "start_time": 1729621150.0657449,
                        "end_time": 1729621150.471858,
                        "http": {
                            "request": {
                                "url": "https://aws.amazon.com/",
                                "method": "GET"
                            },
                            "response": {
                                "status": 200
                            }
                        },
                        "aws": {
                            "span.kind": "CLIENT"
                        },
                        "annotations": {
                            "aws.local.service": "aws-otel-integ-test",
                            "span.name": "GET",
                            "aws.local.operation": "GET /outgoing-http-call",
                            "span.kind": "CLIENT",
                            "aws.remote.service": "aws.amazon.com:443",
                            "aws.remote.operation": "GET /"
                        },
                        "metadata": {
                            "telemetry.distro.version": "1.3.2.dev0-aws",
                            "service.name": "aws-otel-integ-test",
                            "network.protocol.version": "1.1",
                            "telemetry.distro.name": "aws-otel-dotnet-instrumentation",
                            "process.runtime.version": "6.0.35",
                            "process.pid": 1,
                            "container.id": "e0d3c5d8a30e18106c2a461e31288e0bc361ee8284d17c6f8ed107e38b2b43e0",
                            "telemetry.sdk.name": "opentelemetry",
                            "server.address": "aws.amazon.com",
                            "process.owner": "root",
                            "aws.span.kind": "CLIENT",
                            "telemetry.sdk.language": "dotnet",
                            "process.runtime.name": ".NET",
                            "http.request.method": "GET",
                            "server.port": 443,
                            "host.name": "e0d3c5d8a30e",
                            "telemetry.sdk.version": "1.9.0",
                            "process.runtime.description": ".NET 6.0.35",
                            "url.full": "https://aws.amazon.com/",
                            "http.response.status_code": 200
                        },
                        "namespace": "remote"
                    }
                ]
            }
        },
        {
            "Id": "18e3594b1200d064",
            "Document": {
                "id": "18e3594b1200d064",
                "name": "aws.amazon.com:443",
                "start_time": 1729621150.0657449,
                "trace_id": "1-67d23b1e-f9bcd850246514377271dda5",
                "end_time": 1729621150.471858,
                "parent_id": "b6b4a333d0e3226e",
                "inferred": true,
                "http": {
                    "request": {
                        "url": "https://aws.amazon.com/",
                        "method": "GET"
                    },
                    "response": {
                        "status": 200
                    }
                },
                "annotations": {
                    "aws.local.service": "aws.amazon.com:443",
                    "aws.local.operation": "GET /"
                }
            }
        }
    ]
}

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…dotnet-instrumentation

vastin · 2024-10-22T20:54:53Z

What is the format of trace/span id? It is neither W3C nor X-Ray format.

          "traceId": "Z9I7Hvm82FAkZRQ3cnHdpQ==",
         "spanId": "trSjM9DjIm4=",
         "parentSpanId": "AXFO8jW3C3g=",

src/AWS.Distro.OpenTelemetry.AutoInstrumentation/OtlpUdpExporter.cs

src/AWS.Distro.OpenTelemetry.AutoInstrumentation/Plugin.cs

src/AWS.Distro.OpenTelemetry.AutoInstrumentation/opentelemetry/proto/collector/README.md

AsakerMohd · 2024-10-22T22:02:51Z

What is the format of trace/span id? It is neither W3C nor X-Ray format.
          "traceId": "Z9I7Hvm82FAkZRQ3cnHdpQ==",
         "spanId": "trSjM9DjIm4=",
         "parentSpanId": "AXFO8jW3C3g=",

It's in base64 and it's just for printing purposes. This is the ToString function by otel. That is from the raw otel span that is exported to X-Ray. In the X-Ray raw trace data, you can see it's in the correct format. When calling the sample app, the response trace id is {"traceId": "1-67d23b1e-f9bcd850246514377271dda5"} which matches the one in the trace id above.

vastin

LGTM

AsakerMohd added 5 commits October 3, 2024 17:22

Updated release workflow and Nuget README

76a6bdc

Merge branch 'main' of https://github.com/aws-observability/aws-otel-…

f6c87be

…dotnet-instrumentation

Added new flag to allow routeDataParsing for server spans

10b98a9

Merge branch 'main' of https://github.com/aws-observability/aws-otel-…

883ac7a

…dotnet-instrumentation

Added Otlp UDP Exporter for Lambda Support

da050cb

AsakerMohd requested a review from a team as a code owner October 22, 2024 02:39

vastin reviewed Oct 22, 2024

View reviewed changes

Applied comments

7721dd2

vastin approved these changes Oct 22, 2024

View reviewed changes

AsakerMohd merged commit e7dd03d into main Oct 23, 2024
8 checks passed

AsakerMohd deleted the exporterLambdaSupport branch October 23, 2024 18:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Otlp UDP Exporter for Lambda Support #129

Added Otlp UDP Exporter for Lambda Support #129

AsakerMohd commented Oct 22, 2024 •

edited

Loading

vastin commented Oct 22, 2024

AsakerMohd commented Oct 22, 2024 •

edited

Loading

vastin left a comment

Added Otlp UDP Exporter for Lambda Support #129

Added Otlp UDP Exporter for Lambda Support #129

Conversation

AsakerMohd commented Oct 22, 2024 • edited Loading

vastin commented Oct 22, 2024

AsakerMohd commented Oct 22, 2024 • edited Loading

vastin left a comment

Choose a reason for hiding this comment

AsakerMohd commented Oct 22, 2024 •

edited

Loading

AsakerMohd commented Oct 22, 2024 •

edited

Loading