Skip to content

Commit

Permalink
Elasticsearch v8 support (#741)
Browse files Browse the repository at this point in the history 
* Fix function_wrapper calls to module

* Fix wrapper in pika hook

* Revert elasticsearch instrumentation

* Revert some wrap_function_wrappers to orig

* Remove comments/breakpoints

* Fix hooks in elasticsearch

* Add new client methods from v8 and their hooks

* Add elasticsearch v8 to workflow and tox

* Fix indices for elasticsearch01

* Disable xpack security in elasticsearch v8.0

* Start to add try/except blocks in tests

* Add support for v8 transport

* add support for v8 connection

* Add tests-WIP

* Clean up most tests

* Clean up unused instrumentation

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Enriqueta De Leon <deleonenriqueta@users.noreply.github.com>
Co-authored-by: Uma Annamalai <umaannamalai@users.noreply.github.com>
Co-authored-by: Kate Anderson <kanderson250@users.noreply.github.com>
Co-authored-by: Mary Martinez <mary.diana.martinez@gmail.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Remove elastic search source code

* Elasticsearch v8 testing

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Enriqueta De Leon <deleonenriqueta@users.noreply.github.com>
Co-authored-by: Uma Annamalai <umaannamalai@users.noreply.github.com>
Co-authored-by: Kate Anderson <kanderson250@users.noreply.github.com>
Co-authored-by: Mary Martinez <mary.diana.martinez@gmail.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>

* Scope ES fixture

* ES v8 only supports Python3.6+

* Refactor transport tests for v8

Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Uma Annamalai <umaannamalai@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>
Co-authored-by: Kate Anderson <kanderson250@users.noreply.github.com>
Co-authored-by: Enriqueta De Leon <deleonenriqueta@users.noreply.github.com>

* Remove extra comments

* Added perform_request_kwargs to test_transport

* Fix some linter issues

* Remove extra newline

* Group es v7 v8 process modules together

* Add auto signature detection & binding

* Use bind_arguments in ES

* Add test for wrapped function

* Add validator for datastore trace inputs

* Use common bind_arguments for PY3

* Fix tests in starlette v0.23.1 (#752)

* Fix tests in starlette v0.23.1

* Fix conditional tests

* Add comment to bg_task test

* Split below es 8 methods from es 8 methods

Note the previous tests in this file to check whether a method was instrumented, did not test anything because they were checking whether the list
of methods that we instrumented were instrumented instead of whether
there were uninstrumented methods on the es client that we missed.

Because we decided due to lack of reporting of bugs by our customers, to not support the buggy wrapping on previous es versions (below es8), we
only added tests to assert all methods were wrapped from es8+. We also
are only testing es8+ wrapping of methods since the previous versions
wrapping behavior may not have been correct due to the signature of the
methods changing without us detecting it due to lack of tests. Since our customers have not reported any issues, it seems not worth it at this time to go back and fix these bugs.

* Remove signature auto detection implementation

* Fixup: remove signature autodetection

* Fixup: cleanup

* Test method calls on all es versions

* Fixup: don't run some methods on es7

---------

Co-authored-by: Timothy Pansino <11214426+TimPansino@users.noreply.github.com>
Co-authored-by: mary-martinez <mmartinez@newrelic.com>
Co-authored-by: enriqueta <edeleon@newrelic.com>
Co-authored-by: Tim Pansino <timpansino@gmail.com>
Co-authored-by: Lalleh Rafeei <lrafeei@users.noreply.github.com>
Co-authored-by: Enriqueta De Leon <deleonenriqueta@users.noreply.github.com>
Co-authored-by: Uma Annamalai <umaannamalai@users.noreply.github.com>
Co-authored-by: Kate Anderson <kanderson250@users.noreply.github.com>
Co-authored-by: Mary Martinez <mary.diana.martinez@gmail.com>
Co-authored-by: Hannah Stepanek <hmstepanek@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <hstepanek@newrelic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
  • Loading branch information
@lrafeei @TimPansino
@mary-martinez @deleonenriqueta @umaannamalai @kanderson250 @hmstepanek @mergify
13 people authored Feb 9, 2023
1 parent b0d4e73 commit 1bf6a65
Show file tree
Hide file tree
Showing 15 changed files with 1,093 additions and 439 deletions.
15 changes: 8 additions & 7 deletions .github/workflows/tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ jobs:
runs-on: ubuntu-20.04
needs:
- python
- elasticsearchserver01
- elasticsearchserver07
- elasticsearchserver08
- gearman
- grpc
- kafka
Expand Down Expand Up @@ -656,7 +656,7 @@ jobs:
path: ./**/.coverage.*
retention-days: 1

elasticsearchserver01:
elasticsearchserver07:
env:
TOTAL_GROUPS: 1

Expand All @@ -669,8 +669,8 @@ jobs:
timeout-minutes: 30

services:
es01:
image: elasticsearch:1.4.4
es07:
image: elasticsearch:7.17.8
env:
"discovery.type": "single-node"
ports:
Expand Down Expand Up @@ -708,7 +708,7 @@ jobs:
path: ./**/.coverage.*
retention-days: 1

elasticsearchserver07:
elasticsearchserver08:
env:
TOTAL_GROUPS: 1

Expand All @@ -721,9 +721,10 @@ jobs:
timeout-minutes: 30

services:
es01:
image: elasticsearch:7.13.2
es08:
image: elasticsearch:8.6.0
env:
"xpack.security.enabled": "false"
"discovery.type": "single-node"
ports:
- 8080:9200
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
.DS_Store
.DS_Store/

# Linter
megalinter-reports/

Expand Down
87 changes: 85 additions & 2 deletions newrelic/config.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2665,64 +2665,147 @@ def _process_module_builtin_defaults():
"aioredis.connection", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_connection"
)

_process_module_definition("redis.asyncio.client", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client")
_process_module_definition(
"redis.asyncio.client", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client"
)

_process_module_definition("redis.asyncio.commands", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client")
_process_module_definition(
"redis.asyncio.commands", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_client"
)

_process_module_definition(
"redis.asyncio.connection", "newrelic.hooks.datastore_aioredis", "instrument_aioredis_connection"
)

# v7 and below
_process_module_definition(
"elasticsearch.client",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.client.cat",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_cat",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client.cat",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_cat_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.client.cluster",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_cluster",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client.cluster",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_cluster_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.client.indices",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_indices",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client.indices",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_indices_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.client.nodes",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_nodes",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client.nodes",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_nodes_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.client.snapshot",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_snapshot",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client.snapshot",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_snapshot_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.client.tasks",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_tasks",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client.tasks",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_tasks_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.client.ingest",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_ingest",
)
# v8 and above
_process_module_definition(
"elasticsearch._sync.client.ingest",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_client_ingest_v8",
)

# v7 and below
_process_module_definition(
"elasticsearch.connection.base",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_connection_base",
)
# v8 and above
_process_module_definition(
"elastic_transport._node._base",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elastic_transport__node__base",
)

# v7 and below
_process_module_definition(
"elasticsearch.transport",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elasticsearch_transport",
)
# v8 and above
_process_module_definition(
"elastic_transport._transport",
"newrelic.hooks.datastore_elasticsearch",
"instrument_elastic_transport__transport",
)

_process_module_definition("pika.adapters", "newrelic.hooks.messagebroker_pika", "instrument_pika_adapters")
_process_module_definition("pika.channel", "newrelic.hooks.messagebroker_pika", "instrument_pika_channel")
Expand Down
Loading

0 comments on commit 1bf6a65

Please sign in to comment.