Tyk Pump

Tyk Pump is a pluggable analytics purger to move Analytics generated by your Tyk nodes to any back-end.

Table of Contents

• 1. Base Configuration
• 2. Supported Pumps
• 3. Base Pump Configurations
• 4. Developer & Contributer Guide

Pumps / Back ends supported:

• MongoDB & Tyk Dashboard
• CSV
• ElasticSearch (2.0+)
• Graylog
• InfluxDB
• InfluxDB2
• Moesif
• Splunk
• StatsD
• DogStatsD
• Hybrid (Tyk RPC)
• Prometheus
• Logz.io
• Kafka
• Stdout (i.e. for use by Datadog logging agent in Kubernetes)
• Timestream

Configuration:

This will be your base config. We will then add 1 or more Pumps based off our selected data sinks.

JSON / Conf File

Create a pump.conf file:

{
  "analytics_storage_type": "redis",
  "analytics_storage_config": {
    "type": "redis",
    "host": "localhost",
    "port": 6379,
    "hosts": null,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 100,
    "optimisation_max_active": 0,
    "enable_cluster": false,
    "redis_use_ssl": false,
    "redis_ssl_insecure_skip_verify": false
  },
  "log_level":"info",
  "log_format":"text",
  "purge_delay": 1,
  "health_check_endpoint_name": "hello",
  "health_check_endpoint_port": 8083,
  "pumps": {
    "csv": {
      "type": "csv",
      "meta": {
        "csv_dir": "./"
      }
    },
	...
  },
  "dont_purge_uptime_data": true,
  "omit_detailed_recording": false,
  "max_record_size": 1000
}

Env Variables

TYK_PMP_OMITCONFIGFILE=true

TYK_PMP_ANALYTICSSTORAGETYPE=redis
TYK_PMP_ANALYTICSSTORAGECONFIG_TYPE=redis
TYK_PMP_ANALYTICSSTORAGECONFIG_ADDRS=tyk-redis:6379
TYK_PMP_ANALYTICSSTORAGECONFIG_USERNAME=
TYK_PMP_ANALYTICSSTORAGECONFIG_PASSWORD=
TYK_PMP_ANALYTICSSTORAGECONFIG_DATABASE=0
TYK_PMP_ANALYTICSSTORAGECONFIG_MAXIDLE=100
TYK_PMP_ANALYTICSSTORAGECONFIG_MAXACTIVE=100
TYK_PMP_ANALYTICSSTORAGECONFIG_ENABLECLUSTER=false
TYK_PMP_PURGEDELAY=2

TYK_PMP_DONTPURGEUPTIMEDATA=true

Base Configuration Fields Explained

analytics_storage_config

This is the Tyk Pump's primary database which it scrapes Tyk Gateway analytics from. Normally this is redis.

  "analytics_storage_config": {
    "type": "redis",
    "host": "localhost",
    "port": 6379,
    "hosts": null,
    "username": "",
    "password": "",
    "database": 0,
    "optimisation_max_idle": 100,
    "optimisation_max_active": 0,
    "enable_cluster": false,
    "redis_use_ssl": false,
    "redis_ssl_insecure_skip_verify": false
  },

redis_use_ssl - Setting this to true to use SSL when connecting to Redis

redis_ssl_insecure_skip_verify - Set this to true to tell Pump to ignore Redis' cert validation

Purge configuration

purge_delay - The number of seconds the Pump waits between checking for analytics data and purge it from Redis.

purge_chunk - The maximum number of records to pull from Redis at a time. If it's unset or 0, all the analytics records in Redis are pulled. If it's setted, storage_expiration_time is used to reset the analytics record TTL.

storage_expiration_time - The number of seconds for the analytics records TTL. It only works if purge_chunk is enabled. Defaults to 60 seconds.

Logs

log_level - Set the logger details for tyk-pump. The posible values are: info,debug,error and warn. By default, the log level is info.

log_format - Set the logger format. The possible values are: text and json. By default, the log format is text.

Health Check

You can configure the health check endpoint and port for the Tyk Pump:

• health_check_endpoint_name - The default is "health"
• health_check_endpoint_port - The default port is 8083

This returns a HTTP 200 OK response if the Pump is running.

{"status": "ok"}

Pump Configurations

Uptime Data

dont_purge_uptime_data - Setting this to false will create a pump that pushes uptime data to (Mongo) Uptime Pump, so the Tyk Dashboard can read it. Disable by setting to true

Take into account that you can also set log_level field into the uptime_pump_config to debug,info or warning. By default, the SQL logger verbosity is silent.

Mongo Uptime Pump

In uptime_pump_config you can configure a mongo uptime pump. By default, the uptime pump is going to be mongo type, so it's not necessary to specify it here.

The minimum required configurations for uptime pumps are: collection_name - That determines the uptime collection name in mongo. By default, tyk_uptime_analytics. mongo_url - The uptime pump mongo connection url. It is usually something like "mongodb://username:password@{hostname:port},{hostname:port}/{db_name}".

JSON / Conf File

{
  "uptime_pump_config": {
    "collection_name": "tyk_uptime_analytics",
    "mongo_url": "mongodb://username:password@{hostname:port},{hostname:port}/{db_name}",
    "log_level": "info"
  }
}

Env Variables:

TYK_PMP_UPTIMEPUMPCONFIG_COLLECTIONNAME=tyk_uptime_analytics 
TYK_PMP_UPTIMEPUMPCONFIG_MONGOURL=mongodb://tyk-mongo:27017/tyk_analytics 
TYK_PMP_UPTIMEPUMPCONFIG_MAXINSERTBATCHSIZEBYTES=500000 
TYK_PMP_UPTIMEPUMPCONFIG_MAXDOCUMENTSIZEBYTES=200000 
TYK_PMP_UPTIMEPUMPCONFIG_LOGLEVEL=info

SQL Uptime Pump

Supported in Tyk Pump v1.5.0+

In uptime_pump_config you can configure a SQL uptime pump. To do that, you need to add the field uptime_type with sql value. You can also use different types of SQL Uptime pumps, like postgres or sqlite using the type field.

JSON / Conf file Example

    "uptime_pump_config": {
        "uptime_type": "sql",
        "type": "postgres",
        "connection_string": "host=sql_host port=sql_port user=sql_usr dbname=dbname password=sql_pw",
        "table_sharding": false,
	"log_level": "info"
    },

Env Variables:

TYK_PMP_UPTIMEPUMPCONFIG_UPTIMETYPE=sql
TYK_PMP_UPTIMEPUMPCONFIG_TYPE=postgres
TYK_PMP_UPTIMEPUMPCONFIG_CONNECTIONSTRING=host=sql_host port=sql_port user=sql_usr dbname=dbname password=sql_pw
TYK_PMP_UPTIMEPUMPCONFIG_TABLESHARDING=false
TYK_PMP_UPTIMEPUMPCONFIG_LOGLEVEL=info

GrayLog

Example of integrating with GrayLog:

JSON / Conf file Example

"graylog": {
      "type": "graylog",
      "meta": {
        "host": "10.60.6.15",
        "port": 12216,
        "tags": [
          "method",
          "path",
          "response_code",
          "api_key",
          "api_version",
          "api_name",
          "api_id",
          "org_id",
          "oauth_id",
          "raw_request",
          "request_time",
          "raw_response",
          "ip_address"
        ]
      }
    },

Env Variables:

TYK_PMP_PUMPS_GRAYLOG_TYPE=graylog
TYK_PMP_PUMPS_GRAYLOG_META_GRAYLOGHOST=10.60.6.15
TYK_PMP_PUMPS_GRAYLOG_META_GRAYLOGPORT=12216
TYK_PMP_PUMPS_GRAYLOG_META_TAGS=method,path,response_code,api_key,api_version,api_name,api_id,org_id,oauth_id,raw_request,request_time,raw_response,ip_address

StatsD

Example of integrating with StatsD:

JSON / Conf file Example

{
  "pumps": {
    "statsd": {
      "type": "statsd",
      "meta": {
        "address": "localhost:8125",
        "fields": [
          "request_time"
        ],
        "tags": [
          "path",
          "response_code",
          "api_key",
          "api_version",
          "api_name",
          "api_id",
          "raw_request",
          "ip_address",
          "org_id",
          "oauth_id"
        ]
      }
    }
  }
}

By default, StatsD pump will put the analytic record method and path in your path field. From Pump 1.6+ you can set separated_method to true in your Statsd pump meta config in order to have the method attribute in a separated field.

Env Variables:

TYK_PMP_PUMPS_STATSD_TYPE=statsd
TYK_PMP_PUMPS_STATSD_META_ADDRESS="localhost:8125"
TYK_PMP_PUMPS_STATSD_META_FIELDS=request_time
TYK_PMP_PUMPS_STATSD_META_TAGS=path,response_code,api_key,api_version,api_name,api_id,org_id,oauth_id,raw_request,ip_address
TYK_PMP_PUMPS_STATSD_META_SEPARATEDMETHOD=false

Mongo & Tyk Dashboard.

There are 3 mongo pumps. You may use one or multiple depending on the data you want.

The Tyk Dashboard uses various Mongo collections to store and visualize API traffic analytics. Please visit this link for steps on configuration.

JSON / Conf File

{
  ...
  "pumps": {
    "mongo": {
      "type": "mongo",
      "meta": {
        "collection_name": "tyk_analytics",
        "mongo_url": "mongodb://username:password@{hostname:port},{hostname:port}/{db_name}"
      }
    },
    "mongo-pump-aggregate": {
      "type": "mongo-pump-aggregate",
      "meta": {
        "mongo_url": "mongodb://username:password@{hostname:port},{hostname:port}/{db_name}",
        "use_mixed_collection": true,
        "store_analytics_per_minute": false,
        "track_all_paths": false
      },
      "mongo-pump-selective": {
        "name": "mongo-pump-selective",
        "meta": {
          "mongo_url": "mongodb://tyk-mongo:27017/tyk_analytics",
          "use_mixed_collection": true
        }
      }
    }
  }
}

Env Variables

TYK_PMP_PUMPS_MONGO_TYPE=mongo
TYK_PMP_PUMPS_MONGO_META_COLLECTIONNAME=tyk_analytics
TYK_PMP_PUMPS_MONGO_META_MAXINSERTBATCHSIZEBYTES=80000
TYK_PMP_PUMPS_MONGO_META_MAXDOCUMENTSIZEBYTES=20112

TYK_PMP_PUMPS_MONGOAGG_TYPE=mongo-pump-aggregate
TYK_PMP_PUMPS_MONGOAGG_META_USEMIXEDCOLLECTION=true

Elasticsearch Config

"index_name" - The name of the index that all the analytics data will be placed in. Defaults to "tyk_analytics"

"elasticsearch_url" - If sniffing is disabled, the URL that all data will be sent to. Defaults to "http://localhost:9200"

"enable_sniffing" - If sniffing is enabled, the "elasticsearch_url" will be used to make a request to get a list of all the nodes in the cluster, the returned addresses will then be used. Defaults to false

"document_type" - The type of the document that is created in ES. Defaults to "tyk_analytics"

"rolling_index" - Appends the date to the end of the index name, so each days data is split into a different index name. E.g. tyk_analytics-2016.02.28 Defaults to false

"extended_stats" - If set to true will include the following additional fields: Raw Request, Raw Response and User Agent.

"version" - Specifies the ES version. Use "3" for ES 3.X, "5" for ES 5.X, "6" for ES 6.X, "7" for ES 7.X . Defaults to "3".

"disable_bulk" - Disable batch writing. Defaults to false.

bulk_config: Batch writing trigger configuration. Each option is an OR with eachother:

• workers: Number of workers. Defaults to 1.
• flush_interval: Specifies the time in seconds to flush the data and send it to ES. Default disabled.
• bulk_actions: Specifies the number of requests needed to flush the data and send it to ES. Defaults to 1000 requests. If it is needed, can be disabled with -1.
• bulk_size: Specifies the size (in bytes) needed to flush the data and send it to ES. Defaults to 5MB. If it is needed, can be disabled with -1.

Env Variables

TYK_PMP_PUMPS_ELASTICSEARCH_TYPE=elasticsearch
TYK_PMP_PUMPS_ELASTICSEARCH_META_INDEXNAME=tyk_analytics
TYK_PMP_PUMPS_ELASTICSEARCH_META_ELASTICSEARCHURL=http://localhost:9200
TYK_PMP_PUMPS_ELASTICSEARCH_META_ENABLESNIFFING=false
TYK_PMP_PUMPS_ELASTICSEARCH_META_DOCUMENTTYPE=tyk_analytics
TYK_PMP_PUMPS_ELASTICSEARCH_META_ROLLINGINDEX=false
TYK_PMP_PUMPS_ELASTICSEARCH_META_EXTENDEDSTATISTICS=false
TYK_PMP_PUMPS_ELASTICSEARCH_META_VERSION=5
TYK_PMP_PUMPS_ELASTICSEARCH_META_BULKCONFIG_WORKERS=2
TYK_PMP_PUMPS_ELASTICSEARCH_META_BULKCONFIG_FLUSHINTERVAL=60

Moesif Config

Moesif is a user-centric API analytics and monitoring service for APIs. More Info on Moesif for Tyk

• "application_id" - Moesif Application Id. You can find your Moesif Application Id from Moesif Dashboard -> Top Right Menu -> API Keys . Moesif recommends creating separate Application Ids for each environment such as Production, Staging, and Development to keep data isolated.
• "request_header_masks" - (optional) An option to mask a specific request header field. Type: String Array [] string
• "request_body_masks" - (optional) An option to mask a specific - request body field. Type: String Array [] string
• "response_header_masks" - (optional) An option to mask a specific response header field. Type: String Array [] string
• "response_body_masks" - (optional) An option to mask a specific response body field. Type: String Array [] string
• "disable_capture_request_body" - (optional) An option to disable logging of request body. Type: Boolean. Default value is false.
• "disable_capture_response_body" - (optional) An option to disable logging of response body. Type: Boolean. Default value is false.
• "user_id_header" - (optional) An optional field name to identify User from a request or response header. Type: String.
• "company_id_header" - (optional) An optional field name to identify Company (Account) from a request or response header. Type: String.
• "authorization_header_name" - (optional) An optional request header field name to used to identify the User in Moesif. Type: String. Default value is authorization.
• "authorization_user_id_field" - (optional) An optional field name use to parse the User from authorization header in Moesif. Type: String. Default value is sub.
• "enable_bulk" - Set this to true to enable bulk_config.
• "bulk_config"- (optional) Batch writing trigger configuration.
  • "event_queue_size" - (optional) An optional field name which specify the maximum number of events to hold in queue before sending to Moesif. In case of network issues when not able to connect/send event to Moesif, skips adding new events to the queue to prevent memory overflow. Type: int. Default value is 10000.
  • "batch_size" - (optional) An optional field name which specify the maximum batch size when sending to Moesif. Type: int. Default value is 200.
  • "timer_wake_up_seconds" - (optional) An optional field which specifies a time (every n seconds) how often background thread runs to send events to moesif. Type: int. Default value is 2 seconds.

Env Variables

TYK_PMP_PUMPS_MOESIF_TYPE=moesif
TYK_PMP_PUMPS_MOESIF_META_APPLICATIONID=""

Hybrid RPC Config

Hybrid Pump allows you to install Tyk Pump inside Multi-Cloud or MDCB Worker installations. You can configure Tyk Pump to send data to the source of your choice (i.e. ElasticSearch), and in parallel, forward analytics to the Tyk Cloud. Additionally, you can set the aggregated flag to send only aggregated analytics to MDCB or Tyk Cloud, in order to save network bandwidth between DCs.

NOTE: Make sure your tyk.conf has analytics_config.type set to empty string value.

rpc_key - Put your organization ID in this field.

api_key - This the API key of a user used to authenticate and authorise the Gateway’s access through MDCB. The user should be a standard Dashboard user with minimal privileges so as to reduce risk if compromised. The suggested security settings are read for Real-time notifications and the remaining options set to deny.

aggregated - Set this field to true to send only aggregated analytics to MDCB or Tyk Cloud.

connection_string - The MDCB instance or load balancer.

use_ssl - Set this field to true if you need secured connection (default value is false).

ssl_insecure_skip_verify - Set this field to true if you use self signed certificate.

group_id - This is the “zone” that this instance inhabits, e.g. the DC it lives in. It must be unique to each slave cluster / DC.

call_timeout - This is the timeout (in milliseconds) for RPC calls.

rpc_pool_size - This is maximum number of connections to MDCB.

Env Variables

TYK_PMP_PUMPS_HYBRID_TYPE=hybrid
TYK_PMP_PUMPS_HYBRID_META_RPCKEY=5b5fd341e6355b5eb194765e
TYK_PMP_PUMPS_HYBRID_META_APIKEY=008d6d1525104ae77240f687bb866974
TYK_PMP_PUMPS_HYBRID_META_CONNECTIONSTRING=localhost:9090
TYK_PMP_PUMPS_HYBRID_META_AGGREGATED=false
TYK_PMP_PUMPS_HYBRID_META_USESSL=false
TYK_PMP_PUMPS_HYBRID_META_SSLINSECURESKIPVERIFY=false
TYK_PMP_PUMPS_HYBRID_META_GROUPID=""
TYK_PMP_PUMPS_HYBRID_META_CALLTIMEOUT=30
TYK_PMP_PUMPS_HYBRID_META_PINGTIMEOUT=60
TYK_PMP_PUMPS_HYBRID_META_RPCPOOLSIZE=30

Prometheus

Prometheus is an open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

Note - When run as docker image then "listen_address": ":9090"

Tyk expose the following counters:

• tyk_http_status{code, api}
• tyk_http_status_per_path{code, api, path, method}
• tyk_http_status_per_key{code, key}
• tyk_http_status_per_oauth_client{code, client_id}

And the following Histogram for latencies:

• tyk_latency{type, api}

Custom Prometheus metrics

From Pump 1.6+ it's possible to add custom prometheus metrics using the custom_metrics configuration. For example:

"prometheus": {
  "type": "prometheus",
	"meta": {
		"listen_address": "localhost:9090",
		"path": "/metrics",
		"custom_metrics":[
      {
        "name":"tyk_custom_http_status_per_api_name",
        "description":"This is a custom counter",
        "metric_type":"counter",
        "labels":["response_code","api_name"]
      }
    ]
	}
},

This will create a metric for HTTP status code and API name.

There are 2 types of metric_type: counter and histogram.

If you are using histogram, its always going to use the request_time to observe, and you can also set the configuration option buckets where you can define the buckets into which observations are counted. buckets type is an array of float64 and its default value is [1, 2, 5, 7, 10, 15, 20, 25, 30, 40, 50, 60, 70, 80, 90, 100, 200, 300, 400, 500, 1000, 2000, 5000, 10000, 30000, 60000].

The labels configuration determines the label name and value extracted from the analytic record. The available values are: ["host","method", "path", "response_code", "api_key", "time_stamp", "api_version", "api_name", "api_id", "org_id", "oauth_id","request_time", "ip_address"]

JSON / Conf File

{
    ...
    "pumps": {
      "prometheus": {
        "type": "prometheus",
        "meta": {
          "listen_address": "localhost:9090",
          "path": "/metrics"
        }
      }    
    }
}

Env Variables

TYK_PMP_PUMPS_PROMETHEUS_TYPE=prometheus
TYK_PMP_PUMPS_PROMETHEUS_META_ADDR=localhost:9090
TYK_PMP_PUMPS_PROMETHEUS_META_PATH=/metrics
TYK_PMP_PUMPS_PROMETHEUS_META_CUSTOMMETRICS=[]

DogStatsD

• address: address of the datadog agent including host & port
• namespace: prefix for your metrics to datadog
• async_uds: Enable async UDS over UDP https://github.com/Datadog/datadog-go#unix-domain-sockets-client
• async_uds_write_timeout_seconds: Integer write timeout in seconds if async_uds: true
• buffered: Enable buffering of messages
• buffered_max_messages: Max messages in single datagram if buffered: true. Default 16
• sample_rate: default 1 which equates to 100% of requests. To sample at 50%, set to 0.5
• tags: List of tags to be added to the metric. The possible options are listed in the below example

If no tag is specified the fallback behavior is to use the below tags:

• path
• method
• response_code
• api_version
• api_name
• api_id
• org_id
• tracked
• oauth_id

Note that this configuration can generate significant charges due to the unbound nature of the path tag.

"dogstatsd": {
  "type": "dogstatsd",
  "meta": {
    "address": "localhost:8125",
    "namespace": "pump",
    "async_uds": true,
    "async_uds_write_timeout_seconds": 2,
    "buffered": true,
    "buffered_max_messages": 32,
    "sample_rate": 0.5,
    "tags": [
      "method",
      "response_code",
      "api_version",
      "api_name",
      "api_id",
      "org_id",
      "tracked",
      "path",
      "oauth_id"
    ]
  }
},

On startup, you should see the loaded configs when initializing the dogstatsd pump

[May 10 15:23:44]  INFO dogstatsd: initializing pump
[May 10 15:23:44]  INFO dogstatsd: namespace: pump.
[May 10 15:23:44]  INFO dogstatsd: sample_rate: 50%
[May 10 15:23:44]  INFO dogstatsd: buffered: true, max_messages: 32
[May 10 15:23:44]  INFO dogstatsd: async_uds: true, write_timeout: 2s

Env Variables

TYK_PMP_PUMPS_DOGSTATSD_TYPE=dogstatsd
TYK_PMP_PUMPS_DOGSTATSD_META_ADDRESS=localhost:8125
TYK_PMP_PUMPS_DOGSTATSD_META_NAMESPACE=pump
TYK_PMP_PUMPS_DOGSTATSD_META_ASYNCUDS=true
TYK_PMP_PUMPS_DOGSTATSD_META_ASYNCUDSWRITETIMEOUT=2
TYK_PMP_PUMPS_DOGSTATSD_META_BUFFERED=true
TYK_PMP_PUMPS_DOGSTATSD_META_BUFFEREDMAXMESSAGES=32
TYK_PMP_PUMPS_DOGSTATSD_META_TAGS=method,response_code,api_version,api_name,api_id,org_id,tracked,path,oauth_id

Splunk

Setting up Splunk with a HTTP Event Collector

• collector_token: address of the datadog agent including host & port
• collector_url: endpoint the Pump will send analytics too. Should look something like:

https://splunk:8088/services/collector/event

• ssl_insecure_skip_verify: Controls whether the pump client verifies the Splunk server's certificate chain and host name.
• obfuscate_api_keys: (optional) Controls whether the pump client should hide the API key. In case you still need substring of the value, check the next option. Type: Boolean. Default value is false.
• obfuscate_api_keys_length: (optional) Define the number of the characters from the end of the API key. The obfuscate_api_keys should be set to true. Type: Integer. Default value is 0.
• fields: (optional) Define which Analytics fields should participate in the Splunk event. Check the available fields in the example below. Type: String Array [] string. Default value is ["method", "path", "response_code", "api_key", "time_stamp", "api_version", "api_name", "api_id", "org_id", "oauth_id", "raw_request", "request_time", "raw_response", "ip_address"]
• ignore_tag_prefix_list: (optional) Choose which tags to be ignored by the Splunk Pump. Keep in mind that the tag name and value are hyphenated. Type: Type: String Array [] string. Default value is []
• enable_batch: If this is set to true, pump is going to send the analytics records in batch to Splunk. Type: Boolean. Default value is false.
• max_content_length: Max content length in bytes to be sent in batch requests. It should match the max_content_length configured in Splunk. If the purged analytics records size don't reach the amount of bytes, they're send anyways in each purge_loop. Type: Integer. Default value is 838860800 (~ 800 MB), the same default value as Splunk config.

JSON / Conf File

    "splunk": {
      "type": "splunk",
      "meta": {
        "collector_token": "<token>",
        "collector_url": "<url>",
        "ssl_insecure_skip_verify": false,
        "ssl_cert_file": "<cert-path>",
        "ssl_key_file": "<key-path>",
        "ssl_server_name": "<server-name>",
        "obfuscate_api_keys": true,
        "obfuscate_api_keys_length": 10,
        "enable_batch":true,
        "fields": [
          "method",
          "host",
          "path",
          "raw_path",
          "content_length",
          "user_agent",
          "response_code",
          "api_key",
          "time_stamp",
          "api_version",
          "api_name",
          "api_id",
          "org_id",
          "oauth_id",
          "raw_request",
          "request_time",
          "raw_response",
          "ip_address",
          "geo",
          "network",
          "latency",
          "tags",
          "alias",
          "track_path"
        ],
        "ignore_tag_prefix_list": [
          "key-",
          "org-",
          "api-",
          "original-path-",
        ]
      }
    },

Env Variables

TYK_PMP_PUMPS_SPLUNK_TYPE=splunk
TYK_PMP_PUMPS_SPLUNK_META_COLLECTORTOKEN="{TOKEN}"
TYK_PMP_PUMPS_SPLUNK_META_COLLECTORURL="{URL}"
TYK_PMP_PUMPS_SPLUNK_META_SSLINSECURESKIPVERIFY=false
TYK_PMP_PUMPS_SPLUNK_META_SSLCERTFILE="{CERT-PATH}"
TYK_PMP_PUMPS_SPLUNK_META_SSLKEYFILE="{KEY-PATH}"
TYK_PMP_PUMPS_SPLUNK_META_SSLSERVERNAME="{SERVER-NAME}"
TYK_PMP_PUMPS_SPLUNK_META_ENABLEBATCH=true
TYK_PMP_PUMPS_SPLUNK_META_BATCHMAXCONTENTLENGTH="{MAX-CONTENT-LENGTH}"

Logzio Config

Logz.io is a cloud observability platform providing Log Management built on ELK, Infrastructure Monitoring based on Grafana, and an ELK-based Cloud SIEM.

The following configuration values are available:

Example simplest configuration just needs the token for sending data to your logzio account.

JSON / Conf File

"logzio": {
      "type": "logzio",
      "meta": {
        "token": "<YOUR-LOGZ.IO-TOKEN>"
      }
    }

Env Variables

TYK_PMP_PUMPS_LOGZIO_TYPE=logzio
TYK_PMP_PUMPS_LOGZIO_META_TOKEN="{YOUR-LOGZIO-TOKEN}"

More advanced fields:

meta.url - If you do not want to use the default Logzio url i.e. when using a proxy. Default is https://listener.logz.io:8071 meta.queue_dir - The directory for the queue. meta.drain_duration - Set drain duration (flush logs on disk). Default value is 3s meta.disk_threshold - Set disk queue threshold, once the threshold is crossed the sender will not enqueue the received logs. Default value is 98 (percentage of disk). meta.check_disk_space - Set the sender to check if it crosses the maximum allowed disk usage. Default value is true.

Kafka Config

• broker: The list of brokers used to discover the partitions available on the kafka cluster. E.g. "localhost:9092"
• use_ssl: Enables SSL connection.
• ssl_insecure_skip_verify: Controls whether the pump client verifies the kafka server's certificate chain and host name.
• client_id: Unique identifier for client connections established with Kafka.
• topic: The topic that the writer will produce messages to.
• timeout: Timeout is the maximum amount of time will wait for a connect or write to complete.
• compressed: Enable "github.com/golang/snappy" codec to be used to compress Kafka messages. By default is false
• meta_data: Can be used to set custom metadata inside the kafka message
• ssl_cert_file: Can be used to set custom certificate file for authentication with kafka.
• ssl_key_file: Can be used to set custom key file for authentication with kafka.

JSON / Conf File

{
    ...
    "pumps": {
      "kafka": {
      "type": "kafka",
      "meta": {
        "broker": [
            "localhost:9092"
        ],
	"topic": "tyk-pump",
        "use_ssl": true,
        "ssl_insecure_skip_verify": false,
        "client_id": "tyk-pump",
        "timeout": 60,
        "compressed": true,
        "meta_data": {
            "key": "value"
        }
      }
    }
}

Env Variables

TYK_PMP_PUMPS_KAFKA_TYPE=kafka
TYK_PMP_PUMPS_KAFKA_META_BROKER=localhost:9092
TYK_PMP_PUMPS_KAFKA_META_TOPIC=tyk-pump
TYK_PMP_PUMPS_KAFKA_META_USESSL=true
TYK_PMP_PUMPS_KAFKA_META_SSLINSECURESKIPVERIFY=false
TYK_PMP_PUMPS_KAFKA_META_CLIENTID=tyk-pump
TYK_PMP_PUMPS_KAFKA_META_TIMEOUT=60
TYK_PMP_PUMPS_KAFKA_META_COMPRESSED=true
TYK_PMP_PUMPS_KAFKA_META_METADATA_KEY=value

Influx2 Config

Supported in Tyk Pump v1.5.1+

This pump uses the official Go client library for InfluxDB 2.x.

Configuration options:

• "organization" - InfluxDB organization name.
• "bucket" - InfluxDB bucket where the analytic data is going to be stored.
• "create_missing_bucket" - Set this to true if you want to create the bucket if not exists. Defaults to false.
• "new_bucket_config" - If "create_missing_bucket"is true, you can configure the new bucket configuration under "new_bucket_config":
  • "description" - Description of the bucket. This is going to be visible in the Influx UI.
  • "retention_rules"- This is a slice of retention rules for this bucket. An example of this would be:

  "retention_rules":[
  {
  "every_seconds":100000,
  "type":"expires"
  }
  ]

  which would mean that the data in the bucket expires every 100000 seconds.
• "token" - Influx DB Auth token
• "tags" - Which elements should work as a tag for the time series.

JSON / Conf File

"influx2": {
    "type": "influx2",
    "meta": {
      "organization": "my-org",
      "bucket": "my-bucket",
      "address": "http://localhost:8086",
      "token": "my-super-secret-auth-token",
      "fields": ["request_time"],
      "tags": [
        "api_id",
        "api_key",
        "api_name",
        "api_version",
        "ip_address",
        "method",
        "oauth_id",
        "org_id",
        "path",
        "raw_request",
        "response_code"
      ]
    }
}

Env Variables

TYK_PMP_PUMPS_INFLUX_TYPE=influx2
TYK_PMP_PUMPS_INFLUX_META_ORGANIZATION=myorg
TYK_PMP_PUMPS_INFLUX_META_BUCKET=tyk_analytics
TYK_PMP_PUMPS_INFLUX_META_ADDR=http://localhost:8086
TYK_PMP_PUMPS_INFLUX_META_TOKEN=ZzSnH2qRpEJd3ph3A6lPaCcP8BkJfaxeiadFG5DBMO8YIAn3mMzGunMqOQE2uPkAkewXE5Q6Gsye3vQTWmeTiQ==
TYK_PMP_PUMPS_INFLUX_META_CREATEMISSINGBUCKET=true
TYK_PMP_PUMPS_INFLUX_META_NEWBUCKETCONFIG_DESCRIPTION="Tyk gateway requests"
TYK_PMP_PUMPS_INFLUX_META_NEWBUCKETCONFIG_RETENTIONRULES_EVERYSECONDS=3600
TYK_PMP_PUMPS_INFLUX_META_FIELDS=request_time
TYK_PMP_PUMPS_INFLUX_META_TAGS=path,response_code,api_key,api_version,api_name,api_id,raw_request,ip_address,org_id,oauth_id

Syslog

Supported in Tyk Pump v1.0.0+

"transport" - Possible values are udp, tcp, tls in string form

"network_addr" - Host & Port combination of your syslog daemon ie: "localhost:5140"

"log_level" - The severity level, an integer from 0-7, based off the Standard: Syslog Severity Levels

"tag" - Prefix tag

When working with FluentD, you should provide a FluentD Parser based on the OS you are using so that FluentD can correctly read the logs

"syslog": {
  "name": "syslog",
  "meta": {
    "transport": "udp",
    "network_addr": "localhost:5140",
    "log_level": 6,
    "tag": "syslog-pump"
  }

Stdout

log_field_name - Root name of the JSON object the analytics record is nested in

format - Format of the analytics logs. Default is text if json is not explicitly specified. When JSON logging is used all pump logs to stdout will be JSON.

JSON / Conf File

"stdout": {
   "type": "stdout",
    "meta": {
      "log_field_name": "tyk-analytics-record",
      "format": "json"
    }
  }

Env Variables

TYK_PMP_PUMPS_STDOUT_TYPE=stdout
TYK_PMP_PUMPS_STDOUT_META_LOGFIELDNAME=tyk-analytics-record
TYK_PMP_PUMPS_STDOUT_META_FORMAT=json

SQL Pump

Supported in Tyk Pump v1.5.0+

type - The supported and tested types are sqlite and postgres. connection_string - Specifies the connection string to the database. For example, for sqlite it would usually work specifying the path/name of the database and for postgres, specifying the host, port, user, password and dbname. log_level - Specifies the SQL log verbosity. The possible values are: info,error and warning. By default, the value is silent, which means that it won't log any SQL query. table_sharding - Specifies if all the analytics records are going to be stored in one table or in multiple tables (one per day). By default, false. If table_sharding is false, all the records are going to be stored in tyk_analytics table. Instead, if it's true, all the records of the day are going to be stored in tyk_analytics_YYYYMMDD table, where YYYYMMDD is going to change depending on the date. batch_size - Specifies the amount of records that are going to be written each batch. Type int. By default, it writes 1000 records max per batch.

JSON / Conf File

    "sql": {
        "name": "sql",
        "meta": {
            "type": "postgres",
            "connection_string": "host=localhost port=5432 user=admin dbname=postgres_test password=test",
            "table_sharding": false
        }
    }

Env Variables

TYK_PMP_PUMPS_SQL_NAME=sql
TYK_PMP_PUMPS_SQL_META_TYPE=postgres
TYK_PMP_PUMPS_SQL_META_CONNECTIONSTRING="host=sql_host port=sql_port user=sql_usr dbname=dbname password=sql_pw"
TYK_PMP_PUMPS_SQL_META_TABLESHARDING=false

SQL Aggregate Pump

Supported in Tyk Pump v1.5.0+

type - The supported and tested types are sqlite and postgres. connection_string - Specifies the connection string to the database. For example, for sqlite it would usually work specifying the path/name of the database and for postgres, specifying the host, port, user, password and dbname. log_level - Specifies the SQL log verbosity. The possible values are: info,error and warning. By default, the value is silent, which means that it won't log any SQL query. track_all_paths - Specifies if it should store aggregated data for all the endpoints. By default, false which means that only store aggregated data for tracked endpoints. ignore_tag_prefix_list - Specifies prefixes of tags that should be ignored. table_sharding - Specifies if all the analytics records are going to be stored in one table or in multiple tables (one per day). By default, false. If table_sharding is false, all the records are going to be stored in tyk_aggregated table. Instead, if it's true, all the records of the day are going to be stored in tyk_aggregated_YYYYMMDD table, where YYYYMMDD is going to change depending on the date. batch_size - Specifies the amount of records that are going to be written each batch. Type int. By default, it writes 1000 records max per batch.

JSON / Conf File

    "sql_aggregate": {
        "name": "sql_aggregate",
        "meta": {
            "type": "postgres",
            "connection_string": "host=localhost port=5432 user=admin dbname=postgres_test password=test",
            "table_sharding": false
        }
    }

Env Variables

TYK_PMP_PUMPS_SQLAGGREGATE_TYPE=sql_aggregate 
TYK_PMP_PUMPS_SQLAGGREGATE_META_TYPE=postgres 
TYK_PMP_PUMPS_SQLAGGREGATE_META_CONNECTIONSTRING=host=sql_host port=sql_port user=sql_usr dbname=dbname password=sql_pw 
TYK_PMP_PUMPS_SQLAGGREGATE_META_TABLESHARDING=true 

Timestream Config

Authentication & Prerequisite

We must authenticate ourselves by providing credentials to AWS. This pump uses the official AWS GO SDK, so instructions on how to authenticate can be found on their documentation here.

Config Fields

write_rate_limit - Set to true in order to save any of the RateLimit measures, which are extracted from the headers of the raw response.

read_geo_from_request - If set true, we will try to read geo information from the headers of the raw request

write_zero_values - If set to false, numerical values with value zero, won't be recorded

When you initialize a Timestream Pump, the SDK uses its default credential chain to find AWS credentials. This default credential chain looks for credentials in the following order:

• Environment variables.
  • Static Credentials (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
  • Web Identity Token (AWS_WEB_IDENTITY_TOKEN_FILE)
• Shared configuration files.
  • SDK defaults to credentials file under .aws folder that is placed in the home folder on your computer.
• If your application uses an ECS task definition or RunTask API operation, IAM role for tasks.
• If your application is running on an Amazon EC2 instance, IAM role for Amazon EC2.

If no credentials are provided, Timestream Pump won't be able to connect.

JSON / Conf File

    "timestream": {
      "type": "timestream",
      "meta": {
        "aws_region": "us-east-1",
        "timestream_table_name": "tyk-pump-table",
        "timestream_database_name": "tyk-pump",
        "write_rate_limit": true,
        "read_geo_from_request": true,
        "write_zero_values": false,
        "dimensions": [
          "Method",
          "Host",
          "Path",
          "RawPath",
          "APIKey",
          "APIVersion",
          "APIName",
          "APIID",
          "OrgID",
          "OauthID"
        ],
        "measures": [
          "ContentLength",
          "ResponseCode",
          "RequestTime",
          "NetworkStats.OpenConnections",
          "NetworkStats.ClosedConnection",
          "NetworkStats.BytesIn",
          "NetworkStats.BytesOut",
          "Latency.Total",
          "Latency.Upstream",
          "RateLimit.Limit",
          "Ratelimit.Remaining",
          "Ratelimit.Reset",
          "UserAgent",
          "RawRequest",
          "RawResponse",
          "IPAddress",
          "GeoData.Country.ISOCode",
          "GeoData.City.Names",
          "GeoData.Location.TimeZone",
          "GeoData.City.GeoNameID",
          "GeoData.Location.Latitude",
          "GeoData.Location.Longitude"
        ],
        "field_name_mappings":{
          "Path": "path",
          "APIKey": "api_key",
          "APIVersion": "module",
          "APIName": "email",
          "Method": "method",
          "APIID": "account",
          "measure_name": "request_metrics",
          "time": "time",
          "ResponseCode": "response_code",
          "GeoData.Country.ISOCode": "country_code",
          "Latency.Total": "latency_total",
          "RawResponseSize": "sesponse_size",
          "RequestTime": "request_time",
          "GeoData.City.Names": "city",
          "Latency.Upstream": "latency_upstream",
          "UserAgent": "user_agent",
          "IPAddress": "ip_address",
          "RateLimit.Limit": "quota_max",
          "Ratelimit.Remaining": "quota_remaining",
          "Ratelimit.Reset": "quota_renewal_rate"
        }
      }
    },

Env Variables

TYK_PMP_PUMPS_TIMESTREAM_TYPE=timestream
TYK_PMP_PUMPS_TIMESTREAM_META_AWSREGION=us-east-1
TYK_PMP_PUMPS_TIMESTREAM_META_TIMESTREAMTABLENAME=tyk-pump-table
TYK_PMP_PUMPS_TIMESTREAM_META_TIMESTREAMDATABASENAME=tyk-pump
TYK_PMP_PUMPS_TIMESTREAM_META_WRITERATELIMIT=true
TYK_PMP_PUMPS_TIMESTREAM_META_READGEOFROMREQUEST=true
TYK_PMP_PUMPS_TIMESTREAM_META_WRITEZEROVALUES=true
TYK_PMP_PUMPS_TIMESTREAM_META_DIMENSIONS=Method,Host,Path,APIKey
TYK_PMP_PUMPS_TIMESTREAM_META_MEASURES=ResponseCode,RequestTime,Latency.Total,RateLimit.Limit,RateLimit.Remaining,RateLimit.Reset,UserAgent,IPAddress,GeoData.Country.ISOCode,GeoData.City.Names
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_PATH=path
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_APIKEY=api_key
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_METHOD=method
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_HOST=host
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_RESPONSECODE=response_code
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_REQUESTTIME=request_time
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_LATENCY_TOTAL=latency_total
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_GEODATA_COUNTRY_ISOCODE=country_code
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_GEODATA_CITY_NAMES=city
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_USERAGENT=user_agent
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_IPADDRESS=ip_address
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_RATELIMIT_LIMIT=quota_max
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_RATELIMIT_REMAINING=quota_remaining
TYK_PMP_PUMPS_TIMESTREAM_META_FIELDNAMEMAPPINGS_RATELIMIT_RESET=quota_renewal_rate

CSV Config

Enable this Pump to have Tyk Pump create or modify a CSV file to track API Analytics.

JSON / Conf File

    "csv": {
      "type": "csv",
      "meta": {
        "csv_dir": "./"
      }
    },

Env Variables

TYK_PMP_PUMPS_CSV_TYPE=csv
TYK_PMP_PUMPS_CSV_META_CSVDIR=./

Base Pump Configurations

The following configurations can be added to any Pump. Keep reading for an example.

Filter Records

You made add the following config field to each pump called filters and its structure is the following:

"filters":{
  "api_ids":[],
  "org_ids":[],
  "response_codes":[],
  "skip_api_ids":[],
  "skip_org_ids":[],
  "skip_response_codes":[]
}

The fields api_ids, org_ids and response_codes works as allow list (APIs and orgs where we want to send the analytics records) and the fields skip_api_ids, skip_org_ids and skip_response_codes works as block list.

The priority is always block list configurations over allow list.

Here we see how we can take a CSV Pump, and add a filters section to it:

JSON / Conf file Example

"csv": {
 "type": "csv",
 "filters": {
   "api_ids": ["123","789"]
 },
 "meta": {
   "csv_dir": "./bar"
 }
}

Env variables

TYK_PMP_PUMPS_CSV_TYPE=csv 
TYK_PMP_PUMPS_CSV_META_CSVDIR=./bar
TYK_PMP_PUMPS_CSV_FILTERS_APIIDS=123,789

Timeouts

You can configure a different timeout for each pump with the configuration option timeout. Its default value is 0 seconds, which means that the pump will wait for the writing operation forever.

JSON / Conf file Example

"mongo": {
  "type": "mongo",
  "timeout":5,
  "meta": {
    "collection_name": "tyk_analytics",
    "mongo_url": "mongodb://username:password@{hostname:port},{hostname:port}/{db_name}"
  }
}

Env variables

TYK_PMP_PUMPS_MONGO_TYPE=mongo
TYK_PMP_PUMPS_MONGO_TIMEOUT=5
...

In case that any pump doesn't have a configured timeout, and it takes more seconds to write than the value configured for the purge loop in the purge_delay config option, you will see the following warning message: Pump PMP_NAME is taking more time than the value configured of purge_delay. You should try to set a timeout for this pump..

In case that you have a configured timeout, but it still takes more seconds to write than the value configured for the purge loop in the purge_delay config option, you will see the following warning message: Pump PMP_NAME is taking more time than the value configured of purge_delay. You should try lowering the timeout configured for this pump..

Omit Detailed Recording

omit_detailed_recording - Setting this to true on a Pump will avoid writing raw_request and raw_response fields for each request in pumps. Defaults to false.

Max Record Size

max_record_size defines maximum size (in bytes) for Raw Request and Raw Response logs, this value defaults to 0. Is not set then tyk-pump will not trim any data and will store the full information. This can also be set at a pump level. For example:

"csv": {
  "type": "csv",
  "max_record_size":1000,
  "meta": {
    "csv_dir": "./"
  }
}

Compiling & Testing

1. Download dependent packages:

go get -t -d -v ./...

2. Compile:

go build -v ./...

3. Test

go test -v ./...