-
Notifications
You must be signed in to change notification settings - Fork 15
/
dev.txt
119 lines (91 loc) · 5.13 KB
/
dev.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
# comments
https://github.com/cakesolutions/scala-kafka-client/issues/147
https://github.com/confluentinc/schema-registry/issues/789
# TODO add answer
https://stackoverflow.com/questions/39670691/how-to-have-kafkaproducer-to-use-a-mock-schema-registry-for-testing
# TODO
https://github.com/ovotech/kafka-serialization
http://vectos.net/formulation
* multi-version on same topic
- https://github.com/confluentinc/schema-registry/pull/680
- https://softwaremill.com/schema-registry-and-topic-with-multiple-message-types/
* tests: https://github.com/confluentinc/schema-registry/blob/master/client/src/main/java/io/confluent/kafka/schemaregistry/client/MockSchemaRegistryClient.java
# TODO
Schema ID allocation always happen in the master node and they ensure that the Schema IDs are monotonically increasing
Kafka is used as Schema Registry storage backend.
The special Kafka topic <kafkastore.topic> (default _schemas), with a single partition, is used as a highly available write ahead log.
All schemas, subject/version and ID metadata, and compatibility settings are appended as messages to this log.
Schema Registry is designed to work as a distributed service using single master architecture
Only the master is capable of publishing writes to the underlying Kafka log, but all nodes are capable of directly serving read requests.
Slave nodes serve registration requests indirectly by simply forwarding them to the current master, and returning the response supplied by the master.
---
https://github.com/cakesolutions/scala-kafka-client
https://github.com/lightbend/kafka-streams-scala
https://github.com/lightbend/kafka-with-akka-streams-kafka-streams-tutorial
https://github.com/Spinoco/fs2-kafka
---
# fix test java.lang.OutOfMemoryError: Metaspace
https://medium.com/@jan______/sbtconfig-is-deprecated-650d6ff10236
---
https://fd4s.github.io/fs2-kafka
---
version: "3.7"
# https://github.com/confluentinc/cp-docker-images/blob/5.3.2-post/debian/kafka-connect-base/Dockerfile#L59
healthcheck:
test: ["CMD", "/etc/confluent/docker/healthcheck.sh"]
interval: 10s
timeout: 10s
retries: 100
start_period: 2m
---
"key.converter.schemas.enable":"false",
"value.converter.schemas.enable":"true",
"value.converter":"org.apache.kafka.connect.json.JsonConverter"
jdbc-sink-json-schema
https://stackoverflow.com/questions/45928768/kafka-connect-jdbc-sink-connector-not-working/45940013#45940013
https://github.com/apache/kafka/blob/2.4.0/connect/json/src/main/java/org/apache/kafka/connect/json/JsonSchema.java#L39-L59
---
# testing
https://kafka.apache.org/11/documentation/streams/developer-guide/testing.html
# examples
https://github.com/confluentinc/kafka-streams-examples
https://softwaremill.com/hands-on-kafka-streams-in-scala
# ZIO
https://blog.univalence.io/des-microservices-kafka-streams-avec-zio-et-http4s
--- OLD
# connectors
https://softwaremill.com/import-export-through-kafka-connectors/
https://rmoff.net/2019/05/08/when-a-kafka-connect-converter-is-not-a-_converter_/
# kafka
https://hevodata.com/blog/how-to-set-up-kafka-on-ubuntu-16-04/
https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85
https://medium.com/bakdata/queryable-kafka-topics-with-kafka-streams-8d2cca9de33f
# mirror maker
https://engineering.salesforce.com/mirrormaker-performance-tuning-63afaed12c21
https://community.hortonworks.com/articles/79891/kafka-mirror-maker-best-practices.html
https://www.instaclustr.com/support/documentation/kafka/kafka-cluster-management/setting-up-mirror-maker
https://blog.newrelic.com/engineering/kafka-best-practices
# kafka streams
https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple
https://www.confluent.io/blog/hello-world-kafka-connect-kafka-streams
https://www.confluent.io/blog/distributed-real-time-joins-and-aggregations-on-user-activity-events-using-kafka-streams
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka
# avro
https://github.com/akka/alpakka-kafka/issues/342
http://rick-hightower.blogspot.com/2017/05/apache-avro-tutorial.html
https://medium.com/@mykidong/howto-implement-avro-schema-inheritance-757d2897c1ad
https://gist.github.com/cmilfont/2466a4272adcf0d5d20a309002df6799
https://dzone.com/articles/kafka-avro-scala-example
# avro + schema registry
https://docs.confluent.io/current/schema-registry/docs/index.html
https://www.sderosiaux.com/articles/2017/03/02/serializing-data-efficiently-with-apache-avro-and-dealing-with-a-schema-registry
https://web.archive.org/web/20180321112249/https://cloudurable.com/blog/kafka-avro-schema-registry/index.html
https://msayag.github.io/Kafka
http://bytepadding.com/big-data/spark/avro/avro-serialization-de-serialization-using-confluent-schema-registry
https://github.com/sachabarber/KafkaAvroExamples
https://www.madewithtea.com/kafka-streams-in-scala-with-schema-registry.html
https://github.com/lagom/lagom/issues/668
https://docs.confluent.io/current/schema-registry/docs/serializer-formatter.html
https://gist.github.com/avpatel257/0a88d20200661b31ab5f5df7adc42e6f
https://groups.google.com/forum/#!topic/akka-user/42SHwSIM1Xw
http://subprotocol.com/system/apache-spark-ec2-avro.html