Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kafka advanced config defaults: set default/max values for advanced config settings #934

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Kafka advanced config defaults: set default/max values for advanced config settings #934

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6b06180
Kafka advanced config defaults: set default values for advanced confi…
dweinshenker Oct 22, 2024
9e24c55
fix default typo and set max to max of int64
dweinshenker Oct 23, 2024
961dd19
correct compression type default
dweinshenker Oct 23, 2024
cfc2ab9
Merge branch 'main' into dbaas-kafka-auto-topic-create-default
loosla Oct 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 50 additions & 11 deletions specification/resources/databases/models/advanced_config/kafka_advanced_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ properties:
- uncompressed
- producer
example: gzip
default: producer
group_initial_rebalance_delay_ms:
description: >-
The amount of time, in milliseconds, the group coordinator
Expand All @@ -29,6 +30,7 @@ properties:
example: 3000
minimum: 0
maximum: 300000
default: 3000
group_min_session_timeout_ms:
description: >-
The minimum allowed session timeout for registered consumers.
Expand All @@ -38,6 +40,7 @@ properties:
example: 6000
minimum: 0
maximum: 60000
default: 6000
group_max_session_timeout_ms:
description: >-
The maximum allowed session timeout for registered consumers.
Expand All @@ -47,6 +50,7 @@ properties:
example: 1800000
minimum: 0
maximum: 1800000
default: 1800000
connections_max_idle_ms:
description: >-
Idle connections timeout: the server socket processor
Expand All @@ -55,6 +59,7 @@ properties:
minimum: 1000
example: 540000
maximum: 3600000
default: 600000
max_incremental_fetch_session_cache_slots:
description: >-
The maximum number of incremental fetch sessions that the
Expand All @@ -63,27 +68,31 @@ properties:
example: 1000
minimum: 1000
maximum: 10000
default: 1000
message_max_bytes:
description: >-
The maximum size of message that the server can receive.
type: integer
example: 1048588
minimum: 0
maximum: 100001200
default: 1048588
offsets_retention_minutes:
description: >-
Log retention window in minutes for offsets topic
type: integer
example: 10080
minimum: 1
maximum: 2147483647
default: 10080
log_cleaner_delete_retention_ms:
description: >-
How long are delete records retained?
type: integer
minimum: 0
maximum: 315569260000
example: 86400000
default: 86400000
log_cleaner_min_cleanable_ratio:
description: >-
Controls log compactor frequency. Larger value means more
Expand All @@ -94,22 +103,25 @@ properties:
minimum: 0.2
maximum: 0.9
example: 0.5
default: 0.5
log_cleaner_max_compaction_lag_ms:
description: >-
The maximum amount of time message will remain uncompacted.
Only applicable for logs that are being compacted
type: integer
minimum: 30000
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 60000
default: 9223372036854775807
log_cleaner_min_compaction_lag_ms:
description: >-
The minimum time a message will remain uncompacted in the
log. Only applicable for logs that are being compacted.
type: integer
minimum: 0
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 100000
default: 0
log_cleanup_policy:
description: >-
The default cleanup policy for segments beyond the retention
Expand All @@ -120,23 +132,26 @@ properties:
- compact
- compact,delete
example: delete
default: delete
log_flush_interval_messages:
description: >-
The number of messages accumulated on a log partition before
messages are flushed to disk
type: integer
minimum: 1
maximum: 9223372036854776000
example: 9223372036854776000
maximum: 9223372036854775807
example: 9223372036854775807
default: 9223372036854775807
log_flush_interval_ms:
description: >-
The maximum time in ms that a message in any topic is kept
in memory before flushed to disk. If not set, the value in log.flush.scheduler.interval.ms
is used
type: integer
minimum: 0
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 1000000
default: 9223372036854775807
log_index_interval_bytes:
description: >-
The interval with which Kafka adds an entry to the offset
Expand All @@ -145,19 +160,22 @@ properties:
minimum: 0
maximum: 104857600
example: 4096
default: 4096
log_index_size_max_bytes:
description: >-
The maximum size in bytes of the offset index
type: integer
minimum: 1048576
maximum: 104857600
example: 10485760
default: 10485760
log_message_downconversion_enable:
description: >-
This configuration controls whether down-conversion of
message formats is enabled to satisfy consume requests.
type: boolean
example: true
default: true
log_message_timestamp_type:
description: >-
Define whether the timestamp in the message is message
Expand All @@ -167,65 +185,74 @@ properties:
- CreateTime
- LogAppendTime
example: CreateTime
default: CreateTime
log_message_timestamp_difference_max_ms:
description: >-
The maximum difference allowed between the timestamp when
a broker receives a message and the timestamp specified in the message
type: integer
minimum: 0
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 1000000
default: 9223372036854775807
log_preallocate:
description: >-
Controls whether to preallocate a file when creating a new segment
type: boolean
example: false
default: false
log_retention_bytes:
description: >-
The maximum size of the log before deleting messages
type: integer
minimum: -1
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 1000000
default: -1
log_retention_hours:
description: >-
The number of hours to keep a log file before deleting it
type: integer
minimum: -1
maximum: 2147483647
example: 1000000
default: 168
log_retention_ms:
description: >-
The number of milliseconds to keep a log file before deleting
it (in milliseconds), If not set, the value in log.retention.minutes
is used. If set to -1, no time limit is applied.
type: integer
minimum: -1
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 100000000
default: 604800000
log_roll_jitter_ms:
description: >-
The maximum jitter to subtract from logRollTimeMillis (in
milliseconds). If not set, the value in log.roll.jitter.hours is used
type: integer
minimum: 0
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 10000000
default: 604800000
log_roll_ms:
description: >-
The maximum time before a new log segment is rolled out
(in milliseconds).
type: integer
minimum: 1
maximum: 9223372036854776000
maximum: 9223372036854775807
example: 1000000
default: 604800000
log_segment_bytes:
description: >-
The maximum size of a single log file
type: integer
minimum: 10485760
maximum: 1073741824
example: 100000000
default: 1073741824
log_segment_delete_delay_ms:
description: >-
The amount of time to wait before deleting a file from
Expand All @@ -234,11 +261,13 @@ properties:
minimum: 0
maximum: 3600000
example: 60000
default: 60000
auto_create_topics_enable:
description: >-
Enable auto creation of topics
type: boolean
example: true
default: true
min_insync_replicas:
description: >-
When a producer sets acks to 'all' (or '-1'), min_insync_replicas
Expand All @@ -248,20 +277,23 @@ properties:
minimum: 1
maximum: 7
example: 1
default: 1
num_partitions:
description: >-
Number of partitions for autocreated topics
type: integer
minimum: 1
maximum: 1000
example: 10
default: 1
default_replication_factor:
description: >-
Replication factor for autocreated topics
type: integer
minimum: 1
maximum: 10
example: 2
default: 3
replica_fetch_max_bytes:
description: >-
The number of bytes of messages to attempt to fetch for
Expand All @@ -273,6 +305,7 @@ properties:
minimum: 1048576
maximum: 104857600
example: 2097152
default: 1048576
replica_fetch_response_max_bytes:
description: >-
Maximum bytes expected for the entire fetch response (defaults
Expand All @@ -284,6 +317,7 @@ properties:
minimum: 10485760
maximum: 1048576000
example: 20971520
default: 10485760
max_connections_per_ip:
description: >-
The maximum number of connections allowed from each ip
Expand All @@ -292,6 +326,7 @@ properties:
minimum: 256
maximum: 2147483647
example: 512
default: 2147483647
producer_purgatory_purge_interval_requests:
description: >-
The purge interval (in number of requests) of the producer
Expand All @@ -300,6 +335,7 @@ properties:
minimum: 10
maximum: 10000
example: 100
default: 1000
socket_request_max_bytes:
description: >-
The maximum number of bytes in a socket request (defaults
Expand All @@ -308,6 +344,7 @@ properties:
minimum: 10485760
maximum: 209715200
example: 20971520
default: 104857600
transaction_state_log_segment_bytes:
description: >-
The transaction topic segment bytes should be kept relatively
Expand All @@ -317,6 +354,7 @@ properties:
minimum: 1048576
maximum: 2147483647
example: 104857600
default: 104857600
transaction_remove_expired_transaction_cleanup_interval_ms:
description: >-
The interval at which to remove transactions that have
Expand All @@ -325,4 +363,5 @@ properties:
type: integer
minimum: 600000
maximum: 3600000
example: 3600000
example: 3600000
default: 3600000
Loading