Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: generate example.toml from default config #8735

Merged
merged 4 commits into from
Mar 24, 2023
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions Makefile.toml
Original file line number Diff line number Diff line change
Expand Up @@ -1056,3 +1056,14 @@ set -e

cp -a .risingwave/rw-docker/* docker
'''

[tasks.generate-example-config]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about checking whether the toml is the latest on CI?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I had considered about adding it to risedev check but that will make risedev much slower. Let me figure out some alternative ways later.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean the compilation speed? We can move config to a separate crate 😇

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Emmmm that sounds too painful 🤣 I thought example.toml is just an example so no need to keep it exactly matched with config.rs

category = "misc"
description = "Generate default config and write to src/config/example.toml"
script = '''
cat << EOF > src/config/example.toml
# This file is generated by ./risedev generate-example-config
# Check detailed comments in src/common/src/config.rs
EOF
cargo run --bin example-config >> src/config/example.toml
'''
4 changes: 4 additions & 0 deletions src/common/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -111,3 +111,7 @@ harness = false
[[bench]]
name = "bitmap"
harness = false

[[bin]]
name = "example-config"
path = "src/bin/default_config.rs"
21 changes: 21 additions & 0 deletions src/common/src/bin/default_config.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// Copyright 2023 RisingWave Labs
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

use risingwave_common::config::RwConfig;
fuyufjh marked this conversation as resolved.
Show resolved Hide resolved

fn main() {
let config = RwConfig::default();
let toml = toml::to_string(&config).unwrap();
print!("{}", &toml);
}
88 changes: 65 additions & 23 deletions src/config/example.toml
Original file line number Diff line number Diff line change
@@ -1,45 +1,87 @@
# This file is generated by ./risedev generate-example-config
# Check detailed comments in src/common/src/config.rs
[server]
heartbeat_interval_ms = 1000
max_heartbeat_interval_secs = 600
connection_pool_size = 16
metrics_level = 0
telemetry_enabled = true

[meta]
min_sst_retention_time_sec = 604800
collect_gc_watermark_spin_interval_sec = 5
periodic_compaction_interval_sec = 60
vacuum_interval_sec = 30
max_heartbeat_interval_secs = 300
disable_recovery = false
meta_leader_lease_secs = 10
enable_compaction_deterministic = false
enable_committed_sst_sanity_check = false
node_num_monitor_interval_sec = 10
backend = "Mem"
periodic_space_reclaim_compaction_interval_sec = 3600
periodic_ttl_reclaim_compaction_interval_sec = 1800
max_compactor_task_multiplier = 2

[batch]
[batch.developer]
batch_output_channel_size = 64
batch_chunk_size = 1024
stream_enable_executor_row_count = false
stream_connector_message_buffer_size = 16
unsafe_stream_extreme_cache_size = 1024
stream_chunk_size = 1024
stream_exchange_initial_permits = 8192
stream_exchange_batched_permits = 1024
Comment on lines +29 to +34
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems not that batch. Is there something wrong? 🥵


[streaming]
barrier_interval_ms = 1000
in_flight_barrier_nums = 10
checkpoint_frequency = 10
in_flight_barrier_nums = 10000
enable_jaeger_tracing = false
async_stack_trace = "On"
unique_user_stream_errors = 10

[streaming.developer]
batch_output_channel_size = 64
batch_chunk_size = 1024
stream_enable_executor_row_count = false
stream_connector_message_buffer_size = 16
unsafe_stream_extreme_cache_size = 1024
stream_chunk_size = 1024
stream_exchange_initial_permits = 8192
stream_exchange_batched_permits = 1024

[storage]
shared_buffer_capacity_mb = 4096
block_cache_capacity_mb = 4096
meta_cache_capacity_mb = 1024
compactor_memory_limit_mb = 5120
share_buffers_sync_parallelism = 1
share_buffer_compaction_worker_threads_number = 4
shared_buffer_capacity_mb = 1024
write_conflict_detection_enabled = true
block_cache_capacity_mb = 512
meta_cache_capacity_mb = 128
disable_remote_compactor = false
enable_local_spill = true
local_object_store = "tempdisk"
share_buffer_upload_concurrency = 8
compactor_memory_limit_mb = 512
sstable_id_remote_fetch_number = 10
min_sst_size_for_streaming_upload = 33554432
max_sub_compaction = 4
max_concurrent_compaction_task_number = 16

[storage.file_cache]
dir = ""
capacity_mb = 1024
total_buffer_capacity_mb = 128
cache_file_fallocate_unit_mb = 512
cache_meta_fallocate_unit_mb = 16
cache_file_max_write_size_mb = 4

#The configurable parameters in [XXX.developer] subsection are for developers.
#Users are not encouraged to tune or depend on the following parameters.
#Some configurable parameters exist only temporarily.
[batch.developer]
batch_output_channel_size = 64
batch_chunk_size = 1024

[streaming.developer]
stream_enable_executor_row_count = false
stream_connector_message_buffer_size = 16
unsafe_stream_extreme_cache_size = 1024
stream_chunk_size = 1024

[system]
barrier_interval_ms = 1000
checkpoint_frequency = 10
sstable_size_mb = 256
block_size_kb = 1024
block_size_kb = 64
bloom_false_positive = 0.001
data_directory = "hummock_001"
state_store = ""
data_directory = "hummock_001"
backup_storage_url = "memory"
backup_storage_directory = "backup"
telemetry_enabled = true