Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CRASH] When loading module-oss (RediSearch with Coordinator), Valkey Node crashes #1054

Closed
wernermorgenstern opened this issue Sep 19, 2024 · 1 comment · Fixed by #1063
Closed
Assignees

Comments

@wernermorgenstern
Copy link

Crash report

=== VALKEY BUG REPORT START: Cut & paste starting from here ===
 1:M 19 Sep 2024 17:36:10.666 # valkey 8.0.0 crashed by signal: 11, si_code: 1
 1:M 19 Sep 2024 17:36:10.666 # Accessing address: (nil)
 1:M 19 Sep 2024 17:36:10.666 # Crashed running the instruction at: 0xaaaad4294510
 
 ------ STACK TRACE ------
 EIP:
 valkey-server 0.0.0.0:6379 [cluster](clusterCommandSlots+0x20)[0xaaaad4294510]
 
 136 bio_lazy_free
 /lib/aarch64-linux-gnu/libc.so.6(+0x7b698)[0xffffb14cb698]
 /lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x200)[0xffffb14ce1d0]
 valkey-server 0.0.0.0:6379 [cluster](bioProcessBackgroundJobs+0x17c)[0xaaaad42b5060]
 /lib/aarch64-linux-gnu/libc.so.6(+0x7ee30)[0xffffb14cee30]
 /lib/aarch64-linux-gnu/libc.so.6(+0xe7adc)[0xffffb1537adc]
 
 135 bio_aof
 /lib/aarch64-linux-gnu/libc.so.6(+0x7b698)[0xffffb14cb698]
 /lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x200)[0xffffb14ce1d0]
 valkey-server 0.0.0.0:6379 [cluster](bioProcessBackgroundJobs+0x17c)[0xaaaad42b5060]
 /lib/aarch64-linux-gnu/libc.so.6(+0x7ee30)[0xffffb14cee30]
 /lib/aarch64-linux-gnu/libc.so.6(+0xe7adc)[0xffffb1537adc]
 
 134 bio_close_file
 /lib/aarch64-linux-gnu/libc.so.6(+0x7b698)[0xffffb14cb698]
 /lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x200)[0xffffb14ce1d0]
 valkey-server 0.0.0.0:6379 [cluster](bioProcessBackgroundJobs+0x17c)[0xaaaad42b5060]
 /lib/aarch64-linux-gnu/libc.so.6(+0x7ee30)[0xffffb14cee30]
 /lib/aarch64-linux-gnu/libc.so.6(+0xe7adc)[0xffffb1537adc]
 
 1 valkey-server *
 linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffb1c097a0]
 valkey-server 0.0.0.0:6379 [cluster](clusterCommandSlots+0x20)[0xaaaad4294510]
 valkey-server 0.0.0.0:6379 [cluster](call+0x5b8)[0xaaaad41ffa48]
 valkey-server 0.0.0.0:6379 [cluster](VM_Call+0x4cc)[0xaaaad42e8b4c]
 /opt/bitnami/redis/etc/module-oss.so(+0xf850c)[0xffffad0f850c]
 /opt/bitnami/redis/etc/module-oss.so(+0xf8a34)[0xffffad0f8a34]
 valkey-server 0.0.0.0:6379 [cluster](moduleTimerHandler+0x100)[0xaaaad42ed524]
 valkey-server 0.0.0.0:6379 [cluster](+0x89664)[0xaaaad41e9664]
 valkey-server 0.0.0.0:6379 [cluster](aeMain+0x134)[0xaaaad41f2f28]
 valkey-server 0.0.0.0:6379 [cluster](main+0x508)[0xaaaad41e8e08]
 /lib/aarch64-linux-gnu/libc.so.6(+0x27740)[0xffffb1477740]
 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffffb1477818]
 valkey-server 0.0.0.0:6379 [cluster](_start+0x30)[0xaaaad41e94f0]
 
 4/4 expected stacktraces.
 
 ------ STACK TRACE DONE ------
 
 ------ REGISTERS ------
 1:M 19 Sep 2024 17:36:10.678 # 
 X18:0000ffffb126f2f0 X19:0000000020000000
 X20:0000ffffb0651180 X21:0000aaaad44a74b0
 X22:0006227c5bbcfb8d X23:0000aaaad44d3000
 X24:0000000000000001 X25:0000fffff989fdd0
 X26:0000000000000000 X27:0000ffffb0651180
 X28:0000aaaad44e7fd0 X29:0000fffff989fd10
 X30:0000aaaad41ffa48
 pc:0000aaaad4294510 sp:0000fffff989fd10
 pstate:0000000060001000 fault_address:0000000000000000
 
 1:M 19 Sep 2024 17:36:10.678 * hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.
 
 ------ INFO OUTPUT ------
 # Server
 redis_version:7.2.4
 server_name:valkey
 valkey_version:8.0.0
 redis_git_sha1:00000000
 redis_git_dirty:1
 redis_build_id:c145f9e2427621fa
 server_mode:cluster
 os:Linux 6.10.4-linuxkit aarch64
 arch_bits:64
 monotonic_clock:POSIX clock_gettime
 multiplexing_api:epoll
 gcc_version:12.2.0
 process_id:1
 process_supervised:no
 run_id:0b510db652f5d10258c6f28f8f2e5aeb021a87fa
 tcp_port:6379
 server_time_usec:1726767370664389
 uptime_in_seconds:255
 uptime_in_days:0
 hz:10
 configured_hz:10
 lru_clock:15491338
 executable:/valkey-server
 config_file:
 io_threads_active:0
 availability_zone:
 listener0:name=tcp,bind=0.0.0.0,bind=::,port=6379
 
 # Clients
 connected_clients:1
 cluster_connections:10
 maxclients:10000
 client_recent_max_input_buffer:0
 client_recent_max_output_buffer:0
 blocked_clients:0
 tracking_clients:0
 pubsub_clients:0
 watching_clients:0
 clients_in_timeout_table:0
 total_watched_keys:0
 total_blocking_keys:0
 total_blocking_keys_on_nokey:0
 
 # Memory
 used_memory:3122624
 used_memory_human:2.98M
 used_memory_rss:18026496
 used_memory_rss_human:17.19M
 used_memory_peak:3122624
 used_memory_peak_human:2.98M
 used_memory_peak_perc:100.08%
 used_memory_overhead:2773544
 used_memory_startup:2760712
 used_memory_dataset:349080
 used_memory_dataset_perc:96.45%
 allocator_allocated:3528600
 allocator_active:10747904
 allocator_resident:12517376
 allocator_muzzy:0
 total_system_memory:8220094464
 total_system_memory_human:7.66G
 used_memory_lua:31744
 used_memory_vm_eval:31744
 used_memory_lua_human:31.00K
 used_memory_scripts_eval:0
 number_of_cached_scripts:0
 number_of_functions:0
 number_of_libraries:0
 used_memory_vm_functions:33792
 used_memory_vm_total:65536
 used_memory_vm_total_human:64.00K
 used_memory_functions:184
 used_memory_scripts:184
 used_memory_scripts_human:184B
 maxmemory:0
 maxmemory_human:0B
 maxmemory_policy:noeviction
 allocator_frag_ratio:3.05
 allocator_frag_bytes:7219304
 allocator_rss_ratio:1.16
 allocator_rss_bytes:1769472
 rss_overhead_ratio:1.44
 rss_overhead_bytes:5509120
 mem_fragmentation_ratio:5.82
 mem_fragmentation_bytes:14931296
 mem_not_counted_for_evict:8
 mem_replication_backlog:0
 mem_total_replication_buffers:0
 mem_clients_slaves:0
 mem_clients_normal:1920
 mem_cluster_links:10720
 mem_aof_buffer:8
 mem_allocator:jemalloc-5.3.0
 mem_overhead_db_hashtable_rehashing:0
 active_defrag_running:0
 lazyfree_pending_objects:0
 lazyfreed_objects:0
 
 # Persistence
 loading:0
 async_loading:0
 current_cow_peak:0
 current_cow_size:0
 current_cow_size_age:0
 current_fork_perc:0.00
 current_save_keys_processed:0
 current_save_keys_total:0
 rdb_changes_since_last_save:0
 rdb_bgsave_in_progress:0
 rdb_last_save_time:1726767115
 rdb_last_bgsave_status:ok
 rdb_last_bgsave_time_sec:-1
 rdb_current_bgsave_time_sec:-1
 rdb_saves:0
 rdb_last_cow_size:0
 rdb_last_load_keys_expired:0
 rdb_last_load_keys_loaded:0
 aof_enabled:1
 aof_rewrite_in_progress:0
 aof_rewrite_scheduled:0
 aof_last_rewrite_time_sec:-1
 aof_current_rewrite_time_sec:-1
 aof_last_bgrewrite_status:ok
 aof_rewrites:0
 aof_rewrites_consecutive_failures:0
 aof_last_write_status:ok
 aof_last_cow_size:0
 module_fork_in_progress:0
 module_fork_last_cow_size:0
 aof_current_size:89
 aof_base_size:89
 aof_pending_rewrite:0
 aof_buffer_length:0
 aof_pending_bio_fsync:0
 aof_delayed_fsync:0
 
 # Stats
 total_connections_received:2
 total_commands_processed:15
 instantaneous_ops_per_sec:3
 total_net_input_bytes:344
 total_net_output_bytes:428815
 total_net_repl_input_bytes:0
 total_net_repl_output_bytes:0
 instantaneous_input_kbps:0.09
 instantaneous_output_kbps:0.01
 instantaneous_input_repl_kbps:0.00
 instantaneous_output_repl_kbps:0.00
 rejected_connections:0
 sync_full:0
 sync_partial_ok:0
 sync_partial_err:0
 expired_keys:0
 expired_stale_perc:0.00
 expired_time_cap_reached_count:0
 expire_cycle_cpu_milliseconds:7
 evicted_keys:0
 evicted_clients:0
 evicted_scripts:0
 total_eviction_exceeded_time:0
 current_eviction_exceeded_time:0
 keyspace_hits:0
 keyspace_misses:0
 pubsub_channels:0
 pubsub_patterns:0
 pubsubshard_channels:0
 latest_fork_usec:0
 total_forks:0
 migrate_cached_sockets:0
 slave_expires_tracked_keys:0
 active_defrag_hits:0
 active_defrag_misses:0
 active_defrag_key_hits:0
 active_defrag_key_misses:0
 total_active_defrag_time:0
 current_active_defrag_time:0
 tracking_total_keys:0
 tracking_total_items:0
 tracking_total_prefixes:0
 unexpected_error_replies:0
 total_error_replies:0
 dump_payload_sanitizations:0
 total_reads_processed:10
 total_writes_processed:13
 io_threaded_reads_processed:0
 io_threaded_writes_processed:0
 io_threaded_freed_objects:0
 io_threaded_poll_processed:0
 io_threaded_total_prefetch_batches:0
 io_threaded_total_prefetch_entries:0
 client_query_buffer_limit_disconnections:0
 client_output_buffer_limit_disconnections:0
 reply_buffer_shrinks:4
 reply_buffer_expands:2
 eventloop_cycles:3400
 eventloop_duration_sum:1557158
 eventloop_duration_cmd_sum:88518
 instantaneous_eventloop_cycles_per_sec:14
 instantaneous_eventloop_duration_usec:1392
 acl_access_denied_auth:0
 acl_access_denied_cmd:0
 acl_access_denied_key:0
 acl_access_denied_channel:0
 
 # Replication
 role:master
 connected_slaves:0
 replicas_waiting_psync:0
 master_failover_state:no-failover
 master_replid:9c59c6a54419f30080e7ca195b365e458a367013
 master_replid2:0000000000000000000000000000000000000000
 master_repl_offset:0
 second_repl_offset:-1
 repl_backlog_active:0
 repl_backlog_size:10485760
 repl_backlog_first_byte_offset:0
 repl_backlog_histlen:0
 
 # CPU
 used_cpu_sys:0.599317
 used_cpu_user:1.186672
 used_cpu_sys_children:0.071291
 used_cpu_user_children:0.071687
 used_cpu_sys_main_thread:0.587122
 used_cpu_user_main_thread:1.186203
 
 # Modules
 module:name=timeseries,ver=11202,api=1,filters=0,usedby=[],using=[],options=[]
 module:name=search,ver=21008,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]
 
 # Commandstats
 cmdstat_config|get:calls=1,usec=78,usec_per_call=78.00,rejected_calls=0,failed_calls=0
 cmdstat_command|docs:calls=2,usec=5327,usec_per_call=2663.50,rejected_calls=0,failed_calls=0
 cmdstat_cluster|myid:calls=1,usec=53,usec_per_call=53.00,rejected_calls=0,failed_calls=0
 cmdstat_info:calls=6,usec=4493,usec_per_call=748.83,rejected_calls=0,failed_calls=0
 cmdstat_auth:calls=2,usec=305,usec_per_call=152.50,rejected_calls=0,failed_calls=0
 cmdstat_module|list:calls=1,usec=56,usec_per_call=56.00,rejected_calls=0,failed_calls=0
 cmdstat_module|load:calls=2,usec=78509,usec_per_call=39254.50,rejected_calls=0,failed_calls=0
 
 # Errorstats
 
 # Latencystats
 latency_percentiles_usec_config|get:p50=78.335,p99=78.335,p99.9=78.335
 latency_percentiles_usec_command|docs:p50=2506.751,p99=2834.431,p99.9=2834.431
 latency_percentiles_usec_cluster|myid:p50=53.247,p99=53.247,p99.9=53.247
 latency_percentiles_usec_info:p50=13.055,p99=2293.759,p99.9=2293.759
 latency_percentiles_usec_auth:p50=128.511,p99=177.151,p99.9=177.151
 latency_percentiles_usec_module|list:p50=56.063,p99=56.063,p99.9=56.063
 latency_percentiles_usec_module|load:p50=7536.639,p99=71303.167,p99.9=71303.167
 
 # Cluster
 cluster_enabled:1
 
 # Keyspace
 
 # Cluster info
 cluster_state:ok
 cluster_slots_assigned:16384
 cluster_slots_ok:16384
 cluster_slots_pfail:0
 cluster_slots_fail:0
 cluster_known_nodes:6
 cluster_size:6
 cluster_current_epoch:6
 cluster_my_epoch:6
 cluster_stats_messages_ping_sent:249
 cluster_stats_messages_pong_sent:240
 cluster_stats_messages_sent:489
 cluster_stats_messages_ping_received:240
 cluster_stats_messages_pong_received:249
 cluster_stats_messages_received:489
 total_cluster_links_buffer_limit_exceeded:0
 
 ------ CLUSTER NODES OUTPUT ------
 c8d003887817dd5f9ea3b567d20b8c1c557d9701 172.28.1.6:6379@16379,,tls-port=0,shard-id=00a5a7915a1ea7215cc250d290a5a7744f57a0d6 myself,master - 0 1726767115430 6 connected 13653-16383
 0c152bdedab2eb868467ef27490b19364a8dab1c 172.28.1.5:6379@16379,,tls-port=0,shard-id=0b9e199becb1669023dcc304a1df01da7d8916cb master - 0 1726767368000 5 connected 10923-13652
 56a30cca66fed9d8e83707f9e17899c94540e97c 172.28.1.2:6379@16379,,tls-port=0,shard-id=af5ad2e91f5ee99213579fc17ed29a27c4b881d4 master - 0 1726767368000 2 connected 2731-5460
 73a50f792d65cdc391d15f768597ff9f69321152 172.28.1.4:6379@16379,,tls-port=0,shard-id=a4c74dbbcddf38ca325ac18b6cf4da933d675c19 master - 0 1726767369977 4 connected 8192-10922
 11f2dec9b1cbc336f9117ffa87be1e3313639655 172.28.1.3:6379@16379,,tls-port=0,shard-id=ed82cc62bab6f2f0ad44102ead59a61179e9e496 master - 0 1726767368922 3 connected 5461-8191
 eb0624d26a1dfa9ae9c6b58cce4fc02cddbb78bc 172.28.1.1:6379@16379,,tls-port=0,shard-id=dc50db0487e7ee2427b1ce3cd5f1bda479ff67c1 master - 0 1726767367875 1 connected 0-2730
 
 ------ CLIENT LIST OUTPUT ------
 id=5 addr=127.0.0.1:55486 laddr=127.0.0.1:6379 fd=24 name=*redacted* age=72 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=5 obl=0 oll=0 omem=0 tot-mem=1920 events=r cmd=module|load user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=250 tot-net-out=214413 tot-cmds=10
 
 ------ EXECUTING CLIENT INFO ------
 id=8 addr= laddr= fd=-1 name=*redacted* age=1 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17296 events= cmd=cluster|slots user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=0 tot-net-out=0 tot-cmds=0
 argc: '2'
 argv[0]: '"CLUSTER"'
 argv[1]: 5 bytes
 
 ------ MODULES INFO OUTPUT ------
 # search_version
 search_version:2.10.8
 search_redis_version:7.2.4 - oss
 
 # search_index
 search_number_of_indexes:0
 
 # search_fields_statistics
 
 # search_memory
 search_used_memory_indexes:0
 search_used_memory_indexes_human:0
 search_total_indexing_time:0
 
 # search_cursors
 search_global_idle:0
 search_global_total:0
 
 # search_gc
 search_bytes_collected:0
 search_total_cycles:0
 search_total_ms_run:0
 
 # search_dialect_statistics
 search_dialect_1:0
 search_dialect_2:0
 search_dialect_3:0
 search_dialect_4:0
 
 # search_runtime_configurations
 search_enableGC:ON
 search_minimal_term_prefix:2
 search_minimal_stem_length:4
 search_maximal_prefix_expansions:200
 search_query_timeout_ms:500
 search_timeout_policy:return
 search_cursor_read_size:1000
 search_cursor_max_idle_time:300000
 search_max_doc_table_size:1000000
 search_max_search_results:1000000
 search_max_aggregate_results:-1
 search_gc_scan_size:100
 search_min_phonetic_term_length:3
 
 ------ CONFIG DEBUG OUTPUT ------
 repl-diskless-load disabled
 io-threads 1
 io-threads-do-reads yes
 lazyfree-lazy-eviction yes
 replica-read-only yes
 lazyfree-lazy-user-flush yes
 list-compress-depth 0
 proto-max-bulk-len 512mb
 sanitize-dump-payload no
 debug-context ""
 slave-read-only yes
 lazyfree-lazy-user-del yes
 dual-channel-replication-enabled no
 lazyfree-lazy-server-del yes
 repl-diskless-sync yes
 activedefrag no
 client-query-buffer-limit 1gb
 lazyfree-lazy-expire yes
 
 ------ FAST MEMORY TEST ------
 1:M 19 Sep 2024 17:36:10.681 # Bio worker thread #0 terminated
 1:M 19 Sep 2024 17:36:10.681 # Bio worker thread #1 terminated
 1:M 19 Sep 2024 17:36:10.682 # Bio worker thread #2 terminated
 *** Preparing to test memory region aaaad44d4000 (724992 bytes)
 *** Preparing to test memory region aaaaf9329000 (581632 bytes)
 *** Preparing to test memory region ffffaca13000 (12288 bytes)
 *** Preparing to test memory region ffffaceab000 (16384 bytes)
 *** Preparing to test memory region ffffad603000 (20480 bytes)
 *** Preparing to test memory region ffffad810000 (8388608 bytes)
 *** Preparing to test memory region ffffae210000 (8388608 bytes)
 *** Preparing to test memory region ffffaec10000 (8388608 bytes)
 *** Preparing to test memory region ffffaf610000 (8388608 bytes)
 *** Preparing to test memory region ffffb0000000 (7864320 bytes)
 *** Preparing to test memory region ffffb0800000 (3145728 bytes)
 *** Preparing to test memory region ffffb0bef000 (8192 bytes)
 *** Preparing to test memory region ffffb0c00000 (2621440 bytes)
 *** Preparing to test memory region ffffb1000000 (4194304 bytes)
 *** Preparing to test memory region ffffb15f2000 (53248 bytes)
 *** Preparing to test memory region ffffb1a53000 (12288 bytes)
 *** Preparing to test memory region ffffb1bfe000 (20480 bytes)
 *** Preparing to test memory region ffffb1c05000 (8192 bytes)
 .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
 Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
 
 ------ DUMPING CODE AROUND EIP ------
 Symbol: clusterCommandSlots (base: 0xaaaad42944f0)
 Module: valkey-server 0.0.0.0:6379 [cluster] (base 0xaaaad4160000)
 $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
 $ objdump --adjust-vma=0xaaaad42944f0 -D -b binary -m i386:x86-64 /tmp/dump.bin
 ------
 1:M 19 Sep 2024 17:36:11.020 # dump of function (hexdump of 160 bytes):
 fd7bbda9e11100f0fd030091f35301a9f40300aa20884db9f55b02a9820a40f9530040f9000d0034e01100f0e10313aa00c846f91f0013ebd60f00f0d6d641f9f3179f1ae00314aaa10800b4710ffe9700004039811a40b962021f321f6c01715300931a3f0c007160021e321300931a70420094e0040035737e4093731a0b91d57a73f8150500b4c0d647b900070035a2f25f38400800121f08007160050054
 
 === VALKEY BUG REPORT END. Make sure to include from START to END. ===
 
        Please report the crash by opening an issue on github:
 
            https://github.com/valkey-io/valkey/issues
 
   If a module was involved, please open in the module's repo instead.
 
   Suspect RAM error? Use valkey-server --test-memory to verify it.
 
   Some other issues could be detected by valkey-server --check-system
 valkey-cluster 17:36:45.58 INFO  ==> 
 valkey-cluster 17:36:45.58 INFO  ==> Welcome to the Bitnami valkey-cluster container
 valkey-cluster 17:36:45.58 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
 valkey-cluster 17:36:45.58 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
 
 1:C 19 Sep 2024 17:36:45.770 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
 1:C 19 Sep 2024 17:36:45.772 * oO0OoO0OoO0Oo Valkey is starting oO0OoO0OoO0Oo
 1:C 19 Sep 2024 17:36:45.772 * Valkey version=8.0.0, bits=64, commit=00000000, modified=1, pid=1, just started
 1:C 19 Sep 2024 17:36:45.772 * Configuration loaded
 1:M 19 Sep 2024 17:36:45.772 * monotonic clock: POSIX clock_gettime
 1:M 19 Sep 2024 17:36:45.773 * Running mode=cluster, port=6379.
 1:M 19 Sep 2024 17:36:45.773 * Node configuration loaded, I'm c8d003887817dd5f9ea3b567d20b8c1c557d9701
 1:M 19 Sep 2024 17:36:45.774 * Server initialized
 1:M 19 Sep 2024 17:36:45.774 * Reading RDB base file on AOF loading...
 1:M 19 Sep 2024 17:36:45.774 * Loading RDB produced by Valkey version 7.2.6
 1:M 19 Sep 2024 17:36:45.774 * RDB age 1217 seconds
 1:M 19 Sep 2024 17:36:45.774 * RDB memory usage when created 1.56 Mb
 1:M 19 Sep 2024 17:36:45.774 * RDB is base AOF
 1:M 19 Sep 2024 17:36:45.774 * Done loading RDB, keys loaded: 0, keys expired: 0.
 1:M 19 Sep 2024 17:36:45.774 * DB loaded from base file appendonly.aof.1.base.rdb: 0.000 seconds
 1:M 19 Sep 2024 17:36:45.774 * DB loaded from append only file: 0.000 seconds
 valkey-cluster 17:36:45.58 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
 valkey-cluster 17:36:45.59 INFO  ==> 
 valkey-cluster 17:36:45.59 INFO  ==> ** Starting Valkey setup **
 valkey-cluster 17:36:45.61 INFO  ==> Initializing Valkey
 valkey-cluster 17:36:45.63 INFO  ==> Setting Valkey config file
 valkey-cluster 17:36:45.71 INFO  ==> Changing old IP 172.28.1.1 by the new one 172.28.1.1
 valkey-cluster 17:36:45.72 INFO  ==> Changing old IP 172.28.1.2 by the new one 172.28.1.2
 valkey-cluster 17:36:45.72 INFO  ==> Changing old IP 172.28.1.3 by the new one 172.28.1.3
 valkey-cluster 17:36:45.73 INFO  ==> Changing old IP 172.28.1.4 by the new one 172.28.1.4
 valkey-cluster 17:36:45.74 INFO  ==> Changing old IP 172.28.1.5 by the new one 172.28.1.5
 valkey-cluster 17:36:45.75 INFO  ==> Changing old IP 172.28.1.6 by the new one 172.28.1.6
 valkey-cluster 17:36:45.75 INFO  ==> ** Valkey setup finished! **
 1:M 19 Sep 2024 17:36:45.774 * Opening AOF incr file appendonly.aof.1.incr.aof on server start
 1:M 19 Sep 2024 17:36:45.774 * Ready to accept connections tcp
 1:M 19 Sep 2024 17:36:47.861 * Cluster state changed: ok
 1:M 19 Sep 2024 17:36:56.215 * Legacy Redis Module /opt/bitnami/redis/etc/redistimeseries.so found
 1:M 19 Sep 2024 17:36:56.215 * <timeseries> RedisTimeSeries version 11202, git_sha=b992fe30965e1408cd46fd1b66fae42c6b82c4cc
 1:M 19 Sep 2024 17:36:56.216 * <timeseries> Redis version found by RedisTimeSeries : 7.2.4 - oss
 1:M 19 Sep 2024 17:36:56.216 * <timeseries> loaded default CHUNK_SIZE_BYTES policy: 4096
 1:M 19 Sep 2024 17:36:56.216 * <timeseries> loaded server DUPLICATE_POLICY: block
 1:M 19 Sep 2024 17:36:56.216 * <timeseries> loaded default IGNORE_MAX_TIME_DIFF: 0
 1:M 19 Sep 2024 17:36:56.216 * <timeseries> loaded default IGNORE_MAX_VAL_DIFF: 0.000000
 1:M 19 Sep 2024 17:36:56.216 * <timeseries> Setting default series ENCODING to: compressed
 1:M 19 Sep 2024 17:36:56.216 * <timeseries> Detected redis oss
 1:M 19 Sep 2024 17:36:56.217 * Module 'timeseries' loaded from /opt/bitnami/redis/etc/redistimeseries.so
 1:M 19 Sep 2024 17:37:09.527 * Legacy Redis Module /opt/bitnami/redis/etc/module-oss.so found
 1:M 19 Sep 2024 17:37:09.528 * <search> Redis version found by RedisSearch : 7.2.4 - oss
 1:M 19 Sep 2024 17:37:09.529 * <search> RediSearch version 2.10.8 (Git=2.10-ccbef0a74)
 1:M 19 Sep 2024 17:37:09.529 * <search> Low level api version 1 initialized successfully
 1:M 19 Sep 2024 17:37:09.529 * <search> gc: ON, prefix min length: 2, min word length to stem: 4, prefix max expansions: 200, query timeout (ms): 500, timeout policy: return, cursor read size: 1000, cursor max idle (ms): 300000, max doctable size: 1000000, max number of search results:  1000000, 
 1:M 19 Sep 2024 17:37:09.529 * <search> Initialized thread pools!
 1:M 19 Sep 2024 17:37:09.529 * <search> Enabled role change notification
 1:M 19 Sep 2024 17:37:09.530 * <search> Cluster configuration: AUTO partitions, type: 0, coordinator timeout: 0ms
 1:M 19 Sep 2024 17:37:09.530 * <search> Register write commands
 1:M 19 Sep 2024 17:37:09.530 * Module 'search' loaded from /opt/bitnami/redis/etc/module-oss.so
 
 
 === VALKEY BUG REPORT START: Cut & paste starting from here ===
 1:M 19 Sep 2024 17:37:10.533 # valkey 8.0.0 crashed by signal: 11, si_code: 1
 1:M 19 Sep 2024 17:37:10.533 # Accessing address: (nil)
 1:M 19 Sep 2024 17:37:10.533 # Crashed running the instruction at: 0xaaaace5a4510
 
 ------ STACK TRACE ------
 EIP:
 valkey-server 0.0.0.0:6379 [cluster](clusterCommandSlots+0x20)[0xaaaace5a4510]
 
 139 bio_aof
 /lib/aarch64-linux-gnu/libc.so.6(+0x7b698)[0xffffa52cb698]
 /lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x200)[0xffffa52ce1d0]
 valkey-server 0.0.0.0:6379 [cluster](bioProcessBackgroundJobs+0x17c)[0xaaaace5c5060]
 /lib/aarch64-linux-gnu/libc.so.6(+0x7ee30)[0xffffa52cee30]
 /lib/aarch64-linux-gnu/libc.so.6(+0xe7adc)[0xffffa5337adc]
 
 140 bio_lazy_free
 /lib/aarch64-linux-gnu/libc.so.6(+0x7b698)[0xffffa52cb698]
 /lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x200)[0xffffa52ce1d0]
 valkey-server 0.0.0.0:6379 [cluster](bioProcessBackgroundJobs+0x17c)[0xaaaace5c5060]
 /lib/aarch64-linux-gnu/libc.so.6(+0x7ee30)[0xffffa52cee30]
 /lib/aarch64-linux-gnu/libc.so.6(+0xe7adc)[0xffffa5337adc]
 
 138 bio_close_file
 /lib/aarch64-linux-gnu/libc.so.6(+0x7b698)[0xffffa52cb698]
 /lib/aarch64-linux-gnu/libc.so.6(pthread_cond_wait+0x200)[0xffffa52ce1d0]
 valkey-server 0.0.0.0:6379 [cluster](bioProcessBackgroundJobs+0x17c)[0xaaaace5c5060]
 /lib/aarch64-linux-gnu/libc.so.6(+0x7ee30)[0xffffa52cee30]
 /lib/aarch64-linux-gnu/libc.so.6(+0xe7adc)[0xffffa5337adc]
 
 1 valkey-server *
 linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffa5a337a0]
 valkey-server 0.0.0.0:6379 [cluster](clusterCommandSlots+0x20)[0xaaaace5a4510]
 valkey-server 0.0.0.0:6379 [cluster](call+0x5b8)[0xaaaace50fa48]
 valkey-server 0.0.0.0:6379 [cluster](VM_Call+0x4cc)[0xaaaace5f8b4c]
 /opt/bitnami/redis/etc/module-oss.so(+0xf850c)[0xffffa06f850c]
 /opt/bitnami/redis/etc/module-oss.so(+0xf8a34)[0xffffa06f8a34]
 valkey-server 0.0.0.0:6379 [cluster](moduleTimerHandler+0x100)[0xaaaace5fd524]
 valkey-server 0.0.0.0:6379 [cluster](+0x89664)[0xaaaace4f9664]
 valkey-server 0.0.0.0:6379 [cluster](aeMain+0x134)[0xaaaace502f28]
 valkey-server 0.0.0.0:6379 [cluster](main+0x508)[0xaaaace4f8e08]
 /lib/aarch64-linux-gnu/libc.so.6(+0x27740)[0xffffa5277740]
 /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0x98)[0xffffa5277818]
 valkey-server 0.0.0.0:6379 [cluster](_start+0x30)[0xaaaace4f94f0]
 
 4/4 expected stacktraces.
 
 ------ STACK TRACE DONE ------
 
 ------ REGISTERS ------
 1:M 19 Sep 2024 17:37:10.543 # 
 X18:0000ffffa50691f0 X19:0000000020000000
 X20:0000ffffa4451180 X21:0000aaaace7b74b0
 X22:0006227c5f4e7d62 X23:0000aaaace7e3000
 X24:0000000000000001 X25:0000ffffe022cd10
 X26:0000000000000000 X27:0000ffffa4451180
 X28:0000aaaace7f7fd0 X29:0000ffffe022cc50
 X30:0000aaaace50fa48
 pc:0000aaaace5a4510 sp:0000ffffe022cc50
 pstate:0000000060001000 fault_address:0000000000000000
 
 1:M 19 Sep 2024 17:37:10.543 * hide-user-data-from-log is on, skip logging stack content to avoid spilling user data.
 
 ------ INFO OUTPUT ------
 # Server
 redis_version:7.2.4
 server_name:valkey
 valkey_version:8.0.0
 redis_git_sha1:00000000
 redis_git_dirty:1
 redis_build_id:c145f9e2427621fa
 server_mode:cluster
 os:Linux 6.10.4-linuxkit aarch64
 arch_bits:64
 monotonic_clock:POSIX clock_gettime
 multiplexing_api:epoll
 gcc_version:12.2.0
 process_id:1
 process_supervised:no
 run_id:1886edc66995307f1a6e2424029a190d0219f808
 tcp_port:6379
 server_time_usec:1726767430532082
 uptime_in_seconds:25
 uptime_in_days:0
 hz:10
 configured_hz:10
 lru_clock:15491398
 executable:/valkey-server
 config_file:
 io_threads_active:0
 availability_zone:
 listener0:name=tcp,bind=0.0.0.0,bind=::,port=6379
 
 # Clients
 connected_clients:1
 cluster_connections:10
 maxclients:10000
 client_recent_max_input_buffer:0
 client_recent_max_output_buffer:0
 blocked_clients:0
 tracking_clients:0
 pubsub_clients:0
 watching_clients:0
 clients_in_timeout_table:0
 total_watched_keys:0
 total_blocking_keys:0
 total_blocking_keys_on_nokey:0
 
 # Memory
 used_memory:3122512
 used_memory_human:2.98M
 used_memory_rss:18468864
 used_memory_rss_human:17.61M
 used_memory_peak:3122512
 used_memory_peak_human:2.98M
 used_memory_peak_perc:100.08%
 used_memory_overhead:2773576
 used_memory_startup:2760744
 used_memory_dataset:348936
 used_memory_dataset_perc:96.45%
 allocator_allocated:6517184
 allocator_active:12451840
 allocator_resident:14221312
 allocator_muzzy:0
 total_system_memory:8220094464
 total_system_memory_human:7.66G
 used_memory_lua:31744
 used_memory_vm_eval:31744
 used_memory_lua_human:31.00K
 used_memory_scripts_eval:0
 number_of_cached_scripts:0
 number_of_functions:0
 number_of_libraries:0
 used_memory_vm_functions:33792
 used_memory_vm_total:65536
 used_memory_vm_total_human:64.00K
 used_memory_functions:184
 used_memory_scripts:184
 used_memory_scripts_human:184B
 maxmemory:0
 maxmemory_human:0B
 maxmemory_policy:noeviction
 allocator_frag_ratio:1.91
 allocator_frag_bytes:5934656
 allocator_rss_ratio:1.14
 allocator_rss_bytes:1769472
 rss_overhead_ratio:1.30
 rss_overhead_bytes:4247552
 mem_fragmentation_ratio:5.97
 mem_fragmentation_bytes:15373776
 mem_not_counted_for_evict:8
 mem_replication_backlog:0
 mem_total_replication_buffers:0
 mem_clients_slaves:0
 mem_clients_normal:1920
 mem_cluster_links:10720
 mem_aof_buffer:8
 mem_allocator:jemalloc-5.3.0
 mem_overhead_db_hashtable_rehashing:0
 active_defrag_running:0
 lazyfree_pending_objects:0
 lazyfreed_objects:0
 
 # Persistence
 loading:0
 async_loading:0
 current_cow_peak:0
 current_cow_size:0
 current_cow_size_age:0
 current_fork_perc:0.00
 current_save_keys_processed:0
 current_save_keys_total:0
 rdb_changes_since_last_save:0
 rdb_bgsave_in_progress:0
 rdb_last_save_time:1726767405
 rdb_last_bgsave_status:ok
 rdb_last_bgsave_time_sec:-1
 rdb_current_bgsave_time_sec:-1
 rdb_saves:0
 rdb_last_cow_size:0
 rdb_last_load_keys_expired:0
 rdb_last_load_keys_loaded:0
 aof_enabled:1
 aof_rewrite_in_progress:0
 aof_rewrite_scheduled:0
 aof_last_rewrite_time_sec:-1
 aof_current_rewrite_time_sec:-1
 aof_last_bgrewrite_status:ok
 aof_rewrites:0
 aof_rewrites_consecutive_failures:0
 aof_last_write_status:ok
 aof_last_cow_size:0
 module_fork_in_progress:0
 module_fork_last_cow_size:0
 aof_current_size:89
 aof_base_size:89
 aof_pending_rewrite:0
 aof_buffer_length:0
 aof_pending_bio_fsync:0
 aof_delayed_fsync:0
 
 # Stats
 total_connections_received:1
 total_commands_processed:11
 instantaneous_ops_per_sec:2
 total_net_input_bytes:264
 total_net_output_bytes:209261
 total_net_repl_input_bytes:0
 total_net_repl_output_bytes:0
 instantaneous_input_kbps:0.06
 instantaneous_output_kbps:0.00
 instantaneous_input_repl_kbps:0.00
 instantaneous_output_repl_kbps:0.00
 rejected_connections:0
 sync_full:0
 sync_partial_ok:0
 sync_partial_err:0
 expired_keys:0
 expired_stale_perc:0.00
 expired_time_cap_reached_count:0
 expire_cycle_cpu_milliseconds:1
 evicted_keys:0
 evicted_clients:0
 evicted_scripts:0
 total_eviction_exceeded_time:0
 current_eviction_exceeded_time:0
 keyspace_hits:0
 keyspace_misses:0
 pubsub_channels:0
 pubsub_patterns:0
 pubsubshard_channels:0
 latest_fork_usec:0
 total_forks:0
 migrate_cached_sockets:0
 slave_expires_tracked_keys:0
 active_defrag_hits:0
 active_defrag_misses:0
 active_defrag_key_hits:0
 active_defrag_key_misses:0
 total_active_defrag_time:0
 current_active_defrag_time:0
 tracking_total_keys:0
 tracking_total_items:0
 tracking_total_prefixes:0
 unexpected_error_replies:0
 total_error_replies:0
 dump_payload_sanitizations:0
 total_reads_processed:5
 total_writes_processed:7
 io_threaded_reads_processed:0
 io_threaded_writes_processed:0
 io_threaded_freed_objects:0
 io_threaded_poll_processed:0
 io_threaded_total_prefetch_batches:0
 io_threaded_total_prefetch_entries:0
 client_query_buffer_limit_disconnections:0
 client_output_buffer_limit_disconnections:0
 reply_buffer_shrinks:1
 reply_buffer_expands:0
 eventloop_cycles:358
 eventloop_duration_sum:201925
 eventloop_duration_cmd_sum:38754
 instantaneous_eventloop_cycles_per_sec:13
 instantaneous_eventloop_duration_usec:517
 acl_access_denied_auth:0
 acl_access_denied_cmd:0
 acl_access_denied_key:0
 acl_access_denied_channel:0
 
 # Replication
 role:master
 connected_slaves:0
 replicas_waiting_psync:0
 master_failover_state:no-failover
 master_replid:aedbfef125cda52e3d9021ba766ed2d7dee6145b
 master_replid2:0000000000000000000000000000000000000000
 master_repl_offset:0
 second_repl_offset:-1
 repl_backlog_active:0
 repl_backlog_size:10485760
 repl_backlog_first_byte_offset:0
 repl_backlog_histlen:0
 
 # CPU
 used_cpu_sys:0.099880
 used_cpu_user:0.181804
 used_cpu_sys_children:0.109500
 used_cpu_user_children:0.078721
 used_cpu_sys_main_thread:0.096001
 used_cpu_user_main_thread:0.180840
 
 # Modules
 module:name=search,ver=21008,api=1,filters=0,usedby=[],using=[],options=[handle-io-errors]
 module:name=timeseries,ver=11202,api=1,filters=0,usedby=[],using=[],options=[]
 
 # Commandstats
 cmdstat_config|get:calls=1,usec=45,usec_per_call=45.00,rejected_calls=0,failed_calls=0
 cmdstat_auth:calls=1,usec=40,usec_per_call=40.00,rejected_calls=0,failed_calls=0
 cmdstat_module|load:calls=2,usec=22091,usec_per_call=11045.50,rejected_calls=0,failed_calls=0
 cmdstat_cluster|nodes:calls=1,usec=15392,usec_per_call=15392.00,rejected_calls=0,failed_calls=0
 cmdstat_cluster|myid:calls=1,usec=40,usec_per_call=40.00,rejected_calls=0,failed_calls=0
 cmdstat_command|docs:calls=1,usec=1231,usec_per_call=1231.00,rejected_calls=0,failed_calls=0
 cmdstat_info:calls=4,usec=274,usec_per_call=68.50,rejected_calls=0,failed_calls=0
 
 # Errorstats
 
 # Latencystats
 latency_percentiles_usec_config|get:p50=45.055,p99=45.055,p99.9=45.055
 latency_percentiles_usec_auth:p50=40.191,p99=40.191,p99.9=40.191
 latency_percentiles_usec_module|load:p50=10289.151,p99=11862.015,p99.9=11862.015
 latency_percentiles_usec_cluster|nodes:p50=15400.959,p99=15400.959,p99.9=15400.959
 latency_percentiles_usec_cluster|myid:p50=40.191,p99=40.191,p99.9=40.191
 latency_percentiles_usec_command|docs:p50=1236.991,p99=1236.991,p99.9=1236.991
 latency_percentiles_usec_info:p50=5.023,p99=171.007,p99.9=171.007
 
 # Cluster
 cluster_enabled:1
 
 # Keyspace
 
 # Cluster info
 cluster_state:ok
 cluster_slots_assigned:16384
 cluster_slots_ok:16384
 cluster_slots_pfail:0
 cluster_slots_fail:0
 cluster_known_nodes:6
 cluster_size:6
 cluster_current_epoch:6
 cluster_my_epoch:6
 cluster_stats_messages_ping_sent:28
 cluster_stats_messages_pong_sent:30
 cluster_stats_messages_sent:58
 cluster_stats_messages_ping_received:30
 cluster_stats_messages_pong_received:28
 cluster_stats_messages_received:58
 total_cluster_links_buffer_limit_exceeded:0
 
 ------ CLUSTER NODES OUTPUT ------
 eb0624d26a1dfa9ae9c6b58cce4fc02cddbb78bc 172.28.1.1:6379@16379,,tls-port=0,shard-id=dc50db0487e7ee2427b1ce3cd5f1bda479ff67c1 master - 0 1726767429000 1 connected 0-2730
 0c152bdedab2eb868467ef27490b19364a8dab1c 172.28.1.5:6379@16379,,tls-port=0,shard-id=0b9e199becb1669023dcc304a1df01da7d8916cb master - 0 1726767428000 5 connected 10923-13652
 56a30cca66fed9d8e83707f9e17899c94540e97c 172.28.1.2:6379@16379,,tls-port=0,shard-id=af5ad2e91f5ee99213579fc17ed29a27c4b881d4 master - 0 1726767427690 2 connected 2731-5460
 11f2dec9b1cbc336f9117ffa87be1e3313639655 172.28.1.3:6379@16379,,tls-port=0,shard-id=ed82cc62bab6f2f0ad44102ead59a61179e9e496 master - 0 1726767428735 3 connected 5461-8191
 c8d003887817dd5f9ea3b567d20b8c1c557d9701 172.28.1.6:6379@16379,,tls-port=0,shard-id=00a5a7915a1ea7215cc250d290a5a7744f57a0d6 myself,master - 0 1726767405773 6 connected 13653-16383
 73a50f792d65cdc391d15f768597ff9f69321152 172.28.1.4:6379@16379,,tls-port=0,shard-id=a4c74dbbcddf38ca325ac18b6cf4da933d675c19 master - 0 1726767429773 4 connected 8192-10922
 
 ------ CLIENT LIST OUTPUT ------
 id=4 addr=127.0.0.1:47810 laddr=127.0.0.1:6379 fd=24 name=*redacted* age=19 idle=1 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=5 obl=0 oll=0 omem=0 tot-mem=1920 events=r cmd=module|load user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=264 tot-net-out=209261 tot-cmds=10
 
 ------ EXECUTING CLIENT INFO ------
 id=7 addr= laddr= fd=-1 name=*redacted* age=14 idle=14 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=16384 rbp=16384 obl=0 oll=0 omem=0 tot-mem=17296 events= cmd=cluster|slots user=*redacted* redir=-1 resp=2 lib-name= lib-ver= tot-net-in=0 tot-net-out=0 tot-cmds=0
 argc: '2'
 argv[0]: '"CLUSTER"'
 argv[1]: 5 bytes
 
 ------ MODULES INFO OUTPUT ------
 # search_version
 search_version:2.10.8
 search_redis_version:7.2.4 - oss
 
 # search_index
 search_number_of_indexes:0
 
 # search_fields_statistics
 
 # search_memory
 search_used_memory_indexes:0
 search_used_memory_indexes_human:0
 search_total_indexing_time:0
 
 # search_cursors
 search_global_idle:0
 search_global_total:0
 
 # search_gc
 search_bytes_collected:0
 search_total_cycles:0
 search_total_ms_run:0
 
 # search_dialect_statistics
 search_dialect_1:0
 search_dialect_2:0
 search_dialect_3:0
 search_dialect_4:0
 
 # search_runtime_configurations
 search_enableGC:ON
 search_minimal_term_prefix:2
 search_minimal_stem_length:4
 search_maximal_prefix_expansions:200
 search_query_timeout_ms:500
 search_timeout_policy:return
 search_cursor_read_size:1000
 search_cursor_max_idle_time:300000
 search_max_doc_table_size:1000000
 search_max_search_results:1000000
 search_max_aggregate_results:-1
 search_gc_scan_size:100
 search_min_phonetic_term_length:3
 
 ------ CONFIG DEBUG OUTPUT ------
 lazyfree-lazy-expire yes
 lazyfree-lazy-eviction yes
 lazyfree-lazy-user-del yes
 client-query-buffer-limit 1gb
 list-compress-depth 0
 lazyfree-lazy-server-del yes
 io-threads-do-reads yes
 repl-diskless-load disabled
 dual-channel-replication-enabled no
 lazyfree-lazy-user-flush yes
 repl-diskless-sync yes
 slave-read-only yes
 debug-context ""
 sanitize-dump-payload no
 io-threads 1
 replica-read-only yes
 activedefrag no
 proto-max-bulk-len 512mb
 
 ------ FAST MEMORY TEST ------
 1:M 19 Sep 2024 17:37:10.546 # Bio worker thread #0 terminated
 1:M 19 Sep 2024 17:37:10.546 # Bio worker thread #1 terminated
 1:M 19 Sep 2024 17:37:10.546 # Bio worker thread #2 terminated
 *** Preparing to test memory region aaaace7e4000 (724992 bytes)
 *** Preparing to test memory region aaaae2be5000 (585728 bytes)
 *** Preparing to test memory region ffffa0c03000 (20480 bytes)
 *** Preparing to test memory region ffffa1013000 (12288 bytes)
 *** Preparing to test memory region ffffa14ab000 (16384 bytes)
 *** Preparing to test memory region ffffa1610000 (8388608 bytes)
 *** Preparing to test memory region ffffa2010000 (8388608 bytes)
 *** Preparing to test memory region ffffa2a10000 (8388608 bytes)
 *** Preparing to test memory region ffffa3410000 (8388608 bytes)
 *** Preparing to test memory region ffffa3e00000 (7864320 bytes)
 *** Preparing to test memory region ffffa4600000 (3145728 bytes)
 *** Preparing to test memory region ffffa4a00000 (2621440 bytes)
 *** Preparing to test memory region ffffa4def000 (8192 bytes)
 *** Preparing to test memory region ffffa4e00000 (4194304 bytes)
 *** Preparing to test memory region ffffa53f2000 (53248 bytes)
 *** Preparing to test memory region ffffa5853000 (12288 bytes)
 *** Preparing to test memory region ffffa5a28000 (20480 bytes)
 *** Preparing to test memory region ffffa5a2f000 (8192 bytes)
 .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
 Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
 
 ------ DUMPING CODE AROUND EIP ------
 Symbol: clusterCommandSlots (base: 0xaaaace5a44f0)
 Module: valkey-server 0.0.0.0:6379 [cluster] (base 0xaaaace470000)
 $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
 $ objdump --adjust-vma=0xaaaace5a44f0 -D -b binary -m i386:x86-64 /tmp/dump.bin
 ------
 1:M 19 Sep 2024 17:37:10.911 # dump of function (hexdump of 160 bytes):
 fd7bbda9e11100f0fd030091f35301a9f40300aa20884db9f55b02a9820a40f9530040f9000d0034e01100f0e10313aa00c846f91f0013ebd60f00f0d6d641f9f3179f1ae00314aaa10800b4710ffe9700004039811a40b962021f321f6c01715300931a3f0c007160021e321300931a70420094e0040035737e4093731a0b91d57a73f8150500b4c0d647b900070035a2f25f38400800121f08007160050054
 
 === VALKEY BUG REPORT END. Make sure to include from START to END. ===
 
        Please report the crash by opening an issue on github:
 
            https://github.com/valkey-io/valkey/issues
 
   If a module was involved, please open in the module's repo instead.
 
   Suspect RAM error? Use valkey-server --test-memory to verify it.
 
   Some other issues could be detected by valkey-server --check-system

Additional information

OS: MacOs, arm64
Valkey Version: bitnami Valkey Cluster 8.0.0

Created Dockerfile, using Bitnami Valkey Cluster 8.0.0

Created Valkey Cluster with docker-compose on Mac
Used valkey-cli

Ran following commands:
`module load /opt/bitnami/redis/etc/redistimeseries.so`
`module load /opt/bitnami/redis/etc/module-oss.so OSS_GLOBAL_PASSWORD bitnami`


As soon as I loaded the module-oss.so, the Node I was connected to crashes.
@PingXie
Copy link
Member

PingXie commented Sep 20, 2024

The bug is here:

if (c->conn->type == NULL && server.current_client) {

VM_Call uses a fake client without connection to talk to the engine.

 linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffffa5a337a0]
 valkey-server 0.0.0.0:6379 [cluster](clusterCommandSlots+0x20)[0xaaaace5a4510]
 valkey-server 0.0.0.0:6379 [cluster](call+0x5b8)[0xaaaace50fa48]
 valkey-server 0.0.0.0:6379 [cluster](VM_Call+0x4cc)[0xaaaace5f8b4c]

@PingXie PingXie moved this to Todo in Valkey 8.0 Sep 20, 2024
@PingXie PingXie moved this from Todo to To be backported in Valkey 8.0 Sep 20, 2024
@PingXie PingXie moved this from To be backported to Todo in Valkey 8.0 Sep 20, 2024
@enjoy-binbin enjoy-binbin self-assigned this Sep 23, 2024
enjoy-binbin added a commit to enjoy-binbin/valkey that referenced this issue Sep 23, 2024
The reason is VM_Call will use a fake client without connection,
so we also need to check if c->conn is NULL.

Fixes valkey-io#1054.

Signed-off-by: Binbin <binloveplay1314@qq.com>
madolson pushed a commit that referenced this issue Sep 27, 2024
)

moduleTimerHandler is aeTimeProc handler and event loop gets created
with this. However, found that the function return type is int but
actually returns "long long" value(i.e., next_period). and return value
being assigned to int variable in processTimeEvents(where time events
are processed), this might cause an overflow of the timer values. So
changed the return type of the function to long long. And also updated
other callback function return type to be consistent.

I found this when I was checking functions reported in
#1054 issue stacktrace. (FYI,
this is just to update the return type to be consistent and it will not
the fix for the issue reported)

Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
madolson pushed a commit to madolson/valkey that referenced this issue Sep 30, 2024
…sh (valkey-io#1063)

The reason is VM_Call will use a fake client without connection,
so we also need to check if c->conn is NULL.

This also affects scripts. If they are called in the script, the
server will crash. Injecting commands into AOF will also cause
startup failure.

Fixes valkey-io#1054.

Signed-off-by: Binbin <binloveplay1314@qq.com>
naglera pushed a commit to naglera/placeholderkv that referenced this issue Oct 10, 2024
…sh (valkey-io#1063)

The reason is VM_Call will use a fake client without connection,
so we also need to check if c->conn is NULL.

This also affects scripts. If they are called in the script, the
server will crash. Injecting commands into AOF will also cause
startup failure.

Fixes valkey-io#1054.

Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: naglera <anagler123@gmail.com>
naglera pushed a commit to naglera/placeholderkv that referenced this issue Oct 10, 2024
…lkey-io#1057)

moduleTimerHandler is aeTimeProc handler and event loop gets created
with this. However, found that the function return type is int but
actually returns "long long" value(i.e., next_period). and return value
being assigned to int variable in processTimeEvents(where time events
are processed), this might cause an overflow of the timer values. So
changed the return type of the function to long long. And also updated
other callback function return type to be consistent.

I found this when I was checking functions reported in
valkey-io#1054 issue stacktrace. (FYI,
this is just to update the return type to be consistent and it will not
the fix for the issue reported)

Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Signed-off-by: naglera <anagler123@gmail.com>
eifrah-aws pushed a commit to eifrah-aws/valkey that referenced this issue Oct 20, 2024
…lkey-io#1057)

moduleTimerHandler is aeTimeProc handler and event loop gets created
with this. However, found that the function return type is int but
actually returns "long long" value(i.e., next_period). and return value
being assigned to int variable in processTimeEvents(where time events
are processed), this might cause an overflow of the timer values. So
changed the return type of the function to long long. And also updated
other callback function return type to be consistent.

I found this when I was checking functions reported in
valkey-io#1054 issue stacktrace. (FYI,
this is just to update the return type to be consistent and it will not
the fix for the issue reported)

Signed-off-by: Shivshankar-Reddy <shiva.sheri.github@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants