✨ Smarter default pool sizes

[joocer]

Currently the buffer pool is 256mb and the read buffer 256mb/512mb.

In a usual database, the buffer pool is the largest allocation of memory. For our exemplar implementation into 8gb containers 256mb is 1/32th of the memory, this should be closer to 30% of the memory.

We should default the buffer pool to 25% of memory and the read buffer to 10%.

We should also allow larger files to be cached, perhaps 5x the remote cache limit.

import psutil

def configure_memory(allocation):
    """
    Configure the memory allocation for the database based on the input.
    If the allocation is between 0 and 1, it's treated as a percentage of the total system memory.
    If the allocation is greater than 1, it's treated as an absolute value in megabytes.

    Parameters:
        allocation (float|int): Memory allocation value which could be a percentage or an absolute value.

    Returns:
        int: Memory size in megabytes to be allocated.
    """
    total_memory_mb = psutil.virtual_memory().total // (1024 * 1024)  # Convert bytes to megabytes

    if 0 < allocation < 1:  # Treat as a percentage
        return int(total_memory_mb * allocation)
    elif allocation >= 1:  # Treat as an absolute value in MB
        return int(allocation)
    else:
        raise ValueError("Invalid memory allocation value. Must be a positive number.")

# Example usage:
memory_setting = 0.25  # 25% of total memory
# memory_setting = 1024  # 1024 MB
configured_memory = configure_memory(memory_setting)
print(f"Memory allocated: {configured_memory} MB")