Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FIX-#7102: Remove enable_api_only mode in modin logging #7194

Merged
merged 1 commit into from
Apr 17, 2024
Merged

FIX-#7102: Remove enable_api_only mode in modin logging #7194

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 2 additions & 13 deletions docs/usage_guide/advanced_usage/modin_logging.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,7 @@ Modin Logging

Modin logging offers users greater insight into their queries by logging internal Modin API calls, partition metadata,
and profiling system memory. When Modin logging is enabled (default disabled), log files are written to a local ``.modin`` directory at the same
directory level as the notebook/script used to run Modin. It is possible to configure whether to log system memory and additional metadata
in addition to Modin API calls (see the usage examples below).
directory level as the notebook/script used to run Modin.

The logs generated by Modin Logging will be written to a ``.modin/logs/job_<uuid>`` directory, uniquely named after the job uuid.
The logs that contain the Modin API stack traces are named ``trace.log``. The logs that contain the memory utilization metrics are
Expand All @@ -20,17 +19,7 @@ such environments. To resolve this, please run Jupyterlab or other similar servi
Usage examples
--------------

In the example below, we enable logging for internal Modin API calls.

.. code-block:: python

import modin.pandas as pd
from modin.config import LogMode
LogMode.enable_api_only()

# User code goes here

In the next example, we add logging for not only internal Modin API calls, but also for partition metadata and memory profiling.
In the example below, we enable logging for internal Modin API calls, partition metadata and memory profiling.
We can set the granularity (in seconds) at which the system memory utilization is logged using ``LogMemoryInterval``.
We can also set the maximum size of the logs (in MBs) using ``LogFileSize``.

Expand Down
11 changes: 1 addition & 10 deletions modin/config/envvars.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -501,7 +501,7 @@ class LogMode(EnvironmentVariable, type=ExactStr):
"""Set ``LogMode`` value if users want to opt-in."""

varname = "MODIN_LOG_MODE"
choices = ("enable", "disable", "enable_api_only")
choices = ("enable", "disable")
default = "disable"

@classmethod
Expand All @@ -514,15 +514,6 @@ def disable(cls) -> None:
"""Disable logging feature."""
cls.put("disable")

@classmethod
def enable_api_only(cls) -> None:
"""Enable API level logging."""
warnings.warn(
"'enable_api_only' value for LogMode is deprecated and"
+ "will be removed in a future version."
)
cls.put("enable_api_only")


class LogMemoryInterval(EnvironmentVariable, type=int):
"""Interval (in seconds) to profile memory utilization for logging."""
Expand Down
25 changes: 11 additions & 14 deletions modin/logging/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ def configure_logging() -> None:
"modin.logger.default",
job_id,
"trace",
LogLevel.INFO if LogMode.get() == "enable_api_only" else LogLevel.DEBUG,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we losing the ability to configure the debug level?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes the ability to configure would be lost, however the lower-level functionality would be made debug level (instead of info) in a further PR #7184

LogLevel.INFO,
)

logger.info(f"OS Version: {platform.platform()}")
Expand All @@ -181,20 +181,17 @@ def configure_logging() -> None:
logger.info(f"Physical Cores: {num_physical_cores}")
logger.info(f"Total Cores: {num_total_cores}")

if LogMode.get() != "enable_api_only":
mem_sleep = LogMemoryInterval.get()
mem_logger = _create_logger(
"modin_memory.logger", job_id, "memory", LogLevel.DEBUG
)
mem_sleep = LogMemoryInterval.get()
mem_logger = _create_logger("modin_memory.logger", job_id, "memory", LogLevel.DEBUG)

svmem = psutil.virtual_memory()
mem_logger.info(f"Memory Total: {bytes_int_to_str(svmem.total)}")
mem_logger.info(f"Memory Available: {bytes_int_to_str(svmem.available)}")
mem_logger.info(f"Memory Used: {bytes_int_to_str(svmem.used)}")
mem = threading.Thread(
target=memory_thread, args=[mem_logger, mem_sleep], daemon=True
)
mem.start()
svmem = psutil.virtual_memory()
mem_logger.info(f"Memory Total: {bytes_int_to_str(svmem.total)}")
mem_logger.info(f"Memory Available: {bytes_int_to_str(svmem.available)}")
mem_logger.info(f"Memory Used: {bytes_int_to_str(svmem.used)}")
mem = threading.Thread(
target=memory_thread, args=[mem_logger, mem_sleep], daemon=True
)
mem.start()

_create_logger("modin.logger.errors", job_id, "error", LogLevel.INFO)

Expand Down
Loading