Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Qual tool: Print more useful log messages when failures happen downloading dependencies #1292

Merged
merged 4 commits into from
Aug 19, 2024
Merged

Qual tool: Print more useful log messages when failures happen downloading dependencies #1292

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
33e3bb0
checkpoint
tgravescs Aug 15, 2024
16bda39
update errors
tgravescs Aug 15, 2024
47393d7
misc fixes
tgravescs Aug 15, 2024
6dd0466
remove debug code
tgravescs Aug 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 26 additions & 15 deletions user_tools/src/spark_rapids_pytools/rapids/rapids_tool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -151,14 +151,18 @@ def _process_job_submission_args(self):

@phase_banner('Process-Arguments')
def _process_arguments(self):
# 0- process the output location
self._process_output_args()
# 1- process any arguments to be passed to the RAPIDS tool
self._process_rapids_args()
# 2- we need to process the arguments of the CLI
self._process_custom_args()
# 3- process submission arguments
self._process_job_submission_args()
try:
# 0- process the output location
self._process_output_args()
# 1- process any arguments to be passed to the RAPIDS tool
self._process_rapids_args()
# 2- we need to process the arguments of the CLI
self._process_custom_args()
# 3- process submission arguments
self._process_job_submission_args()
except Exception as ex: # pylint: disable=broad-except
self.logger.error('Failed in processing arguments')
raise ex

@phase_banner('Initialization')
def _init_tool(self):
Expand Down Expand Up @@ -387,14 +391,21 @@ class RapidsJarTool(RapidsTool):
"""

def _process_jar_arg(self):
jar_path = ''
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems redundant because the exception is re-raised if the jars are not downloaded successfully.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you clarify what you think is redundant here? Just the setting of jar_path? the main additions in this are the messages being printed not the exception stacks

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I meant we can remove setting the jar_path explicitly here. Sorry, it wasn't clear in my last comment.

tools_jar_url = self.wrapper_options.get('toolsJar')
if tools_jar_url is None:
tools_jar_url = self.ctxt.get_rapids_jar_url()
# download the jar
jar_path = self.ctxt.platform.storage.download_resource(tools_jar_url,
self.ctxt.get_local_work_dir(),
fail_ok=False,
create_dir=True)
try:
if tools_jar_url is None:
tools_jar_url = self.ctxt.get_rapids_jar_url()
# download the jar
self.logger.info('Downloading the tools jars %s', tools_jar_url)
jar_path = self.ctxt.platform.storage.download_resource(tools_jar_url,
parthosa marked this conversation as resolved.
Show resolved Hide resolved
self.ctxt.get_local_work_dir(),
fail_ok=False,
create_dir=True)
except Exception as e: # pylint: disable=broad-except
self.logger.exception('Exception occurred downloading jar %s', tools_jar_url)
raise e

self.logger.info('RAPIDS accelerator tools jar is downloaded to work_dir %s', jar_path)
# get the jar file name
jar_file_name = FSUtil.get_resource_name(jar_path)
Expand Down
2 changes: 2 additions & 0 deletions user_tools/src/spark_rapids_pytools/rapids/tool_ctxt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,13 +169,15 @@ def get_local_work_dir(self) -> str:
return self.get_local('depFolder')

def get_rapids_jar_url(self) -> str:
self.logger.info('Fetching the Rapids Jar URL')
# get the version from the package, instead of the yaml file
# jar_version = self.get_value('sparkRapids', 'version')
if self.is_fatwheel_mode():
offline_path_regex = FSUtil.build_path(self.get_cache_folder(), 'rapids-4-spark-tools_*.jar')
matching_files = glob(offline_path_regex)
if not matching_files:
raise FileNotFoundError('In Fat Mode. No matching JAR files found.')
self.logger.info('Using jar from wheel file %s', matching_files[0])
return matching_files[0]
mvn_base_url = self.get_value('sparkRapids', 'mvnUrl')
jar_version = Utilities.get_latest_mvn_jar_from_metadata(mvn_base_url)
Expand Down