Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] do not fail to start analytics process if memory estimate is lower than predicted memory usage #1465

Merged
merged 2 commits into from
Sep 1, 2020
Merged

[ML] do not fail to start analytics process if memory estimate is lower than predicted memory usage #1465

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c694515
[ML] do not fail to start analytics process of memory estimate is low
benwtrent Aug 31, 2020
c384d9f
fixing test
benwtrent Sep 1, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions lib/api/CDataFrameAnalysisRunner.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,14 +99,14 @@ void CDataFrameAnalysisRunner::computeAndSaveExecutionStrategy() {
auto roundMb = [](std::size_t memory) {
return 0.01 * static_cast<double>((100 * memory) / BYTES_IN_MB);
};

// Report rounded up to the nearest MB.
HANDLE_FATAL(<< "Input error: memory limit " << roundMb(memoryLimit)
<< "MB is too low to perform analysis. You need to give the process"
<< " at least " << std::ceil(roundMb(memoryUsage))
<< "MB, but preferably more.");

} else if (m_NumberPartitions > 1) {
// Simply log the limit being configured too low.
// If we exceed the limit during the process, we will fail and the user
// will have to update the limit and attempt to re-run
LOG_DEBUG(<< "Memory limit " << roundMb(memoryLimit) << "MB is configured lower than estimate "
<< std::ceil(roundMb(memoryUsage)) << "MB."
<< "Analytics process may fail due to low memory limit");
}
if (m_NumberPartitions > 1) {
// The maximum number of rows is found by binary search in the interval
// [numberRows / m_NumberPartitions, numberRows / (m_NumberPartitions - 1)).

Expand Down
8 changes: 2 additions & 6 deletions lib/api/unittest/CDataFrameAnalysisRunnerTest.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,8 @@ BOOST_AUTO_TEST_CASE(testComputeAndSaveExecutionStrategyDiskUsageFlag) {
.diskUsageAllowed(false)
.outlierSpec();

// single error is registered that the memory limit is to low
LOG_DEBUG(<< "errors = " << core::CContainerPrinter::print(errors));
core::CRegex re;
re.init("Input error: memory limit.*");
BOOST_REQUIRE_EQUAL(1, static_cast<int>(errors.size()));
BOOST_TEST_REQUIRE(re.matches(errors[0]));
// no error should be registered
BOOST_REQUIRE_EQUAL(0, static_cast<int>(errors.size()));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Casting to int is a leftover from CppUnit, which required the LHS and RHS types to be exactly the same. Boost.Test doesn't, so the cast is just noise now. But no need to make another commit just to change this, as it had the cast before too.

}

// Test large memory requirement with disk usage
Expand Down