Skip to content

Commit

Permalink
HPCC-33150 Avoid padding logical files with empty parts
Browse files Browse the repository at this point in the history
When targeting a cluster/plane that was wider than the Thor
instance, Thor used to ensure the target file width matched the
width of the target cluster (or plane), by in part, creating
empty physical files.
This behaviour becomes more common in a containerized world,
where the target plane is normally the default data plane with
a width that doesn't relate to a Thor instance.

Turn off this behaviour by default, allow it to be re-enabled
in configuration or on a per output basis as a fail-safe.

Signed-off-by: Jake Smith <jake.smith@lexisnexisrisk.com>
  • Loading branch information
jakesmith committed Dec 23, 2024
1 parent b959a45 commit 0eddb1e
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions thorlcr/mfilemanager/thmfilemanager.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,9 @@ class CFileManager : public CSimpleInterface, implements IThorFileManager
}
else
{
// NB: it is very unlikely this code has ever been used.
// The whole of fixTotal should be deleted at some point (see HPCC-33150)

// Check options to shrink 'large' tgtfile to this cluster size
const char *wideDestOptStr = globals->queryProp("@wideDestOpt");
if (wideDestOptStr)
Expand Down Expand Up @@ -489,12 +492,16 @@ class CFileManager : public CSimpleInterface, implements IThorFileManager
break;
};
#endif
unsigned offset = 0;
unsigned total;
if (restrictedWidth)
total = restrictedWidth;
else
else if (job.getOptBool("legacyLFNTargetWidth"))
{
unsigned offset; // set by fixTotal (always 0), but never used.
total = fixTotal(job, groups, offset);
}
else
total = job.querySlaves();
if (nonLocalIndex)
++total;
StringBuffer dir;
Expand Down

0 comments on commit 0eddb1e

Please sign in to comment.