Uses merlin-dataloader package #547

edknv · 2022-11-22T07:24:20Z

Fixes NVIDIA-Merlin/dataloader#16

Goals ⚽

Replace nvtabular dataloader with the new merlin-dataloader.
User can specify the argument data_loader_engine=‘merlin’ in the T4RecTrainingArguments object to use the merlin data loader. It supports GPU and CPU environments.

Implementation Details 🚧

NVTabular was updated a while ago to use the new merlin-dataloader package, and and as a result, T4rec has already been using merlin-dataloader via NVTabular. The new dataloader had a number of changes on dtypes, etc. and most of the work necessary to make it work with T4rec had already been completed by @sararb in Fix CI test based on the requirements of the new merlin loader #536, Fix dtype mismatch in CLM masking class due to new data loader changes #539, Fix failing ci error related to sparse_names containing features that are not part of the model's schema #541, and Specify output dtype for Normalize op in ETL example to match model expectations #523. This PR proposes to create a loader class that inherits from merlin.loader.torch.Loader. Since NVTabular is using merlin.loader.torch.Loader under the hood, replacing NVTabularDataLoader with MerlinDataLoader pretty much amounts to a name change. There should be no difference in functionality, but this change has the advantage that we make the dependency explicit and it becomes less susceptible to breaking changes in NVTabular.
See Update Transformers4Rec to use new dataloader package dataloader#16 for more details on dtypes and the new dataloader using all the columns in the dataset schema. See also [Task] Use the same logic of the new merlin DataLoader to set the features types in T4Rec #524 for potential future work. It is not clear if we should commit to more work on T4Rec when session-based models have been implemented in Merlin Models and there is plan to support Torch in Models.

Testing Details 🔍

tox -e test-gpu
./ci/test_integration.sh

nvidia-merlin-bot · 2022-11-22T07:25:22Z

Click to view CI Results

GitHub pull request #547 of commit 55a20b0e6d0288679f831815f2a6ed25a325767a, no merge conflicts.
Running as SYSTEM
Setting status of 55a20b0e6d0288679f831815f2a6ed25a325767a to PENDING with url http://merlin-infra1.nvidia.com:8080/job/transformers4rec_tests/322/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/transformers4rec_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git init /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/pull/547/*:refs/remotes/origin/pr/547/* # timeout=10
 > git rev-parse 55a20b0e6d0288679f831815f2a6ed25a325767a^{commit} # timeout=10
Checking out Revision 55a20b0e6d0288679f831815f2a6ed25a325767a (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 55a20b0e6d0288679f831815f2a6ed25a325767a # timeout=10
Commit message: "Uses merlin-dataloader package"
 > git rev-list --no-walk 1b3ba5b18cad2c7b41ed9fd97d85384582a399d6 # timeout=10
[transformers4rec_tests] $ /bin/bash /tmp/jenkins8358601063209756351.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-xax78ggg
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-xax78ggg
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit e5b7351deb9e4885c4038aa0bbc9f146d8477a0e
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (0.6.0+1.g5926fcf)
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (0.0.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (1.8.1)
Requirement already satisfied: distributed>=2022.3.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.1)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.0)
Requirement already satisfied: numba>=0.54 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.56.2)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (7.0.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.64.1)
Requirement already satisfied: dask>=2022.3.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.10.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.19.5)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.3.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /usr/local/lib/python3.8/dist-packages (from scipy->nvtabular==1.6.0+6.ge5b7351d) (1.22.4)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.3.0)
Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.12.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.4)
Requirement already satisfied: psutil>=5.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (5.9.2)
Requirement already satisfied: tblib>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.2.0)
Requirement already satisfied: tornado>=6.0.3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.2)
Requirement already satisfied: urllib3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.26.12)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.4.0)
Requirement already satisfied: locket>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.0)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (8.1.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.1.2)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /usr/local/lib/python3.8/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.39.1)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.8/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.12.0)
Requirement already satisfied: setuptools<60 in /usr/lib/python3/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (45.2.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.8.2)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.52.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.15.0)
Requirement already satisfied: heapdict in /usr/local/lib/python3.8/dist-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.1.0)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata->numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.8.1)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.0.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec
plugins: anyio-3.6.1, xdist-3.0.2, cov-4.0.0
collected 1 item
tests/unit/test_notebooks.py F                                           [100%]
=================================== FAILURES ===================================

_________________________________ test_session _________________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-43/test_session0')
@pytest.mark.skipif(importlib.util.find_spec("cudf") is None, reason="needs cudf")
def test_session(tmpdir):
    BASE_PATH = os.path.join(dirname(TEST_PATH), SESSION_PATH)
    os.environ["INPUT_DATA_DIR"] = "/tmp/data/"
    # Run ETL
    nb_path = os.path.join(BASE_PATH, "01-ETL-with-NVTabular.ipynb")
    _run_notebook(tmpdir, nb_path)

    # Run session based
    torch = importlib.util.find_spec("torch")
    if torch is not None:
        os.environ["INPUT_SCHEMA_PATH"] = BASE_PATH + "schema.pb"
        nb_path = os.path.join(BASE_PATH, "02-session-based-XLNet-with-PyT.ipynb")


      _run_notebook(tmpdir, nb_path)


tests/unit/test_notebooks.py:44:

tests/unit/test_notebooks.py:66: in _run_notebook

subprocess.check_output([sys.executable, script_path])

/usr/lib/python3.8/subprocess.py:415: in check_output

return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

input = None, capture_output = False, timeout = None, check = True

popenargs = (['/usr/bin/python', '/tmp/pytest-of-jenkins/pytest-43/test_session0/notebook.py'],)

kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f1d51d1ec70>

stdout = b"['/tmp/data//sessions_by_day/1/train.parquet']\n********************\nLaunch training for day 1 are:\n********************\n\n"

stderr = None, retcode = 1
def run(*popenargs,
        input=None, capture_output=False, timeout=None, check=False, **kwargs):
    """Run command with arguments and return a CompletedProcess instance.

    The returned instance will have attributes args, returncode, stdout and
    stderr. By default, stdout and stderr are not captured, and those attributes
    will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

    If check is True and the exit code was non-zero, it raises a
    CalledProcessError. The CalledProcessError object will have the return code
    in the returncode attribute, and output & stderr attributes if those streams
    were captured.

    If timeout is given, and the process takes too long, a TimeoutExpired
    exception will be raised.

    There is an optional argument "input", allowing you to
    pass bytes or a string to the subprocess's stdin.  If you use this argument
    you may not also use the Popen constructor's "stdin" argument, as
    it will be used internally.

    By default, all communication is in bytes, and therefore any "input" should
    be bytes, and the stdout and stderr will be bytes. If in text mode, any
    "input" should be a string, and stdout and stderr will be strings decoded
    according to locale encoding, or by "encoding" if set. Text mode is
    triggered by setting any of text, encoding, errors or universal_newlines.

    The other arguments are the same as for the Popen constructor.
    """
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE

    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used '
                             'with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE

    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                # Windows accumulates the output in a single blocking
                # read() call run on child threads, with the timeout
                # being done in a join() on those threads.  communicate()
                # _after_ kill() is required to collect that and add it
                # to the exception.
                exc.stdout, exc.stderr = process.communicate()
            else:
                # POSIX _communicate already populated the output so
                # far into the TimeoutExpired exception.
                process.wait()
            raise
        except:  # Including KeyboardInterrupt, communicate handled that.
            process.kill()
            # We don't call process.wait() as .__exit__ does that for us.
            raise
        retcode = process.poll()
        if check and retcode:


          raise CalledProcessError(retcode, process.args,


                                     output=stdout, stderr=stderr)

E               subprocess.CalledProcessError: Command '['/usr/bin/python', '/tmp/pytest-of-jenkins/pytest-43/test_session0/notebook.py']' returned non-zero exit status 1.
/usr/lib/python3.8/subprocess.py:516: CalledProcessError

----------------------------- Captured stderr call -----------------------------

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
Creating time-based splits:   0%|          | 0/9 [00:00<?, ?it/s]

Creating time-based splits:  11%|█         | 1/9 [00:00<00:01,  5.49it/s]

Creating time-based splits:  22%|██▏       | 2/9 [00:00<00:01,  6.34it/s]

Creating time-based splits:  33%|███▎      | 3/9 [00:00<00:00,  7.02it/s]

Creating time-based splits:  44%|████▍     | 4/9 [00:00<00:00,  7.14it/s]

Creating time-based splits:  56%|█████▌    | 5/9 [00:00<00:00,  7.22it/s]

Creating time-based splits:  67%|██████▋   | 6/9 [00:00<00:00,  7.58it/s]

Creating time-based splits:  78%|███████▊  | 7/9 [00:00<00:00,  7.51it/s]

Creating time-based splits:  89%|████████▉ | 8/9 [00:01<00:00,  7.97it/s]

Creating time-based splits: 100%|██████████| 9/9 [00:01<00:00,  7.84it/s]

Creating time-based splits: 100%|██████████| 9/9 [00:01<00:00,  7.41it/s]

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

Traceback (most recent call last):

File "/tmp/pytest-of-jenkins/pytest-43/test_session0/notebook.py", line 202, in 

trainer.train()

File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1115, in train

train_dataloader = self.get_train_dataloader()

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/trainer.py", line 156, in get_train_dataloader

return T4RecDataLoader.parse(self.args.data_loader_engine).from_schema(

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/utils/data_utils.py", line 332, in from_schema

loader = cls(

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/utils/data_utils.py", line 257, in init

loader = Loader(

TypeError: init() got an unexpected keyword argument 'device'

============================== 1 failed in 21.01s ==============================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=2 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Transformers4Rec/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[transformers4rec_tests] $ /bin/bash /tmp/jenkins6672984936992492400.sh

nvidia-merlin-bot · 2022-11-22T07:26:14Z

Click to view CI Results

GitHub pull request #547 of commit 5fe06bce07f34dc373c8b754998ff827c83f6d96, no merge conflicts.
Running as SYSTEM
Setting status of 5fe06bce07f34dc373c8b754998ff827c83f6d96 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/transformers4rec_tests/323/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/transformers4rec_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git init /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/pull/547/*:refs/remotes/origin/pr/547/* # timeout=10
 > git rev-parse 5fe06bce07f34dc373c8b754998ff827c83f6d96^{commit} # timeout=10
Checking out Revision 5fe06bce07f34dc373c8b754998ff827c83f6d96 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5fe06bce07f34dc373c8b754998ff827c83f6d96 # timeout=10
Commit message: "Merge branch 'main' into merlin_dataloader"
 > git rev-list --no-walk 55a20b0e6d0288679f831815f2a6ed25a325767a # timeout=10
[transformers4rec_tests] $ /bin/bash /tmp/jenkins9547213277815100852.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-doqxi72n
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-doqxi72n
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit e5b7351deb9e4885c4038aa0bbc9f146d8477a0e
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (0.0.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (1.8.1)
Requirement already satisfied: merlin-core>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (0.6.0+1.g5926fcf)
Requirement already satisfied: distributed>=2022.3.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.1)
Requirement already satisfied: numba>=0.54 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.56.2)
Requirement already satisfied: dask>=2022.3.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.1)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.64.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (21.3)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.3.5)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.19.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.0)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.10.0)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /usr/local/lib/python3.8/dist-packages (from scipy->nvtabular==1.6.0+6.ge5b7351d) (1.22.4)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.4.3)
Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (5.4.1)
Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.3.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.2.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.4.0)
Requirement already satisfied: tblib>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.7.0)
Requirement already satisfied: tornado>=6.0.3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.2)
Requirement already satisfied: locket>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.0)
Requirement already satisfied: urllib3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.26.12)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (8.1.3)
Requirement already satisfied: psutil>=5.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (5.9.2)
Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.2.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.4)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.1.2)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.8/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.12.0)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /usr/local/lib/python3.8/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.39.1)
Requirement already satisfied: setuptools<60 in /usr/lib/python3/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (45.2.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.0.9)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.2.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.8.2)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.52.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.15.0)
Requirement already satisfied: heapdict in /usr/local/lib/python3.8/dist-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.0.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata->numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.8.1)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.1.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.0.0)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.0.1)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec
plugins: anyio-3.6.1, xdist-3.0.2, cov-4.0.0
collected 1 item
tests/unit/test_notebooks.py F                                           [100%]
=================================== FAILURES ===================================

_________________________________ test_session _________________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-44/test_session0')
@pytest.mark.skipif(importlib.util.find_spec("cudf") is None, reason="needs cudf")
def test_session(tmpdir):
    BASE_PATH = os.path.join(dirname(TEST_PATH), SESSION_PATH)
    os.environ["INPUT_DATA_DIR"] = "/tmp/data/"
    # Run ETL
    nb_path = os.path.join(BASE_PATH, "01-ETL-with-NVTabular.ipynb")
    _run_notebook(tmpdir, nb_path)

    # Run session based
    torch = importlib.util.find_spec("torch")
    if torch is not None:
        os.environ["INPUT_SCHEMA_PATH"] = BASE_PATH + "schema.pb"
        nb_path = os.path.join(BASE_PATH, "02-session-based-XLNet-with-PyT.ipynb")


      _run_notebook(tmpdir, nb_path)


tests/unit/test_notebooks.py:44:

tests/unit/test_notebooks.py:66: in _run_notebook

subprocess.check_output([sys.executable, script_path])

/usr/lib/python3.8/subprocess.py:415: in check_output

return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

input = None, capture_output = False, timeout = None, check = True

popenargs = (['/usr/bin/python', '/tmp/pytest-of-jenkins/pytest-44/test_session0/notebook.py'],)

kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f58346454f0>

stdout = b"['/tmp/data//sessions_by_day/1/train.parquet']\n********************\nLaunch training for day 1 are:\n********************\n\n"

stderr = None, retcode = 1
def run(*popenargs,
        input=None, capture_output=False, timeout=None, check=False, **kwargs):
    """Run command with arguments and return a CompletedProcess instance.

    The returned instance will have attributes args, returncode, stdout and
    stderr. By default, stdout and stderr are not captured, and those attributes
    will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

    If check is True and the exit code was non-zero, it raises a
    CalledProcessError. The CalledProcessError object will have the return code
    in the returncode attribute, and output & stderr attributes if those streams
    were captured.

    If timeout is given, and the process takes too long, a TimeoutExpired
    exception will be raised.

    There is an optional argument "input", allowing you to
    pass bytes or a string to the subprocess's stdin.  If you use this argument
    you may not also use the Popen constructor's "stdin" argument, as
    it will be used internally.

    By default, all communication is in bytes, and therefore any "input" should
    be bytes, and the stdout and stderr will be bytes. If in text mode, any
    "input" should be a string, and stdout and stderr will be strings decoded
    according to locale encoding, or by "encoding" if set. Text mode is
    triggered by setting any of text, encoding, errors or universal_newlines.

    The other arguments are the same as for the Popen constructor.
    """
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE

    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used '
                             'with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE

    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                # Windows accumulates the output in a single blocking
                # read() call run on child threads, with the timeout
                # being done in a join() on those threads.  communicate()
                # _after_ kill() is required to collect that and add it
                # to the exception.
                exc.stdout, exc.stderr = process.communicate()
            else:
                # POSIX _communicate already populated the output so
                # far into the TimeoutExpired exception.
                process.wait()
            raise
        except:  # Including KeyboardInterrupt, communicate handled that.
            process.kill()
            # We don't call process.wait() as .__exit__ does that for us.
            raise
        retcode = process.poll()
        if check and retcode:


          raise CalledProcessError(retcode, process.args,


                                     output=stdout, stderr=stderr)

E               subprocess.CalledProcessError: Command '['/usr/bin/python', '/tmp/pytest-of-jenkins/pytest-44/test_session0/notebook.py']' returned non-zero exit status 1.
/usr/lib/python3.8/subprocess.py:516: CalledProcessError

----------------------------- Captured stderr call -----------------------------

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
Creating time-based splits:   0%|          | 0/9 [00:00<?, ?it/s]

Creating time-based splits:  11%|█         | 1/9 [00:00<00:01,  6.22it/s]

Creating time-based splits:  22%|██▏       | 2/9 [00:00<00:01,  6.58it/s]

Creating time-based splits:  33%|███▎      | 3/9 [00:00<00:00,  7.31it/s]

Creating time-based splits:  44%|████▍     | 4/9 [00:00<00:00,  7.46it/s]

Creating time-based splits:  56%|█████▌    | 5/9 [00:00<00:00,  7.72it/s]

Creating time-based splits:  67%|██████▋   | 6/9 [00:00<00:00,  7.78it/s]

Creating time-based splits:  78%|███████▊  | 7/9 [00:00<00:00,  7.48it/s]

Creating time-based splits:  89%|████████▉ | 8/9 [00:01<00:00,  7.68it/s]

Creating time-based splits: 100%|██████████| 9/9 [00:01<00:00,  7.97it/s]

Creating time-based splits: 100%|██████████| 9/9 [00:01<00:00,  7.59it/s]

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

Traceback (most recent call last):

File "/tmp/pytest-of-jenkins/pytest-44/test_session0/notebook.py", line 202, in 

trainer.train()

File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1115, in train

train_dataloader = self.get_train_dataloader()

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/trainer.py", line 156, in get_train_dataloader

return T4RecDataLoader.parse(self.args.data_loader_engine).from_schema(

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/utils/data_utils.py", line 332, in from_schema

loader = cls(

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/utils/data_utils.py", line 257, in init

loader = Loader(

TypeError: init() got an unexpected keyword argument 'device'

============================== 1 failed in 21.04s ==============================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=2 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Transformers4Rec/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[transformers4rec_tests] $ /bin/bash /tmp/jenkins1862245947604633010.sh

nvidia-merlin-bot · 2022-11-22T08:01:42Z

Click to view CI Results

GitHub pull request #547 of commit 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48, no merge conflicts.
Running as SYSTEM
Setting status of 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/transformers4rec_tests/324/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/transformers4rec_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git init /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/pull/547/*:refs/remotes/origin/pr/547/* # timeout=10
 > git rev-parse 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48^{commit} # timeout=10
Checking out Revision 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48 # timeout=10
Commit message: "Adds missing requirements file"
 > git rev-list --no-walk 5fe06bce07f34dc373c8b754998ff827c83f6d96 # timeout=10
[transformers4rec_tests] $ /bin/bash /tmp/jenkins15472683788389627609.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/NVTabular.git
  Cloning https://github.com/NVIDIA-Merlin/NVTabular.git to /tmp/pip-req-build-oud5wucs
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/NVTabular.git /tmp/pip-req-build-oud5wucs
  Resolved https://github.com/NVIDIA-Merlin/NVTabular.git to commit e5b7351deb9e4885c4038aa0bbc9f146d8477a0e
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (0.6.0+1.g5926fcf)
Requirement already satisfied: merlin-dataloader>=0.0.2 in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (0.0.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+6.ge5b7351d) (1.8.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (7.0.0)
Requirement already satisfied: distributed>=2022.3.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.1)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (21.3)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.19.5)
Requirement already satisfied: fsspec==2022.5.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.5.0)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.3.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.64.1)
Requirement already satisfied: numba>=0.54 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.56.2)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.5)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /usr/local/lib/python3.8/dist-packages (from scipy->nvtabular==1.6.0+6.ge5b7351d) (1.22.4)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.3.0)
Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (5.4.1)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.12.0)
Requirement already satisfied: locket>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.0)
Requirement already satisfied: tornado>=6.0.3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.2)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (8.1.3)
Requirement already satisfied: urllib3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.26.12)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (5.9.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.1.2)
Requirement already satisfied: tblib>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.4)
Requirement already satisfied: zict>=0.1.3 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.2.0)
Requirement already satisfied: setuptools<60 in /usr/lib/python3/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (45.2.0)
Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.8/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.12.0)
Requirement already satisfied: llvmlite<0.40,>=0.39.0dev0 in /usr/local/lib/python3.8/dist-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (0.39.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.52.0)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.15.0)
Requirement already satisfied: heapdict in /usr/local/lib/python3.8/dist-packages (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.0.2)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata->numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (3.8.1)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.8/dist-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (2.1.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+6.ge5b7351d) (4.0.0)
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
rootdir: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec
plugins: anyio-3.6.1, xdist-3.0.2, cov-4.0.0
collected 1 item
tests/unit/test_notebooks.py F                                           [100%]
=================================== FAILURES ===================================

_________________________________ test_session _________________________________
tmpdir = local('/tmp/pytest-of-jenkins/pytest-45/test_session0')
@pytest.mark.skipif(importlib.util.find_spec("cudf") is None, reason="needs cudf")
def test_session(tmpdir):
    BASE_PATH = os.path.join(dirname(TEST_PATH), SESSION_PATH)
    os.environ["INPUT_DATA_DIR"] = "/tmp/data/"
    # Run ETL
    nb_path = os.path.join(BASE_PATH, "01-ETL-with-NVTabular.ipynb")
    _run_notebook(tmpdir, nb_path)

    # Run session based
    torch = importlib.util.find_spec("torch")
    if torch is not None:
        os.environ["INPUT_SCHEMA_PATH"] = BASE_PATH + "schema.pb"
        nb_path = os.path.join(BASE_PATH, "02-session-based-XLNet-with-PyT.ipynb")


      _run_notebook(tmpdir, nb_path)


tests/unit/test_notebooks.py:44:

tests/unit/test_notebooks.py:66: in _run_notebook

subprocess.check_output([sys.executable, script_path])

/usr/lib/python3.8/subprocess.py:415: in check_output

return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,

input = None, capture_output = False, timeout = None, check = True

popenargs = (['/usr/bin/python', '/tmp/pytest-of-jenkins/pytest-45/test_session0/notebook.py'],)

kwargs = {'stdout': -1}, process = <subprocess.Popen object at 0x7f759d4043d0>

stdout = b"['/tmp/data//sessions_by_day/1/train.parquet']\n********************\nLaunch training for day 1 are:\n********************\n\n"

stderr = None, retcode = 1
def run(*popenargs,
        input=None, capture_output=False, timeout=None, check=False, **kwargs):
    """Run command with arguments and return a CompletedProcess instance.

    The returned instance will have attributes args, returncode, stdout and
    stderr. By default, stdout and stderr are not captured, and those attributes
    will be None. Pass stdout=PIPE and/or stderr=PIPE in order to capture them.

    If check is True and the exit code was non-zero, it raises a
    CalledProcessError. The CalledProcessError object will have the return code
    in the returncode attribute, and output & stderr attributes if those streams
    were captured.

    If timeout is given, and the process takes too long, a TimeoutExpired
    exception will be raised.

    There is an optional argument "input", allowing you to
    pass bytes or a string to the subprocess's stdin.  If you use this argument
    you may not also use the Popen constructor's "stdin" argument, as
    it will be used internally.

    By default, all communication is in bytes, and therefore any "input" should
    be bytes, and the stdout and stderr will be bytes. If in text mode, any
    "input" should be a string, and stdout and stderr will be strings decoded
    according to locale encoding, or by "encoding" if set. Text mode is
    triggered by setting any of text, encoding, errors or universal_newlines.

    The other arguments are the same as for the Popen constructor.
    """
    if input is not None:
        if kwargs.get('stdin') is not None:
            raise ValueError('stdin and input arguments may not both be used.')
        kwargs['stdin'] = PIPE

    if capture_output:
        if kwargs.get('stdout') is not None or kwargs.get('stderr') is not None:
            raise ValueError('stdout and stderr arguments may not be used '
                             'with capture_output.')
        kwargs['stdout'] = PIPE
        kwargs['stderr'] = PIPE

    with Popen(*popenargs, **kwargs) as process:
        try:
            stdout, stderr = process.communicate(input, timeout=timeout)
        except TimeoutExpired as exc:
            process.kill()
            if _mswindows:
                # Windows accumulates the output in a single blocking
                # read() call run on child threads, with the timeout
                # being done in a join() on those threads.  communicate()
                # _after_ kill() is required to collect that and add it
                # to the exception.
                exc.stdout, exc.stderr = process.communicate()
            else:
                # POSIX _communicate already populated the output so
                # far into the TimeoutExpired exception.
                process.wait()
            raise
        except:  # Including KeyboardInterrupt, communicate handled that.
            process.kill()
            # We don't call process.wait() as .__exit__ does that for us.
            raise
        retcode = process.poll()
        if check and retcode:


          raise CalledProcessError(retcode, process.args,


                                     output=stdout, stderr=stderr)

E               subprocess.CalledProcessError: Command '['/usr/bin/python', '/tmp/pytest-of-jenkins/pytest-45/test_session0/notebook.py']' returned non-zero exit status 1.
/usr/lib/python3.8/subprocess.py:516: CalledProcessError

----------------------------- Captured stderr call -----------------------------

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(

/usr/local/lib/python3.8/dist-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
Creating time-based splits:   0%|          | 0/9 [00:00<?, ?it/s]

Creating time-based splits:  11%|█         | 1/9 [00:00<00:01,  6.09it/s]

Creating time-based splits:  22%|██▏       | 2/9 [00:00<00:01,  6.92it/s]

Creating time-based splits:  33%|███▎      | 3/9 [00:00<00:00,  7.26it/s]

Creating time-based splits:  44%|████▍     | 4/9 [00:00<00:00,  7.84it/s]

Creating time-based splits:  56%|█████▌    | 5/9 [00:00<00:00,  7.71it/s]

Creating time-based splits:  67%|██████▋   | 6/9 [00:00<00:00,  7.94it/s]

Creating time-based splits:  78%|███████▊  | 7/9 [00:00<00:00,  7.77it/s]

Creating time-based splits:  89%|████████▉ | 8/9 [00:01<00:00,  7.82it/s]

Creating time-based splits: 100%|██████████| 9/9 [00:01<00:00,  8.26it/s]

Creating time-based splits: 100%|██████████| 9/9 [00:01<00:00,  7.79it/s]

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "

Traceback (most recent call last):

File "/tmp/pytest-of-jenkins/pytest-45/test_session0/notebook.py", line 202, in 

trainer.train()

File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 1115, in train

train_dataloader = self.get_train_dataloader()

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/trainer.py", line 156, in get_train_dataloader

return T4RecDataLoader.parse(self.args.data_loader_engine).from_schema(

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/utils/data_utils.py", line 332, in from_schema

loader = cls(

File "/var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/transformers4rec/torch/utils/data_utils.py", line 257, in init

loader = Loader(

TypeError: init() got an unexpected keyword argument 'device'

============================== 1 failed in 21.18s ==============================

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=2 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Transformers4Rec/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[transformers4rec_tests] $ /bin/bash /tmp/jenkins2814616248376582975.sh

github-actions · 2022-11-22T08:10:08Z

Documentation preview

https://nvidia-merlin.github.io/Transformers4Rec/review/pr-547

edknv · 2022-11-22T08:12:07Z

rerun tests

nvidia-merlin-bot · 2022-11-22T08:13:11Z

Click to view CI Results

GitHub pull request #547 of commit 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48, no merge conflicts.
GitHub pull request #547 of commit 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48, no merge conflicts.
Running as SYSTEM
Setting status of 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/transformers4rec_tests/325/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/transformers4rec_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git init /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/pull/547/*:refs/remotes/origin/pr/547/* # timeout=10
 > git rev-parse 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48^{commit} # timeout=10
Checking out Revision 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48 # timeout=10
Commit message: "Adds missing requirements file"
 > git rev-list --no-walk 3f96aadff3ea92bcc4e9f4cc68b3dc7222e74c48 # timeout=10
[transformers4rec_tests] $ /bin/bash /tmp/jenkins2096972900455356331.sh
GLOB sdist-make: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/setup.py
py38-gpu create: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/.tox/py38-gpu
py38-gpu inst: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/.tox/.tmp/package/1/transformers4rec-0.1.14+19.g3f96aadf.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py38-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.13,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,bleach==5.0.1,boto3==1.24.75,botocore==1.29.13,Brotli==1.0.9,cachetools==5.2.0,certifi==2022.9.24,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.5.1,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,docker-pycreds==0.4.0,docutils==0.16,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==3.4,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,merlin-core==0.6.0+1.g5926fcf,merlin-dataloader==0.0.2,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular @ git+https://github.com/NVIDIA-Merlin/NVTabular.git@e5b7351deb9e4885c4038aa0bbc9f146d8477a0e,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.7.2,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.16.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,tabulate==0.8.10,tblib==1.7.0,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.14+19.g3f96aadf,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
py38-gpu run-test-pre: PYTHONHASHSEED='3054815666'
py38-gpu run-test: commands[0] | pip install --upgrade pip
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in ./.tox/py38-gpu/lib/python3.8/site-packages (22.3.1)
py38-gpu run-test: commands[1] | pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: betterproto<2.0.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+19.g3f96aadf) (1.2.5)
Requirement already satisfied: numpy>=1.17.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+19.g3f96aadf) (1.23.5)
Requirement already satisfied: pyarrow>=1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+19.g3f96aadf) (10.0.0)
Requirement already satisfied: tensorflow-metadata in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+19.g3f96aadf) (1.11.0)
Requirement already satisfied: transformers<4.19 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+19.g3f96aadf) (4.18.0)
Requirement already satisfied: tqdm>=4.27 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+19.g3f96aadf) (4.64.1)
Requirement already satisfied: stringcase in ./.tox/py38-gpu/lib/python3.8/site-packages (from betterproto<2.0.0->transformers4rec==0.1.14+19.g3f96aadf) (1.2.0)
Requirement already satisfied: grpclib in ./.tox/py38-gpu/lib/python3.8/site-packages (from betterproto<2.0.0->transformers4rec==0.1.14+19.g3f96aadf) (0.4.3)
Requirement already satisfied: regex!=2019.12.17 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (2022.10.31)
Requirement already satisfied: requests in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (2.28.1)
Requirement already satisfied: pyyaml>=5.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (6.0)
Requirement already satisfied: filelock in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (3.8.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (0.11.0)
Requirement already satisfied: sacremoses in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (0.0.53)
Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (0.12.1)
Requirement already satisfied: packaging>=20.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (21.3)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+19.g3f96aadf) (1.3.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+19.g3f96aadf) (1.57.0)
Requirement already satisfied: protobuf<4,>=3.13 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+19.g3f96aadf) (3.20.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.tox/py38-gpu/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (4.4.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/py38-gpu/lib/python3.8/site-packages (from packaging>=20.0->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (3.0.9)
Requirement already satisfied: multidict in ./.tox/py38-gpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->transformers4rec==0.1.14+19.g3f96aadf) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->transformers4rec==0.1.14+19.g3f96aadf) (4.1.0)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (3.4)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (2022.9.24)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (1.26.12)
Requirement already satisfied: joblib in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (1.2.0)
Requirement already satisfied: click in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (8.1.3)
Requirement already satisfied: six in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+19.g3f96aadf) (1.16.0)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->transformers4rec==0.1.14+19.g3f96aadf) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->transformers4rec==0.1.14+19.g3f96aadf) (4.0.0)
Building wheels for collected packages: transformers4rec
  Building wheel for transformers4rec (pyproject.toml): started
  Building wheel for transformers4rec (pyproject.toml): finished with status 'done'
  Created wheel for transformers4rec: filename=transformers4rec-0.1.14+19.g3f96aadf-py3-none-any.whl size=217698 sha256=4bd915e2acb9e07f7b7508aaaed1c1ae913fabcf3874fb3aec304656b4e09ca2
  Stored in directory: /tmp/pip-ephem-wheel-cache-gjr_ps_2/wheels/cb/5d/b4/e081835ae498194a418e957657f998bdff0fa2bd103855a861
Successfully built transformers4rec
Installing collected packages: transformers4rec
  Attempting uninstall: transformers4rec
    Found existing installation: transformers4rec 0.1.14+19.g3f96aadf
    Uninstalling transformers4rec-0.1.14+19.g3f96aadf:
      Successfully uninstalled transformers4rec-0.1.14+19.g3f96aadf
Successfully installed transformers4rec-0.1.14+19.g3f96aadf
___________________________________ summary ____________________________________
  py38-gpu: commands succeeded
  congratulations :)
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=2 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Transformers4Rec/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[transformers4rec_tests] $ /bin/bash /tmp/jenkins4711668789356957653.sh

edknv · 2022-11-23T11:31:55Z

The naming convention in dataloader has been changed from merlin.loader to merlin.dataloader: NVIDIA-Merlin/dataloader#57. I'll update the PR.

oliverholworthy · 2022-11-23T16:35:23Z

merlin_standard_lib/utils/misc_utils.py

+        cs = schema[col]
+        properties = cs.properties
+        if sparse_max and col in sparse_max:
+            properties["value_count"] = {"min": sparse_max[col], "max": sparse_max[col]}


the min here can be removed. Made some changes in core recenlty to make that possible. (NVIDIA-Merlin/core#171)

sararb

Thank you for your PR! This makes the t4rec loader much easier to maintain with the merlin loader package.
I have just left two comments about the _augment_schema method

sararb · 2022-11-23T23:34:11Z

merlin_standard_lib/utils/misc_utils.py

    return Dataset(files, engine=engine, **reader_kwargs)
+
+
+def _augment_schema(


In the T4Rec data loader, we have a filtering step happening here to filter out the columns that are not used by the model before passing the dataset to the loader. This step is required to make sure that the data loader is returning only the input features required by the model .

so perhaps we can add this filtering step to the _augment_schema method to keep all the schema processing logic in one place.

sararb · 2022-11-23T23:49:59Z

merlin_standard_lib/utils/misc_utils.py

+    for col in sparse_names or []:
+        cs = schema[col]
+        properties = cs.properties
+        if sparse_max and col in sparse_max:


From what I understand: sparse_max is a dict where keys are the sparse feature names and values are the max-length to use for padding, and sparse_names is the list of sparse feature names. So is there a use case where the list sparse_names is different from the list of the sparse_max keys?

This logic was ported over from nvtabular, but I think there are some cases where sparse_names and sparse_max might be different (or sparse_max is under-specified). For example, when we want the value count of a feature to be unbounded, we would leave out sparse_max but keep sparse_names.

nvidia-merlin-bot · 2022-11-28T18:38:56Z

Click to view CI Results

GitHub pull request #547 of commit bfeff1e589d7689c161d6b2c2649a113883c1783, has merge conflicts.
Running as SYSTEM
Setting status of bfeff1e589d7689c161d6b2c2649a113883c1783 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/transformers4rec_tests/354/ and message: 'Build started for original commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/transformers4rec_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git init /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/pull/547/*:refs/remotes/origin/pr/547/* # timeout=10
 > git rev-parse bfeff1e589d7689c161d6b2c2649a113883c1783^{commit} # timeout=10
Checking out Revision bfeff1e589d7689c161d6b2c2649a113883c1783 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bfeff1e589d7689c161d6b2c2649a113883c1783 # timeout=10
Commit message: "Rename to dataloader; Move column selection; Use only max value count"
 > git rev-list --no-walk 4cbf9d64604be23494b026a7b345f65d630659ea # timeout=10
[transformers4rec_tests] $ /bin/bash /tmp/jenkins3889521142059852045.sh
GLOB sdist-make: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/setup.py
py38-gpu create: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/.tox/py38-gpu
py38-gpu inst: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/.tox/.tmp/package/1/transformers4rec-0.1.14+20.gbfeff1e5.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py38-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.17,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,bleach==5.0.1,boto3==1.24.75,botocore==1.29.17,Brotli==1.0.9,cachetools==5.2.0,certifi==2022.9.24,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,docker-pycreds==0.4.0,docutils==0.16,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==3.4,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular==1.4.0+8.g95e12d347,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.7.2,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.16.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,tabulate==0.8.10,tblib==1.7.0,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.14+20.gbfeff1e5,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
py38-gpu run-test-pre: PYTHONHASHSEED='1881429804'
py38-gpu run-test: commands[0] | pip install --upgrade pip
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in ./.tox/py38-gpu/lib/python3.8/site-packages (22.3.1)
py38-gpu run-test: commands[1] | pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tqdm>=4.27 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+20.gbfeff1e5) (4.64.1)
Requirement already satisfied: transformers<4.19 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+20.gbfeff1e5) (4.18.0)
Requirement already satisfied: betterproto<2.0.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+20.gbfeff1e5) (1.2.5)
Requirement already satisfied: numpy>=1.17.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+20.gbfeff1e5) (1.23.5)
Requirement already satisfied: pyarrow>=1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+20.gbfeff1e5) (10.0.1)
Requirement already satisfied: tensorflow-metadata in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+20.gbfeff1e5) (1.11.0)
Requirement already satisfied: stringcase in ./.tox/py38-gpu/lib/python3.8/site-packages (from betterproto<2.0.0->transformers4rec==0.1.14+20.gbfeff1e5) (1.2.0)
Requirement already satisfied: grpclib in ./.tox/py38-gpu/lib/python3.8/site-packages (from betterproto<2.0.0->transformers4rec==0.1.14+20.gbfeff1e5) (0.4.3)
Requirement already satisfied: sacremoses in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (0.0.53)
Requirement already satisfied: filelock in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (3.8.0)
Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (0.12.1)
Requirement already satisfied: requests in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (2.28.1)
Requirement already satisfied: pyyaml>=5.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (6.0)
Requirement already satisfied: packaging>=20.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (21.3)
Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (0.11.1)
Requirement already satisfied: regex!=2019.12.17 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (2022.10.31)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+20.gbfeff1e5) (1.57.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+20.gbfeff1e5) (1.3.0)
Requirement already satisfied: protobuf<4,>=3.13 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+20.gbfeff1e5) (3.20.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.tox/py38-gpu/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (4.4.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/py38-gpu/lib/python3.8/site-packages (from packaging>=20.0->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (3.0.9)
Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->transformers4rec==0.1.14+20.gbfeff1e5) (4.1.0)
Requirement already satisfied: multidict in ./.tox/py38-gpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->transformers4rec==0.1.14+20.gbfeff1e5) (6.0.2)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (2022.9.24)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (1.26.13)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (3.4)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (2.1.1)
Requirement already satisfied: joblib in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (1.2.0)
Requirement already satisfied: six in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (1.16.0)
Requirement already satisfied: click in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+20.gbfeff1e5) (8.1.3)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->transformers4rec==0.1.14+20.gbfeff1e5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->transformers4rec==0.1.14+20.gbfeff1e5) (4.0.0)
Building wheels for collected packages: transformers4rec
  Building wheel for transformers4rec (pyproject.toml): started
  Building wheel for transformers4rec (pyproject.toml): finished with status 'done'
  Created wheel for transformers4rec: filename=transformers4rec-0.1.14+20.gbfeff1e5-py3-none-any.whl size=217706 sha256=26e21bbc46a9e5c18912b4d61476f9d3b6f7bd6a71dd2e2f9eb7c9f09701f457
  Stored in directory: /tmp/pip-ephem-wheel-cache-yywnxg4f/wheels/cb/5d/b4/e081835ae498194a418e957657f998bdff0fa2bd103855a861
Successfully built transformers4rec
Installing collected packages: transformers4rec
  Attempting uninstall: transformers4rec
    Found existing installation: transformers4rec 0.1.14+20.gbfeff1e5
    Uninstalling transformers4rec-0.1.14+20.gbfeff1e5:
      Successfully uninstalled transformers4rec-0.1.14+20.gbfeff1e5
Successfully installed transformers4rec-0.1.14+20.gbfeff1e5
___________________________________ summary ____________________________________
  py38-gpu: commands succeeded
  congratulations :)
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=2 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Transformers4Rec/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[transformers4rec_tests] $ /bin/bash /tmp/jenkins5051764584264130128.sh

nvidia-merlin-bot · 2022-11-28T18:44:26Z

Click to view CI Results

GitHub pull request #547 of commit e9d78237b34cb6e03c7b47d62bdeb86bef23875d, no merge conflicts.
Running as SYSTEM
Setting status of e9d78237b34cb6e03c7b47d62bdeb86bef23875d to PENDING with url http://merlin-infra1.nvidia.com:8080/job/transformers4rec_tests/355/ and message: 'Build started for merge commit.'
Using context: Jenkins Unit Test Run
Building on master in workspace /var/jenkins_home/workspace/transformers4rec_tests
using credential nvidia-merlin-bot
Cloning the remote Git repository
Cloning repository https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git init /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/Transformers4Rec.git # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/Transformers4Rec.git
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/Transformers4Rec.git +refs/pull/547/*:refs/remotes/origin/pr/547/* # timeout=10
 > git rev-parse e9d78237b34cb6e03c7b47d62bdeb86bef23875d^{commit} # timeout=10
Checking out Revision e9d78237b34cb6e03c7b47d62bdeb86bef23875d (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e9d78237b34cb6e03c7b47d62bdeb86bef23875d # timeout=10
Commit message: "Merge branch 'main' into merlin_dataloader"
 > git rev-list --no-walk bfeff1e589d7689c161d6b2c2649a113883c1783 # timeout=10
[transformers4rec_tests] $ /bin/bash /tmp/jenkins10393458489005852344.sh
GLOB sdist-make: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/setup.py
py38-gpu create: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/.tox/py38-gpu
py38-gpu inst: /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec/.tox/.tmp/package/1/transformers4rec-0.1.14+26.ge9d78237.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py38-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.17,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,bleach==5.0.1,boto3==1.24.75,botocore==1.29.17,Brotli==1.0.9,cachetools==5.2.0,certifi==2022.9.24,cffi==1.15.1,charset-normalizer==2.1.1,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,docker-pycreds==0.4.0,docutils==0.16,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==3.4,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.7.0+11.g280956aa4,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy-extensions==0.4.3,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,nvtabular==1.4.0+8.g95e12d347,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.3,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathtools==0.1.2,pbr==5.11.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,Pygments==2.13.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,regex==2022.9.13,requests==2.28.1,requests-oauthlib==1.3.1,rsa==4.7.2,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.16.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,tabulate==0.8.10,tblib==1.7.0,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.14+26.ge9d78237,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typing-inspect==0.8.0,typing_extensions==4.3.0,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,wrapt==1.12.1,xgboost==1.6.2,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
py38-gpu run-test-pre: PYTHONHASHSEED='5750704'
py38-gpu run-test: commands[0] | pip install --upgrade pip
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: pip in ./.tox/py38-gpu/lib/python3.8/site-packages (22.3.1)
py38-gpu run-test: commands[1] | pip install .
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /var/jenkins_home/workspace/transformers4rec_tests/transformers4rec
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: transformers<4.19 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+26.ge9d78237) (4.18.0)
Requirement already satisfied: tensorflow-metadata in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+26.ge9d78237) (1.11.0)
Requirement already satisfied: pyarrow>=1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+26.ge9d78237) (10.0.1)
Requirement already satisfied: tqdm>=4.27 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+26.ge9d78237) (4.64.1)
Requirement already satisfied: numpy>=1.17.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+26.ge9d78237) (1.23.5)
Requirement already satisfied: betterproto<2.0.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers4rec==0.1.14+26.ge9d78237) (1.2.5)
Requirement already satisfied: grpclib in ./.tox/py38-gpu/lib/python3.8/site-packages (from betterproto<2.0.0->transformers4rec==0.1.14+26.ge9d78237) (0.4.3)
Requirement already satisfied: stringcase in ./.tox/py38-gpu/lib/python3.8/site-packages (from betterproto<2.0.0->transformers4rec==0.1.14+26.ge9d78237) (1.2.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (0.11.1)
Requirement already satisfied: sacremoses in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (0.0.53)
Requirement already satisfied: regex!=2019.12.17 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (2022.10.31)
Requirement already satisfied: filelock in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (3.8.0)
Requirement already satisfied: packaging>=20.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (21.3)
Requirement already satisfied: requests in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (2.28.1)
Requirement already satisfied: tokenizers!=0.11.3,<0.13,>=0.11.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (0.12.1)
Requirement already satisfied: pyyaml>=5.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (6.0)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+26.ge9d78237) (1.3.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+26.ge9d78237) (1.57.0)
Requirement already satisfied: protobuf<4,>=3.13 in ./.tox/py38-gpu/lib/python3.8/site-packages (from tensorflow-metadata->transformers4rec==0.1.14+26.ge9d78237) (3.20.3)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.tox/py38-gpu/lib/python3.8/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (4.4.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.tox/py38-gpu/lib/python3.8/site-packages (from packaging>=20.0->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (3.0.9)
Requirement already satisfied: h2<5,>=3.1.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->transformers4rec==0.1.14+26.ge9d78237) (4.1.0)
Requirement already satisfied: multidict in ./.tox/py38-gpu/lib/python3.8/site-packages (from grpclib->betterproto<2.0.0->transformers4rec==0.1.14+26.ge9d78237) (6.0.2)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (1.26.13)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (2022.9.24)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/py38-gpu/lib/python3.8/site-packages (from requests->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (3.4)
Requirement already satisfied: joblib in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (1.2.0)
Requirement already satisfied: six in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (1.16.0)
Requirement already satisfied: click in ./.tox/py38-gpu/lib/python3.8/site-packages (from sacremoses->transformers<4.19->transformers4rec==0.1.14+26.ge9d78237) (8.1.3)
Requirement already satisfied: hyperframe<7,>=6.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->transformers4rec==0.1.14+26.ge9d78237) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->transformers4rec==0.1.14+26.ge9d78237) (4.0.0)
Building wheels for collected packages: transformers4rec
  Building wheel for transformers4rec (pyproject.toml): started
  Building wheel for transformers4rec (pyproject.toml): finished with status 'done'
  Created wheel for transformers4rec: filename=transformers4rec-0.1.14+26.ge9d78237-py3-none-any.whl size=478088 sha256=b5673e2599b572a755dcd6a2310dbadf8147b7060026285e848b9641e4cbcffb
  Stored in directory: /tmp/pip-ephem-wheel-cache-32slx93y/wheels/cb/5d/b4/e081835ae498194a418e957657f998bdff0fa2bd103855a861
Successfully built transformers4rec
Installing collected packages: transformers4rec
  Attempting uninstall: transformers4rec
    Found existing installation: transformers4rec 0.1.14+26.ge9d78237
    Uninstalling transformers4rec-0.1.14+26.ge9d78237:
      Successfully uninstalled transformers4rec-0.1.14+26.ge9d78237
Successfully installed transformers4rec-0.1.14+26.ge9d78237
___________________________________ summary ____________________________________
  py38-gpu: commands succeeded
  congratulations :)
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash
cd /var/jenkins_home/
CUDA_VISIBLE_DEVICES=2 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/Transformers4Rec/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log" 
[transformers4rec_tests] $ /bin/bash /tmp/jenkins7288390718275752726.sh

sararb

Thank you @edknv for the updates! I pulled your PR locally and run the ci tests with the latest core and dataloader packages. Everything runs fine without any issues.

* Uses merlin-dataloader package * Adds missing requirements file * Rename to dataloader; Move column selection; Use only max value count

* remove max_steps 20 in XLNet test * update main script to include simulated inference score * fix masked language modeling * fix the trainer.prediction_step during inference mode * clean up redundant code * update masking tests * set the maximum sequence length inside the transformers configs * fix naming * update recall method * Uses merlin-dataloader package (#547) * Uses merlin-dataloader package * Adds missing requirements file * Rename to dataloader; Move column selection; Use only max value count * update the code to use merlin dataloader Co-authored-by: edknv <109497216+edknv@users.noreply.github.com>

Uses merlin-dataloader package

55a20b0

edknv added the chore Maintenance for the repository label Nov 22, 2022

edknv requested a review from sararb November 22, 2022 07:24

edknv self-assigned this Nov 22, 2022

Merge branch 'main' into merlin_dataloader

5fe06bc

Adds missing requirements file

3f96aad

rnyak requested a review from gabrielspmoreira November 22, 2022 15:06

oliverholworthy reviewed Nov 23, 2022

View reviewed changes

sararb reviewed Nov 23, 2022

View reviewed changes

Rename to dataloader; Move column selection; Use only max value count

bfeff1e

Merge branch 'main' into merlin_dataloader

e9d7823

edknv requested review from sararb and oliverholworthy November 28, 2022 18:55

sararb approved these changes Nov 30, 2022

View reviewed changes

sararb mentioned this pull request Nov 30, 2022

Transformers4Rec Docs Restructure #428

Merged

edknv merged commit dcc353e into NVIDIA-Merlin:main Nov 30, 2022

sararb pushed a commit that referenced this pull request Nov 30, 2022

Uses merlin-dataloader package (#547)

aebc66c

* Uses merlin-dataloader package * Adds missing requirements file * Rename to dataloader; Move column selection; Use only max value count

oliverholworthy mentioned this pull request Dec 6, 2022

Support multiple label columns NVIDIA-Merlin/dataloader#80

Merged

sararb added the breaking Breaking change label Dec 15, 2022

edknv mentioned this pull request Dec 15, 2022

Change data_loader_engine to 'merlin' in examples #580

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uses merlin-dataloader package #547

Uses merlin-dataloader package #547

edknv commented Nov 22, 2022 •

edited by sararb

Loading

nvidia-merlin-bot commented Nov 22, 2022

nvidia-merlin-bot commented Nov 22, 2022

nvidia-merlin-bot commented Nov 22, 2022

github-actions bot commented Nov 22, 2022

edknv commented Nov 22, 2022

nvidia-merlin-bot commented Nov 22, 2022

edknv commented Nov 23, 2022

oliverholworthy Nov 23, 2022

sararb left a comment •

edited

Loading

sararb Nov 23, 2022

sararb Nov 23, 2022

edknv Nov 28, 2022

nvidia-merlin-bot commented Nov 28, 2022

nvidia-merlin-bot commented Nov 28, 2022

sararb left a comment

		return Dataset(files, engine=engine, **reader_kwargs)


		def _augment_schema(

Uses merlin-dataloader package #547

Uses merlin-dataloader package #547

Conversation

edknv commented Nov 22, 2022 • edited by sararb Loading

Goals ⚽

Implementation Details 🚧

Testing Details 🔍

nvidia-merlin-bot commented Nov 22, 2022

nvidia-merlin-bot commented Nov 22, 2022

nvidia-merlin-bot commented Nov 22, 2022

github-actions bot commented Nov 22, 2022

Documentation preview

edknv commented Nov 22, 2022

nvidia-merlin-bot commented Nov 22, 2022

edknv commented Nov 23, 2022

oliverholworthy Nov 23, 2022

Choose a reason for hiding this comment

sararb left a comment • edited Loading

Choose a reason for hiding this comment

sararb Nov 23, 2022

Choose a reason for hiding this comment

sararb Nov 23, 2022

Choose a reason for hiding this comment

edknv Nov 28, 2022

Choose a reason for hiding this comment

nvidia-merlin-bot commented Nov 28, 2022

nvidia-merlin-bot commented Nov 28, 2022

sararb left a comment

Choose a reason for hiding this comment

edknv commented Nov 22, 2022 •

edited by sararb

Loading

sararb left a comment •

edited

Loading