Fix the inference of transformer-based models trained with masked language modeling #909

sararb · 2022-12-01T16:15:51Z

Fixes #907

Goals ⚽

This PR implements two blocks needed to support the inference of transformer-based next-item prediction models in Merlin Models.
For that, the user should specify two specific blocks: SequenceMaskLastInference and inference_hidden_state as follows:

BertBlock(
            d_model=48,
            n_head=8,
            n_layer=2,
            pre=mm.SequentialBlock([SequenceMaskLastInference(), mm.ReplaceMaskedEmbeddings()]),
            post="inference_hidden_state",
)

Implementation Details 🚧

Implement the SequenceMaskLastInference block that generates the boolean mask to indicate the position of the next-item during inference.
Implement the block InferenceHiddenState that extracts the hidden representation related to the next-item, from the mask generated by SequenceMaskLastInference.

Testing Details 🔍

update the test test_transformer_with_masked_language_modeling to showcase the inference logic.

merlin/models/tf/transforms/sequence.py

sararb · 2022-12-01T16:23:15Z

merlin/models/tf/transformers/block.py

@@ -91,7 +91,7 @@ def __init__(
            self.transformer = get_tf_main_layer(transformer)
        else:
            self.transformer = transformer
-
+        self.transformer.supports_masking = True


set supports_masking=True so that the transformer layer can forward the mask tensor created by SequenceMaskLastInference

nvidia-merlin-bot · 2022-12-01T17:14:40Z

Click to view CI Results

GitHub pull request #909 of commit 6a5e5820e135c20afea50853526ace50d385f28e, no merge conflicts.
Running as SYSTEM
Setting status of 6a5e5820e135c20afea50853526ace50d385f28e to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_models/1981/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_models/workspace
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/909/*:refs/remotes/origin/pr/909/* # timeout=10
 > git rev-parse 6a5e5820e135c20afea50853526ace50d385f28e^{commit} # timeout=10
Checking out Revision 6a5e5820e135c20afea50853526ace50d385f28e (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 6a5e5820e135c20afea50853526ace50d385f28e # timeout=10
Commit message: "first solution for inference support"
 > git rev-list --no-walk 25d61d1a6a9996c526aac030b4397aefa5c30dc0 # timeout=10
[workspace] $ /bin/bash /tmp/jenkins4632028146081311140.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: traitlets>=5.2.2 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (5.4.0)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
GLOB sdist-make: /var/jenkins_home/workspace/merlin_models/models/setup.py
py38-gpu recreate: /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu
py38-gpu installdeps: -rrequirements/dev.txt, tensorflow<2.10
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py38-gpu inst: /var/jenkins_home/workspace/merlin_models/models/.tox/.tmp/package/4/merlin-models-0.9.0+63.g6a5e5820.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py38-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,appdirs==1.4.4,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==20.8b1,bleach==5.0.1,bokeh==3.0.2,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,build==0.9.0,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,check-manifest==0.48,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flake8==6.0.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,interrogate==1.5.0,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,isort==5.10.1,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter==1.0.0,jupyter-cache==0.4.3,jupyter-console==6.4.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mccabe==0.7.0,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.9.0+63.g6a5e5820,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy==0.991,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathspec==0.10.2,pathtools==0.1.2,pbr==5.11.0,pep517==0.13.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycodestyle==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,pyflakes==3.0.1,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,qtconsole==5.4.0,QtPy==2.3.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typed-ast==1.5.4,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,xyzservices==2022.9.0,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
py38-gpu run-test-pre: PYTHONHASHSEED='3140736048'
py38-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-6z2evmyw
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-6z2evmyw
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in ./.tox/py38-gpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: setuptools in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=d94627cef632f8b78307633e9d04c5f2542e00f3c198437229c8b1dca8cf3bd3
  Stored in directory: /tmp/pip-ephem-wheel-cache-fvhv9732/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
py38-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/nvtabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/nvtabular.git
  Cloning https://github.com/NVIDIA-Merlin/nvtabular.git to /tmp/pip-req-build-ek7wl4gp
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/nvtabular.git /tmp/pip-req-build-ek7wl4gp
  Resolved https://github.com/NVIDIA-Merlin/nvtabular.git to commit 3bcd555a7fdc1cdaca47379676c2f111750352a8
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from nvtabular==1.6.0+12.g3bcd555a) (0.9.0+14.g4f73ff5)
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+12.g3bcd555a) (1.8.1)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.7 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.10.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.3.0)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.5)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.55.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.3.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.5.0)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (7.0.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.3.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (3.19.5)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (4.64.1)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+12.g3bcd555a) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.12.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (5.4.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.2.0)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (5.8.0)
Requirement already satisfied: tornado>=6.0.3 in ./.tox/py38-gpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (6.2)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.7.0)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.0.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (8.1.3)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.4.0)
Requirement already satisfied: setuptools in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (1.0.1)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (6.0.2)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (4.1.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (2.0.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+12.g3bcd555a) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+12.g3bcd555a-cp38-cp38-linux_x86_64.whl size=257601 sha256=d9a39fae034e2f40b0aeaf3fb167947fce69b6c81ae6bd32202a8be66cbf81e8
  Stored in directory: /tmp/pip-ephem-wheel-cache-rj0dx4vo/wheels/8f/d9/f9/30f2cdc5bf8787fae6fdfe55afd6e1b493e619ec32c32ec40b
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=87caba767c64d94674ec9e4c1f7223487765aace18a0fe68e0ef9d0366b6bd8c
  Stored in directory: /tmp/pip-ephem-wheel-cache-rj0dx4vo/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+12.g3bcd555a
py38-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/py38-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 886 items
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  0%]

tests/unit/datasets/test_ecommerce.py ..sss.s                            [  1%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  2%]

tests/unit/datasets/test_social.py .                                     [  2%]

tests/unit/datasets/test_synthetic.py ..........                         [  3%]

tests/unit/implicit/test_implicit.py .                                   [  3%]

tests/unit/lightfm/test_lightfm.py .                                     [  3%]

tests/unit/tf/test_core.py ......                                        [  4%]

tests/unit/tf/test_loader.py ................                            [  6%]

tests/unit/tf/test_public_api.py .                                       [  6%]

tests/unit/tf/blocks/test_cross.py ...........                           [  7%]

tests/unit/tf/blocks/test_dlrm.py ..........                             [  8%]

tests/unit/tf/blocks/test_interactions.py ...                            [  9%]

tests/unit/tf/blocks/test_mlp.py ....................................... [ 13%]

............................                                             [ 16%]

tests/unit/tf/blocks/test_optimizer.py s................................ [ 20%]

.....................                                                    [ 22%]

tests/unit/tf/blocks/retrieval/test_base.py .                            [ 22%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 23%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ............            [ 24%]

tests/unit/tf/blocks/sampling/test_cross_batch.py .                      [ 24%]

tests/unit/tf/blocks/sampling/test_in_batch.py .                         [ 24%]

tests/unit/tf/core/test_aggregation.py .........                         [ 25%]

tests/unit/tf/core/test_base.py ..                                       [ 25%]

tests/unit/tf/core/test_combinators.py s.....................            [ 28%]

tests/unit/tf/core/test_encoder.py ..                                    [ 28%]

tests/unit/tf/core/test_index.py ...                                     [ 29%]

tests/unit/tf/core/test_prediction.py ..                                 [ 29%]

tests/unit/tf/core/test_tabular.py ......                                [ 29%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 30%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 30%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 30%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 30%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 30%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 30%]

tests/unit/tf/examples/test_07_train_traditional_models.py .             [ 30%]

tests/unit/tf/examples/test_usecase_accelerate_training_by_lazyadam.py . [ 30%]

[ 30%]

tests/unit/tf/examples/test_usecase_data_parallel.py .                   [ 30%]

tests/unit/tf/examples/test_usecase_ecommerce_session_based.py .         [ 31%]

tests/unit/tf/examples/test_usecase_incremental_training_layer_freezing.py . [ 31%]

[ 31%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 31%]

tests/unit/tf/examples/test_usecase_retrieval_with_hpo.py .              [ 31%]

tests/unit/tf/examples/test_usecase_transformers_next_item_prediction.py . [ 31%]

[ 31%]

tests/unit/tf/horovod/test_horovod.py ...                                [ 31%]

tests/unit/tf/inputs/test_base.py .                                      [ 31%]

tests/unit/tf/inputs/test_continuous.py ........                         [ 32%]

tests/unit/tf/inputs/test_embedding.py ................................. [ 36%]

........                                                                 [ 37%]

tests/unit/tf/inputs/test_tabular.py ..................                  [ 39%]

tests/unit/tf/layers/test_queue.py ..............                        [ 41%]

tests/unit/tf/losses/test_losses.py .......................              [ 43%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 44%]

tests/unit/tf/metrics/test_metrics_topk.py .........................     [ 47%]

tests/unit/tf/models/test_base.py s.........................             [ 50%]

tests/unit/tf/models/test_benchmark.py ..                                [ 50%]

tests/unit/tf/models/test_ranking.py ..................................  [ 54%]

tests/unit/tf/models/test_retrieval.py ................................. [ 57%]

.............................................                            [ 62%]

tests/unit/tf/outputs/test_base.py ......                                [ 63%]

tests/unit/tf/outputs/test_classification.py ......                      [ 64%]

tests/unit/tf/outputs/test_contrastive.py ..............                 [ 65%]

tests/unit/tf/outputs/test_regression.py ..                              [ 66%]

tests/unit/tf/outputs/test_sampling.py ....                              [ 66%]

tests/unit/tf/outputs/test_topk.py .                                     [ 66%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 66%]

tests/unit/tf/prediction_tasks/test_multi_task.py ................       [ 68%]

tests/unit/tf/prediction_tasks/test_next_item.py .....                   [ 69%]

tests/unit/tf/prediction_tasks/test_regression.py .....                  [ 69%]

tests/unit/tf/prediction_tasks/test_retrieval.py .                       [ 69%]

tests/unit/tf/prediction_tasks/test_sampling.py ......                   [ 70%]

tests/unit/tf/transformers/test_block.py ..................FF..          [ 73%]

tests/unit/tf/transformers/test_transforms.py ..........                 [ 74%]

tests/unit/tf/transforms/test_bias.py ..                                 [ 74%]

tests/unit/tf/transforms/test_features.py s............................. [ 77%]

.......................s......                                           [ 81%]

tests/unit/tf/transforms/test_negative_sampling.py .........             [ 82%]

tests/unit/tf/transforms/test_noise.py .....                             [ 82%]

tests/unit/tf/transforms/test_sequence.py ....................           [ 84%]

tests/unit/tf/transforms/test_tensor.py ...                              [ 85%]

tests/unit/tf/utils/test_batch.py ....                                   [ 85%]

tests/unit/tf/utils/test_dataset.py ..                                   [ 86%]

tests/unit/tf/utils/test_tf_utils.py .....                               [ 86%]

tests/unit/torch/test_dataset.py .........                               [ 87%]

tests/unit/torch/test_public_api.py .                                    [ 87%]

tests/unit/torch/block/test_base.py ....                                 [ 88%]

tests/unit/torch/block/test_mlp.py .                                     [ 88%]

tests/unit/torch/features/test_continuous.py ..                          [ 88%]

tests/unit/torch/features/test_embedding.py ..............               [ 90%]

tests/unit/torch/features/test_tabular.py ....                           [ 90%]

tests/unit/torch/model/test_head.py ............                         [ 91%]

tests/unit/torch/model/test_model.py ..                                  [ 92%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 93%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 93%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 94%]

tests/unit/utils/test_schema_utils.py ................................   [ 97%]

tests/unit/xgb/test_xgboost.py ....................                      [100%]
=================================== FAILURES ===================================

_____________ test_transformer_with_masked_language_modeling[True] _____________
sequence_testing_data = <merlin.io.dataset.Dataset object at 0x7f9f47463400>

run_eagerly = True
@pytest.mark.parametrize("run_eagerly", [True, False])
def test_transformer_with_masked_language_modeling(sequence_testing_data: Dataset, run_eagerly):

    seq_schema = sequence_testing_data.schema.select_by_tag(Tags.SEQUENCE).select_by_tag(
        Tags.CATEGORICAL
    )
    target = sequence_testing_data.schema.select_by_tag(Tags.ITEM_ID).column_names[0]

    loader = Loader(sequence_testing_data, batch_size=8, shuffle=False)
    model = mm.Model(
        mm.InputBlockV2(
            seq_schema,
            categorical=mm.Embeddings(
                seq_schema.select_by_tag(Tags.CATEGORICAL), sequence_combiner=None
            ),
        ),
        BertBlock(
            d_model=48,
            n_head=8,
            n_layer=2,
            pre=mm.SequentialBlock([SequenceMaskLastInference(), mm.ReplaceMaskedEmbeddings()]),
            post="inference_hidden_state",
        ),
        mm.CategoricalOutput(
            seq_schema.select_by_name(target),
            default_loss="categorical_crossentropy",
        ),
    )
    seq_mask_random = mm.SequenceMaskRandom(schema=seq_schema, target=target, masking_prob=0.3)

    inputs, targets = next(iter(loader))
    outputs = model(inputs, targets=targets, training=True)
    assert list(outputs.shape) == [8, 4, 51997]


  testing_utils.model_test(


        model,
        loader,
        run_eagerly=run_eagerly,
        reload_model=True,
        fit_kwargs={"pre": seq_mask_random},
    )

tests/unit/tf/transformers/test_block.py:302:

merlin/models/tf/utils/testing_utils.py:97: in model_test

model.save(tmpdir)

merlin/models/tf/models/base.py:1155: in save

super().save(

merlin/models/tf/models/base.py:1100: in save

super().save(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:2435: in save

save.save_model(self, filepath, overwrite, include_optimizer, save_format,

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/saving/save.py:153: in save_model

saved_model_save.save(model, filepath, overwrite, include_optimizer,

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save.py:93: in save

saved_nodes, node_paths = save_lib.save_and_return_nodes(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1325: in save_and_return_nodes

build_meta_graph(obj, signatures, options, meta_graph_def))

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1491: in build_meta_graph

return build_meta_graph_impl(obj, signatures, options, meta_graph_def)

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1433: in build_meta_graph_impl

signatures = signature_serialization.find_function_to_export(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/signature_serialization.py:103: in find_function_to_export

for name, child in children:

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:143: in list_children

for name, child in super(AugmentedGraphView, self).list_children(

../../../.local/lib/python3.8/site-packages/tensorflow/python/training/tracking/graph_view.py:203: in list_children

in obj.trackable_children(save_type, **kwargs).items()]

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:3201: in trackable_children

children = super(Model, self).trackable_children(save_type, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:3174: in trackable_children

children = self.trackable_saved_model_saver.trackable_children(cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/base_serialization.py:59: in trackable_children

children = self.objects_to_serialize(serialization_cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:68: in objects_to_serialize

return (self.get_serialized_attributes(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:88: in get_serialized_attributes

object_dict, function_dict = self.get_serialized_attributes_internal(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/model_serialization.py:51: in get_serialized_attributes_internal

default_signature = save_impl.default_save_signature(self.obj)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save_impl.py:208: in default_save_signature

fn = saving_utils.trace_model_call(layer)

../../../.local/lib/python3.8/site-packages/keras/saving/saving_utils.py:148: in trace_model_call

return wrapped_model.get_concrete_function(*model_args, **model_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1239: in get_concrete_function

concrete = self.get_concrete_function_garbage_collected(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1219: in get_concrete_function_garbage_collected

self.initialize(args, kwargs, add_initializers_to=initializers)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:785: in initialize

self.stateful_fn.get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2480: in get_concrete_function_internal_garbage_collected

graph_function, _ = self.maybe_define_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2711: in maybe_define_function

graph_function = self.create_graph_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2627: in create_graph_function

func_graph_module.func_graph_from_py_func(

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1141: in func_graph_from_py_func

func_outputs = python_func(*func_args, **func_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:677: in wrapped_fn

out = weak_wrapped_fn().wrapped(*args, **kwds)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1127: in autograph_handler

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1116: in autograph_handler

return autograph.converted_call(

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filebhqb1yva.py:14: in tf___wrapped_model

outputs = ag.converted_call(ag.ld(model), tuple(ag.ld(args)), dict(**ag.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:377: in converted_call

return call_unconverted(f, args, kwargs, options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:490: in call

return super().call(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file9w5w3fj0.py:42: in tf__call

ag.for_stmt(ag.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:449: in for_stmt

py_for_stmt(iter, extra_test, body, None, None)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:498: in py_for_stmt

body(target)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:464: in protected_body

original_body(protected_iter)

/tmp/autograph_generated_file9w5w3fj0.py:40: in loop_body

(outputs, context) = ag.converted_call(ag.ld(self).call_child, (ag.ld(block), ag.ld(outputs), ag.ld(context)), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:441: in converted_call

result = converted_f(*effective_args)

/tmp/autograph_generated_filei0bok8gt.py:25: in tf___call_child

outputs = ag.converted_call(ag.ld(call_layer), (ag.ld(child), ag.ld(inputs)), dict(**ag.ld(call_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_fileznees3_m.py:50: in tf__call_layer

retval = ag.converted_call(ag.ld(layer), ((ag.ld(inputs),) + tuple(ag_.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filemgv9evaz.py:14: in tf____call

retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filehdw_blhh.py:12: in tf__call

transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:459: in call_unconverted

return f(*args)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file_u19m308.py:10: in tf__call

inputs = ag.converted_call(ag.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file6tjr73tu.py:83: in tf__input_processing

ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1341: in if_stmt

_py_if_stmt(cond, body, orelse)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1394: in _py_if_stmt

return body() if cond else orelse()

def if_body_4():


  raise ag__.converted_call(ag__.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)


E       ValueError: in user code:

E

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/saving/saving_utils.py", line 138, in wrapped_model  *

E               outputs = model(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/training.py", line 490, in call

E               return super().call(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file9w5w3fj0.py", line 42, in tf__call  **

E               ag.for_stmt(ag_.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 449, in for_stmt

E               py_for_stmt(iter, extra_test, body, None, None)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 498, in py_for_stmt

E               body(target)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 464, in protected_body

E               original_body(protected_iter)

E           File "/tmp/autograph_generated_file9w5w3fj0.py", line 40, in loop_body

E               (outputs, context) = ag.converted_call(ag_.ld(self).call_child, (ag_.ld(block), ag__.ld(outputs), ag__.ld(context)), None, fscope)

E           File "/tmp/autograph_generated_filei0bok8gt.py", line 25, in tf___call_child  **

E               outputs = ag.converted_call(ag__.ld(call_layer), (ag__.ld(child), ag__.ld(inputs)), dict(**ag__.ld(call_kwargs)), fscope)

E           File "/tmp/autograph_generated_fileznees3_m.py", line 50, in tf__call_layer  **

E               retval = ag_.converted_call(ag__.ld(layer), ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

E           File "/tmp/autograph_generated_filemgv9evaz.py", line 14, in tf____call  **

E               retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_filehdw_blhh.py", line 12, in tf__call  **

E               transformer = ag.converted_call(ag__.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file_u19m308.py", line 10, in tf__call  **

E               inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E           File "/tmp/autograph_generated_file6tjr73tu.py", line 83, in tf__input_processing  **

E               ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E               py_if_stmt(cond, body, orelse)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E               return body() if cond else orelse()

E           File "/tmp/autograph_generated_file6tjr73tu.py", line 79, in if_body_4

E               raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E           ValueError: Exception encountered when calling layer "model" (type Model).

E

E           in user code:

E

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1236, in call  *

E                   outputs, context = self.call_child(block, outputs, context)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1265, in call_child  *

E                   outputs = call_layer(child, inputs, **call_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py", line 437, in call_layer  *

E                   return layer(inputs, *args, **filtered_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/config/schema.py", line 58, in call  *

E                   return super().call(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_filehdw_blhh.py", line 12, in tf__call  **

E                   transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_file_u19m308.py", line 10, in tf__call  **

E                   inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E               File "/tmp/autograph_generated_file6tjr73tu.py", line 83, in tf__input_processing  **

E                   ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                   py_if_stmt(cond, body, orelse)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                   return body() if cond else orelse()

E               File "/tmp/autograph_generated_file6tjr73tu.py", line 79, in if_body_4

E                   raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E               ValueError: Exception encountered when calling layer "bert_block" (type BertBlock).

E

E               in user code:

E

E                   File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transformers/block.py", line 131, in call  *

E                       transformer = self.transformer(pre)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                       return fn(*args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                       outputs = call_fn(inputs, *args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                       raise new_e.with_traceback(e.traceback) from None

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                       return fn(*args, **kwargs)

E                   File "/tmp/autograph_generated_file_u19m308.py", line 10, in tf__call  **

E                       inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E                   File "/tmp/autograph_generated_file6tjr73tu.py", line 83, in tf__input_processing  **

E                       ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                       py_if_stmt(cond, body, orelse)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                       return body() if cond else orelse()

E                   File "/tmp/autograph_generated_file6tjr73tu.py", line 79, in if_body_4

E                       raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E                   ValueError: Exception encountered when calling layer "bert" (type TFBertMainLayer).

E

E                   in user code:

E

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_tf_bert.py", line 743, in call  *

E                           inputs = input_processing(

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_tf_utils.py", line 387, in input_processing  *

E                           raise ValueError(

E

E                       ValueError: The following keyword arguments are not supported by this model: ['mask'].

E

E

E                   Call arguments received by layer "bert" (type TFBertMainLayer):

E                     • input_ids={'inputs_embeds': 'tf.Tensor(shape=(None, None, 48), dtype=float32)'}

E                     • attention_mask=None

E                     • token_type_ids=None

E                     • position_ids=None

E                     • head_mask=None

E                     • inputs_embeds=None

E                     • encoder_hidden_states=None

E                     • encoder_attention_mask=None

E                     • past_key_values=None

E                     • use_cache=None

E                     • output_attentions=None

E                     • output_hidden_states=None

E                     • return_dict=None

E                     • training=False

E                     • kwargs={'mask': {'inputs_embeds': 'tf.RaggedTensor(values=Tensor("model/bert_block/sequential_block/sequence_mask_last_inference/RaggedMask/boolean_mask/GatherV2:0", shape=(None,), dtype=bool), row_splits=Tensor("model/bert_block/sequential_block/sequence_mask_last_inference/RaggedMask/RaggedFromRowLengths/RowPartitionFromRowLengths/concat:0", shape=(None,), dtype=int64))'}}

E

E

E               Call arguments received by layer "bert_block" (type BertBlock):

E                 • inputs=tf.RaggedTensor(values=Tensor("model/concat_features/RaggedConcat/concat:0", shape=(None, 48), dtype=float32), row_splits=Tensor("model/parallel_block/embeddings/categories/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))

E                 • kwargs={'features': {'item_id_seq': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))', 'categories': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_1/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths_1/control_dependency:0", shape=(None,), dtype=int32))', 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_2/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_2/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_3/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_3/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_4/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_4/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_5/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_5/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_6/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_6/control_dependency:0", shape=(None,), dtype=int32))', 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}, 'training': 'False', 'testing': 'False'}

E

E

E           Call arguments received by layer "model" (type Model):

E             • inputs={'item_id_seq': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'categories': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}

E             • targets=None

E             • training=False

E             • testing=False

E             • output_context=False
/tmp/__autograph_generated_file6tjr73tu.py:79: ValueError

----------------------------- Captured stdout call -----------------------------
1/1 [==============================] - ETA: 0s - loss: 3.7263 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 3.7263������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

1/1 [==============================] - 1s 1s/step - loss: 3.7263 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 3.7263

----------------------------- Captured stderr call -----------------------------

WARNING:tensorflow:Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:utils.py:76 Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

____________ test_transformer_with_masked_language_modeling[False] _____________
sequence_testing_data = <merlin.io.dataset.Dataset object at 0x7f9f3f72c460>

run_eagerly = False
@pytest.mark.parametrize("run_eagerly", [True, False])
def test_transformer_with_masked_language_modeling(sequence_testing_data: Dataset, run_eagerly):

    seq_schema = sequence_testing_data.schema.select_by_tag(Tags.SEQUENCE).select_by_tag(
        Tags.CATEGORICAL
    )
    target = sequence_testing_data.schema.select_by_tag(Tags.ITEM_ID).column_names[0]

    loader = Loader(sequence_testing_data, batch_size=8, shuffle=False)
    model = mm.Model(
        mm.InputBlockV2(
            seq_schema,
            categorical=mm.Embeddings(
                seq_schema.select_by_tag(Tags.CATEGORICAL), sequence_combiner=None
            ),
        ),
        BertBlock(
            d_model=48,
            n_head=8,
            n_layer=2,
            pre=mm.SequentialBlock([SequenceMaskLastInference(), mm.ReplaceMaskedEmbeddings()]),
            post="inference_hidden_state",
        ),
        mm.CategoricalOutput(
            seq_schema.select_by_name(target),
            default_loss="categorical_crossentropy",
        ),
    )
    seq_mask_random = mm.SequenceMaskRandom(schema=seq_schema, target=target, masking_prob=0.3)

    inputs, targets = next(iter(loader))
    outputs = model(inputs, targets=targets, training=True)
    assert list(outputs.shape) == [8, 4, 51997]


  testing_utils.model_test(


        model,
        loader,
        run_eagerly=run_eagerly,
        reload_model=True,
        fit_kwargs={"pre": seq_mask_random},
    )

tests/unit/tf/transformers/test_block.py:302:

merlin/models/tf/utils/testing_utils.py:97: in model_test

model.save(tmpdir)

merlin/models/tf/models/base.py:1155: in save

super().save(

merlin/models/tf/models/base.py:1100: in save

super().save(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:2435: in save

save.save_model(self, filepath, overwrite, include_optimizer, save_format,

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/saving/save.py:153: in save_model

saved_model_save.save(model, filepath, overwrite, include_optimizer,

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save.py:93: in save

saved_nodes, node_paths = save_lib.save_and_return_nodes(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1325: in save_and_return_nodes

build_meta_graph(obj, signatures, options, meta_graph_def))

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1491: in build_meta_graph

return build_meta_graph_impl(obj, signatures, options, meta_graph_def)

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1433: in build_meta_graph_impl

signatures = signature_serialization.find_function_to_export(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/signature_serialization.py:103: in find_function_to_export

for name, child in children:

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:143: in list_children

for name, child in super(AugmentedGraphView, self).list_children(

../../../.local/lib/python3.8/site-packages/tensorflow/python/training/tracking/graph_view.py:203: in list_children

in obj.trackable_children(save_type, **kwargs).items()]

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:3201: in trackable_children

children = super(Model, self).trackable_children(save_type, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:3174: in trackable_children

children = self.trackable_saved_model_saver.trackable_children(cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/base_serialization.py:59: in trackable_children

children = self.objects_to_serialize(serialization_cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:68: in objects_to_serialize

return (self.get_serialized_attributes(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:88: in get_serialized_attributes

object_dict, function_dict = self.get_serialized_attributes_internal(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/model_serialization.py:51: in get_serialized_attributes_internal

default_signature = save_impl.default_save_signature(self.obj)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save_impl.py:208: in default_save_signature

fn = saving_utils.trace_model_call(layer)

../../../.local/lib/python3.8/site-packages/keras/saving/saving_utils.py:148: in trace_model_call

return wrapped_model.get_concrete_function(*model_args, **model_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1239: in get_concrete_function

concrete = self.get_concrete_function_garbage_collected(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1219: in get_concrete_function_garbage_collected

self.initialize(args, kwargs, add_initializers_to=initializers)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:785: in initialize

self.stateful_fn.get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2480: in get_concrete_function_internal_garbage_collected

graph_function, _ = self.maybe_define_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2711: in maybe_define_function

graph_function = self.create_graph_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2627: in create_graph_function

func_graph_module.func_graph_from_py_func(

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1141: in func_graph_from_py_func

func_outputs = python_func(*func_args, **func_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:677: in wrapped_fn

out = weak_wrapped_fn().wrapped(*args, **kwds)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1127: in autograph_handler

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1116: in autograph_handler

return autograph.converted_call(

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filebhqb1yva.py:14: in tf___wrapped_model

outputs = ag.converted_call(ag.ld(model), tuple(ag.ld(args)), dict(**ag.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:377: in converted_call

return call_unconverted(f, args, kwargs, options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:490: in call

return super().call(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file9w5w3fj0.py:42: in tf__call

ag.for_stmt(ag.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:449: in for_stmt

py_for_stmt(iter, extra_test, body, None, None)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:498: in py_for_stmt

body(target)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:464: in protected_body

original_body(protected_iter)

/tmp/autograph_generated_file9w5w3fj0.py:40: in loop_body

(outputs, context) = ag.converted_call(ag.ld(self).call_child, (ag.ld(block), ag.ld(outputs), ag.ld(context)), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:441: in converted_call

result = converted_f(*effective_args)

/tmp/autograph_generated_filei0bok8gt.py:25: in tf___call_child

outputs = ag.converted_call(ag.ld(call_layer), (ag.ld(child), ag.ld(inputs)), dict(**ag.ld(call_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_fileznees3_m.py:50: in tf__call_layer

retval = ag.converted_call(ag.ld(layer), ((ag.ld(inputs),) + tuple(ag_.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filemgv9evaz.py:14: in tf____call

retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filehdw_blhh.py:12: in tf__call

transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:459: in call_unconverted

return f(*args)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file_u19m308.py:10: in tf__call

inputs = ag.converted_call(ag.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file6tjr73tu.py:83: in tf__input_processing

ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1341: in if_stmt

_py_if_stmt(cond, body, orelse)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1394: in _py_if_stmt

return body() if cond else orelse()

def if_body_4():


  raise ag__.converted_call(ag__.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)


E       ValueError: in user code:

E

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/saving/saving_utils.py", line 138, in wrapped_model  *

E               outputs = model(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/training.py", line 490, in call

E               return super().call(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file9w5w3fj0.py", line 42, in tf__call  **

E               ag.for_stmt(ag_.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 449, in for_stmt

E               py_for_stmt(iter, extra_test, body, None, None)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 498, in py_for_stmt

E               body(target)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 464, in protected_body

E               original_body(protected_iter)

E           File "/tmp/autograph_generated_file9w5w3fj0.py", line 40, in loop_body

E               (outputs, context) = ag.converted_call(ag_.ld(self).call_child, (ag_.ld(block), ag__.ld(outputs), ag__.ld(context)), None, fscope)

E           File "/tmp/autograph_generated_filei0bok8gt.py", line 25, in tf___call_child  **

E               outputs = ag.converted_call(ag__.ld(call_layer), (ag__.ld(child), ag__.ld(inputs)), dict(**ag__.ld(call_kwargs)), fscope)

E           File "/tmp/autograph_generated_fileznees3_m.py", line 50, in tf__call_layer  **

E               retval = ag_.converted_call(ag__.ld(layer), ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

E           File "/tmp/autograph_generated_filemgv9evaz.py", line 14, in tf____call  **

E               retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_filehdw_blhh.py", line 12, in tf__call  **

E               transformer = ag.converted_call(ag__.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file_u19m308.py", line 10, in tf__call  **

E               inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E           File "/tmp/autograph_generated_file6tjr73tu.py", line 83, in tf__input_processing  **

E               ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E               py_if_stmt(cond, body, orelse)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E               return body() if cond else orelse()

E           File "/tmp/autograph_generated_file6tjr73tu.py", line 79, in if_body_4

E               raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E           ValueError: Exception encountered when calling layer "model" (type Model).

E

E           in user code:

E

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1236, in call  *

E                   outputs, context = self.call_child(block, outputs, context)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1265, in call_child  *

E                   outputs = call_layer(child, inputs, **call_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py", line 437, in call_layer  *

E                   return layer(inputs, *args, **filtered_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/config/schema.py", line 58, in call  *

E                   return super().call(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_filehdw_blhh.py", line 12, in tf__call  **

E                   transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_file_u19m308.py", line 10, in tf__call  **

E                   inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E               File "/tmp/autograph_generated_file6tjr73tu.py", line 83, in tf__input_processing  **

E                   ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                   py_if_stmt(cond, body, orelse)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                   return body() if cond else orelse()

E               File "/tmp/autograph_generated_file6tjr73tu.py", line 79, in if_body_4

E                   raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E               ValueError: Exception encountered when calling layer "bert_block" (type BertBlock).

E

E               in user code:

E

E                   File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transformers/block.py", line 131, in call  *

E                       transformer = self.transformer(pre)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                       return fn(*args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                       outputs = call_fn(inputs, *args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                       raise new_e.with_traceback(e.traceback) from None

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                       return fn(*args, **kwargs)

E                   File "/tmp/autograph_generated_file_u19m308.py", line 10, in tf__call  **

E                       inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E                   File "/tmp/autograph_generated_file6tjr73tu.py", line 83, in tf__input_processing  **

E                       ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                       py_if_stmt(cond, body, orelse)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                       return body() if cond else orelse()

E                   File "/tmp/autograph_generated_file6tjr73tu.py", line 79, in if_body_4

E                       raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E                   ValueError: Exception encountered when calling layer "bert" (type TFBertMainLayer).

E

E                   in user code:

E

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_tf_bert.py", line 743, in call  *

E                           inputs = input_processing(

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_tf_utils.py", line 387, in input_processing  *

E                           raise ValueError(

E

E                       ValueError: The following keyword arguments are not supported by this model: ['mask'].

E

E

E                   Call arguments received by layer "bert" (type TFBertMainLayer):

E                     • input_ids={'inputs_embeds': 'tf.Tensor(shape=(None, None, 48), dtype=float32)'}

E                     • attention_mask=None

E                     • token_type_ids=None

E                     • position_ids=None

E                     • head_mask=None

E                     • inputs_embeds=None

E                     • encoder_hidden_states=None

E                     • encoder_attention_mask=None

E                     • past_key_values=None

E                     • use_cache=None

E                     • output_attentions=None

E                     • output_hidden_states=None

E                     • return_dict=None

E                     • training=False

E                     • kwargs={'mask': {'inputs_embeds': 'tf.RaggedTensor(values=Tensor("model/bert_block/sequential_block/sequence_mask_last_inference/RaggedMask/boolean_mask/GatherV2:0", shape=(None,), dtype=bool), row_splits=Tensor("model/bert_block/sequential_block/sequence_mask_last_inference/RaggedMask/RaggedFromRowLengths/RowPartitionFromRowLengths/concat:0", shape=(None,), dtype=int64))'}}

E

E

E               Call arguments received by layer "bert_block" (type BertBlock):

E                 • inputs=tf.RaggedTensor(values=Tensor("model/concat_features/RaggedConcat/concat:0", shape=(None, 48), dtype=float32), row_splits=Tensor("model/parallel_block/embeddings/categories/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))

E                 • kwargs={'features': {'item_id_seq': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))', 'categories': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_1/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths_1/control_dependency:0", shape=(None,), dtype=int32))', 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_2/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_2/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_3/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_3/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_4/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_4/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_5/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_5/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_6/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_6/control_dependency:0", shape=(None,), dtype=int32))', 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}, 'training': 'False', 'testing': 'False'}

E

E

E           Call arguments received by layer "model" (type Model):

E             • inputs={'item_id_seq': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'categories': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}

E             • targets=None

E             • training=False

E             • testing=False

E             • output_context=False
/tmp/__autograph_generated_file6tjr73tu.py:79: ValueError

----------------------------- Captured stdout call -----------------------------
1/1 [==============================] - ETA: 0s - loss: 4.4090 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 4.4090������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

1/1 [==============================] - 9s 9s/step - loss: 4.4090 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 4.4090

----------------------------- Captured stderr call -----------------------------

WARNING:tensorflow:Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

WARNING:tensorflow:Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

2022-12-01 17:11:13.878603: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:907] Skipping loop optimization for Merge node with control input: model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Assert/AssertGuard/branch_executed/_9

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:utils.py:76 Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

WARNING  tensorflow:utils.py:76 Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning

tests/unit/datasets/test_ecommerce.py: 3 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 10 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 60 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 6 warnings

tests/unit/tf/core/test_index.py: 8 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/horovod/test_horovod.py: 6 warnings

tests/unit/tf/inputs/test_base.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 7 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 28 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 38 warnings

tests/unit/tf/models/test_retrieval.py: 123 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning

tests/unit/tf/transformers/test_block.py: 14 warnings

tests/unit/tf/transforms/test_bias.py: 2 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_noise.py: 1 warning

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 9 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 3 warnings

tests/unit/xgb/test_xgboost.py: 18 warnings

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_ecommerce.py: 3 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 9 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 60 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 7 warnings

tests/unit/tf/core/test_index.py: 3 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/horovod/test_horovod.py: 1 warning

tests/unit/tf/inputs/test_base.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 7 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 28 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 36 warnings

tests/unit/tf/models/test_retrieval.py: 91 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/transformers/test_block.py: 10 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 7 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 2 warnings

tests/unit/xgb/test_xgboost.py: 17 warnings

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_ecommerce.py: 1 warning

tests/unit/datasets/test_entertainment.py: 1 warning

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 2 warnings

tests/unit/tf/core/test_prediction.py: 1 warning

tests/unit/tf/horovod/test_horovod.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 4 warnings

tests/unit/tf/inputs/test_embedding.py: 9 warnings

tests/unit/tf/inputs/test_tabular.py: 8 warnings

tests/unit/tf/models/test_ranking.py: 20 warnings

tests/unit/tf/models/test_retrieval.py: 10 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 3 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 9 warnings

tests/unit/xgb/test_xgboost.py: 12 warnings

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:970: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/blocks/retrieval/test_two_tower.py: 1 warning

tests/unit/tf/core/test_index.py: 4 warnings

tests/unit/tf/horovod/test_horovod.py: 3 warnings

tests/unit/tf/models/test_retrieval.py: 63 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings

tests/unit/tf/utils/test_batch.py: 2 warnings

/tmp/autograph_generated_filenvnibx6.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag_.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/core/test_combinators.py::test_parallel_block_select_by_tags

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/core/tabular.py:602: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working

elif isinstance(self.feature_names, collections.Sequence):
tests/unit/tf/core/test_encoder.py: 1 warning

tests/unit/tf/core/test_index.py: 5 warnings

tests/unit/tf/models/test_retrieval.py: 30 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

tests/unit/tf/utils/test_dataset.py: 1 warning

/var/jenkins_home/workspace/merlin_models/models/merlin/models/utils/dataset.py:75: DeprecationWarning: unique_rows_by_features is deprecated and will be removed in a future version. Please use unique_by_tag instead.

warnings.warn(
tests/unit/tf/models/test_base.py::test_model_pre_post[True]

tests/unit/tf/models/test_base.py::test_model_pre_post[False]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.1]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.3]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.5]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.7]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]

tests/unit/tf/models/test_base.py::test_freeze_sequential_block

tests/unit/tf/models/test_base.py::test_freeze_unfreeze

tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.

super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_base.py::test_retrieval_model_query

tests/unit/tf/models/test_base.py::test_retrieval_model_query

tests/unit/tf/models/test_retrieval.py::test_youtube_dnn_v2_export_embeddings

tests/unit/tf/models/test_retrieval.py::test_youtube_dnn_topk_evaluation[True]

tests/unit/tf/models/test_retrieval.py::test_youtube_dnn_topk_evaluation[False]

tests/unit/tf/transformers/test_block.py::test_retrieval_transformer[True]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py:298: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

tensor_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(tensor)))
tests/unit/tf/models/test_ranking.py::test_deepfm_model_only_categ_feats[False]

tests/unit/tf/models/test_ranking.py::test_deepfm_model_categ_and_continuous_feats[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transforms/features.py:569: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_retrieval_transformer[True]

tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[True]

tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py:83: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.

config[key] = tf.keras.utils.serialize_keras_object(maybe_value)
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:112: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.

return generic_utils.serialize_keras_object(obj)
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask/GatherV2:0", shape=(None, 48), dtype=float32), dense_shape=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Reshape_3:0", shape=(None,), dtype=int64), values=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Reshape_2:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask/GatherV2:0", shape=(None, 48), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_3:0", shape=(None,), dtype=int64), values=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_2:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:344: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first

self.make_current()
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective

/var/jenkins_home/.local/lib/python3.8/site-packages/distributed/node.py:180: UserWarning: Port 8787 is already in use.

Perhaps you already have a cluster running?

Hosting the HTTP server on port 36051 instead

warnings.warn(
tests/unit/xgb/test_xgboost.py: 14 warnings

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/xgb/test_xgboost.py: 11 warnings

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1183: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

mask = pd.Series(mask)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                 Stmts   Miss  Cover
merlin/datasets/init.py                                              2      0   100%

merlin/datasets/advertising/init.py                                  2      0   100%

merlin/datasets/advertising/criteo/init.py                           0      0   100%

merlin/datasets/advertising/criteo/dataset.py                           79     49    38%

merlin/datasets/advertising/criteo/transformed/init.py               0      0   100%

merlin/datasets/ecommerce/init.py                                    4      0   100%

merlin/datasets/ecommerce/aliccp/init.py                             0      0   100%

merlin/datasets/ecommerce/aliccp/dataset.py                            141     87    38%

merlin/datasets/ecommerce/aliccp/raw/init.py                         0      0   100%

merlin/datasets/ecommerce/aliccp/transformed/init.py                 0      0   100%

merlin/datasets/ecommerce/booking/init.py                            0      0   100%

merlin/datasets/ecommerce/booking/dataset.py                           127    100    21%

merlin/datasets/ecommerce/booking/raw/init.py                        0      0   100%

merlin/datasets/ecommerce/booking/transformed/init.py                0      0   100%

merlin/datasets/ecommerce/dressipi/init.py                           0      0   100%

merlin/datasets/ecommerce/dressipi/dataset.py                           45     37    18%

merlin/datasets/ecommerce/dressipi/preprocessed/init.py              0      0   100%

merlin/datasets/ecommerce/large/init.py                              0      0   100%

merlin/datasets/ecommerce/small/init.py                              0      0   100%

merlin/datasets/ecommerce/transactions/init.py                       0      0   100%

merlin/datasets/entertainment/init.py                                2      0   100%

merlin/datasets/entertainment/movielens/1m-raw/init.py               0      0   100%

merlin/datasets/entertainment/movielens/1m-raw/ratings/init.py       0      0   100%

merlin/datasets/entertainment/movielens/1m/init.py                   0      0   100%

merlin/datasets/entertainment/movielens/25m/init.py                  0      0   100%

merlin/datasets/entertainment/movielens/100k/init.py                 0      0   100%

merlin/datasets/entertainment/movielens/init.py                      0      0   100%

merlin/datasets/entertainment/movielens/dataset.py                     152    122    20%

merlin/datasets/entertainment/music_streaming/init.py                0      0   100%

merlin/datasets/social/init.py                                       0      0   100%

merlin/datasets/synthetic.py                                           147      8    95%

merlin/datasets/testing/init.py                                      0      0   100%

merlin/datasets/testing/sequence_testing/init.py                     0      0   100%

merlin/models/init.py                                                2      0   100%

merlin/models/_version.py                                              354    205    42%

merlin/models/api.py                                                    14      5    64%

merlin/models/config/init.py                                         0      0   100%

merlin/models/config/schema.py                                          62      0   100%

merlin/models/implicit/init.py                                      27      4    85%

merlin/models/io.py                                                     15      0   100%

merlin/models/lightfm/init.py                                       23      0   100%

merlin/models/loader/init.py                                         0      0   100%

merlin/models/loader/backend.py                                        379     41    89%

merlin/models/loader/dataframe_iter.py                                  21     17    19%

merlin/models/loader/tf_utils.py                                        57     27    53%

merlin/models/loader/utils.py                                           40     15    62%

merlin/models/tf/init.py                                            70      0   100%

merlin/models/tf/blocks/init.py                                      0      0   100%

merlin/models/tf/blocks/cross.py                                        44      0   100%

merlin/models/tf/blocks/dlrm.py                                         49      2    96%

merlin/models/tf/blocks/experts.py                                      99     17    83%

merlin/models/tf/blocks/interaction.py                                 108     40    63%

merlin/models/tf/blocks/mlp.py                                         117      7    94%

merlin/models/tf/blocks/optimizer.py                                   173     12    93%

merlin/models/tf/blocks/retrieval/init.py                            0      0   100%

merlin/models/tf/blocks/retrieval/base.py                              175     62    65%

merlin/models/tf/blocks/retrieval/matrix_factorization.py               35      1    97%

merlin/models/tf/blocks/retrieval/two_tower.py                          30      0   100%

merlin/models/tf/blocks/sampling/init.py                             0      0   100%

merlin/models/tf/blocks/sampling/base.py                                29      2    93%

merlin/models/tf/blocks/sampling/cross_batch.py                         46      2    96%

merlin/models/tf/blocks/sampling/in_batch.py                            35      0   100%

merlin/models/tf/blocks/sampling/queue.py                              115     12    90%

merlin/models/tf/core/init.py                                        0      0   100%

merlin/models/tf/core/aggregation.py                                   241     45    81%

merlin/models/tf/core/base.py                                          242     51    79%

merlin/models/tf/core/combinators.py                                   426     53    88%

merlin/models/tf/core/encoder.py                                       182     28    85%

merlin/models/tf/core/index.py                                         106     16    85%

merlin/models/tf/core/prediction.py                                     50      1    98%

merlin/models/tf/core/tabular.py                                       280     29    90%

merlin/models/tf/distributed/init.py                                 0      0   100%

merlin/models/tf/distributed/backend.py                                  9      2    78%

merlin/models/tf/inputs/init.py                                      0      0   100%

merlin/models/tf/inputs/base.py                                         64     20    69%

merlin/models/tf/inputs/continuous.py                                   39      3    92%

merlin/models/tf/inputs/embedding.py                                   458     51    89%

merlin/models/tf/loader.py                                             268     94    65%

merlin/models/tf/losses/init.py                                      4      0   100%

merlin/models/tf/losses/base.py                                          9      0   100%

merlin/models/tf/losses/listwise.py                                     13      0   100%

merlin/models/tf/losses/pairwise.py                                    115      1    99%

merlin/models/tf/metrics/init.py                                     2      0   100%

merlin/models/tf/metrics/evaluation.py                                 105     48    54%

merlin/models/tf/metrics/topk.py                                       198     48    76%

merlin/models/tf/models/init.py                                      0      0   100%

merlin/models/tf/models/base.py                                        772    101    87%

merlin/models/tf/models/benchmark.py                                    16      0   100%

merlin/models/tf/models/ranking.py                                      67      3    96%

merlin/models/tf/models/retrieval.py                                    78      4    95%

merlin/models/tf/models/utils.py                                        10      1    90%

merlin/models/tf/outputs/init.py                                     0      0   100%

merlin/models/tf/outputs/base.py                                       123     17    86%

merlin/models/tf/outputs/classification.py                              91      1    99%

merlin/models/tf/outputs/contrastive.py                                147     10    93%

merlin/models/tf/outputs/regression.py                                   9      0   100%

merlin/models/tf/outputs/sampling/init.py                            0      0   100%

merlin/models/tf/outputs/sampling/base.py                               78     21    73%

merlin/models/tf/outputs/sampling/in_batch.py                           37      1    97%

merlin/models/tf/outputs/sampling/popularity.py                         27      1    96%

merlin/models/tf/outputs/topk.py                                        98      6    94%

merlin/models/tf/prediction_tasks/init.py                            0      0   100%

merlin/models/tf/prediction_tasks/base.py                              207     37    82%

merlin/models/tf/prediction_tasks/classification.py                     68     17    75%

merlin/models/tf/prediction_tasks/multi.py                               7      0   100%

merlin/models/tf/prediction_tasks/next_item.py                          59     17    71%

merlin/models/tf/prediction_tasks/regression.py                         35      2    94%

merlin/models/tf/prediction_tasks/retrieval.py                          73      3    96%

merlin/models/tf/transformers/init.py                                0      0   100%

merlin/models/tf/transformers/block.py                                 102      2    98%

merlin/models/tf/transformers/transforms.py                             82      4    95%

merlin/models/tf/transforms/init.py                                  0      0   100%

merlin/models/tf/transforms/bias.py                                    111      9    92%

merlin/models/tf/transforms/features.py                                435     42    90%

merlin/models/tf/transforms/negative_sampling.py                        77      3    96%

merlin/models/tf/transforms/noise.py                                    43      1    98%

merlin/models/tf/transforms/regularization.py                           17      1    94%

merlin/models/tf/transforms/sequence.py                                304     44    86%

merlin/models/tf/transforms/tensor.py                                  158     13    92%

merlin/models/tf/typing.py                                               7      0   100%

merlin/models/tf/utils/init.py                                       0      0   100%

merlin/models/tf/utils/batch_utils.py                                   85      4    95%

merlin/models/tf/utils/repr_utils.py                                    69      4    94%

merlin/models/tf/utils/search_utils.py                                  34     22    35%

merlin/models/tf/utils/testing_utils.py                                206     36    83%

merlin/models/tf/utils/tf_utils.py                                     209     43    79%

merlin/models/torch/init.py                                         12      0   100%

merlin/models/torch/block/init.py                                    0      0   100%

merlin/models/torch/block/base.py                                      167     32    81%

merlin/models/torch/block/mlp.py                                        38      5    87%

merlin/models/torch/dataset.py                                          68      5    93%

merlin/models/torch/features/init.py                                 0      0   100%

merlin/models/torch/features/base.py                                     4      0   100%

merlin/models/torch/features/continuous.py                              22      0   100%

merlin/models/torch/features/embedding.py                              165     12    93%

merlin/models/torch/features/tabular.py                                 65      8    88%

merlin/models/torch/losses.py                                           28     28     0%

merlin/models/torch/model/init.py                                    0      0   100%

merlin/models/torch/model/base.py                                      286     65    77%

merlin/models/torch/model/prediction_task.py                            24      0   100%

merlin/models/torch/tabular/init.py                                  0      0   100%

merlin/models/torch/tabular/aggregation.py                              75      0   100%

merlin/models/torch/tabular/base.py                                    247     39    84%

merlin/models/torch/tabular/transformations.py                          67      3    96%

merlin/models/torch/typing.py                                            6      0   100%

merlin/models/torch/utils/init.py                                    0      0   100%

merlin/models/torch/utils/data_utils.py                                117    117     0%

merlin/models/torch/utils/examples_utils.py                             39     39     0%

merlin/models/torch/utils/torch_utils.py                                80     22    72%

merlin/models/utils/init.py                                          0      0   100%

merlin/models/utils/constants.py                                         3      0   100%

merlin/models/utils/dataset.py                                          38      4    89%

merlin/models/utils/dependencies.py                                     26     19    27%

merlin/models/utils/doc_utils.py                                        10      0   100%

merlin/models/utils/example_utils.py                                    31      2    94%

merlin/models/utils/misc_utils.py                                      118     90    24%

merlin/models/utils/nvt_utils.py                                        27     24    11%

merlin/models/utils/registry.py                                        101     29    71%

merlin/models/utils/schema_utils.py                                     90     19    79%

merlin/models/xgb/init.py                                          124      4    97%
TOTAL                                                                11650   2400    79%
=========================== short test summary info ============================

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:64: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:80: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:94: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:115: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [5] ../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/test_util.py:2746: Not a test.

==== 2 failed, 871 passed, 13 skipped, 1475 warnings in 1994.89s (0:33:14) =====

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   py38-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins3296120362848603266.sh

jperez999 · 2022-12-01T17:16:58Z

rerun tests

jperez999 · 2022-12-01T17:29:27Z

rerun tests

jperez999 · 2022-12-01T17:34:14Z

rerun tests

jperez999 · 2022-12-01T17:36:29Z

rerun tests

jperez999 · 2022-12-01T17:42:07Z

rerun tasks

jperez999 · 2022-12-01T17:42:39Z

rerun tests

gabrielspmoreira · 2022-12-01T21:28:03Z

merlin/models/tf/transforms/sequence.py

        Returns
        -------
        Union[tf.Tensor, tf.RaggedTensor]
            If training, returns a tensor with the masked inputs replaced by the dummy embedding
        """
+        if not training and not testing:


Don't we need to replace the masked (selected) item with the trainable embedding even in inference mode, to avoid that issue you found in Transformers4Rec when you don't replace the input by the trainable embedding?

You're right! I updated the logic to replace with the mask embeddings

merlin/models/tf/transforms/sequence.py

merlin/models/tf/transformers/transforms.py

merlin/models/tf/transforms/sequence.py

nvidia-merlin-bot · 2022-12-02T00:59:29Z

Click to view CI Results

GitHub pull request #909 of commit a135fe2afb8713f7f254bed90841106c2406d0b7, no merge conflicts.
Running as SYSTEM
Setting status of a135fe2afb8713f7f254bed90841106c2406d0b7 to PENDING with url http://merlin-infra1.nvidia.com:8080/job/merlin_models/1992/ and message: 'Pending'
Using context: Jenkins
Building on the built-in node in workspace /var/jenkins_home/jobs/merlin_models/workspace
using credential nvidia-merlin-bot
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/NVIDIA-Merlin/models/ # timeout=10
Fetching upstream changes from https://github.com/NVIDIA-Merlin/models/
 > git --version # timeout=10
using GIT_ASKPASS to set credentials This is the bot credentials for our CI/CD
 > git fetch --tags --force --progress -- https://github.com/NVIDIA-Merlin/models/ +refs/pull/909/*:refs/remotes/origin/pr/909/* # timeout=10
 > git rev-parse a135fe2afb8713f7f254bed90841106c2406d0b7^{commit} # timeout=10
Checking out Revision a135fe2afb8713f7f254bed90841106c2406d0b7 (detached)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f a135fe2afb8713f7f254bed90841106c2406d0b7 # timeout=10
Commit message: "updates based on PR comments"
 > git rev-list --no-walk 93706b6be396c2951b7d02f13ea456eb5f9ce10d # timeout=10
[workspace] $ /bin/bash /tmp/jenkins17955862001468041862.sh
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Requirement already satisfied: testbook in /usr/local/lib/python3.8/dist-packages (0.4.2)
Requirement already satisfied: nbformat>=5.0.4 in /usr/local/lib/python3.8/dist-packages (from testbook) (5.5.0)
Requirement already satisfied: nbclient>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from testbook) (0.6.8)
Requirement already satisfied: nest-asyncio in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (1.5.5)
Requirement already satisfied: jupyter-client>=6.1.5 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (7.3.5)
Requirement already satisfied: traitlets>=5.2.2 in /usr/local/lib/python3.8/dist-packages (from nbclient>=0.4.0->testbook) (5.4.0)
Requirement already satisfied: jupyter_core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.11.1)
Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (4.16.0)
Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.0.4->testbook) (2.16.1)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (22.1.0)
Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (1.3.10)
Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (5.9.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.0.4->testbook) (0.18.1)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (2.8.2)
Requirement already satisfied: tornado>=6.2 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (6.2)
Requirement already satisfied: entrypoints in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (0.4)
Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (24.0.0)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.0.4->testbook) (3.8.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.5->nbclient>=0.4.0->testbook) (1.15.0)
GLOB sdist-make: /var/jenkins_home/workspace/merlin_models/models/setup.py
py38-gpu recreate: /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu
py38-gpu installdeps: -rrequirements/dev.txt, tensorflow<2.10
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py38-gpu inst: /var/jenkins_home/workspace/merlin_models/models/.tox/.tmp/package/4/merlin-models-0.9.0+64.ga135fe2a.zip
WARNING: Discarding $PYTHONPATH from environment, to override specify PYTHONPATH in 'passenv' in your configuration.
py38-gpu installed: absl-py==1.2.0,aiohttp==3.8.1,aiosignal==1.2.0,alabaster==0.7.12,alembic==1.8.1,anyio==3.6.1,appdirs==1.4.4,argon2-cffi==21.3.0,argon2-cffi-bindings==21.2.0,astroid==2.5.6,asttokens==2.0.8,astunparse==1.6.3,asv==0.5.1,asvdb==0.4.2,async-timeout==4.0.2,attrs==22.1.0,autopage==0.5.1,awscli==1.27.20,Babel==2.10.3,backcall==0.2.0,beautifulsoup4==4.11.1,betterproto==1.2.5,black==20.8b1,bleach==5.0.1,bokeh==3.0.2,boto3==1.24.75,botocore==1.29.20,Brotli==1.0.9,build==0.9.0,cachetools==5.2.0,certifi==2019.11.28,cffi==1.15.1,chardet==3.0.4,charset-normalizer==2.1.1,check-manifest==0.48,clang==5.0,click==8.1.3,cliff==4.1.0,cloudpickle==2.2.0,cmaes==0.9.0,cmake==3.24.1.1,cmd2==2.4.2,colorama==0.4.4,colorlog==6.7.0,contourpy==1.0.5,coverage==6.5.0,cuda-python==11.7.1,cupy-cuda117==10.6.0,cycler==0.11.0,Cython==0.29.32,dask==2022.1.1,dbus-python==1.2.16,debugpy==1.6.3,decorator==5.1.1,defusedxml==0.7.1,dill==0.3.5.1,distlib==0.3.6,distributed==2022.5.1,distro==1.7.0,dm-tree==0.1.6,docker-pycreds==0.4.0,docutils==0.16,emoji==1.7.0,entrypoints==0.4,execnet==1.9.0,executing==1.0.0,faiss==1.7.2,faiss-gpu==1.7.2,fastai==2.7.9,fastapi==0.85.0,fastavro==1.6.1,fastcore==1.5.27,fastdownload==0.0.7,fastjsonschema==2.16.1,fastprogress==1.0.3,fastrlock==0.8,feast==0.19.4,fiddle==0.2.2,filelock==3.8.0,flake8==6.0.0,flatbuffers==1.12,fonttools==4.37.3,frozenlist==1.3.1,fsspec==2022.5.0,gast==0.4.0,gevent==21.12.0,geventhttpclient==2.0.2,gitdb==4.0.9,GitPython==3.1.27,google==3.0.0,google-api-core==2.10.1,google-auth==2.11.1,google-auth-oauthlib==0.4.6,google-pasta==0.2.0,googleapis-common-protos==1.52.0,graphviz==0.20.1,greenlet==1.1.3,grpcio==1.41.0,grpcio-channelz==1.49.0,grpcio-reflection==1.48.1,grpclib==0.4.3,h11==0.13.0,h2==4.1.0,h5py==3.7.0,HeapDict==1.0.1,horovod==0.26.1,hpack==4.0.0,httptools==0.5.0,hugectr2onnx==0.0.0,huggingface-hub==0.9.1,hyperframe==6.0.1,idna==2.8,imagesize==1.4.1,implicit==0.6.1,importlib-metadata==4.12.0,importlib-resources==5.9.0,iniconfig==1.1.1,interrogate==1.5.0,ipykernel==6.15.3,ipython==8.5.0,ipython-genutils==0.2.0,ipywidgets==7.7.0,isort==5.10.1,jedi==0.18.1,Jinja2==3.1.2,jmespath==1.0.1,joblib==1.2.0,json5==0.9.10,jsonschema==4.16.0,jupyter==1.0.0,jupyter-cache==0.4.3,jupyter-console==6.4.4,jupyter-core==4.11.1,jupyter-server==1.18.1,jupyter-server-mathjax==0.2.5,jupyter-sphinx==0.3.2,jupyter_client==7.3.5,jupyterlab==3.4.7,jupyterlab-pygments==0.2.2,jupyterlab-widgets==1.1.0,jupyterlab_server==2.15.1,keras==2.9.0,Keras-Preprocessing==1.1.2,kiwisolver==1.4.4,lazy-object-proxy==1.8.0,libclang==14.0.6,libcst==0.4.7,lightfm==1.16,lightgbm==3.3.2,linkify-it-py==1.0.3,llvmlite==0.39.1,locket==1.0.0,lxml==4.9.1,Mako==1.2.4,Markdown==3.4.1,markdown-it-py==1.1.0,MarkupSafe==2.1.1,matplotlib==3.6.0,matplotlib-inline==0.1.6,mccabe==0.7.0,mdit-py-plugins==0.2.8,merlin-core==0.6.0+1.g5926fcf,merlin-models==0.9.0+64.ga135fe2a,merlin-systems==0.5.0+4.g15074ad,mistune==2.0.4,mmh3==3.0.0,mpi4py==3.1.3,msgpack==1.0.4,multidict==6.0.2,mypy==0.991,mypy-extensions==0.4.3,myst-nb==0.13.2,myst-parser==0.15.2,natsort==8.1.0,nbclassic==0.4.3,nbclient==0.6.8,nbconvert==7.0.0,nbdime==3.1.1,nbformat==5.5.0,nest-asyncio==1.5.5,ninja==1.10.2.3,notebook==6.4.12,notebook-shim==0.1.0,numba==0.56.2,numpy==1.22.4,nvidia-pyindex==1.0.9,# Editable install with no version control (nvtabular==1.4.0+8.g95e12d347),-e /usr/local/lib/python3.8/dist-packages,nvtx==0.2.5,oauthlib==3.2.1,oldest-supported-numpy==2022.8.16,onnx==1.12.0,onnxruntime==1.11.1,opt-einsum==3.3.0,optuna==3.0.4,packaging==21.3,pandas==1.3.5,pandavro==1.5.2,pandocfilters==1.5.0,parso==0.8.3,partd==1.3.0,pathspec==0.10.2,pathtools==0.1.2,pbr==5.11.0,pep517==0.13.0,pexpect==4.8.0,pickleshare==0.7.5,Pillow==9.2.0,pkgutil_resolve_name==1.3.10,platformdirs==2.5.2,plotly==5.11.0,pluggy==1.0.0,prettytable==3.5.0,prometheus-client==0.14.1,promise==2.3,prompt-toolkit==3.0.31,proto-plus==1.19.6,protobuf==3.19.5,psutil==5.9.2,ptyprocess==0.7.0,pure-eval==0.2.2,py==1.11.0,pyarrow==7.0.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pybind11==2.10.0,pycodestyle==2.10.0,pycparser==2.21,pydantic==1.10.2,pydot==1.4.2,pyflakes==3.0.1,Pygments==2.13.0,PyGObject==3.36.0,pynvml==11.4.1,pyparsing==3.0.9,pyperclip==1.8.2,pyrsistent==0.18.1,pytest==7.1.3,pytest-cov==4.0.0,pytest-xdist==3.0.2,python-apt==2.0.0+ubuntu0.20.4.8,python-dateutil==2.8.2,python-dotenv==0.21.0,python-rapidjson==1.8,pytz==2022.2.1,PyYAML==5.4.1,pyzmq==24.0.0,qtconsole==5.4.0,QtPy==2.3.0,regex==2022.9.13,requests==2.22.0,requests-oauthlib==1.3.1,requests-unixsocket==0.2.0,rsa==4.7.2,s3fs==2022.2.0,s3transfer==0.6.0,sacremoses==0.0.53,scikit-build==0.15.0,scikit-learn==1.1.2,scipy==1.8.1,seedir==0.3.0,Send2Trash==1.8.0,sentry-sdk==1.9.8,setproctitle==1.3.2,setuptools-scm==7.0.5,shortuuid==1.0.9,six==1.15.0,sklearn==0.0,smmap==5.0.0,sniffio==1.3.0,snowballstemmer==2.2.0,sortedcontainers==2.4.0,soupsieve==2.3.2.post1,Sphinx==5.3.0,sphinx-multiversion==0.2.4,sphinx-togglebutton==0.3.1,sphinx_external_toc==0.3.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-copydirs @ git+https://github.com/mikemckiernan/sphinxcontrib-copydirs.git@bd8c5d79b3f91cf5f1bb0d6995aeca3fe84b670e,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==2.0.0,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.5,SQLAlchemy==1.4.44,stack-data==0.5.0,starlette==0.20.4,stevedore==4.1.1,stringcase==1.2.0,supervisor==4.1.0,tabulate==0.8.10,tblib==1.7.0,tdqm==0.0.1,tenacity==8.0.1,tensorboard==2.9.1,tensorboard-data-server==0.6.1,tensorboard-plugin-wit==1.8.1,tensorflow==2.9.2,tensorflow-estimator==2.9.0,tensorflow-gpu==2.9.2,tensorflow-io-gcs-filesystem==0.27.0,tensorflow-metadata==1.10.0,termcolor==2.0.1,terminado==0.15.0,testbook==0.4.2,threadpoolctl==3.1.0,tinycss2==1.1.1,tokenizers==0.10.3,toml==0.10.2,tomli==2.0.1,toolz==0.12.0,torch==1.12.1+cu113,torchmetrics==0.3.2,tornado==6.2,tox==3.26.0,tqdm==4.64.1,traitlets==5.4.0,transformers==4.12.0,transformers4rec==0.1.12+2.gbcc939255,treelite==2.3.0,treelite-runtime==2.3.0,tritonclient==2.25.0,typed-ast==1.5.4,typing-inspect==0.8.0,typing_extensions==4.3.0,uc-micro-py==1.0.1,urllib3==1.26.12,uvicorn==0.18.3,uvloop==0.17.0,versioneer==0.20,virtualenv==20.16.5,wandb==0.13.3,watchfiles==0.17.0,wcwidth==0.2.5,webencodings==0.5.1,websocket-client==1.4.1,websockets==10.3,Werkzeug==2.2.2,widgetsnbextension==3.6.0,wrapt==1.12.1,xgboost==1.6.2,xyzservices==2022.9.0,yarl==1.8.1,zict==2.2.0,zipp==3.8.1,zope.event==4.5.0,zope.interface==5.4.0
py38-gpu run-test-pre: PYTHONHASHSEED='3125791795'
py38-gpu run-test: commands[0] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/core.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/core.git
  Cloning https://github.com/NVIDIA-Merlin/core.git to /tmp/pip-req-build-w56sw4j7
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/core.git /tmp/pip-req-build-w56sw4j7
  Resolved https://github.com/NVIDIA-Merlin/core.git to commit 4f73ff5bd4121c1acaabdc01a123af4f986ffc78
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (0.55.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core==0.9.0+14.g4f73ff5) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core==0.9.0+14.g4f73ff5) (2022.5.0)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.4.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in ./.tox/py38-gpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (6.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (0.38.1)
Requirement already satisfied: numpy<1.22,>=1.18 in /var/jenkins_home/.local/lib/python3.8/site-packages (from numba>=0.54->merlin-core==0.9.0+14.g4f73ff5) (1.20.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core==0.9.0+14.g4f73ff5) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core==0.9.0+14.g4f73ff5) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core==0.9.0+14.g4f73ff5) (2.0.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core==0.9.0+14.g4f73ff5) (4.0.0)
Building wheels for collected packages: merlin-core
  Building wheel for merlin-core (pyproject.toml): started
  Building wheel for merlin-core (pyproject.toml): finished with status 'done'
  Created wheel for merlin-core: filename=merlin_core-0.9.0+14.g4f73ff5-py3-none-any.whl size=119010 sha256=4e737ab9bb33e570725347bed81c7152421b9faff20c9245ee11725fca233cb7
  Stored in directory: /tmp/pip-ephem-wheel-cache-gnld4r40/wheels/c8/38/16/a6968787eafcec5fa772148af8408b089562f71af0752e8e84
Successfully built merlin-core
Installing collected packages: merlin-core
  Attempting uninstall: merlin-core
    Found existing installation: merlin-core 0.3.0+12.g78ecddd
    Not uninstalling merlin-core at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu
    Can't uninstall 'merlin-core'. No files were found to uninstall.
Successfully installed merlin-core-0.9.0+14.g4f73ff5
py38-gpu run-test: commands[1] | python -m pip install --upgrade git+https://github.com/NVIDIA-Merlin/nvtabular.git
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting git+https://github.com/NVIDIA-Merlin/nvtabular.git
  Cloning https://github.com/NVIDIA-Merlin/nvtabular.git to /tmp/pip-req-build-6p2yo3a0
  Running command git clone --filter=blob:none --quiet https://github.com/NVIDIA-Merlin/nvtabular.git /tmp/pip-req-build-6p2yo3a0
  Resolved https://github.com/NVIDIA-Merlin/nvtabular.git to commit 51af616069689b3ba57e8842a6f4a20377795df7
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: merlin-core>=0.2.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from nvtabular==1.6.0+13.g51af6160) (0.9.0+14.g4f73ff5)
Collecting merlin-dataloader>=0.0.2
  Downloading merlin-dataloader-0.0.3.tar.gz (48 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.3/48.3 kB 1.6 MB/s eta 0:00:00
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: scipy in /usr/local/lib/python3.8/dist-packages (from nvtabular==1.6.0+13.g51af6160) (1.8.1)
Requirement already satisfied: pyarrow>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (7.0.0)
Requirement already satisfied: protobuf>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.19.5)
Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (21.3)
Requirement already satisfied: distributed>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.3.0)
Requirement already satisfied: dask>=2022.3.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.3.0)
Requirement already satisfied: tqdm>=4.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.64.1)
Requirement already satisfied: tensorflow-metadata>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.10.0)
Requirement already satisfied: numba>=0.54 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.55.1)
Requirement already satisfied: pandas<1.4.0dev0,>=1.2.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.3.5)
Requirement already satisfied: betterproto<2.0.0 in /usr/local/lib/python3.8/dist-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.5)
Requirement already satisfied: fsspec==2022.5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages (from merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.5.0)
Requirement already satisfied: numpy<1.25.0,>=1.17.3 in /var/jenkins_home/.local/lib/python3.8/site-packages (from scipy->nvtabular==1.6.0+13.g51af6160) (1.20.3)
Requirement already satisfied: grpclib in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.4.3)
Requirement already satisfied: stringcase in /usr/local/lib/python3.8/dist-packages (from betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: toolz>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.12.0)
Requirement already satisfied: cloudpickle>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.2.0)
Requirement already satisfied: partd>=0.3.10 in /var/jenkins_home/.local/lib/python3.8/site-packages/partd-1.2.0-py3.8.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: pyyaml>=5.3.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/PyYAML-5.4.1-py3.8-linux-x86_64.egg (from dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (5.4.1)
Requirement already satisfied: zict>=0.1.3 in /var/jenkins_home/.local/lib/python3.8/site-packages/zict-2.0.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.0.0)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /var/jenkins_home/.local/lib/python3.8/site-packages/sortedcontainers-2.4.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.4.0)
Requirement already satisfied: tornado>=6.0.3 in ./.tox/py38-gpu/lib/python3.8/site-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.2)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.1.2)
Requirement already satisfied: psutil>=5.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/psutil-5.8.0-py3.8-linux-x86_64.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (5.8.0)
Requirement already satisfied: tblib>=1.6.0 in /var/jenkins_home/.local/lib/python3.8/site-packages/tblib-1.7.0-py3.8.egg (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.0.4)
Requirement already satisfied: click>=6.6 in /usr/local/lib/python3.8/dist-packages (from distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (8.1.3)
Requirement already satisfied: setuptools in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (65.5.1)
Requirement already satisfied: llvmlite<0.39,>=0.38.0rc1 in ./.tox/py38-gpu/lib/python3.8/site-packages (from numba>=0.54->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.38.1)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2022.2.1)
Requirement already satisfied: absl-py<2.0.0,>=0.9 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.2.0)
Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-metadata>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.52.0)
Requirement already satisfied: locket in /var/jenkins_home/.local/lib/python3.8/site-packages/locket-0.2.1-py3.8.egg (from partd>=0.3.10->dask>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (0.2.1)
Requirement already satisfied: six>=1.5 in /var/jenkins_home/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas<1.4.0dev0,>=1.2.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.15.0)
Requirement already satisfied: heapdict in /var/jenkins_home/.local/lib/python3.8/site-packages/HeapDict-1.0.1-py3.8.egg (from zict>=0.1.3->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (1.0.1)
Requirement already satisfied: h2<5,>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.1.0)
Requirement already satisfied: multidict in /usr/local/lib/python3.8/dist-packages (from grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.0.2)
Requirement already satisfied: MarkupSafe>=2.0 in ./.tox/py38-gpu/lib/python3.8/site-packages (from jinja2->distributed>=2022.3.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (2.0.1)
Requirement already satisfied: hyperframe<7,>=6.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (6.0.1)
Requirement already satisfied: hpack<5,>=4.0 in /usr/local/lib/python3.8/dist-packages (from h2<5,>=3.1.0->grpclib->betterproto<2.0.0->merlin-core>=0.2.0->nvtabular==1.6.0+13.g51af6160) (4.0.0)
Building wheels for collected packages: nvtabular, merlin-dataloader
  Building wheel for nvtabular (pyproject.toml): started
  Building wheel for nvtabular (pyproject.toml): finished with status 'done'
  Created wheel for nvtabular: filename=nvtabular-1.6.0+13.g51af6160-cp38-cp38-linux_x86_64.whl size=257601 sha256=951ff0436f79ce975b46d696f76d5005b72a04acf4c23341aea45e21a140750e
  Stored in directory: /tmp/pip-ephem-wheel-cache-bmdzypjn/wheels/8f/d9/f9/30f2cdc5bf8787fae6fdfe55afd6e1b493e619ec32c32ec40b
  Building wheel for merlin-dataloader (pyproject.toml): started
  Building wheel for merlin-dataloader (pyproject.toml): finished with status 'done'
  Created wheel for merlin-dataloader: filename=merlin_dataloader-0.0.3-py3-none-any.whl size=37647 sha256=310bd5183dfafcfebac5076e81b4165d4df3fae6805b76ff719e9013ad4ed2f7
  Stored in directory: /tmp/pip-ephem-wheel-cache-bmdzypjn/wheels/1c/a3/4a/0feebb30e0c8cb7ba7046544390b43c7017a2195232f5305a1
Successfully built nvtabular merlin-dataloader
Installing collected packages: merlin-dataloader, nvtabular
  Attempting uninstall: nvtabular
    Found existing installation: nvtabular 1.1.1
    Not uninstalling nvtabular at /var/jenkins_home/.local/lib/python3.8/site-packages, outside environment /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu
    Can't uninstall 'nvtabular'. No files were found to uninstall.
Successfully installed merlin-dataloader-0.0.3 nvtabular-1.6.0+13.g51af6160
py38-gpu run-test: commands[2] | python -m pytest --cov-report term --cov merlin -rxs tests/unit
============================= test session starts ==============================
platform linux -- Python 3.8.10, pytest-7.1.3, pluggy-1.0.0
cachedir: .tox/py38-gpu/.pytest_cache
rootdir: /var/jenkins_home/workspace/merlin_models/models, configfile: pyproject.toml
plugins: anyio-3.5.0, xdist-3.0.2, cov-4.0.0
collected 886 items
tests/unit/config/test_schema.py ....                                    [  0%]

tests/unit/datasets/test_advertising.py .s                               [  0%]

tests/unit/datasets/test_ecommerce.py ..sss.s                            [  1%]

tests/unit/datasets/test_entertainment.py ....sss.                       [  2%]

tests/unit/datasets/test_social.py .                                     [  2%]

tests/unit/datasets/test_synthetic.py ..........                         [  3%]

tests/unit/implicit/test_implicit.py .                                   [  3%]

tests/unit/lightfm/test_lightfm.py .                                     [  3%]

tests/unit/tf/test_core.py ......                                        [  4%]

tests/unit/tf/test_loader.py ................                            [  6%]

tests/unit/tf/test_public_api.py .                                       [  6%]

tests/unit/tf/blocks/test_cross.py ...........                           [  7%]

tests/unit/tf/blocks/test_dlrm.py ..........                             [  8%]

tests/unit/tf/blocks/test_interactions.py ...                            [  9%]

tests/unit/tf/blocks/test_mlp.py ....................................... [ 13%]

............................                                             [ 16%]

tests/unit/tf/blocks/test_optimizer.py s................................ [ 20%]

.....................                                                    [ 22%]

tests/unit/tf/blocks/retrieval/test_base.py .                            [ 22%]

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py ..           [ 23%]

tests/unit/tf/blocks/retrieval/test_two_tower.py ............            [ 24%]

tests/unit/tf/blocks/sampling/test_cross_batch.py .                      [ 24%]

tests/unit/tf/blocks/sampling/test_in_batch.py .                         [ 24%]

tests/unit/tf/core/test_aggregation.py .........                         [ 25%]

tests/unit/tf/core/test_base.py ..                                       [ 25%]

tests/unit/tf/core/test_combinators.py s.....................            [ 28%]

tests/unit/tf/core/test_encoder.py ..                                    [ 28%]

tests/unit/tf/core/test_index.py ...                                     [ 29%]

tests/unit/tf/core/test_prediction.py ..                                 [ 29%]

tests/unit/tf/core/test_tabular.py ......                                [ 29%]

tests/unit/tf/examples/test_01_getting_started.py .                      [ 30%]

tests/unit/tf/examples/test_02_dataschema.py .                           [ 30%]

tests/unit/tf/examples/test_03_exploring_different_models.py .           [ 30%]

tests/unit/tf/examples/test_04_export_ranking_models.py .                [ 30%]

tests/unit/tf/examples/test_05_export_retrieval_model.py .               [ 30%]

tests/unit/tf/examples/test_06_advanced_own_architecture.py .            [ 30%]

tests/unit/tf/examples/test_07_train_traditional_models.py .             [ 30%]

tests/unit/tf/examples/test_usecase_accelerate_training_by_lazyadam.py . [ 30%]

[ 30%]

tests/unit/tf/examples/test_usecase_data_parallel.py .                   [ 30%]

tests/unit/tf/examples/test_usecase_ecommerce_session_based.py .         [ 31%]

tests/unit/tf/examples/test_usecase_incremental_training_layer_freezing.py . [ 31%]

[ 31%]

tests/unit/tf/examples/test_usecase_pretrained_embeddings.py .           [ 31%]

tests/unit/tf/examples/test_usecase_retrieval_with_hpo.py .              [ 31%]

tests/unit/tf/examples/test_usecase_transformers_next_item_prediction.py . [ 31%]

[ 31%]

tests/unit/tf/horovod/test_horovod.py ...                                [ 31%]

tests/unit/tf/inputs/test_base.py .                                      [ 31%]

tests/unit/tf/inputs/test_continuous.py ........                         [ 32%]

tests/unit/tf/inputs/test_embedding.py ................................. [ 36%]

........                                                                 [ 37%]

tests/unit/tf/inputs/test_tabular.py ..................                  [ 39%]

tests/unit/tf/layers/test_queue.py ..............                        [ 41%]

tests/unit/tf/losses/test_losses.py .......................              [ 43%]

tests/unit/tf/metrics/test_metrics_popularity.py .....                   [ 44%]

tests/unit/tf/metrics/test_metrics_topk.py .........................     [ 47%]

tests/unit/tf/models/test_base.py s.........................             [ 50%]

tests/unit/tf/models/test_benchmark.py ..                                [ 50%]

tests/unit/tf/models/test_ranking.py ..................................  [ 54%]

tests/unit/tf/models/test_retrieval.py ................................. [ 57%]

.............................................                            [ 62%]

tests/unit/tf/outputs/test_base.py ......                                [ 63%]

tests/unit/tf/outputs/test_classification.py ......                      [ 64%]

tests/unit/tf/outputs/test_contrastive.py ..............                 [ 65%]

tests/unit/tf/outputs/test_regression.py ..                              [ 66%]

tests/unit/tf/outputs/test_sampling.py ....                              [ 66%]

tests/unit/tf/outputs/test_topk.py .                                     [ 66%]

tests/unit/tf/prediction_tasks/test_classification.py ..                 [ 66%]

tests/unit/tf/prediction_tasks/test_multi_task.py ................       [ 68%]

tests/unit/tf/prediction_tasks/test_next_item.py .....                   [ 69%]

tests/unit/tf/prediction_tasks/test_regression.py .....                  [ 69%]

tests/unit/tf/prediction_tasks/test_retrieval.py .                       [ 69%]

tests/unit/tf/prediction_tasks/test_sampling.py ......                   [ 70%]

tests/unit/tf/transformers/test_block.py ..................FF..          [ 73%]

tests/unit/tf/transformers/test_transforms.py ..........                 [ 74%]

tests/unit/tf/transforms/test_bias.py ..                                 [ 74%]

tests/unit/tf/transforms/test_features.py s............................. [ 77%]

.......................s......                                           [ 81%]

tests/unit/tf/transforms/test_negative_sampling.py .........             [ 82%]

tests/unit/tf/transforms/test_noise.py .....                             [ 82%]

tests/unit/tf/transforms/test_sequence.py ....................           [ 84%]

tests/unit/tf/transforms/test_tensor.py ...                              [ 85%]

tests/unit/tf/utils/test_batch.py ....                                   [ 85%]

tests/unit/tf/utils/test_dataset.py ..                                   [ 86%]

tests/unit/tf/utils/test_tf_utils.py .....                               [ 86%]

tests/unit/torch/test_dataset.py .........                               [ 87%]

tests/unit/torch/test_public_api.py .                                    [ 87%]

tests/unit/torch/block/test_base.py ....                                 [ 88%]

tests/unit/torch/block/test_mlp.py .                                     [ 88%]

tests/unit/torch/features/test_continuous.py ..                          [ 88%]

tests/unit/torch/features/test_embedding.py ..............               [ 90%]

tests/unit/torch/features/test_tabular.py ....                           [ 90%]

tests/unit/torch/model/test_head.py ............                         [ 91%]

tests/unit/torch/model/test_model.py ..                                  [ 92%]

tests/unit/torch/tabular/test_aggregation.py ........                    [ 93%]

tests/unit/torch/tabular/test_tabular.py ...                             [ 93%]

tests/unit/torch/tabular/test_transformations.py .......                 [ 94%]

tests/unit/utils/test_schema_utils.py ................................   [ 97%]

tests/unit/xgb/test_xgboost.py ....................                      [100%]
=================================== FAILURES ===================================

_____________ test_transformer_with_masked_language_modeling[True] _____________
sequence_testing_data = <merlin.io.dataset.Dataset object at 0x7f0d4ac15490>

run_eagerly = True
@pytest.mark.parametrize("run_eagerly", [True, False])
def test_transformer_with_masked_language_modeling(sequence_testing_data: Dataset, run_eagerly):

    seq_schema = sequence_testing_data.schema.select_by_tag(Tags.SEQUENCE).select_by_tag(
        Tags.CATEGORICAL
    )
    target = sequence_testing_data.schema.select_by_tag(Tags.ITEM_ID).column_names[0]

    loader = Loader(sequence_testing_data, batch_size=8, shuffle=False)
    model = mm.Model(
        mm.InputBlockV2(
            seq_schema,
            categorical=mm.Embeddings(
                seq_schema.select_by_tag(Tags.CATEGORICAL), sequence_combiner=None
            ),
        ),
        BertBlock(
            d_model=48,
            n_head=8,
            n_layer=2,
            pre=mm.SequentialBlock([mm.SequenceMaskLastInference(), mm.ReplaceMaskedEmbeddings()]),
            post="inference_hidden_state",
        ),
        mm.CategoricalOutput(
            seq_schema.select_by_name(target),
            default_loss="categorical_crossentropy",
        ),
    )
    seq_mask_random = mm.SequenceMaskRandom(schema=seq_schema, target=target, masking_prob=0.3)

    inputs, targets = next(iter(loader))
    outputs = model(inputs, targets=targets, training=True)
    assert list(outputs.shape) == [8, 4, 51997]


  testing_utils.model_test(


        model,
        loader,
        run_eagerly=run_eagerly,
        reload_model=True,
        fit_kwargs={"pre": seq_mask_random},
    )

tests/unit/tf/transformers/test_block.py:301:

merlin/models/tf/utils/testing_utils.py:97: in model_test

model.save(tmpdir)

merlin/models/tf/models/base.py:1155: in save

super().save(

merlin/models/tf/models/base.py:1100: in save

super().save(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:2435: in save

save.save_model(self, filepath, overwrite, include_optimizer, save_format,

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/saving/save.py:153: in save_model

saved_model_save.save(model, filepath, overwrite, include_optimizer,

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save.py:93: in save

saved_nodes, node_paths = save_lib.save_and_return_nodes(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1325: in save_and_return_nodes

build_meta_graph(obj, signatures, options, meta_graph_def))

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1491: in build_meta_graph

return build_meta_graph_impl(obj, signatures, options, meta_graph_def)

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1433: in build_meta_graph_impl

signatures = signature_serialization.find_function_to_export(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/signature_serialization.py:103: in find_function_to_export

for name, child in children:

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:143: in list_children

for name, child in super(AugmentedGraphView, self).list_children(

../../../.local/lib/python3.8/site-packages/tensorflow/python/training/tracking/graph_view.py:203: in list_children

in obj.trackable_children(save_type, **kwargs).items()]

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:3201: in trackable_children

children = super(Model, self).trackable_children(save_type, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:3174: in trackable_children

children = self.trackable_saved_model_saver.trackable_children(cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/base_serialization.py:59: in trackable_children

children = self.objects_to_serialize(serialization_cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:68: in objects_to_serialize

return (self.get_serialized_attributes(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:88: in get_serialized_attributes

object_dict, function_dict = self.get_serialized_attributes_internal(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/model_serialization.py:51: in get_serialized_attributes_internal

default_signature = save_impl.default_save_signature(self.obj)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save_impl.py:208: in default_save_signature

fn = saving_utils.trace_model_call(layer)

../../../.local/lib/python3.8/site-packages/keras/saving/saving_utils.py:148: in trace_model_call

return wrapped_model.get_concrete_function(*model_args, **model_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1239: in get_concrete_function

concrete = self.get_concrete_function_garbage_collected(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1219: in get_concrete_function_garbage_collected

self.initialize(args, kwargs, add_initializers_to=initializers)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:785: in initialize

self.stateful_fn.get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2480: in get_concrete_function_internal_garbage_collected

graph_function, _ = self.maybe_define_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2711: in maybe_define_function

graph_function = self.create_graph_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2627: in create_graph_function

func_graph_module.func_graph_from_py_func(

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1141: in func_graph_from_py_func

func_outputs = python_func(*func_args, **func_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:677: in wrapped_fn

out = weak_wrapped_fn().wrapped(*args, **kwds)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1127: in autograph_handler

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1116: in autograph_handler

return autograph.converted_call(

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filecexucevj.py:14: in tf___wrapped_model

outputs = ag.converted_call(ag.ld(model), tuple(ag.ld(args)), dict(**ag.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:377: in converted_call

return call_unconverted(f, args, kwargs, options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:490: in call

return super().call(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filexenb4ya2.py:42: in tf__call

ag.for_stmt(ag.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:449: in for_stmt

py_for_stmt(iter, extra_test, body, None, None)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:498: in py_for_stmt

body(target)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:464: in protected_body

original_body(protected_iter)

/tmp/autograph_generated_filexenb4ya2.py:40: in loop_body

(outputs, context) = ag.converted_call(ag.ld(self).call_child, (ag.ld(block), ag.ld(outputs), ag.ld(context)), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:441: in converted_call

result = converted_f(*effective_args)

/tmp/autograph_generated_fileg45vl_uj.py:25: in tf___call_child

outputs = ag.converted_call(ag.ld(call_layer), (ag.ld(child), ag.ld(inputs)), dict(**ag.ld(call_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_fileymmumnvc.py:50: in tf__call_layer

retval = ag.converted_call(ag.ld(layer), ((ag.ld(inputs),) + tuple(ag_.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_fileainhchfl.py:14: in tf____call

retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file8dwrq7uo.py:12: in tf__call

transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:459: in call_unconverted

return f(*args)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file1yic3w2n.py:10: in tf__call

inputs = ag.converted_call(ag.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file237fjfzg.py:83: in tf__input_processing

ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1341: in if_stmt

_py_if_stmt(cond, body, orelse)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1394: in _py_if_stmt

return body() if cond else orelse()

def if_body_4():


  raise ag__.converted_call(ag__.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)


E       ValueError: in user code:

E

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/saving/saving_utils.py", line 138, in wrapped_model  *

E               outputs = model(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/training.py", line 490, in call

E               return super().call(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_filexenb4ya2.py", line 42, in tf__call  **

E               ag.for_stmt(ag_.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 449, in for_stmt

E               py_for_stmt(iter, extra_test, body, None, None)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 498, in py_for_stmt

E               body(target)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 464, in protected_body

E               original_body(protected_iter)

E           File "/tmp/autograph_generated_filexenb4ya2.py", line 40, in loop_body

E               (outputs, context) = ag.converted_call(ag_.ld(self).call_child, (ag_.ld(block), ag__.ld(outputs), ag__.ld(context)), None, fscope)

E           File "/tmp/autograph_generated_fileg45vl_uj.py", line 25, in tf___call_child  **

E               outputs = ag.converted_call(ag__.ld(call_layer), (ag__.ld(child), ag__.ld(inputs)), dict(**ag__.ld(call_kwargs)), fscope)

E           File "/tmp/autograph_generated_fileymmumnvc.py", line 50, in tf__call_layer  **

E               retval = ag_.converted_call(ag__.ld(layer), ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

E           File "/tmp/autograph_generated_fileainhchfl.py", line 14, in tf____call  **

E               retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file8dwrq7uo.py", line 12, in tf__call  **

E               transformer = ag.converted_call(ag__.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file1yic3w2n.py", line 10, in tf__call  **

E               inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E           File "/tmp/autograph_generated_file237fjfzg.py", line 83, in tf__input_processing  **

E               ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E               py_if_stmt(cond, body, orelse)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E               return body() if cond else orelse()

E           File "/tmp/autograph_generated_file237fjfzg.py", line 79, in if_body_4

E               raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E           ValueError: Exception encountered when calling layer "model" (type Model).

E

E           in user code:

E

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1236, in call  *

E                   outputs, context = self.call_child(block, outputs, context)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1265, in call_child  *

E                   outputs = call_layer(child, inputs, **call_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py", line 437, in call_layer  *

E                   return layer(inputs, *args, **filtered_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/config/schema.py", line 58, in call  *

E                   return super().call(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_file8dwrq7uo.py", line 12, in tf__call  **

E                   transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_file1yic3w2n.py", line 10, in tf__call  **

E                   inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E               File "/tmp/autograph_generated_file237fjfzg.py", line 83, in tf__input_processing  **

E                   ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                   py_if_stmt(cond, body, orelse)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                   return body() if cond else orelse()

E               File "/tmp/autograph_generated_file237fjfzg.py", line 79, in if_body_4

E                   raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E               ValueError: Exception encountered when calling layer "bert_block" (type BertBlock).

E

E               in user code:

E

E                   File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transformers/block.py", line 131, in call  *

E                       transformer = self.transformer(pre)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                       return fn(*args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                       outputs = call_fn(inputs, *args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                       raise new_e.with_traceback(e.traceback) from None

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                       return fn(*args, **kwargs)

E                   File "/tmp/autograph_generated_file1yic3w2n.py", line 10, in tf__call  **

E                       inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E                   File "/tmp/autograph_generated_file237fjfzg.py", line 83, in tf__input_processing  **

E                       ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                       py_if_stmt(cond, body, orelse)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                       return body() if cond else orelse()

E                   File "/tmp/autograph_generated_file237fjfzg.py", line 79, in if_body_4

E                       raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E                   ValueError: Exception encountered when calling layer "bert" (type TFBertMainLayer).

E

E                   in user code:

E

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_tf_bert.py", line 743, in call  *

E                           inputs = input_processing(

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_tf_utils.py", line 387, in input_processing  *

E                           raise ValueError(

E

E                       ValueError: The following keyword arguments are not supported by this model: ['mask'].

E

E

E                   Call arguments received by layer "bert" (type TFBertMainLayer):

E                     • input_ids={'inputs_embeds': 'tf.Tensor(shape=(None, None, 48), dtype=float32)'}

E                     • attention_mask=None

E                     • token_type_ids=None

E                     • position_ids=None

E                     • head_mask=None

E                     • inputs_embeds=None

E                     • encoder_hidden_states=None

E                     • encoder_attention_mask=None

E                     • past_key_values=None

E                     • use_cache=None

E                     • output_attentions=None

E                     • output_hidden_states=None

E                     • return_dict=None

E                     • training=False

E                     • kwargs={'mask': {'inputs_embeds': 'tf.Tensor(shape=(None, None), dtype=bool)'}}

E

E

E               Call arguments received by layer "bert_block" (type BertBlock):

E                 • inputs=tf.RaggedTensor(values=Tensor("model/concat_features/RaggedConcat/concat:0", shape=(None, 48), dtype=float32), row_splits=Tensor("model/parallel_block/embeddings/categories/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))

E                 • kwargs={'features': {'item_id_seq': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))', 'categories': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_1/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths_1/control_dependency:0", shape=(None,), dtype=int32))', 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_2/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_2/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_3/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_3/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_4/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_4/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_5/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_5/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_6/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_6/control_dependency:0", shape=(None,), dtype=int32))', 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}, 'training': 'False', 'testing': 'False'}

E

E

E           Call arguments received by layer "model" (type Model):

E             • inputs={'item_id_seq': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'categories': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}

E             • targets=None

E             • training=False

E             • testing=False

E             • output_context=False
/tmp/__autograph_generated_file237fjfzg.py:79: ValueError

----------------------------- Captured stdout call -----------------------------
1/1 [==============================] - ETA: 0s - loss: 3.7263 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 3.7263������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

1/1 [==============================] - 1s 1s/step - loss: 3.7263 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 3.7263

----------------------------- Captured stderr call -----------------------------

WARNING:tensorflow:Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:utils.py:76 Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

____________ test_transformer_with_masked_language_modeling[False] _____________
sequence_testing_data = <merlin.io.dataset.Dataset object at 0x7f0d23b20220>

run_eagerly = False
@pytest.mark.parametrize("run_eagerly", [True, False])
def test_transformer_with_masked_language_modeling(sequence_testing_data: Dataset, run_eagerly):

    seq_schema = sequence_testing_data.schema.select_by_tag(Tags.SEQUENCE).select_by_tag(
        Tags.CATEGORICAL
    )
    target = sequence_testing_data.schema.select_by_tag(Tags.ITEM_ID).column_names[0]

    loader = Loader(sequence_testing_data, batch_size=8, shuffle=False)
    model = mm.Model(
        mm.InputBlockV2(
            seq_schema,
            categorical=mm.Embeddings(
                seq_schema.select_by_tag(Tags.CATEGORICAL), sequence_combiner=None
            ),
        ),
        BertBlock(
            d_model=48,
            n_head=8,
            n_layer=2,
            pre=mm.SequentialBlock([mm.SequenceMaskLastInference(), mm.ReplaceMaskedEmbeddings()]),
            post="inference_hidden_state",
        ),
        mm.CategoricalOutput(
            seq_schema.select_by_name(target),
            default_loss="categorical_crossentropy",
        ),
    )
    seq_mask_random = mm.SequenceMaskRandom(schema=seq_schema, target=target, masking_prob=0.3)

    inputs, targets = next(iter(loader))
    outputs = model(inputs, targets=targets, training=True)
    assert list(outputs.shape) == [8, 4, 51997]


  testing_utils.model_test(


        model,
        loader,
        run_eagerly=run_eagerly,
        reload_model=True,
        fit_kwargs={"pre": seq_mask_random},
    )

tests/unit/tf/transformers/test_block.py:301:

merlin/models/tf/utils/testing_utils.py:97: in model_test

model.save(tmpdir)

merlin/models/tf/models/base.py:1155: in save

super().save(

merlin/models/tf/models/base.py:1100: in save

super().save(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:2435: in save

save.save_model(self, filepath, overwrite, include_optimizer, save_format,

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/saving/save.py:153: in save_model

saved_model_save.save(model, filepath, overwrite, include_optimizer,

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save.py:93: in save

saved_nodes, node_paths = save_lib.save_and_return_nodes(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1325: in save_and_return_nodes

build_meta_graph(obj, signatures, options, meta_graph_def))

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1491: in build_meta_graph

return build_meta_graph_impl(obj, signatures, options, meta_graph_def)

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:1433: in build_meta_graph_impl

signatures = signature_serialization.find_function_to_export(

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/signature_serialization.py:103: in find_function_to_export

for name, child in children:

../../../.local/lib/python3.8/site-packages/tensorflow/python/saved_model/save.py:143: in list_children

for name, child in super(AugmentedGraphView, self).list_children(

../../../.local/lib/python3.8/site-packages/tensorflow/python/training/tracking/graph_view.py:203: in list_children

in obj.trackable_children(save_type, **kwargs).items()]

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:3201: in trackable_children

children = super(Model, self).trackable_children(save_type, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:3174: in trackable_children

children = self.trackable_saved_model_saver.trackable_children(cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/base_serialization.py:59: in trackable_children

children = self.objects_to_serialize(serialization_cache)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:68: in objects_to_serialize

return (self.get_serialized_attributes(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:88: in get_serialized_attributes

object_dict, function_dict = self.get_serialized_attributes_internal(

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/model_serialization.py:51: in get_serialized_attributes_internal

default_signature = save_impl.default_save_signature(self.obj)

../../../.local/lib/python3.8/site-packages/keras/saving/saved_model/save_impl.py:208: in default_save_signature

fn = saving_utils.trace_model_call(layer)

../../../.local/lib/python3.8/site-packages/keras/saving/saving_utils.py:148: in trace_model_call

return wrapped_model.get_concrete_function(*model_args, **model_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1239: in get_concrete_function

concrete = self.get_concrete_function_garbage_collected(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:1219: in get_concrete_function_garbage_collected

self.initialize(args, kwargs, add_initializers_to=initializers)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:785: in initialize

self.stateful_fn.get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2480: in get_concrete_function_internal_garbage_collected

graph_function, _ = self.maybe_define_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2711: in maybe_define_function

graph_function = self.create_graph_function(args, kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/function.py:2627: in create_graph_function

func_graph_module.func_graph_from_py_func(

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1141: in func_graph_from_py_func

func_outputs = python_func(*func_args, **func_kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py:677: in wrapped_fn

out = weak_wrapped_fn().wrapped(*args, **kwds)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1127: in autograph_handler

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py:1116: in autograph_handler

return autograph.converted_call(

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filecexucevj.py:14: in tf___wrapped_model

outputs = ag.converted_call(ag.ld(model), tuple(ag.ld(args)), dict(**ag.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:377: in converted_call

return call_unconverted(f, args, kwargs, options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/training.py:490: in call

return super().call(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_filexenb4ya2.py:42: in tf__call

ag.for_stmt(ag.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:449: in for_stmt

py_for_stmt(iter, extra_test, body, None, None)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:498: in py_for_stmt

body(target)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:464: in protected_body

original_body(protected_iter)

/tmp/autograph_generated_filexenb4ya2.py:40: in loop_body

(outputs, context) = ag.converted_call(ag.ld(self).call_child, (ag.ld(block), ag.ld(outputs), ag.ld(context)), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:441: in converted_call

result = converted_f(*effective_args)

/tmp/autograph_generated_fileg45vl_uj.py:25: in tf___call_child

outputs = ag.converted_call(ag.ld(call_layer), (ag.ld(child), ag.ld(inputs)), dict(**ag.ld(call_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_fileymmumnvc.py:50: in tf__call_layer

retval = ag.converted_call(ag.ld(layer), ((ag.ld(inputs),) + tuple(ag_.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_fileainhchfl.py:14: in tf____call

retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:458: in call_unconverted

return f(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file8dwrq7uo.py:12: in tf__call

transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:331: in converted_call

return call_unconverted(f, args, kwargs, options, False)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:459: in call_unconverted

return f(*args)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:60: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/engine/base_layer.py:1014: in call

outputs = call_fn(inputs, *args, **kwargs)

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:146: in error_handler

raise new_e.with_traceback(e.traceback) from None

../../../.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py:92: in error_handler

return fn(*args, **kwargs)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:692: in wrapper

raise e.ag_error_metadata.to_exception(e)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:689: in wrapper

return converted_call(f, args, kwargs, options=options)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file1yic3w2n.py:10: in tf__call

inputs = ag.converted_call(ag.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:439: in converted_call

result = converted_f(*effective_args, **kwargs)

/tmp/autograph_generated_file237fjfzg.py:83: in tf__input_processing

ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1341: in if_stmt

_py_if_stmt(cond, body, orelse)

../../../.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py:1394: in _py_if_stmt

return body() if cond else orelse()

def if_body_4():


  raise ag__.converted_call(ag__.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)


E       ValueError: in user code:

E

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/saving/saving_utils.py", line 138, in wrapped_model  *

E               outputs = model(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/training.py", line 490, in call

E               return super().call(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_filexenb4ya2.py", line 42, in tf__call  **

E               ag.for_stmt(ag_.ld(self).blocks, None, loop_body, get_state_1, set_state_1, ('context', 'outputs'), {'iterate_names': 'block'})

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 449, in for_stmt

E               py_for_stmt(iter, extra_test, body, None, None)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 498, in py_for_stmt

E               body(target)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 464, in protected_body

E               original_body(protected_iter)

E           File "/tmp/autograph_generated_filexenb4ya2.py", line 40, in loop_body

E               (outputs, context) = ag.converted_call(ag_.ld(self).call_child, (ag_.ld(block), ag__.ld(outputs), ag__.ld(context)), None, fscope)

E           File "/tmp/autograph_generated_fileg45vl_uj.py", line 25, in tf___call_child  **

E               outputs = ag.converted_call(ag__.ld(call_layer), (ag__.ld(child), ag__.ld(inputs)), dict(**ag__.ld(call_kwargs)), fscope)

E           File "/tmp/autograph_generated_fileymmumnvc.py", line 50, in tf__call_layer  **

E               retval = ag_.converted_call(ag__.ld(layer), ((ag__.ld(inputs),) + tuple(ag__.ld(args))), dict(**ag__.ld(filtered_kwargs)), fscope)

E           File "/tmp/autograph_generated_fileainhchfl.py", line 14, in tf____call  **

E               retval_ = ag__.converted_call(ag__.converted_call(ag__.ld(super), (), None, fscope).call, tuple(ag__.ld(args)), dict(**ag__.ld(kwargs)), fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file8dwrq7uo.py", line 12, in tf__call  **

E               transformer = ag.converted_call(ag__.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E               return fn(*args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E               outputs = call_fn(inputs, *args, **kwargs)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E               raise new_e.with_traceback(e.traceback) from None

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E               return fn(*args, **kwargs)

E           File "/tmp/autograph_generated_file1yic3w2n.py", line 10, in tf__call  **

E               inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E           File "/tmp/autograph_generated_file237fjfzg.py", line 83, in tf__input_processing  **

E               ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E               py_if_stmt(cond, body, orelse)

E           File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E               return body() if cond else orelse()

E           File "/tmp/autograph_generated_file237fjfzg.py", line 79, in if_body_4

E               raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E           ValueError: Exception encountered when calling layer "model" (type Model).

E

E           in user code:

E

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1236, in call  *

E                   outputs, context = self.call_child(block, outputs, context)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/models/base.py", line 1265, in call_child  *

E                   outputs = call_layer(child, inputs, **call_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py", line 437, in call_layer  *

E                   return layer(inputs, *args, **filtered_kwargs)

E               File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/config/schema.py", line 58, in call  *

E                   return super().call(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_file8dwrq7uo.py", line 12, in tf__call  **

E                   transformer = ag.converted_call(ag.ld(self).transformer, (ag__.ld(pre),), None, fscope)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                   return fn(*args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                   outputs = call_fn(inputs, *args, **kwargs)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                   raise new_e.with_traceback(e.traceback) from None

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                   return fn(*args, **kwargs)

E               File "/tmp/autograph_generated_file1yic3w2n.py", line 10, in tf__call  **

E                   inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E               File "/tmp/autograph_generated_file237fjfzg.py", line 83, in tf__input_processing  **

E                   ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                   py_if_stmt(cond, body, orelse)

E               File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                   return body() if cond else orelse()

E               File "/tmp/autograph_generated_file237fjfzg.py", line 79, in if_body_4

E                   raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E               ValueError: Exception encountered when calling layer "bert_block" (type BertBlock).

E

E               in user code:

E

E                   File "/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transformers/block.py", line 131, in call  *

E                       transformer = self.transformer(pre)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 60, in error_handler  **

E                       return fn(*args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in call

E                       outputs = call_fn(inputs, *args, **kwargs)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 146, in error_handler

E                       raise new_e.with_traceback(e.traceback) from None

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler

E                       return fn(*args, **kwargs)

E                   File "/tmp/autograph_generated_file1yic3w2n.py", line 10, in tf__call  **

E                       inputs = ag.converted_call(ag__.ld(input_processing), (), dict(func=ag__.ld(self).call, config=ag__.ld(self).config, input_ids=ag__.ld(input_ids), attention_mask=ag__.ld(attention_mask), token_type_ids=ag__.ld(token_type_ids), position_ids=ag__.ld(position_ids), head_mask=ag__.ld(head_mask), inputs_embeds=ag__.ld(inputs_embeds), encoder_hidden_states=ag__.ld(encoder_hidden_states), encoder_attention_mask=ag__.ld(encoder_attention_mask), past_key_values=ag__.ld(past_key_values), use_cache=ag__.ld(use_cache), output_attentions=ag__.ld(output_attentions), output_hidden_states=ag__.ld(output_hidden_states), return_dict=ag__.ld(return_dict), training=ag__.ld(training), kwargs_call=ag__.ld(kwargs)), fscope)

E                   File "/tmp/autograph_generated_file237fjfzg.py", line 83, in tf__input_processing  **

E                       ag.if_stmt((ag__.converted_call(ag__.ld(len), (ag__.ld(kwargs)['kwargs_call'],), None, fscope) > 0), if_body_4, else_body_4, get_state_4, set_state_4, (), 0)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1341, in if_stmt

E                       py_if_stmt(cond, body, orelse)

E                   File "/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/operators/control_flow.py", line 1394, in py_if_stmt

E                       return body() if cond else orelse()

E                   File "/tmp/autograph_generated_file237fjfzg.py", line 79, in if_body_4

E                       raise ag.converted_call(ag.ld(ValueError), (f"The following keyword arguments are not supported by this model: {ag__.converted_call(ag__.ld(list), (ag__.converted_call(ag__.ld(kwargs)['kwargs_call'].keys, (), None, fscope),), None, fscope)}.",), None, fscope)

E

E                   ValueError: Exception encountered when calling layer "bert" (type TFBertMainLayer).

E

E                   in user code:

E

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/models/bert/modeling_tf_bert.py", line 743, in call  *

E                           inputs = input_processing(

E                       File "/usr/local/lib/python3.8/dist-packages/transformers/modeling_tf_utils.py", line 387, in input_processing  *

E                           raise ValueError(

E

E                       ValueError: The following keyword arguments are not supported by this model: ['mask'].

E

E

E                   Call arguments received by layer "bert" (type TFBertMainLayer):

E                     • input_ids={'inputs_embeds': 'tf.Tensor(shape=(None, None, 48), dtype=float32)'}

E                     • attention_mask=None

E                     • token_type_ids=None

E                     • position_ids=None

E                     • head_mask=None

E                     • inputs_embeds=None

E                     • encoder_hidden_states=None

E                     • encoder_attention_mask=None

E                     • past_key_values=None

E                     • use_cache=None

E                     • output_attentions=None

E                     • output_hidden_states=None

E                     • return_dict=None

E                     • training=False

E                     • kwargs={'mask': {'inputs_embeds': 'tf.Tensor(shape=(None, None), dtype=bool)'}}

E

E

E               Call arguments received by layer "bert_block" (type BertBlock):

E                 • inputs=tf.RaggedTensor(values=Tensor("model/concat_features/RaggedConcat/concat:0", shape=(None, 48), dtype=float32), row_splits=Tensor("model/parallel_block/embeddings/categories/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))

E                 • kwargs={'features': {'item_id_seq': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths/control_dependency:0", shape=(None,), dtype=int32))', 'categories': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_1/ExpandDims:0", shape=(None, 1), dtype=int64), row_splits=Tensor("model/process_list/RaggedFromRowLengths_1/control_dependency:0", shape=(None,), dtype=int32))', 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_2/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_2/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_3/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_3/control_dependency:0", shape=(None,), dtype=int32))', 'event_hour_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_4/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_4/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_sin': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_5/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_5/control_dependency:0", shape=(None,), dtype=int32))', 'event_weekday_cos': 'tf.RaggedTensor(values=Tensor("model/process_list/RaggedExpandDims_6/ExpandDims:0", shape=(None, 1), dtype=float32), row_splits=Tensor("model/process_list/RaggedFromRowLengths_6/control_dependency:0", shape=(None,), dtype=int32))', 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}, 'training': 'False', 'testing': 'False'}

E

E

E           Call arguments received by layer "model" (type Model):

E             • inputs={'item_id_seq': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'categories': ('tf.Tensor(shape=(None, 1), dtype=int64)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'test_user_id': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'user_country': 'tf.Tensor(shape=(None, 1), dtype=int64)', 'item_age_days_norm': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_hour_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_sin': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'event_weekday_cos': ('tf.Tensor(shape=(None, 1), dtype=float32)', 'tf.Tensor(shape=(None, 1), dtype=int32)'), 'user_age': 'tf.Tensor(shape=(None, 1), dtype=float32)'}

E             • targets=None

E             • training=False

E             • testing=False

E             • output_context=False
/tmp/__autograph_generated_file237fjfzg.py:79: ValueError

----------------------------- Captured stdout call -----------------------------
1/1 [==============================] - ETA: 0s - loss: 4.4090 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 4.4090������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

1/1 [==============================] - 8s 8s/step - loss: 4.4090 - recall_at_10: 0.0000e+00 - mrr_at_10: 0.0000e+00 - ndcg_at_10: 0.0000e+00 - map_at_10: 0.0000e+00 - precision_at_10: 0.0000e+00 - regularization_loss: 0.0000e+00 - loss_batch: 4.4090

----------------------------- Captured stderr call -----------------------------

WARNING:tensorflow:Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

WARNING:tensorflow:Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

2022-12-02 00:56:05.399886: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:907] Skipping loop optimization for Merge node with control input: model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Assert/AssertGuard/branch_executed/_9

------------------------------ Captured log call -------------------------------

WARNING  tensorflow:utils.py:76 Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

WARNING  tensorflow:utils.py:76 Gradients do not exist for variables ['bert/embeddings/word_embeddings/weight:0', 'bert/pooler/dense/kernel:0', 'bert/pooler/dense/bias:0'] when minimizing the loss. If you're using model.compile(), did you forget to provide a lossargument?

=============================== warnings summary ===============================

../../../../../usr/lib/python3/dist-packages/requests/init.py:89

/usr/lib/python3/dist-packages/requests/init.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!

warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
../../../.local/lib/python3.8/site-packages/flatbuffers/compat.py:19

/var/jenkins_home/.local/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses

import imp
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.

'nearest': pil_image.NEAREST,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.

'bilinear': pil_image.BILINEAR,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.

'bicubic': pil_image.BICUBIC,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.

'hamming': pil_image.HAMMING,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.

'box': pil_image.BOX,
../../../.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.

'lanczos': pil_image.LANCZOS,
tests/unit/datasets/test_advertising.py: 1 warning

tests/unit/datasets/test_ecommerce.py: 3 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 10 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 60 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 6 warnings

tests/unit/tf/core/test_index.py: 8 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/horovod/test_horovod.py: 6 warnings

tests/unit/tf/inputs/test_base.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 7 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 28 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 38 warnings

tests/unit/tf/models/test_retrieval.py: 123 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/prediction_tasks/test_retrieval.py: 1 warning

tests/unit/tf/transformers/test_block.py: 14 warnings

tests/unit/tf/transforms/test_bias.py: 2 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_noise.py: 1 warning

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 9 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 3 warnings

tests/unit/xgb/test_xgboost.py: 18 warnings

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.ITEM_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.ITEM: 'item'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_ecommerce.py: 3 warnings

tests/unit/datasets/test_entertainment.py: 4 warnings

tests/unit/datasets/test_social.py: 1 warning

tests/unit/datasets/test_synthetic.py: 9 warnings

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_core.py: 6 warnings

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/test_cross.py: 5 warnings

tests/unit/tf/blocks/test_dlrm.py: 9 warnings

tests/unit/tf/blocks/test_interactions.py: 2 warnings

tests/unit/tf/blocks/test_mlp.py: 60 warnings

tests/unit/tf/blocks/test_optimizer.py: 30 warnings

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 11 warnings

tests/unit/tf/core/test_aggregation.py: 6 warnings

tests/unit/tf/core/test_base.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 7 warnings

tests/unit/tf/core/test_index.py: 3 warnings

tests/unit/tf/core/test_prediction.py: 2 warnings

tests/unit/tf/horovod/test_horovod.py: 1 warning

tests/unit/tf/inputs/test_base.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 7 warnings

tests/unit/tf/inputs/test_embedding.py: 20 warnings

tests/unit/tf/inputs/test_tabular.py: 18 warnings

tests/unit/tf/models/test_base.py: 28 warnings

tests/unit/tf/models/test_benchmark.py: 2 warnings

tests/unit/tf/models/test_ranking.py: 36 warnings

tests/unit/tf/models/test_retrieval.py: 91 warnings

tests/unit/tf/outputs/test_base.py: 6 warnings

tests/unit/tf/outputs/test_classification.py: 6 warnings

tests/unit/tf/outputs/test_contrastive.py: 19 warnings

tests/unit/tf/outputs/test_regression.py: 2 warnings

tests/unit/tf/prediction_tasks/test_classification.py: 2 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 5 warnings

tests/unit/tf/transformers/test_block.py: 10 warnings

tests/unit/tf/transforms/test_features.py: 10 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 10 warnings

tests/unit/tf/transforms/test_sequence.py: 15 warnings

tests/unit/tf/utils/test_batch.py: 7 warnings

tests/unit/tf/utils/test_dataset.py: 2 warnings

tests/unit/torch/block/test_base.py: 4 warnings

tests/unit/torch/block/test_mlp.py: 1 warning

tests/unit/torch/features/test_continuous.py: 1 warning

tests/unit/torch/features/test_embedding.py: 4 warnings

tests/unit/torch/features/test_tabular.py: 4 warnings

tests/unit/torch/model/test_head.py: 12 warnings

tests/unit/torch/model/test_model.py: 2 warnings

tests/unit/torch/tabular/test_aggregation.py: 6 warnings

tests/unit/torch/tabular/test_transformations.py: 2 warnings

tests/unit/xgb/test_xgboost.py: 17 warnings

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.USER_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.USER: 'user'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/datasets/test_ecommerce.py: 1 warning

tests/unit/datasets/test_entertainment.py: 1 warning

tests/unit/implicit/test_implicit.py: 1 warning

tests/unit/lightfm/test_lightfm.py: 1 warning

tests/unit/tf/test_loader.py: 1 warning

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py: 2 warnings

tests/unit/tf/blocks/retrieval/test_two_tower.py: 2 warnings

tests/unit/tf/core/test_combinators.py: 11 warnings

tests/unit/tf/core/test_encoder.py: 2 warnings

tests/unit/tf/core/test_prediction.py: 1 warning

tests/unit/tf/horovod/test_horovod.py: 1 warning

tests/unit/tf/inputs/test_continuous.py: 4 warnings

tests/unit/tf/inputs/test_embedding.py: 9 warnings

tests/unit/tf/inputs/test_tabular.py: 8 warnings

tests/unit/tf/models/test_ranking.py: 20 warnings

tests/unit/tf/models/test_retrieval.py: 10 warnings

tests/unit/tf/prediction_tasks/test_multi_task.py: 16 warnings

tests/unit/tf/prediction_tasks/test_regression.py: 3 warnings

tests/unit/tf/transforms/test_negative_sampling.py: 9 warnings

tests/unit/xgb/test_xgboost.py: 12 warnings

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/merlin/schema/tags.py:148: UserWarning: Compound tags like Tags.SESSION_ID have been deprecated and will be removed in a future version. Please use the atomic versions of these tags, like [<Tags.SESSION: 'session'>, <Tags.ID: 'id'>].

warnings.warn(
tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_matrix_factorization.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/blocks/retrieval/test_two_tower.py::test_matrix_factorization_embedding_export

tests/unit/tf/inputs/test_embedding.py::test_embedding_features_exporting_and_loading_pretrained_initializer

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/inputs/embedding.py:970: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

embeddings_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(embeddings)))
tests/unit/tf/blocks/retrieval/test_two_tower.py: 1 warning

tests/unit/tf/core/test_index.py: 4 warnings

tests/unit/tf/horovod/test_horovod.py: 3 warnings

tests/unit/tf/models/test_retrieval.py: 63 warnings

tests/unit/tf/prediction_tasks/test_next_item.py: 3 warnings

tests/unit/tf/utils/test_batch.py: 2 warnings

/tmp/autograph_generated_filefpzqgkss.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead

ag.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)
tests/unit/tf/core/test_combinators.py::test_parallel_block_select_by_tags

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/core/tabular.py:602: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working

elif isinstance(self.feature_names, collections.Sequence):
tests/unit/tf/core/test_encoder.py: 1 warning

tests/unit/tf/core/test_index.py: 5 warnings

tests/unit/tf/models/test_retrieval.py: 30 warnings

tests/unit/tf/utils/test_batch.py: 4 warnings

tests/unit/tf/utils/test_dataset.py: 1 warning

/var/jenkins_home/workspace/merlin_models/models/merlin/models/utils/dataset.py:75: DeprecationWarning: unique_rows_by_features is deprecated and will be removed in a future version. Please use unique_by_tag instead.

warnings.warn(
tests/unit/tf/models/test_base.py::test_model_pre_post[True]

tests/unit/tf/models/test_base.py::test_model_pre_post[False]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.1]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.3]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.5]

tests/unit/tf/transforms/test_noise.py::test_stochastic_swap_noise[0.7]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py:1082: UserWarning: tf.keras.backend.random_binomial is deprecated, and will be removed in a future version.Please use tf.keras.backend.random_bernoulli instead.

return dispatch_target(*args, **kwargs)
tests/unit/tf/models/test_base.py::test_freeze_parallel_block[True]

tests/unit/tf/models/test_base.py::test_freeze_sequential_block

tests/unit/tf/models/test_base.py::test_freeze_unfreeze

tests/unit/tf/models/test_base.py::test_unfreeze_all_blocks

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/optimizers/optimizer_v2/gradient_descent.py:108: UserWarning: The lr argument is deprecated, use learning_rate instead.

super(SGD, self).init(name, **kwargs)
tests/unit/tf/models/test_base.py::test_retrieval_model_query

tests/unit/tf/models/test_base.py::test_retrieval_model_query

tests/unit/tf/models/test_retrieval.py::test_youtube_dnn_v2_export_embeddings

tests/unit/tf/models/test_retrieval.py::test_youtube_dnn_topk_evaluation[True]

tests/unit/tf/models/test_retrieval.py::test_youtube_dnn_topk_evaluation[False]

tests/unit/tf/transformers/test_block.py::test_retrieval_transformer[True]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py:298: DeprecationWarning: This function is deprecated in favor of cupy.from_dlpack

tensor_cupy = cupy.fromDlpack(to_dlpack(tf.convert_to_tensor(tensor)))
tests/unit/tf/models/test_ranking.py::test_deepfm_model_only_categ_feats[False]

tests/unit/tf/models/test_ranking.py::test_deepfm_model_categ_and_continuous_feats[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_3/parallel_block_2/sequential_block_3/sequential_block_2/private__dense_1/dense_1/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_categorical_one_hot[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_model_hashed_cross[False]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_2/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[True]

tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/transforms/features.py:569: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_embedding_custom_inputblock[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:371: UserWarning: Please make sure input features to be categorical, detect user_age has no categorical tag

return py_builtins.overload_of(f)(*args)
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_onehot_multihot_feature_interaction[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_5/sequential_block_9/sequential_block_8/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/models/test_ranking.py::test_wide_deep_model_wide_feature_interaction_multi_optimizer[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Reshape:0", shape=(None, 1), dtype=float32), dense_shape=Tensor("gradient_tape/model/parallel_block_4/sequential_block_6/sequential_block_5/private__dense_3/dense_3/embedding_lookup_sparse/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_retrieval_transformer[True]

tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[True]

tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/workspace/merlin_models/models/merlin/models/tf/utils/tf_utils.py:83: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.

config[key] = tf.keras.utils.serialize_keras_object(maybe_value)
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_as_classfication_model[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[True]

tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/keras/saving/saved_model/layer_serialization.py:112: CustomMaskWarning: Custom mask layers require a config and must override get_config. When loading, the custom mask layer must be passed to the custom_objects argument.

return generic_utils.serialize_keras_object(obj)
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_causal_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/boolean_mask/GatherV2:0", shape=(None, 48), dtype=float32), dense_shape=Tensor("gradient_tape/model/bert_block/prepare_transformer_inputs_1/RaggedToTensor/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Reshape_3:0", shape=(None,), dtype=int64), values=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Reshape_2:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/bert_block/sequential_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask_1/GatherV2:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/boolean_mask/GatherV2:0", shape=(None, 48), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/prepare_transformer_inputs_5/RaggedToTensor/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_3:0", shape=(None,), dtype=int64), values=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Reshape_2:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_1:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/tf/transformers/test_block.py::test_transformer_with_masked_language_modeling_check_eval_masked[False]

/var/jenkins_home/.local/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:444: UserWarning: Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradient_tape/model/gpt2_block/replace_masked_embeddings/RaggedWhere/RaggedTile_2/Reshape_3:0", shape=(None,), dtype=int32), values=Tensor("gradient_tape/model/concat_features/RaggedConcat/Slice_3:0", shape=(None, None), dtype=float32), dense_shape=Tensor("gradient_tape/model/concat_features/RaggedConcat/Shape_1:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

warnings.warn(
tests/unit/torch/block/test_mlp.py::test_mlp_block

/var/jenkins_home/workspace/merlin_models/models/tests/unit/torch/_conftest.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:201.)

return {key: torch.tensor(value) for key, value in data.items()}
tests/unit/xgb/test_xgboost.py::test_without_dask_client

tests/unit/xgb/test_xgboost.py::TestXGBoost::test_music_regression

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs0-DaskDeviceQuantileDMatrix]

tests/unit/xgb/test_xgboost.py::test_gpu_hist_dmatrix[fit_kwargs1-DaskDMatrix]

tests/unit/xgb/test_xgboost.py::TestEvals::test_multiple

tests/unit/xgb/test_xgboost.py::TestEvals::test_default

tests/unit/xgb/test_xgboost.py::TestEvals::test_train_and_valid

tests/unit/xgb/test_xgboost.py::TestEvals::test_invalid_data

/var/jenkins_home/workspace/merlin_models/models/merlin/models/xgb/init.py:344: UserWarning: Ignoring list columns as inputs to XGBoost model: ['item_genres', 'user_genres'].

warnings.warn(f"Ignoring list columns as inputs to XGBoost model: {list_column_names}.")
tests/unit/xgb/test_xgboost.py::TestXGBoost::test_unsupported_objective

/var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/lib/python3.8/site-packages/tornado/ioloop.py:350: DeprecationWarning: make_current is deprecated; start the event loop first

self.make_current()
tests/unit/xgb/test_xgboost.py: 14 warnings

/usr/local/lib/python3.8/dist-packages/xgboost/dask.py:884: RuntimeWarning: coroutine 'Client._wait_for_workers' was never awaited

client.wait_for_workers(n_workers)

Enable tracemalloc to get traceback where the object was allocated.

See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
tests/unit/xgb/test_xgboost.py: 11 warnings

/usr/local/lib/python3.8/dist-packages/cudf/core/dataframe.py:1183: DeprecationWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning.

mask = pd.Series(mask)
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
---------- coverage: platform linux, python 3.8.10-final-0 -----------

Name                                                                 Stmts   Miss  Cover
merlin/datasets/init.py                                              2      0   100%

merlin/datasets/advertising/init.py                                  2      0   100%

merlin/datasets/advertising/criteo/init.py                           0      0   100%

merlin/datasets/advertising/criteo/dataset.py                           79     49    38%

merlin/datasets/advertising/criteo/transformed/init.py               0      0   100%

merlin/datasets/ecommerce/init.py                                    4      0   100%

merlin/datasets/ecommerce/aliccp/init.py                             0      0   100%

merlin/datasets/ecommerce/aliccp/dataset.py                            141     87    38%

merlin/datasets/ecommerce/aliccp/raw/init.py                         0      0   100%

merlin/datasets/ecommerce/aliccp/transformed/init.py                 0      0   100%

merlin/datasets/ecommerce/booking/init.py                            0      0   100%

merlin/datasets/ecommerce/booking/dataset.py                           127    100    21%

merlin/datasets/ecommerce/booking/raw/init.py                        0      0   100%

merlin/datasets/ecommerce/booking/transformed/init.py                0      0   100%

merlin/datasets/ecommerce/dressipi/init.py                           0      0   100%

merlin/datasets/ecommerce/dressipi/dataset.py                           45     37    18%

merlin/datasets/ecommerce/dressipi/preprocessed/init.py              0      0   100%

merlin/datasets/ecommerce/large/init.py                              0      0   100%

merlin/datasets/ecommerce/small/init.py                              0      0   100%

merlin/datasets/ecommerce/transactions/init.py                       0      0   100%

merlin/datasets/entertainment/init.py                                2      0   100%

merlin/datasets/entertainment/movielens/1m-raw/init.py               0      0   100%

merlin/datasets/entertainment/movielens/1m-raw/ratings/init.py       0      0   100%

merlin/datasets/entertainment/movielens/1m/init.py                   0      0   100%

merlin/datasets/entertainment/movielens/25m/init.py                  0      0   100%

merlin/datasets/entertainment/movielens/100k/init.py                 0      0   100%

merlin/datasets/entertainment/movielens/init.py                      0      0   100%

merlin/datasets/entertainment/movielens/dataset.py                     152    122    20%

merlin/datasets/entertainment/music_streaming/init.py                0      0   100%

merlin/datasets/social/init.py                                       0      0   100%

merlin/datasets/synthetic.py                                           147      8    95%

merlin/datasets/testing/init.py                                      0      0   100%

merlin/datasets/testing/sequence_testing/init.py                     0      0   100%

merlin/models/init.py                                                2      0   100%

merlin/models/_version.py                                              354    205    42%

merlin/models/api.py                                                    14      5    64%

merlin/models/config/init.py                                         0      0   100%

merlin/models/config/schema.py                                          62      0   100%

merlin/models/implicit/init.py                                      27      4    85%

merlin/models/io.py                                                     15      0   100%

merlin/models/lightfm/init.py                                       23      0   100%

merlin/models/loader/init.py                                         0      0   100%

merlin/models/loader/backend.py                                        379     41    89%

merlin/models/loader/dataframe_iter.py                                  21     17    19%

merlin/models/loader/tf_utils.py                                        57     27    53%

merlin/models/loader/utils.py                                           40     15    62%

merlin/models/tf/init.py                                            70      0   100%

merlin/models/tf/blocks/init.py                                      0      0   100%

merlin/models/tf/blocks/cross.py                                        44      0   100%

merlin/models/tf/blocks/dlrm.py                                         49      2    96%

merlin/models/tf/blocks/experts.py                                      99     17    83%

merlin/models/tf/blocks/interaction.py                                 108     40    63%

merlin/models/tf/blocks/mlp.py                                         117      7    94%

merlin/models/tf/blocks/optimizer.py                                   173     12    93%

merlin/models/tf/blocks/retrieval/init.py                            0      0   100%

merlin/models/tf/blocks/retrieval/base.py                              175     62    65%

merlin/models/tf/blocks/retrieval/matrix_factorization.py               35      1    97%

merlin/models/tf/blocks/retrieval/two_tower.py                          30      0   100%

merlin/models/tf/blocks/sampling/init.py                             0      0   100%

merlin/models/tf/blocks/sampling/base.py                                29      2    93%

merlin/models/tf/blocks/sampling/cross_batch.py                         46      2    96%

merlin/models/tf/blocks/sampling/in_batch.py                            35      0   100%

merlin/models/tf/blocks/sampling/queue.py                              115     12    90%

merlin/models/tf/core/init.py                                        0      0   100%

merlin/models/tf/core/aggregation.py                                   241     45    81%

merlin/models/tf/core/base.py                                          242     51    79%

merlin/models/tf/core/combinators.py                                   426     53    88%

merlin/models/tf/core/encoder.py                                       182     28    85%

merlin/models/tf/core/index.py                                         106     16    85%

merlin/models/tf/core/prediction.py                                     50      1    98%

merlin/models/tf/core/tabular.py                                       280     29    90%

merlin/models/tf/distributed/init.py                                 0      0   100%

merlin/models/tf/distributed/backend.py                                  9      2    78%

merlin/models/tf/inputs/init.py                                      0      0   100%

merlin/models/tf/inputs/base.py                                         64     20    69%

merlin/models/tf/inputs/continuous.py                                   39      3    92%

merlin/models/tf/inputs/embedding.py                                   458     51    89%

merlin/models/tf/loader.py                                             268     94    65%

merlin/models/tf/losses/init.py                                      4      0   100%

merlin/models/tf/losses/base.py                                          9      0   100%

merlin/models/tf/losses/listwise.py                                     13      0   100%

merlin/models/tf/losses/pairwise.py                                    115      1    99%

merlin/models/tf/metrics/init.py                                     2      0   100%

merlin/models/tf/metrics/evaluation.py                                 105     48    54%

merlin/models/tf/metrics/topk.py                                       198     48    76%

merlin/models/tf/models/init.py                                      0      0   100%

merlin/models/tf/models/base.py                                        772    101    87%

merlin/models/tf/models/benchmark.py                                    16      0   100%

merlin/models/tf/models/ranking.py                                      67      3    96%

merlin/models/tf/models/retrieval.py                                    78      4    95%

merlin/models/tf/models/utils.py                                        10      1    90%

merlin/models/tf/outputs/init.py                                     0      0   100%

merlin/models/tf/outputs/base.py                                       123     17    86%

merlin/models/tf/outputs/classification.py                              91      1    99%

merlin/models/tf/outputs/contrastive.py                                147     10    93%

merlin/models/tf/outputs/regression.py                                   9      0   100%

merlin/models/tf/outputs/sampling/init.py                            0      0   100%

merlin/models/tf/outputs/sampling/base.py                               78     21    73%

merlin/models/tf/outputs/sampling/in_batch.py                           37      1    97%

merlin/models/tf/outputs/sampling/popularity.py                         27      1    96%

merlin/models/tf/outputs/topk.py                                        98      6    94%

merlin/models/tf/prediction_tasks/init.py                            0      0   100%

merlin/models/tf/prediction_tasks/base.py                              207     37    82%

merlin/models/tf/prediction_tasks/classification.py                     68     17    75%

merlin/models/tf/prediction_tasks/multi.py                               7      0   100%

merlin/models/tf/prediction_tasks/next_item.py                          59     17    71%

merlin/models/tf/prediction_tasks/regression.py                         35      2    94%

merlin/models/tf/prediction_tasks/retrieval.py                          73      3    96%

merlin/models/tf/transformers/init.py                                0      0   100%

merlin/models/tf/transformers/block.py                                 102      2    98%

merlin/models/tf/transformers/transforms.py                             85      4    95%

merlin/models/tf/transforms/init.py                                  0      0   100%

merlin/models/tf/transforms/bias.py                                    111      9    92%

merlin/models/tf/transforms/features.py                                435     42    90%

merlin/models/tf/transforms/negative_sampling.py                        77      3    96%

merlin/models/tf/transforms/noise.py                                    43      1    98%

merlin/models/tf/transforms/regularization.py                           17      1    94%

merlin/models/tf/transforms/sequence.py                                302     44    85%

merlin/models/tf/transforms/tensor.py                                  158     13    92%

merlin/models/tf/typing.py                                               7      0   100%

merlin/models/tf/utils/init.py                                       0      0   100%

merlin/models/tf/utils/batch_utils.py                                   85      4    95%

merlin/models/tf/utils/repr_utils.py                                    69      4    94%

merlin/models/tf/utils/search_utils.py                                  34     22    35%

merlin/models/tf/utils/testing_utils.py                                206     36    83%

merlin/models/tf/utils/tf_utils.py                                     209     43    79%

merlin/models/torch/init.py                                         12      0   100%

merlin/models/torch/block/init.py                                    0      0   100%

merlin/models/torch/block/base.py                                      167     32    81%

merlin/models/torch/block/mlp.py                                        38      5    87%

merlin/models/torch/dataset.py                                          68      5    93%

merlin/models/torch/features/init.py                                 0      0   100%

merlin/models/torch/features/base.py                                     4      0   100%

merlin/models/torch/features/continuous.py                              22      0   100%

merlin/models/torch/features/embedding.py                              165     12    93%

merlin/models/torch/features/tabular.py                                 65      8    88%

merlin/models/torch/losses.py                                           28     28     0%

merlin/models/torch/model/init.py                                    0      0   100%

merlin/models/torch/model/base.py                                      286     65    77%

merlin/models/torch/model/prediction_task.py                            24      0   100%

merlin/models/torch/tabular/init.py                                  0      0   100%

merlin/models/torch/tabular/aggregation.py                              75      0   100%

merlin/models/torch/tabular/base.py                                    247     39    84%

merlin/models/torch/tabular/transformations.py                          67      3    96%

merlin/models/torch/typing.py                                            6      0   100%

merlin/models/torch/utils/init.py                                    0      0   100%

merlin/models/torch/utils/data_utils.py                                117    117     0%

merlin/models/torch/utils/examples_utils.py                             39     39     0%

merlin/models/torch/utils/torch_utils.py                                80     22    72%

merlin/models/utils/init.py                                          0      0   100%

merlin/models/utils/constants.py                                         3      0   100%

merlin/models/utils/dataset.py                                          38      4    89%

merlin/models/utils/dependencies.py                                     26     19    27%

merlin/models/utils/doc_utils.py                                        10      0   100%

merlin/models/utils/example_utils.py                                    31      2    94%

merlin/models/utils/misc_utils.py                                      118     90    24%

merlin/models/utils/nvt_utils.py                                        27     24    11%

merlin/models/utils/registry.py                                        101     29    71%

merlin/models/utils/schema_utils.py                                     90     19    79%

merlin/models/xgb/init.py                                          124      4    97%
TOTAL                                                                11651   2400    79%
=========================== short test summary info ============================

SKIPPED [1] tests/unit/datasets/test_advertising.py:20: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:64: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:80: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:94: ALI-CCP data is not available, pass it through env variable $DATA_PATH_ALICCP

SKIPPED [1] tests/unit/datasets/test_ecommerce.py:115: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [3] tests/unit/datasets/test_entertainment.py:44: No data-dir available, pass it through env variable $INPUT_DATA_DIR

SKIPPED [5] ../../../.local/lib/python3.8/site-packages/tensorflow/python/framework/test_util.py:2746: Not a test.

==== 2 failed, 871 passed, 13 skipped, 1474 warnings in 1881.81s (0:31:21) =====

ERROR: InvocationError for command /var/jenkins_home/workspace/merlin_models/models/.tox/py38-gpu/bin/python -m pytest --cov-report term --cov merlin -rxs tests/unit (exited with code 1)

___________________________________ summary ____________________________________

ERROR:   py38-gpu: commands failed

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash

cd /var/jenkins_home/

CUDA_VISIBLE_DEVICES=1 python test_res_push.py "https://api.GitHub.com/repos/NVIDIA-Merlin/models/issues/$ghprbPullId/comments" "/var/jenkins_home/jobs/$JOB_NAME/builds/$BUILD_NUMBER/log"

[workspace] $ /bin/bash /tmp/jenkins2046654639003195310.sh

gabrielspmoreira

I undestand this PR is focused in Masked Language Modeling. After reviewing, I am wondering if for Causal Language Modeling we shouldn't return during inference the last non-padded prediction per sequence. The thing is that we don't use `_keras_mask` for causal language modeling.
Thoughts?

merlin/models/tf/transformers/transforms.py

gabrielspmoreira · 2022-12-12T21:06:58Z

merlin/models/tf/transformers/transforms.py

+        """
+        if not training and not testing:
+            if getattr(inputs, "_keras_mask", None) is not None:
+                inputs = tf.reshape(


I think we can add an assert here to check if the resulting inputs after boolean_mask() has the 1st dim as before. Because we are trusting here that during inference only one position item (the last one) is masked per example, otherwise the batch size will differ after this conversion. Adding such assert should help users to understand this is an assumption when using this block.

Co-authored-by: Gabriel Moreira <gmoreira@nvidia.com>

* Use merlin-dataloader package * remove torch.dataset in favor of merlin.loader.torch * update dressipi notebook * minor clean up * Completely removes models DataLoader * Installs merlin-dataloader in github actions * Adds back the stop method * dataloader can produce sparse tensors using value counts * remove data files * fix torch tests * add missing target to dlrm test * use loader.peek() * add some comments to help understand horovod tests * make sparse tensors optional * cleanup * fix spelling * fix merge * replace while loop with for loop in horovod test * use loader context mananger * Update according to dataloader changes #80 * restore tox.ini * restore gh workflow * revert generator changes

- Change Python 3.9.7 to 3.8. - Update the versions of the GH actions. - Update pre-commit config file to get flake8 from GitHub instead of GitLab.

…his (#890) * Support `tuple` return typee from `pre` arg to `evaluate`, `predict` * Update CLM transformer test to use `pre` instead of Loader `transform` * Update youtube dnn tests to use transform as model fit pre * Add `pre` to ModelBlock fit/evaluate * Revert "Add `pre` to ModelBlock fit/evaluate" This reverts commit 1eef7b8. * Raise exception if ragged/sparse tensors are passed at training time. * Update model_test helper to avoid passing ragged tensors to `fit` * Handle x and y in model_test * Change process_lists param to False by default * Convert to tuple in test loader * Move order of ragged tensor assertion to before train_pre call * expand dims in test_classification * pass transform as pre in test in batch negatives * Update continuous and retrieval tests * Remove test of sequence predict functions with loader * Update error message about ragged tensors for clarity * Add explanation about why the input types are restricted * Rename dataset to dataloader in model_test Co-authored-by: rnyak <ronayak@hotmail.com>

github-actions · 2022-12-13T16:18:44Z

Documentation preview

https://nvidia-merlin.github.io/models/review/pr-909

review-notebook-app · 2022-12-13T17:27:51Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

sararb added 2 commits December 1, 2022 10:58

add the inference fix to ReplaceMaskedEmbeddings

ef64992

first solution for inference support

6a5e582

sararb added the bug Something isn't working label Dec 1, 2022

sararb added this to the Merlin 22.12 milestone Dec 1, 2022

sararb self-assigned this Dec 1, 2022

sararb commented Dec 1, 2022

View reviewed changes

merlin/models/tf/transforms/sequence.py Show resolved Hide resolved

sararb commented Dec 1, 2022

View reviewed changes

sararb changed the title ~~Tf/fix inference~~ Fix the inference of transformer-based models trained with masked language modeling Dec 1, 2022

gabrielspmoreira reviewed Dec 1, 2022

View reviewed changes

updates based on PR comments

a135fe2

sararb requested a review from gabrielspmoreira December 2, 2022 00:47

gabrielspmoreira reviewed Dec 12, 2022

View reviewed changes

sararb and others added 4 commits December 13, 2022 11:14

Apply suggestions from code review

2cd8fd2

Co-authored-by: Gabriel Moreira <gmoreira@nvidia.com>

Restore documentation build (#916)

d2286f6

- Change Python 3.9.7 to 3.8. - Update the versions of the GH actions. - Update pre-commit config file to get flake8 from GitHub instead of GitLab.

add assertion check to TransformerInferenceHiddenState

2983433

Merge branch 'main' into tf/fix-inference

8607a9b

sararb mentioned this pull request Dec 13, 2022

Support masking for causal language modeling #918

Closed

gabrielspmoreira approved these changes Dec 13, 2022

View reviewed changes

sararb merged commit df84c81 into main Dec 13, 2022

rnyak deleted the tf/fix-inference branch February 8, 2023 20:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix the inference of transformer-based models trained with masked language modeling #909

Fix the inference of transformer-based models trained with masked language modeling #909

sararb commented Dec 1, 2022 •

edited

Loading

sararb Dec 1, 2022

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

gabrielspmoreira Dec 1, 2022

sararb Dec 1, 2022 •

edited

Loading

nvidia-merlin-bot commented Dec 2, 2022

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

gabrielspmoreira left a comment

gabrielspmoreira Dec 12, 2022

github-actions bot commented Dec 13, 2022

review-notebook-app bot commented Dec 13, 2022

Fix the inference of transformer-based models trained with masked language modeling #909

Fix the inference of transformer-based models trained with masked language modeling #909

Conversation

sararb commented Dec 1, 2022 • edited Loading

Goals ⚽

Implementation Details 🚧

Testing Details 🔍

sararb Dec 1, 2022

Choose a reason for hiding this comment

nvidia-merlin-bot commented Dec 1, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

jperez999 commented Dec 1, 2022

gabrielspmoreira Dec 1, 2022

Choose a reason for hiding this comment

sararb Dec 1, 2022 • edited Loading

Choose a reason for hiding this comment

nvidia-merlin-bot commented Dec 2, 2022

---------- coverage: platform linux, python 3.8.10-final-0 ----------- Name Stmts Miss Cover

gabrielspmoreira left a comment

Choose a reason for hiding this comment

I undestand this PR is focused in Masked Language Modeling. After reviewing, I am wondering if for Causal Language Modeling we shouldn't return during inference the last non-padded prediction per sequence. The thing is that we don't use _keras_mask for causal language modeling. Thoughts?

gabrielspmoreira Dec 12, 2022

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2022

Documentation preview

review-notebook-app bot commented Dec 13, 2022

sararb commented Dec 1, 2022 •

edited

Loading

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

sararb Dec 1, 2022 •

edited

Loading

---------- coverage: platform linux, python 3.8.10-final-0 -----------
Name Stmts Miss Cover

I undestand this PR is focused in Masked Language Modeling. After reviewing, I am wondering if for Causal Language Modeling we shouldn't return during inference the last non-padded prediction per sequence. The thing is that we don't use `_keras_mask` for causal language modeling.
Thoughts?