RWKV4neo #20737

ArEnSc · 2022-12-12T17:07:58Z

Model description

RWKV - Receptance Weighted Key Value

RWKV is a Sequence to Sequence Model that takes the best features of Generative PreTraining (GPT) and Recurrent Nueral Networks (RNN) that performs Language Modelling (LM). This is used to generate text Auto Regressive manner (AR).

This is a hybrid model.

It has Transformer Level Performance without the quadratic attention mechanism. It borrows ideas from Attention Free Transformers, meaning the attention is a linear in complexity. Allowing for infinite context through the hidden state in RWKV_RNN.

There are two models for RWKV, they are refered to as modes.

RWKV_RNN: This mode is designed for running inference quickly.
RWKV_GPT: This mode is for training or fine tuning your model quickly.

In the first pass we will be implementing RWKV_RNN Although we can weight share to have RWKV_GPT generate the inital context for RWKV_RNN.

Open source status

The model implementation is available
The model weights are available
Scaffolding
API Discussion

Provide useful links for the implementation

More from the Research and Development Repository: https://github.com/BlinkDL/RWKV-LM

ArEnSc · 2022-12-12T17:10:15Z

@ArthurZucker
@younesbelkada
#17230 (comment)

younesbelkada · 2022-12-12T17:14:36Z

This is super cool !! 🔥
Really looking forward to it
Also as discussed with @leondz , we'll probably also include a blogpost explaining this new architecture!
Happy to help for the implementation and the blogpost!

ArEnSc · 2022-12-12T23:20:51Z

Ok I am gonna just set this up on my linux machine m1 setup isn't ready I spent 2 hours on this gonna try again tomorrow sorry D:

ArEnSc · 2022-12-14T02:12:19Z

Ok I am gonna just set this up on my linux machine m1 setup isn't ready I spent 2 hours on this gonna try again tomorrow sorry D:

scratch that, it also didn't work, mainly because of my limited harddisk space. Gonna retry mac...

ArEnSc · 2022-12-14T06:52:14Z

Building wheels for collected packages: transformers, onnx
  Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.26.0.dev0-0.editable-py3-none-any.whl size=31899 sha256=d4b123bfb17b8f11ab811da95d253a066a98638c492d76f4d00afa509dfc6d71
  Stored in directory: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-ephem-wheel-cache-w5ucgfda/wheels/52/2c/02/9b0e2ee52910e61c69011870086c52ab4eaaa554c34005f48f
  Building wheel for onnx (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [76 lines of output]
      
      You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      running bdist_wheel
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']
      -- The C compiler identification is unknown
      -- The CXX compiler identification is unknown
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - failed
      -- Check for working C compiler: /usr/bin/cc
      -- Check for working C compiler: /usr/bin/cc - broken
      CMake Error at /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/cmake/data/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
        The C compiler
      
          "/usr/bin/cc"
      
        is not able to compile a simple test program.
      
        It fails with the following output:
      
          Change Dir: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeScratch/TryCompile-cd2Kpc
      
          Run Build Command(s):/usr/bin/make -f Makefile cmTC_a6349/fast &&
          You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      
      
      
      
      
        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:17 (project)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for onnx
  Running setup.py clean for onnx
Successfully built transformers
Failed to build onnx
Installing collected packages: onnx, tf2onnx, nbformat, matplotlib, markdown, jaxlib, jax, huggingface-hub, gql, google-auth, GitPython, Flask, docker, cryptography, cliff, botocore, arrow, alembic, aiohttp, accelerate, transformers, timm, s3transfer, pyOpenSSL, optuna, librosa, kubernetes, jinja2-time, ipython, google-auth-oauthlib, dash, csvw, chex, APScheduler, tensorboard, optax, hf-doc-builder, datasets, dash-bootstrap-components, cookiecutter, clldutils, boto3, tensorflow-macos, sigopt, segments, flax, evaluate, codecarbon, phonemizer
  Attempting uninstall: onnx
    Found existing installation: onnx 1.13.0
    Uninstalling onnx-1.13.0:
      Successfully uninstalled onnx-1.13.0
  Running setup.py install for onnx ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for onnx did not run successfully.
  │ exit code: 1
  ╰─> [78 lines of output]
      
      You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      running install
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']
      -- The C compiler identification is unknown
      -- The CXX compiler identification is unknown
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - failed
      -- Check for working C compiler: /usr/bin/cc
      -- Check for working C compiler: /usr/bin/cc - broken
      CMake Error at /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/cmake/data/share/cmake-3.25/Modules/CMakeTestCCompiler.cmake:70 (message):
        The C compiler
      
          "/usr/bin/cc"
      
        is not able to compile a simple test program.
      
        It fails with the following output:
      
          Change Dir: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeScratch/TryCompile-WGiugk
      
          Run Build Command(s):/usr/bin/make -f Makefile cmTC_3370b/fast &&
          You have not agreed to the Xcode license agreements, please run 'sudo xcodebuild -license' from within a Terminal window to review and agree to the Xcode license agreements.
      
      
      
      
      
        CMake will not be able to correctly generate this project.
      Call Stack (most recent call first):
        CMakeLists.txt:17 (project)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
          return orig.install.run(self)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/install.py", line 546, in run
          self.run_command('build')
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-ppsum_cw/onnx_d98e5ffd9f474a8b9f3cf5ca58551b5f']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of onnx
  Moving to /Users/michaelchung/Code/transformers/.env/bin/backend-test-tools
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/backend-test-tools
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-model
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/check-model
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-node
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-rs3gpves/check-node
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx-1.13.0.dist-info/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx-1.13.0.dist-info
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> onnx

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Fails at building wheels for transformers and onnx
I installed onnx manually using pip3

absl-py==1.3.0
aiosignal==1.3.1
appdirs==1.4.4
appnope==0.1.3
asttokens==2.2.1
astunparse==1.6.3
async-timeout==4.0.2
attrs==22.1.0
audioread==3.0.0
autopage==0.5.1
Babel==2.11.0
backcall==0.2.0
backoff==1.11.1
beautifulsoup4==4.11.1
binaryornot==0.4.4
black==22.3.0
cachetools==5.2.0
certifi==2022.12.7
cffi==1.15.1
chardet==5.1.0
charset-normalizer==2.1.1
click==8.1.3
cmaes==0.9.0
cmake==3.25.0
cmd2==2.4.2
colorama==0.4.6
coloredlogs==15.0.1
colorlog==6.7.0
commonmark==0.9.1
contourpy==1.0.6
cycler==0.11.0
dash-core-components==2.0.0
dash-html-components==2.0.0
dash-table==5.0.0
decorator==5.1.1
dill==0.3.4
distlib==0.3.6
dlinfo==1.2.1
dm-tree==0.1.7
exceptiongroup==1.0.4
execnet==1.9.0
executing==1.2.0
faiss-cpu==1.7.3
fastjsonschema==2.16.2
filelock==3.8.2
fire==0.5.0
flake8==6.0.0
flatbuffers==2.0.7
fonttools==4.38.0
frozenlist==1.3.3
fsspec==2022.11.0
fugashi==1.1.2a6
gast==0.4.0
gitdb==4.0.10
google-pasta==0.2.0
graphql-core==3.2.3
grpcio==1.51.1
h5py==3.7.0
humanfriendly==10.0
hypothesis==6.61.0
idna==3.4
importlib-metadata==4.13.0
iniconfig==1.1.1
ipadic==1.0.0
isodate==0.6.1
isort==5.11.2
itsdangerous==2.1.2
jedi==0.18.2
Jinja2==3.1.2
jmespath==1.0.1
joblib==1.2.0
jsonschema==4.17.3
jupyter_core==5.1.0
kenlm==0.1
keras==2.11.0
keras-nlp==0.3.1
kiwisolver==1.4.4
language-tags==1.1.0
libclang==14.0.6
llvmlite==0.39.1
lxml==4.9.2
Mako==1.2.4
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
mccabe==0.7.0
mpmath==1.2.1
msgpack==1.0.4
multidict==6.0.3
multiprocess==0.70.12.2
mypy-extensions==0.4.3
nltk==3.8
numba==0.56.4
numpy==1.23.5
oauthlib==3.2.2
onnx==1.13.0
onnxconverter-common==1.13.0
onnxruntime==1.13.1
onnxruntime-tools==1.7.0
opt-einsum==3.3.0
packaging==22.0
pandas==1.5.2
parameterized==0.8.1
parso==0.8.3
pathspec==0.10.3
pbr==5.11.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.3.0
Pint==0.16.1
plac==1.3.5
platformdirs==2.6.0
plotly==5.11.0
pluggy==1.0.0
pooch==1.6.0
portalocker==2.0.0
poyo==0.5.0
prettytable==3.5.0
prompt-toolkit==3.0.36
protobuf==3.19.6
psutil==5.9.4
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
py3nvml==0.2.7
pyarrow==10.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.10.0
pycparser==2.21
pyctcdecode==0.4.0
pyflakes==3.0.1
Pygments==2.13.0
pygtrie==2.5.0
pyknp==0.6.1
pylatexenc==2.10
pynvml==11.4.1
pyparsing==3.0.9
pyperclip==1.8.2
pypng==0.20220715.0
pyrsistent==0.19.2
pytest==7.2.0
pytest-timeout==2.1.0
pytest-xdist==3.1.0
python-dateutil==2.8.2
python-slugify==7.0.0
pytz==2022.6
pytz-deprecation-shim==0.1.0.post0
PyYAML==5.4.1
ray==2.2.0
rdflib==6.2.0
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
requests-toolbelt==0.10.1
resampy==0.4.2
responses==0.18.0
rfc3986==1.5.0
rich==11.2.0
rjieba==0.1.11
rouge-score==0.1.2
rsa==4.9
sacrebleu==1.5.1
sacremoses==0.0.53
safetensors==0.2.6
scikit-learn==1.2.0
scipy==1.8.1
sentencepiece==0.1.97
six==1.16.0
smmap==5.0.0
sortedcontainers==2.4.0
soundfile==0.11.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.45
stack-data==0.6.2
stevedore==4.1.1
SudachiDict-core==20221021
SudachiPy==0.6.6
sympy==1.11.1
tabulate==0.9.0
tenacity==8.1.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorboardX==2.5.1
tensorflow-estimator==2.11.0
tensorflow-metal==0.7.0
tensorstore==0.1.28
termcolor==2.1.1
text-unidecode==1.3
threadpoolctl==3.1.0
timeout-decorator==0.5.0
tokenizers==0.13.2
tomli==2.0.1
toolz==0.12.0
torch==1.13.0
torchaudio==0.13.0
torchvision==0.14.0
tqdm==4.64.1
traitlets==5.7.1
typing_extensions==4.4.0
tzdata==2022.7
tzlocal==4.2
unidic==1.1.0
unidic-lite==1.0.8
uritemplate==4.1.1
urllib3==1.26.13
virtualenv==20.17.1
wasabi==0.10.1
wcwidth==0.2.5
websocket-client==1.4.2
Werkzeug==2.2.2
wrapt==1.14.1
xmltodict==0.13.0
xxhash==3.1.0
yarl==1.8.2
zipp==3.11.0

I am on 3.9.11

when running
conda install -c conda-forge onnxruntime
it says all the components were installed

ArEnSc · 2022-12-14T06:52:33Z

hmmm I accepted the xcode license agreement.
It's built transformers I think it fails to build onnx, but whats odd is it uninstalls onnx. Maybe I can do with out it and remove it out of setup.py?

Building wheels for collected packages: transformers, onnx
  Building editable for transformers (pyproject.toml) ... done
  Created wheel for transformers: filename=transformers-4.26.0.dev0-0.editable-py3-none-any.whl size=31899 sha256=e63397c8685e05971af9d4767e872200eb8393ed081e37582f8d24760daaca08
  Stored in directory: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-ephem-wheel-cache-w75hif9g/wheels/52/2c/02/9b0e2ee52910e61c69011870086c52ab4eaaa554c34005f48f
  Building wheel for onnx (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [67 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      running bdist_wheel
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']
      -- The C compiler identification is AppleClang 14.0.0.14000029
      -- The CXX compiler identification is AppleClang 14.0.0.14000029
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found PythonInterp: /Users/michaelchung/Code/transformers/.env/bin/python3 (found version "3.9.11")
      -- Found PythonLibs: /opt/homebrew/Frameworks/Python.framework/Versions/3.11/lib/libpython3.11.dylib (found version "3.9.11")
      Generated: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/onnx/onnx-ml.proto
      CMake Error at CMakeLists.txt:299 (message):
        Protobuf compiler not found
      Call Stack (most recent call first):
        CMakeLists.txt:330 (relative_protobuf_generate_cpp)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for onnx
  Running setup.py clean for onnx
Successfully built transformers
Failed to build onnx
Installing collected packages: onnx, tf2onnx, nbformat, matplotlib, markdown, jaxlib, jax, huggingface-hub, gql, google-auth, GitPython, Flask, docker, cryptography, cliff, botocore, arrow, alembic, aiohttp, accelerate, transformers, timm, s3transfer, pyOpenSSL, optuna, librosa, kubernetes, jinja2-time, ipython, google-auth-oauthlib, dash, csvw, chex, APScheduler, tensorboard, optax, hf-doc-builder, datasets, dash-bootstrap-components, cookiecutter, clldutils, boto3, tensorflow-macos, sigopt, segments, flax, evaluate, codecarbon, phonemizer
  Attempting uninstall: onnx
    Found existing installation: onnx 1.13.0
    Uninstalling onnx-1.13.0:
      Successfully uninstalled onnx-1.13.0
  Running setup.py install for onnx ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for onnx did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      fatal: not a git repository (or any of the parent directories): .git
      running install
      running build
      running build_py
      running create_version
      running cmake_build
      Using cmake args: ['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']
      Generated: /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/onnx/onnx-ml.proto
      CMake Error at CMakeLists.txt:299 (message):
        Protobuf compiler not found
      Call Stack (most recent call first):
        CMakeLists.txt:330 (relative_protobuf_generate_cpp)
      
      
      -- Configuring incomplete, errors occurred!
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeOutput.log".
      See also "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/.setuptools-cmake-build/CMakeFiles/CMakeError.log".
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 332, in <module>
          setuptools.setup(
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/__init__.py", line 153, in setup
          return distutils.core.setup(**attrs)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/core.py", line 148, in setup
          dist.run_commands()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 966, in run_commands
          self.run_command(cmd)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/setuptools/command/install.py", line 61, in run
          return orig.install.run(self)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/install.py", line 546, in run
          self.run_command('build')
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 223, in run
          self.run_command("cmake_build")
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/cmd.py", line 313, in run_command
          self.distribution.run_command(command)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/distutils/dist.py", line 985, in run_command
          cmd_obj.run()
        File "/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928/setup.py", line 209, in run
          subprocess.check_call(cmake_args)
        File "/Users/michaelchung/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 373, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/Users/michaelchung/Code/transformers/.env/bin/cmake', '-DPYTHON_INCLUDE_DIR=/Users/michaelchung/.pyenv/versions/3.9.11/include/python3.9', '-DPYTHON_EXECUTABLE=/Users/michaelchung/Code/transformers/.env/bin/python3', '-DBUILD_ONNX_PYTHON=ON', '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON', '-DONNX_NAMESPACE=onnx', '-DPY_EXT_SUFFIX=.cpython-39-darwin.so', '-DCMAKE_BUILD_TYPE=Release', '-DONNX_ML=1', '/private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-install-9aulk13y/onnx_331e37bbfc1440c0bc09391c545d3928']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of onnx
  Moving to /Users/michaelchung/Code/transformers/.env/bin/backend-test-tools
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/backend-test-tools
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-model
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/check-model
  Moving to /Users/michaelchung/Code/transformers/.env/bin/check-node
   from /private/var/folders/jn/8d33s3c55jv5pctdc6wdnm2h0000gn/T/pip-uninstall-1ylzipwt/check-node
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx-1.13.0.dist-info/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx-1.13.0.dist-info
  Moving to /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/onnx/
   from /Users/michaelchung/Code/transformers/.env/lib/python3.9/site-packages/~nnx
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> onnx

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

ArEnSc · 2022-12-17T20:18:55Z

Alright dev environment installed, the key here was not to use conda but miniforge

ArEnSc · 2022-12-17T20:51:20Z

Ok draft created

vonjackustc · 2023-03-29T05:26:49Z

Do you know how to convert .pth model to config.json/pytorch_model.bin for RWKV4neo?

tensorpro · 2023-04-21T03:09:26Z

I have a conversion script + draft that has consistent logit ordering with the official implementation here:

conversion script: https://github.com/tensorpro/transformers/blob/rwkv_draft/src/transformers/models/rwkv4_neo/convert_rwkv_original_pytorch_checkpoint_to_pytorch.py
model in torch: https://github.com/tensorpro/transformers/blob/rwkv_draft/src/transformers/models/rwkv4_neo/modeling_rwkv4_neo.py

I can clean it up and turn it into a PR if that would help?

ArthurZucker · 2023-04-21T09:29:27Z

Sure, there is also #22797 that should be in a pretty good state! I'm about to review it but feel free to add your touch to it if you feel like it!

tensorpro · 2023-04-21T18:06:10Z

Oh cool, that one looks awesome!

YovaKem · 2023-04-25T12:12:29Z

I have a conversion script + draft that has consistent logit ordering with the official implementation here:

conversion script: https://github.com/tensorpro/transformers/blob/main/src/transformers/models/rwkv4_neo/convert_rwkv_original_pytorch_checkpoint_to_pytorch.py model in torch: https://github.com/tensorpro/transformers/blob/main/src/transformers/models/rwkv4_neo/modeling_rwkv4_neo.py

I can clean it up and turn it into a PR if that would help?

@tensorpro I could really use your scripts but I get 404 when I try to access those links. :/

tensorpro · 2023-04-25T17:23:20Z

Ah sorry, the links broke when I changed the branch I was working in. I edited the comment to point to the right branch

That said, you may want to use the code in #22797 since it will be closer to the official HF version and already supports CUDA accelerated WKV.

ArEnSc added the New model label Dec 12, 2022

ArEnSc mentioned this issue Dec 12, 2022

Add RWKV2 (fast) #17230

Closed

2 tasks

ArEnSc mentioned this issue Dec 17, 2022

[WIP] RWKV4Neo the RNN and GPT Hybrid Model #20809

Closed

5 tasks

KerfuffleV2 mentioned this issue Mar 26, 2023

Support for RWKV rustformers/llm#75

Open

younesbelkada mentioned this issue May 3, 2023

Add RWKV-4 #22797

Merged

7 tasks

sgugger closed this as completed in #22797 May 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RWKV4neo #20737

RWKV4neo #20737

ArEnSc commented Dec 12, 2022 •

edited

Loading

ArEnSc commented Dec 12, 2022

younesbelkada commented Dec 12, 2022

ArEnSc commented Dec 12, 2022

ArEnSc commented Dec 14, 2022

ArEnSc commented Dec 14, 2022

ArEnSc commented Dec 14, 2022 •

edited

Loading

ArEnSc commented Dec 17, 2022

ArEnSc commented Dec 17, 2022

vonjackustc commented Mar 29, 2023

tensorpro commented Apr 21, 2023 •

edited

Loading

ArthurZucker commented Apr 21, 2023

tensorpro commented Apr 21, 2023

YovaKem commented Apr 25, 2023

tensorpro commented Apr 25, 2023

RWKV4neo #20737

RWKV4neo #20737

Comments

ArEnSc commented Dec 12, 2022 • edited Loading

Model description

Open source status

Provide useful links for the implementation

ArEnSc commented Dec 12, 2022

younesbelkada commented Dec 12, 2022

ArEnSc commented Dec 12, 2022

ArEnSc commented Dec 14, 2022

ArEnSc commented Dec 14, 2022

ArEnSc commented Dec 14, 2022 • edited Loading

ArEnSc commented Dec 17, 2022

ArEnSc commented Dec 17, 2022

vonjackustc commented Mar 29, 2023

tensorpro commented Apr 21, 2023 • edited Loading

ArthurZucker commented Apr 21, 2023

tensorpro commented Apr 21, 2023

YovaKem commented Apr 25, 2023

tensorpro commented Apr 25, 2023

ArEnSc commented Dec 12, 2022 •

edited

Loading

ArEnSc commented Dec 14, 2022 •

edited

Loading

tensorpro commented Apr 21, 2023 •

edited

Loading