Skip to content

DMWM Packaging with PyPi

Jump to bottom
Erik Gough edited this page Mar 28, 2022 · 19 revisions

This document outlines the transition of DMWM software to the Python Package Index, https://pypi.org/

Account Information

Developers should create their own personal PyPI accounts for maintaining DMWM software packages in PyPI. DMWM uses PyPI collaborator roles to make all projects owned by the user cms-oc-dmwm, with maintainer access going to individual developers. More info about roles can be found here: https://pypi.org/help/#collaborator-roles

DMWM PyPi project owner account: cms-oc-dmwm

Projects: https://pypi.org/user/cms-oc-dmwm/

Procedure for obtaining the Maintainer role for a DMWM PyPI project

This should be used when a developer wants access to manage DMWM PyPI projects.

  • Email cms-oc-dmwm-pypi E-Group and request to be added as a maintainer of a project
  • Once the role is configured, you will receive a notification email about the new role.

Procedure for transitioning ownership of an existing PyPI project to the cms-oc-dmwm account

This should be used if a developer has created a new PyPI project and wants to combine it with other DMWM PyPI projects managed by the cms-oc-dmwm account.

  • Initial upload of packages to PyPI should be performed by the developer under their account
  • Once you have uploaded, give the cms-oc-dmwm account the "Owner" role under the "Collaboration" settings
  • The cms-oc-dmwm-pypi E-Group members will be notified of the change in role via email
  • We will change your role to "Maintainer", allowing you to release new versions, but not delete the project, etc.

General Information

General docs for packaging projects: https://packaging.python.org/tutorials/packaging-projects/

Test packages location: https://test.pypi.org/user/cms-oc-dmwm/

Production packages location: https://www.pypi.org/user/cms-oc-dmwm/

Packaging and uploading WMCore/WMAgent to PyPI

This was all documented using lxplus. Instructions might vary depending on your OS and python version.

Create a virtualenv for building packages. Need setuptools and twine.

$ virtualenv pypibuild
$ source pypibuild/bin/activate
(pypibuild) $ pip install twine

Make a ~/.pypirc file with required contents for uploading to PyPI.

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: <your username>

Clone the WMCore repo and fetch the set of tags

$ git clone https://github.com/dmwm/WMCore.git
$ cd WMCore
$ git fetch && git fetch --tags

Checkout what tag you want

$ git checkout <tag>

Run the PyPI build script to build and upload the version you checked out. Enter the cms-oc-dmwm PyPI user password when prompted.

$ sh tools/build_pypi_packages.sh <wmagent|wmcore|all>

(pypibuild) [username@lxplus789 WMCore]$ sh tools/build_pypi_packages.sh wmcore
Building wmcore package
/afs/cern.ch/user/u/username/pypibuild/lib/python2.7/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '1.3.3.pre2' to '1.3.3rc2'
  normalized_version,
running sdist
<snip>
Writing wmagent-1.3.3rc2/setup.cfg
Creating tar archive
removing 'wmagent-1.3.3rc2' (and everything under it)
Enter your password: 
Uploading distributions to https://upload.pypi.org/legacy/
Uploading wmagent-1.3.3rc2.tar.gz
View at:
https://pypi.org/project/wmagent/1.3.3rc1/

Check the cms-oc-dmwm PyPI page to find the new uploaded version: https://pypi.org/project/wmcore/#history

Install WMCore via PyPI

[username@lxplus789 WMCore]$ virtualenv wmcore-pypi
New python executable in /afs/cern.ch/user/u/username/WMCore/wmcore-pypi/bin/python
Please make sure you remove any previous custom paths from your /afs/cern.ch/user/u/username/.pydistutils.cfg file.
Installing setuptools, pip, wheel...done.

[username@lxplus789 WMCore]$ source wmcore-pypi/bin/activate
(wmcore-pypi) [username@lxplus789 WMCore]$ pip install wmcore
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting wmcore
  Downloading wmcore-1.3.5.tar.gz (2.2 MB)
<snip>
Successfully built wmcore
Installing collected packages: wmcore
Successfully installed wmcore-1.3.5

If you want a specific version:

pip install wmcore==1.3.6.pre3

Dependencies (non-python) needed for WMAgent

This is the list of dependencies that are required before one can install wmagent via pip. Python dependencies are automatically pulled when the wmagent package is installed.

  • MariaDB
  • CouchDB
  • Oracle Client
  • Condor
  • libcurl-devel
  • jemalloc
  • libuuid
  • yui (might not be necessary)
  • pystack (might not be necessary)
  • gcc

If any of these packages are installed in a non-standard location, you will need to set some environment variables as well.

  • Add mariadb bin and gcc bin directory to PATH
  • Add gcc lib directory to LD_LIBRARY_PATH
  • Set ORACLE_HOME to Oracle client location

Download and install WMAgent

Create a wmagent virtualenv

$ virtualenv wmagent
$ source wmagent/bin/activate
(wmagent) $ pip install wmagent

Packaging DBS Applications

Instructions on how to build and upload DBS Client and Pycurl Client to PyPi.

One time setup for users wanting to upload new releases to PyPi

Create a .pypirc file in your home directory with the following contents:

[distutils]
index-servers =
  pypi

[pypi]
repository: https://upload.pypi.org/legacy/
username: <your account>

Changes to be made at release time

The following changes should be made when a new version of DBS is released. Some version numbers will have to be updated manually for now. This will (hopefully) be automated in the future.

  1. dbs-client version in setup_dbs_client.py, L10
  2. pycurl-client version in setup_pycurl_client.py, L10
  3. pycurl-client version in requirements.dbs_client.py, L1

Build and upload to PyPi

Clone the DBS repo and checkout the latest tag:

$ git clone https://github.com/dmwm/DBS.git
$ git fetch && git fetch --tags
$ git checkout <tag>

Run the build_pypi_packages.sh script build and upload the packages. The script takes the following options, but you will usually want to build "all".

  all              Build all DBS packages
  dbs-client       Build dbs-client package
  pycurl-client    Build pycurl-client package

$ sh Tools/build_pypi_packages.sh all

Enter the cms-oc-dmwm account password when prompted. This can also be set in the .pypirc file if you don't mind storing passwords in clear text.

Check the details of the new build here: https://pypi.org/user/cms-oc-dmwm/

Installing DBS client and Pycurl client via PyPI on lxplus7

Source a CMSSW environment.

[user@lxplusXXX ~]$ export SCRAM_ARCH=<arch>
[user@lxplusXXX ~]$ cmsrel CMSSW_X_Y_Z
[user@lxplusXXX ~]$ cd CMSSW_X_Y_Z/src
[user@lxplusXXX src]$ cmsenv

To use DBS client, one needs to install it via pip and then activate a CMSSW envrionment. Using DBS client with user certificates and proxies requires a curl shared library built with special configuration options. This curl library is pulled in when setting the CMSSW environment. Using virtual python environments (virtualenv) in conjunction with the CMSSW environment gets messy real quick, so it is best to just install DBS client into your home directory with the --user option.

[user@lxplusXXX ~]$ pip install --user dbs-client
Collecting dbs-client
<snip>
Successfully installed dbs-client-3.7.8 pycurl-7.19.0.3 pycurl-client-3.7.8 python-cjson-1.2.1

Create a proxy

[user@lxplusXXX ~]$ voms-proxy-init -rfc -voms cms

Test access with a simple python script

[user@lxplusXXX ~]$ cat dbs-test.py
from dbs.apis.dbsClient import DbsApi
dbs = DbsApi('https://cmsweb.cern.ch/dbs/prod/global/DBSReader')
ds = dbs.listDatasets(dataset = '/MinBias/Summer*/*')
print(ds)

[user@lxplusXXX ~]$ python dbs-test.py 
[{'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa1E31-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3-v1_HLT_AlCa8E29-v1/RAW'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer09-MC_31X_V3_7TeV_CS_DiJetAve15U-v1/USER'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM'}, {'dataset': '/MinBias/Summer11-DESIGN42_V11_428_SLHChcal-v1/GEN-SIM'}]
Clone this wiki locally