Skip to content

All in one test

Jump to bottom
ticoann edited this page May 19, 2016 · 51 revisions

Brief Summary (for Linux) : this set up is not complete so submitting request might not work. Need to improve

  • This is the set up for quick integration test. This set up is NOT providing the same environment as real system.
  • It doesn't contain apatch frontend, no authentication.
  • reqmgr and workqueue is set as component rather than the cherrypy daemon.
  • To make wmstats working there will be manual changes required (not added in this tutorial)
  • don't use this set up if you want to test cmsweb application (reqmgr, workqueue, reqmon), use this only for agent side integration test.
  • This tutorial assumes condor is installed in the testing machine and proper cert is available to submit the jobs

Setting up all in one agent (wmagent, reqmgr, reqmon, workqueue)

1. setting up the directory structure.

(follows wmagent deployment directory structure - change to your convenient location )

$ export ADMIN_DIR=/data/admin/wmagent

$ export DEPLOY_DIR=/data/srv/wmagent

$ mkdir -p $ADMIN_DIR # for env script.

2. get deployment script

$ cd /data

$ git clone https://github.com/dmwm/deployment.git

$ git checkout branch/tag (optional if need specific tag or branch)

3. Deploy wmagent rpm (example shows 1.0.0.patch4 version)

$ cd /data/deployment

$ ./Deploy -r comp=comp.pre -s prep -A slc6_amd64_gcc481 -t v1.0.0.patch4 $DEPLOY_DIR wmagent@1.0.0.patch4

$ ./Deploy -r comp=comp.pre -s sw -A slc6_amd64_gcc481 -t v1.0.0.patch4 $DEPLOY_DIR wmagent@1.0.0.patch4

$ ./Deploy -r comp=comp.pre -s post -A slc6_amd64_gcc481 -t v1.0.0.patch4 $DEPLOY_DIR wmagent@1.0.0.patch4

Setting up environment

1. create secrete file (set db password and port)

$ vi $ADMIN_DIR/WMAgent.secrets

   type following information (change user/password and port - don't use port 9999,
   it is used by WMBSService component - need to be changed to getting in from secrete file)

   MYSQL_USER=devrecoagent
   MYSQL_PASS=passwd
   COUCH_USER=devrecoagent
   COUCH_PASS=passwd
   COUCH_PORT=5984
   COUCH_HOST=0.0.0.0 (WARNING: this has to be ip address not hostname)
   GLOBAL_WORKQUEUE_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/globalwq
   GLOBAL_WORKQUEUE_DBNAME=globalwq
   LOCAL_WORKQUEUE_DBNAME=localwq
   WORKLOAD_SUMMARY_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/workloadsummary
   REQMGR_HOSTNAME=131.225.206.104
   REQMGR_PORT=8687
   WMSTATS_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/wmstats
   COUCH_CERT_FILE=/data/certs/servicecert.pem
   COUCH_KEY_FILE=/data/certs/servicekey.pem
   REQMGR_URL=http://cmssrv95.fnal.gov:8687/reqmgr/rest
   REQUESTCOUCH_URL=http://cmssrv95.fnal.gov:5984/reqmgr_workload_cache
   ACDC_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/acdcserver
   DBS3_URL=https://cmsweb-testbed.cern.ch/dbs/int/global/DBSWriter
   PHEDEX_URL=https://cmsweb-testbed.cern.ch/phedex/datasvc/json/prod/
   DQM_URL=https://cmsweb-testbed.cern.ch/dqm/offline/
   DASHBOARD_URL==http://dashb-ssb.cern.ch/dashboard
   REQMGR2_URL=http://cmssrv95.fnal.gov:8688/reqmgr2
   CENTRAL_LOGDB_URL=https://cmsweb-testbed.cern.ch/couchdb/wmstats_logdb

   for Oracle version of unittest, env.sh file need to contain oracle variables instead of mysql 

   ORACLE_USER=(oracle user name)
   ORACLE_PASS=(oracle password)
   ORACLE_TNS=(tns name)
   COUCH_USER=devrecoagent
   COUCH_PASS=passwd
   COUCH_PORT=5984
   COUCH_HOST=0.0.0.0 (WARNING: this has to be ip address not hostname)
   GLOBAL_WORKQUEUE_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/globalwq
   GLOBAL_WORKQUEUE_DBNAME=globalwq
   LOCAL_WORKQUEUE_DBNAME=localwq
   WORKLOAD_SUMMARY_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/workloadsummary
   REQMGR_HOSTNAME=131.225.206.104
   REQMGR_PORT=8687
   WMSTATS_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/wmstats
   COUCH_CERT_FILE=/data/certs/servicecert.pem
   COUCH_KEY_FILE=/data/certs/servicekey.pem
   REQMGR_URL=http://cmssrv95.fnal.gov:8687/reqmgr/rest
   REQUESTCOUCH_URL=http://cmssrv95.fnal.gov:5984/reqmgr_workload_cache
   ACDC_URL=http://devrecoagent:passwd@cmssrv95.fnal.gov:5984/acdcserver
   DBS3_URL=https://cmsweb-testbed.cern.ch/dbs/int/global/DBSWriter
   PHEDEX_URL=https://cmsweb-testbed.cern.ch/phedex/datasvc/json/prod/
   DQM_URL=https://cmsweb-testbed.cern.ch/dqm/offline/
   DASHBOARD_URL==http://dashb-ssb.cern.ch/dashboard
   REQMGR2_URL=http://cmssrv95.fnal.gov:8688/reqmgr2
   CENTRAL_LOGDB_URL=https://cmsweb-testbed.cern.ch/couchdb/wmstats_logdb

2. set the environment script

servicecert and servicekey can be your own cert. but make sure key is not encrypted.

$ vi $ADMIN_DIR/env.sh

    #change this block according to your directory setting
    export DEPLOY_DIR=/data/srv/wmagent

    # Necessary variables
    export WMAGENT_SECRETS_LOCATION=/data/admin/wmagent/WMAgent.secrets
    export X509_HOST_CERT=/data/certs/servicecert.pem
    export X509_HOST_KEY=/data/certs/servicekey.pem
    export X509_USER_CERT=/data/certs/servicecert.pem 
    export X509_USER_KEY=/data/certs/servicekey.pem
    export X509_USER_PROXY=/data/certs/myproxy.pem #(need to set proxy for condor)

    #Very convenient variables
    export install=$DEPLOY_DIR/current/install/wmagent
    export config=$DEPLOY_DIR/current/config/wmagent
    export manage=$config/manage

Start components and setting up other thing for request submission.

1. start server and create the db

  $ source $ADMIN_DIR/env.sh

  $ $manage activate-agent

  $ $manage activate-reqmgr

  $ $manage activate-workqueue

  $ $manage start-services
  
  $ $manage init-agent

  if you change config file manually need to remove init files and start agent again
  i.e rm $install/.init; rm $install/../workqueue/.init; rm $install/../reqmgr/.init

  Setting up threshold so you can submit the job
  $ $manage execute-agent wmagent-resource-control --add-all-sites --plugin=PyCondorPlugin --pending-slots=10 --running-slots=10
  
  Or set up one T1 or T2s
  $ $manage execute-agent wmagent-resource-control --add-T1s --plugin=PyCondorPlugin --pending-slots=50 --running-slots=50
  
  $ $manage execute-agent wmagent-resource-control --add-T2s --plugin=PyCondorPlugin --pending-slots=50 --running-slots=50

1.1 step to modify agent

 To be able to abort workflow
 in reqmgr db reqmgr_assigned_prodmgr table workqueue url should include the passwd.
 update reqmgr_assigned_prodmgr set prodmgr_id='http://devrecoagent:passwd@cmssrv95.fnal.gov:5984';
 TODO: fix this in secure way.

2. start agent

  $ $manage start-agent

3. check the status (url and port is from secrete file)

  agent and db
  $ $manage status 
  
  request manager
  http://cmssrv95.fnal.gov:8687/reqmgr/

  couchdb
  http://cmssrv95.fnal.gov:5984/_utils/

  wmstats (There need some manual changes to make inner link to work)
  http://cmssrv95.fnal.gov:5984/wmstats/_design/WMStats/index.html

3. add user, group team (this should be exactly like this unless you want to change config.py)

  add 'fbloggs' as user (this is the default user for wmagent test)
  http://cmssrv95.fnal.gov:8687/reqmgr/admin/users/
  
  add DATAOPS as group (this should be matched with your request)
  http://cmssrv95.fnal.gov:8687/reqmgr/admin/groups/
  
  add DATAOPS group to the user
  http://cmssrv95.fnal.gov:8687/reqmgr/admin/user/fbloggs

  add cmsdataops as team
  http://cmssrv95.fnal.gov:8687/reqmgr/admin/teams/

  add cmssw versions (click update from tc button)
  http://cmssrv95.fnal.gov:8687/reqmgr/admin/versions/

3. Create the Request and submit to ReqMgr

  This doesn't have to be in the same machine (unless the test request manager is under firewall)
  Just assume python is installed.
  in any directory
  
  $ git clone git@github.com:dmwm/WMCore.git
  
  $ cd WMCore

  change parameters on the test. (i.e. MonteCarlo test)
  make config cache exist in the config cache db (default config cache db is pointed to the 
  cmsweb-testbed which should work)
   
  $ vi test/data/ReqMgr/requests/MonteCarlo.json
  
    "Team": "Team--OVERRIDE-ME", should be "Team": "cmsdataops"
    "Requestor": fbloggs
    other OVERRIDE-ME needs to be over written whatever string you want
    
    "Campaign": "Test_sryu"
    "RequestString": "sryu_test"
    "AcquisitionEra": "AcquisitionEra-test",
    "ProcessingString" : "TestSryu",
    "SiteWhitelist": ["T1_US_FNAL"],
    "dashboard": "mc"

  get right python version
  $ source /data/srv/wmagent/current/apps/wmagent/etc/profile.d/init.sh
  
  inject the request
  $ python test/data/ReqMgr/reqmgr.py --reqMgrUrl http://cmssrv101.fnal.gov:8687 -f test/data/ReqMgr/requests/MonteCarlo.json -j '{"createRequest" : {"RequestString" : "test_MC_Files","Campaign": "Test_Campaign"}}' --createRequest --assignRequests

Clone this wiki locally