trouble shooting

WMAgent trouble shoot

Workqueue is not acquiring request: acquired status but not moving to running open status.

1. Check WorkQueueManager Component Log.
2. If the error message is showing json parse error, wq view is corrupted.

   ERROR:WorkQueueManagerWMBSFileFeeder:Error in wmbs inject loop: unterminated array starting at position 0:
   

   1. shutdown the agent, and shutdown couch server
   2. remove the view from /data1/database/.workqueue_design
   3. start the couch server and rebuild the view by triggering one of the view

      curl http://localhost:5984/workqueue/_design/WorkQueue/_view/availableByPriority
      

   4. If the view rebuild is finished, start the agent.

Note, however, in case this issue is happening to all the agents, then it probably is an issue with central workqueue.

Datasets were produced with None either for AcquisitionEra or ProcessingString.

1. In these cases these blocks and files will bug DBS3Upload all the time since it does not pass the Lexicon validation in DBS. So, what we need to do is basically to close those blocks and mark them (and its files) as injected in dbsbuffer tables. 1. Shutdown PhEDExInjector and DBS3Upload 2. Gather a list of workflow names and their bad output dataset names: SELECT DISTINCT dbsbuffer_workflow.name, dbsbuffer_dataset.path FROM dbsbuffer_dataset INNER JOIN dbsbuffer_dataset_subscription ON dbsbuffer_dataset.id = dbsbuffer_dataset_subscription.dataset_id INNER JOIN dbsbuffer_block ON dbsbuffer_block.dataset_id = dbsbuffer_dataset_subscription.dataset_id INNER JOIN dbsbuffer_file ON dbsbuffer_file.block_id = dbsbuffer_block.id INNER JOIN dbsbuffer_workflow ON dbsbuffer_workflow.id = dbsbuffer_file.workflow WHERE dbsbuffer_block.blockname LIKE '/%/None-%' AND dbsbuffer_block.status != 'Closed'; 3. Find the blocks and files that needs manual intervention (just to keep record) SELECT * FROM dbsbuffer_block WHERE blockname LIKE '/%/None-%' AND status!='Closed'; SELECT * FROM dbsbuffer_file WHERE lfn LIKE '/store/%/None/%' AND status!='InDBS'; 4. Close and mark them as injected UPDATE dbsbuffer_block SET status='Closed' WHERE blockname LIKE '/%/None-%' AND status!='Closed'; UPDATE dbsbuffer_file SET status='InDBS', in_phedex='1' WHERE lfn LIKE '/store/%/None/%' AND status!='InDBS'; 5. elog them in the workflow team.