Skip to content

Timing information

Jump to bottom
Javier Moldon edited this page Sep 4, 2017 · 4 revisions

Here we can include summaries or plots that shows how long it takes to process each step of the pipeline and see what can we make more efficient if possible.

Setting MS order with setToCasaOrder?

General conclusions:

  • Don't use setToCasaOrder. It takes a lot of time and in fact it does not make things faster. Even UVFIX immediately after loading the data take exactly the same ammount of time.
  • In general, currently the slowest step is aoflagger. It operates in write mode because there is not enough RAM.
  • After splitting the data to 128 chan/spw and 2s the calibration is really fast. About 40 minutes including plotting the calibration tables with plotcal.

Details:

The test was conducted on richards (12 CPU @ 3.50GHz, 64 GB of RAM). The data set is a 12h L-band observation, 512 chan/spw and 1 sec integration time. The original file size is 184 GB in fits IDI format. In the split step the data are averaged to 128 chan/spw and 2 sec integration time. Task ms2mms was not used, so everything is run on a MS (not an MMS).

Times to run different steps with setToCasaOrder: summary_test2_1

Times to run different steps with NO setToCasaOrder: summary_test2_3

Test conclusions:

  • setToCasaOrder takes a long time. About 3h spent on setting the order of a 184 GB file.
  • Most of the following steps with unaveraged data take the same ammount of time.
  • Aoflagger seems to take much longer on ordered data. Making it more than 1h slower.
  • The split step (including averaging to 128chan/spw and 2sec) is also slower when applying setOrder.
  • The rest of the steps after averaging does not make a difference.
  • I see the same behaviour on unaveraged data.

Notes on Aoflagger:

  • Aoflagger is very slow if the data to be flagged cannot be loaded with the available RAM, because it has to write everything on disk. It is very fast if it can process the data in RAM.
  • In this case, the flagging times for sources 1331+305, 1407+284, 0319+415 (about 1h or data each) were about 5 minutes.
  • The flagging time for the phase calibrator was about 18 min.
  • The flagging time for the target source was 4.5 hours. Probably these part of the data was too big.

Home

Pipeline Meeting Minutes

Coding Practices

Test Data

Common Problems

Clone this wiki locally