Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Effective area file sizes #148

Closed
GernotMaier opened this issue Dec 16, 2021 · 5 comments
Closed

Effective area file sizes #148

GernotMaier opened this issue Dec 16, 2021 · 5 comments
Assignees
@GernotMaier

Comments

@GernotMaier
Copy link
Member

Effective area files are very big, dominated by histograms to be used by gammapy analysis.
@GernotMaier GernotMaier self-assigned this Dec 16, 2021
@GernotMaier
Copy link
Member Author

Suggest to have a anasum only version of the effective areas with the following tree entries. Please have a look and confirm that this is fine:

******************************************************************************
*Tree    :fEffArea  : effective area values                                  *
*Entries :   302940 : Total =       819185992 bytes  File  Size =  152026730 *
*        :          : Tree compression factor =   5.39                       *
******************************************************************************
*Br    0 :ze        : ze/D                                                   *
*Entries :   302940 : Total  Size=    2426708 bytes  File Size  =      16281 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 148.99     *
*............................................................................*
*Br    1 :az        : az/I                                                   *
*Entries :   302940 : Total  Size=    1214103 bytes  File Size  =      11990 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression= 101.20     *
*............................................................................*
*Br    2 :azMin     : azMin/D                                                *
*Entries :   302940 : Total  Size=    2426813 bytes  File Size  =      39492 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  61.43     *
*............................................................................*
*Br    3 :azMax     : azMax/D                                                *
*Entries :   302940 : Total  Size=    2426813 bytes  File Size  =      34532 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  70.25     *
*............................................................................*
*Br    4 :Xoff      : Xoff/D                                                 *
*Entries :   302940 : Total  Size=    2426778 bytes  File Size  =      21158 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 114.65     *
*............................................................................*
*Br    5 :Yoff      : Yoff/D                                                 *
*Entries :   302940 : Total  Size=    2426778 bytes  File Size  =      22712 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 106.81     *
*............................................................................*
*Br    6 :Woff      : Woff/D                                                 *
*Entries :   302940 : Total  Size=    2426778 bytes  File Size  =      22236 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression= 109.10     *
*............................................................................*
*Br    7 :noise     : noise/I                                                *
*Entries :   302940 : Total  Size=    1214181 bytes  File Size  =      12625 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression=  96.11     *
*............................................................................*
*Br    8 :pedvar    : pedvar/D                                               *
*Entries :   302940 : Total  Size=    2426848 bytes  File Size  =      27828 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  87.18     *
*............................................................................*
*Br    9 :index     : index/D                                                *
*Entries :   302940 : Total  Size=    2426813 bytes  File Size  =      58855 *
*Baskets :       31 : Basket Size=     163328 bytes  Compression=  41.22     *
*............................................................................*
*Br   10 :nbins     : nbins/I                                                *
*Entries :   302940 : Total  Size=    1214181 bytes  File Size  =      34504 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression=  35.17     *
*............................................................................*
*Br   11 :e0        : e0[nbins]/D                                            *
*Entries :   302940 : Total  Size=   85784285 bytes  File Size  =    2153436 *
*Baskets :      749 : Basket Size=    6243840 bytes  Compression=  39.83     *
*............................................................................*
*Br   12 :eff       : eff[nbins]/D                                           *
*Entries :   302940 : Total  Size=   85785038 bytes  File Size  =   72214376 *
*Baskets :      749 : Basket Size=    6244352 bytes  Compression=   1.19     *
*............................................................................*
*Br   13 :Rec_nbins : Rec_nbins/I                                            *
*Entries :   302940 : Total  Size=    1214285 bytes  File Size  =      52558 *
*Baskets :       22 : Basket Size=      81408 bytes  Compression=  23.09     *
*............................................................................*
*Br   14 :Rec_e0    : Rec_e0[Rec_nbins]/D                                    *
*Entries :   302940 : Total  Size=   71873603 bytes  File Size  =    1770038 *
*Baskets :      655 : Basket Size=    5445632 bytes  Compression=  40.60     *
*............................................................................*
*Br   15 :Rec_eff   : Rec_eff[Rec_nbins]/D                                   *
*Entries :   302940 : Total  Size=   71874262 bytes  File Size  =   60079301 *
*Baskets :      655 : Basket Size=    5445632 bytes  Compression=   1.20     *
*............................................................................*
*Br   16 :esys_rel  : esys_rel[nbins]/F                                      *
*Entries :   302940 : Total  Size=   43502142 bytes  File Size  =    2660131 *
*Baskets :      390 : Basket Size=    3163648 bytes  Compression=  16.35     *
*............................................................................*
*Br   17 :hEsysMCRelative : TProfile                                         *
*Entries :   302940 : Total  Size=  436095470 bytes  File Size  =   12753304 *
*Baskets :     1738 : Basket Size=   25600000 bytes  Compression=  34.19     *
*............................................................................*

Removed are:

*Br   13 :effNoTh2  : effNoTh2[nbins]/D                                      *
*Entries :   302940 : Total  Size=   85746962 bytes  File Size  =   71742586 *
*Baskets :      358 : Basket Size=     312320 bytes  Compression=   1.20     *
*............................................................................*
*Br   17 :Rec_effNoTh2 : Rec_effNoTh2[Rec_nbins]/D                           *
*Entries :   302940 : Total  Size=   71842588 bytes  File Size  =   59881588 *
*Baskets :      340 : Basket Size=     271360 bytes  Compression=   1.20     *
*Br   19 :Rec_angRes_p68 : Rec_angRes_p68[Rec_nbins]/F                       *
*Entries :   302940 : Total  Size=   36545705 bytes  File Size  =    2842198 *
*Baskets :      329 : Basket Size=     137728 bytes  Compression=  12.86     *
*............................................................................*
*Br   20 :Rec_angRes_p80 : Rec_angRes_p80[Rec_nbins]/F                       *
*Entries :   302940 : Total  Size=   36545705 bytes  File Size  =    2854665 *
*Baskets :      329 : Basket Size=     137728 bytes  Compression=  12.80     *
*............................................................................*
*Br   21 :hEsysMCRelative : TProfile                                         *
*Entries :   302940 : Total  Size=  435931990 bytes  File Size  =   12155646 *
*Baskets :      304 : Basket Size=    2840576 bytes  Compression=  35.86     *
*............................................................................*
*Br   22 :hEsysMCRelative2D : TH2F                                           *
*Entries :302940 : Total  Size= 9153905056 bytes  File Size  = 101062485 *
*Baskets :     2615 : Basket Size=   25600000 bytes  Compression=  90.58     *
*............................................................................*
*Br   23 :hEsysMCRelative2DNoDirectionCut : TH2F                             *
*Entries :302940 : Total  Size= 9163366508 bytes  File Size  = 613688375 *
*Baskets :     2615 : Basket Size=   25600000 bytes  Compression=  14.93     *
*............................................................................*
*Br   24 :hAngularLogDiffEmc_2D : TH2F                                       *
*Entries :302940 : Total  Size= 5083231848 bytes  File Size  = 105916139 *
*Baskets :     1677 : Basket Size=   25600000 bytes  Compression=  47.99     *
*............................................................................*

GernotMaier added a commit that referenced this issue Dec 16, 2021
@GernotMaier
File size reduction options for effective areas (issue #148)
468f60a 
Allow to write entries required for anasum analysis only
@GernotMaier GernotMaier mentioned this issue Dec 16, 2021
Merged
@steob92
Copy link
Contributor

steob92 commented Dec 16, 2021

The ED likelihood spectral analysis also won't work with the slimmed down effective area files. Like the gammapy analysis they use hEsysMCRelative2D.

@GernotMaier
Copy link
Member Author

Yes, I had this in mind but forgot to mention. Using the likelihood spectral analysis would require to download the full IRFs.

@GernotMaier
Copy link
Member Author

Removing the branches above reduces the effective area file size by a factor (!) 6-8.

@GernotMaier
Copy link
Member Author

fixed and release with v486b

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@GernotMaier GernotMaier

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@steob92 @GernotMaier