Add support for special dither scheme when for TOO observations

[soares-santos]

We are using the code from @ehneilsen to modify the opsim db produced by the project and we want to run the make_simblib.py script from @rbiswas4 to turn them into simblib files for SNANA. However, we are running into trouble with the fact that we don't have ditherfiles for the new db's which were modified for TOO observations. One possibility is to turn off the dithering altogether when we do TOO observations, or use a special scheme that optimizes the probability of discovery of the source. This could be done, we think, by adding a propID that this specific for TOO observations.

[rbiswas4]

The plan @soares-santos and I discussed for this is:

1. I make sure there is something so that this can run with no -ditherfile, ignoring dithers everywhere.
2. I add something to take into account TOO as a new proposal. This should have entroes in the proposalId column of the summary table which is an integer distinct from the used ones. I think 7 would do for now , and a new row in the Proposal table. describing this proopsal. The opsim like databases should come from @ehneilsen . Please do let me know if this is possible. I will add code to use this.

[soares-santos]

I talked to @ehneilsen about this and he already has the feature we need in his code. He uses proposalID = 6, but it is trivial to change it to another number. @rbiswas4, if you can make it so that there is an option such as "--noditherpropidnum" we can then be safe in case the opsim people come up with other poposalID = 7 and we have to bump ours.

[ehneilsen]

In principle, it might be better to use proposal name rather than proposal id, so that proposal ids can be arbitrarily renumbered and changed in the future. The Proposal table in the sqlite3 database maps id to name. The owsim code that generates the database looks in the reference database (the one it overwrites) for the next available id in the old Proposal table, and uses it is the id for gw followup. If in the future if owsim is run on a different database with a different number of proposals, such that owsim end up assigning a different number to GW followup, then things will break if id is used to specify which proposals should or should not be dithered, but if the proposal name is used, things will continue to work. This does mean, however, code will occasionally need to join the summary table with the proposal table when doing queries on the database. On Aug 2, 2018, at 9:10 AM, soares-santos <notifications@github.com<mailto:notifications@github.com>> wrote: I talked to @ehneilsen<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ehneilsen&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=C1pT7tUhW3PFDWNL7ANIOQ&m=X0b_8Gj6KOyh1pybk6AN5XHoI7HHecDxkYiE6yFEVEM&s=ycsZMXzdHUh83gJ0DArQnt-1dcnbsvRWFvUs_X4Jwz8&e=> about this and he already has the feature we need in his code. He uses proposalID = 6, but it is trivial to change it to another number. @rbiswas4<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rbiswas4&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=C1pT7tUhW3PFDWNL7ANIOQ&m=X0b_8Gj6KOyh1pybk6AN5XHoI7HHecDxkYiE6yFEVEM&s=G72IzVrEv71vzVtF7-cMSduI0x775siTVYAaXB5HvNE&e=>, if you can make it so that there is an option such as "--noditherpropidnum" we can then be safe in case the opsim people come up with other poposalID = 7 and we have to bump ours. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rbiswas4_OpSimSummary_issues_254-23issuecomment-2D409939703&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=C1pT7tUhW3PFDWNL7ANIOQ&m=X0b_8Gj6KOyh1pybk6AN5XHoI7HHecDxkYiE6yFEVEM&s=QzWo6lLoAwAC9KmFlIQi2NdAkD7dNEpYAzYKcR0hO8o&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMCH3yL4xZOAM0iziwbYT-2DIw0-5Ff5Z-5Fwrks5uMwhCgaJpZM4Vqpc9&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=C1pT7tUhW3PFDWNL7ANIOQ&m=X0b_8Gj6KOyh1pybk6AN5XHoI7HHecDxkYiE6yFEVEM&s=o5EMqdIfeufxP5DklLbkfX7Z6zbFYZi_gs2NQ1CTsh8&e=>.

[rbiswas4]

@soares-santos @ehneilsen : I agree. So, combining your comments I think the plan is :

The proposal table should contain an extra row with an id distinct from each of the propIDs already in that table which usually looks like.

propId|Session_sessionId|propName|propType
1|2022|NorthEclipticSpur|General
2|2022|SouthCelestialPole|General
3|2022|WideFastDeep|General
4|2022|GalacticPlane|General
5|2022|DeepDrillingCosmology1|Sequence

Perhaps

propId|Session_sessionId|propName|propType
7|2022|GWTOO|Sequence

and all rows representing TOO observations in the summaryAllProps table should have
ProposalId values of 7.
Note, as long as these are consistent and distinct in that particular database, (7 could be replaced by 6 in one database and 15 in another, you don't need to tell me) my code will work.

• The additional information
  I do need from @ehneilsen is what his propName will be (eg. GWTOO?). I would need you to stick with preferably the same name in all databases you create and want to use my code for.

• I also expect that your replaced observations will have ra and dec in degrees (to be consistent with conventions in opsim v4) in fieldRA and fieldDec columns.

• A question for @ehneilsen :
  OpSim tends to have (have not checked v4) duplicate entries for the same observation assigned to DDF and WFD (ie. the same pointing serves both needs) for a miniscule fraction of cases). I am assuming that if you have picked any such observation to replace, you have also removed the duplicate. Could you lease confirm that this a safe assumption, and I don't need to add code to deal with this consistently?

[ehneilsen]

On Aug 2, 2018, at 7:03 PM, rbiswas4 ***@***.***> wrote:

...

Note, as long as these are consistent and distinct in that particular database, (7 could be replaced by 6 in one database and 15 in another, you don't need to tell me) my code will work. The additional information - I do need from @ehneilsen is what his propName will be (eg. GWTOO?). I would need you to stick with preferably the same name in all databases you create and want to use my code for.

The propName I have been using is “GWFollowup”. It would be trivial for me to change.

• I also expect that your replaced observations will have ra and dec in degrees (in opsim v4 ) in fieldRA and fieldDec columns. A question for @ehneilsen : OpSim tends to have (have not checked v4) duplicate entries for the same observation assigned to DDF and WFD (ie. the same pointing serves both needs) for a miniscule fraction of cases). I am assuming that if you have picked any such observation to replace, you have also removed the duplicate. Could you lease confirm that this a safe assumption, and I don't need to add code to deal with this consistently?

This seems to be the case in v4 as well. I remove all exposures in the time range occupied by the ToO from the reference database, including multiple rows if there were multiple exposures. I expect that the exposures I use for the ToO will also be useful for WFD (they are presently designed to be), so I add two rows for them: one with the WFD proposal id, the other with the GWfollowup one. -Eric

[rbiswas4]

@ehneilsen

The propName I have been using is “GWFollowup”. It would be trivial for me to change.

No need. I just need to know what you are using. I will take that as given.

I expect that the exposures I use for the ToO will also be useful for WFD (they are presently designed to be), so I add two rows for them: one with the WFD proposal id, the other with the GWfollowup one.

I wonder if you would be willing to remove the extra WFD, at least temporarily. This would be a bit of simplification for me, in order to get this up to speed. (I am willing to work on accommodating the extra WFD in a while, but with too many deadlines, it would be really helpful to get started without this. Let me add that at least for SN studies that I am going to help in, this will not mean that the GWFollowup epochs are missed in any SN study.

[ehneilsen]

The list of exposures to be associated with each overwriting exposure is in the configuration file for followup.py, followup.conf: the keyword is “proposals”. I’ll update it to be nothing but “GWFollowup” and rerun.

…

-Eric On Aug 3, 2018, at 10:46 AM, rbiswas4 <notifications@github.com<mailto:notifications@github.com>> wrote: @ehneilsen<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ehneilsen&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=C1pT7tUhW3PFDWNL7ANIOQ&m=5SOi6N3QXnjpnBVSgL4e-ntIpLvY9r1iURirAI9ZUY8&s=gya5sJ0WTrF0SPhOhf4v10rKjK0FkNyHG68-r1n2vlY&e=> The propName I have been using is “GWFollowup”. It would be trivial for me to change. No need. I just need to know what you are using. I will take that as given. I expect that the exposures I use for the ToO will also be useful for WFD (they are presently designed to be), so I add two rows for them: one with the WFD proposal id, the other with the GWfollowup one. I wonder if you would be willing to remove the extra WFD, at least temporarily. This would be a bit of simplification for me, in order to get this up to speed. (I am willing to work on accommodating the extra WFD in a while, but with too many deadlines, it would be really helpful to get started without this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_rbiswas4_OpSimSummary_issues_254-23issuecomment-2D410294748&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=C1pT7tUhW3PFDWNL7ANIOQ&m=5SOi6N3QXnjpnBVSgL4e-ntIpLvY9r1iURirAI9ZUY8&s=5edSHc4X80xt8Brx8vHaOIZzQoS985Rg7--e-hLWwD0&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMCH38mxHKWMJWCdEgXDuOwEzxVPy3jzks5uNHBrgaJpZM4Vqpc9&d=DwMFaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=C1pT7tUhW3PFDWNL7ANIOQ&m=5SOi6N3QXnjpnBVSgL4e-ntIpLvY9r1iURirAI9ZUY8&s=toRvOMuD2jj122AfchhRa9SS_w6T7iO6RYCfSIcMGhU&e=>.